Background

Abnormal skin pigmentation — including freckles, sunspots, melasma, cafe-au-lait spots, acquired nevus of Ota, age spots, and post-inflammatory hyperpigmentation (PIH) — is among the most common skin concerns worldwide. Beyond cosmetic impact, pigmentation is closely linked to photodamage, skin aging, and skin cancer risk. Precise identification and quantification of different pigmentation types is essential for skin health management, cosmetic intervention, and disease risk assessment.

Traditional pigmentation assessment relies on clinical visual inspection and standardised photography, which suffers from subjectivity, poor reproducibility, and limited quantification. Current automated tools mostly focus on segmentation or classification using supervised discriminative models, lacking deep semantic understanding of pigmentation morphology, type diversity, and clinical interpretability.

Project Overview

This project combines multimodal visual foundation models for precise pigmentation segmentation and feature extraction with large language models for semantic understanding and reasoning. The resulting system can not only detect pigmentation but also understand its morphology and describe results in natural language, producing structured and clinically interpretable reports.

Research Objectives

• Data collection and annotation: Recruit 500 subjects with facial pigmentation, capturing high-resolution 3D facial images and multispectral data. Four experienced dermatologists annotate each sample with boundary segmentation, type classification, severity grading, and natural language descriptions.
• AI pigmentation model: Develop a multi-task AI model that performs simultaneous segmentation, classification, and severity grading, with a multimodal reasoning module that generates structured, clinically interpretable reports.
• Validation: Comprehensive internal and external validation to ensure accuracy, robustness, and cross-population generalisability.
• Genetic discovery: Apply the validated model to generate pigmentation phenotype data for genome-wide association studies (GWAS), identifying genetic loci linked to different pigmentation types.

Collaborating Institutions

• Monash Suzhou Research Institute — AI model development and validation
• CAS Institute of Nutrition and Health (Shanghai) — Subject recruitment, data collection, dermatologist annotation, and GWAS analysis