Is AI skin cancer detection reliable on dark skin? Not yet, as a rule. Public training datasets rarely record skin tone, and a 2022 Stanford study found leading dermatology models performed substantially worse on darker skin. Before deployment, a tool should show pathology-confirmed accuracy for each skin-tone group, with enough cancers in each to make the numbers meaningful.
A primary-care doctor in a busy urban practice sees a dark, irregular spot on the sole of a patient’s foot. She has an AI app on her phone that scores skin lesions. It says low risk. The question she should be asking is not “how accurate is this app?” but “how many lesions like this one, on skin like this, was it ever tested on?” For most dermatology AI on the market, the honest answer is: very few, and nobody wrote down which.
“Biased data” has become the standard phrase for this problem, and it has become too comfortable. Everyone agrees that dermatology datasets skew towards lighter skin. Far fewer people can say what, concretely, a dermatology AI should have to demonstrate before it is used on a population that includes darker skin tones.
That is the practical gap. The thesis here is that fixing it is less about collecting more images in general and more about three specific proofs: stratified accuracy with enough cases to be meaningful, ground truth that does not inherit human bias, and evidence about the doctor using the tool, not just the tool.
Is AI for skin cancer detection less accurate on dark skin?
The best evidence says yes, for many models, and that the problem starts upstream of the algorithm.
A systematic review published in The Lancet Digital Health in November 2021 examined 21 open-access skin image datasets containing 106,950 lesion images. Only 2,236 images, about 2.1%, had Fitzpatrick skin type recorded, and only 1,415, about 1.3%, had ethnicity recorded. Most datasets that reported a country of origin came from Europe, North America or Oceania. The authors described substantial under-representation of darker skin types. You cannot measure a model’s performance on dark skin if the dataset does not say which images show dark skin.
A year later, a Stanford-led team built something the field lacked. The Diverse Dermatology Images dataset, published in Science Advances in August 2022 by Roxana Daneshjou and colleagues, was expertly curated, pathologically confirmed and deliberately diverse in skin tone. When state-of-the-art dermatology models were tested on it, they showed substantial limitations, particularly on dark skin tones and uncommon diseases. Two further findings matter just as much. Dermatologists, who often label the images used to train AI, also performed worse on dark skin tones. And fine-tuning the models on the diverse images closed the performance gap.
That last result is encouraging and easy to misread. It shows the gap is fixable with the right data. It does not show that any given commercial product has been fixed.
Why “more diverse data” is not enough
Three problems survive even a well-intentioned push for diversity.
The labels can carry the bias. If the ground truth for a training image is a dermatologist’s visual diagnosis, and dermatologists are less accurate on dark skin, then the model learns from noisier labels on exactly the images where accuracy matters most. The Stanford dataset avoided this by using biopsy-confirmed diagnoses. Many training sets do not.
The diseases differ, not just the skin. Skin cancer in people with darker skin more often presents in places and forms that general datasets underweight, such as acral melanoma on palms, soles and nails. A model can be fair across skin tones on the lesion types it has seen and still be weak on the lesion types that matter most for a particular population.
The human-AI pair behaves differently from the AI. A large experiment published in Nature Medicine in February 2024 by Matthew Groh and colleagues asked 389 board-certified dermatologists and 459 primary-care physicians from 39 countries to diagnose 364 images across 46 skin diseases. Specialists were right 38% of the time and generalists 19%, and both were about four percentage points less accurate on dark skin. Decision support from a deep-learning system improved accuracy for both groups by more than a third. But for generalists, it widened the gap between light and dark skin. Overall accuracy went up; fairness got worse.
That result deserves more attention than it gets. Most dermatology AI is aimed at non-specialists, because that is where the access problem is. The Groh experiment suggests the setting where AI adds most value is also the setting where it can amplify disparities.
What should a dermatology AI prove before deployment in a diverse population?
A useful checklist for any buyer, regulator or clinical lead has five items.
- Stratified accuracy with confidence intervals. Sensitivity and specificity reported separately for lighter and darker skin groups, using a recognised scale, on biopsy-confirmed cases.
- Enough cancers in each group. Not enough images, enough malignancies. Sensitivity is measured on cancers, and dark-skin cancers are where the numbers are thinnest.
- The right lesion mix. Evidence on the presentations common in the target population, including acral and nail lesions.
- Human-plus-AI results by group. Whether the intended users, usually primary-care clinicians, get better on every group or only on average.
- Capture conditions. Performance on images taken the way they will be taken in practice: phone cameras, clinic lighting, with and without dermoscopy.
Item two is where most submissions quietly fail, and it is worth a worked example.
A worked example: when 50 images is not evidence
Hypothetical, with illustrative numbers. A vendor validates its skin-cancer app on 1,000 biopsy-confirmed lesions. It reports 90% sensitivity overall and proudly notes that 5% of images are from darker skin types, Fitzpatrick V and VI. That is 50 images. If one in ten is malignant, the dark-skin group contains five cancers.
Suppose the model catches four of them. Its sensitivity on dark skin is reported as 80%. But with only five cancers, the 95% confidence interval runs from roughly 28% to more than 99%. The data are consistent with a model that misses nearly three-quarters of cancers on dark skin and with one that misses almost none. The honest summary is “unknown.”
To estimate sensitivity for that group within about ten percentage points either side, the vendor would need something like 35 to 50 confirmed cancers on dark skin, meaning hundreds of dark-skin lesions in total at typical malignancy rates. With 45 of 50 cancers detected, for example, the interval narrows to roughly 78% to 97%. That is a result a clinical lead can act on.
The lesson is not to demand perfection. It is to ask for the count of cancers per skin-tone group before looking at any percentage.
Wouldn’t strict requirements keep useful tools from the people who need them most?
This is the strongest counterargument. Daneshjou’s team noted that an estimated 3 billion people lack access to dermatological care. Many of them have darker skin. Skin cancer is less common in darker skin, so collecting enough biopsy-confirmed cancers is genuinely slow and expensive. If regulators and buyers demand large stratified datasets before deployment, the effect could be to deny populations a helpful tool while lighter-skinned populations enjoy it, which is its own kind of inequity.
There is real force in this. And the answer is not to block deployment until the perfect dataset exists. It is to be precise about claims and to design deployment as evidence collection. A tool can be labelled for the populations where it has proven performance. It can be deployed elsewhere with explicit monitoring, where every AI-assessed lesion that is later biopsied feeds back into a registry by skin tone. Multi-site data pooling can reach the needed numbers far faster than any single vendor. What should not happen is the current default, where a product validated mostly on lighter skin is marketed as general-purpose and nobody tracks what happens next.
We have made a broader version of this argument in bias in clinical algorithms: the fix that matters most is the one we rarely do. Dermatology is simply the specialty where the missing measurement is most visible, because it is literally on the surface.
What this means for regulators, buyers, clinicians and developers
For regulators, the progress so far is real but uneven. In January 2024 the FDA granted de novo authorisation to DermaSensor, a handheld device that uses spectroscopy and an algorithm to help physicians evaluate lesions that may be cancerous. As more tools reach primary care, regulators can push further by making skin-tone-stratified performance with case counts a standard part of public summaries, not an optional appendix. The same principle applies to subgroup reporting in drug trials, as we noted in women are in the trials now, the data still isn’t.
For health systems and buyers, put the five-point checklist into procurement. Ask for the number of biopsy-confirmed cancers by skin-tone group, not just the percentage of diverse images. Require post-deployment monitoring that links AI scores to biopsy results.
For clinicians, especially in primary care, treat a low-risk score on darker skin with extra caution, and particularly on acral and nail lesions. The tool’s confidence is not evidence of its competence on a lesion type it may rarely have seen.
For developers, the Stanford result is good news: targeted, biopsy-confirmed diverse data narrows the gap. Building it through partnerships with clinics serving diverse populations is slower than scraping atlases, but it creates the only kind of evidence that will survive scrutiny.
For patients, the advice is simple and not specific to AI: any new, changing or unusual spot, including on palms, soles or under nails, deserves a clinician’s review regardless of what an app says.
Frequently asked questions
Why does skin cancer AI perform worse on darker skin?
Mainly because the images used to build and test many models come overwhelmingly from people with lighter skin, and many datasets do not record skin tone at all. Labels can also be less reliable, since dermatologists themselves are less accurate on dark-skin images. Some lesion types more common in darker skin, such as acral melanoma, are also under-represented.
What is the Fitzpatrick scale?
The Fitzpatrick scale classifies skin into six types, from I, which always burns and never tans, to VI, deeply pigmented skin that rarely burns. It was designed to estimate sun sensitivity, not to describe skin colour, which is why some researchers prefer broader skin-tone scales. It remains the most common way dermatology AI studies report skin tone.
Can dermatology AI be made fair across skin tones?
Evidence suggests the gap can be narrowed. In the 2022 Diverse Dermatology Images study, fine-tuning models on diverse, biopsy-confirmed images closed the performance gap between light and dark skin. But fairness has to be proven for each product, and a 2024 Nature Medicine study showed AI support can still widen gaps among the physicians who use it.
Is there an FDA-authorised AI device for skin cancer?
Yes. In January 2024 the FDA granted de novo authorisation to DermaSensor, a handheld device that uses light-scattering measurements and an algorithm to help physicians evaluate skin lesions that may be cancerous. It is an adjunct to a physician’s assessment rather than a standalone diagnosis, and other tools are available in some markets outside the US.
Should I use a skin cancer app to check my moles?
An app can prompt you to look more carefully, but it cannot rule out cancer, and its accuracy may be lower on darker skin or unusual locations. Any new, changing, bleeding or unusual spot, including on the palms, soles or under a nail, should be checked by a clinician whatever an app reports.
Fair dermatology AI will not arrive through better intentions about data. It will arrive when buyers stop accepting “diverse images” as evidence and start asking how many confirmed cancers, on which skin, the tool has actually been tested on.
Related on the GHC blog: Why Universal Health Coverage Stalls After the Pilot Phase and What Health Ministries Get Wrong About Airline Safety Culture.
If you have deployed or evaluated a skin AI tool, did anyone ever show you its results by skin tone?
