Datasets, Clinical Studies, and External Validation in Dermatology AI

In this talk, Albert Chu discusses the role of datasets and clinical studies in evaluating algorithms used in dermatology, particularly in the context of AI. He highlights the importance of understanding how algorithms perform under real-world conditions compared to their performance on curated retrospective datasets. While algorithms can show promising results in experimental settings, moving to practical applications reveals a different scenario. Chu emphasizes the need for prospective data sets that capture the variability and challenges encountered in clinical practice, as retrospective data may not represent the full spectrum of conditions encountered in patients. The talk presents case studies, illustrating how algorithms can experience significant performance degradation when deployed in real-world settings, particularly when the data used differs from training data. An example includes a melanoma classifier that performed well under controlled testing but faltered when faced with actual patient presentations. The session encourages careful evaluation of algorithm performance across diverse datasets to understand their clinical utility, urging practitioners to remain mindful of how an algorithm's original task definition affects its adaptability. Ultimately, Chu advocates for robust external datasets and local validations to ensure algorithms are clinically relevant and capable of supporting dermatological care effectively.