Vision Language Models for Derm: How Do They Work TODAY?

The presentation explores the use of vision language models in dermatology, emphasizing their experimental nature and the caution needed in their application. The speaker discusses the importance of structured prompts and context for obtaining accurate model responses. Various cases are demonstrated using images sourced from the ISIK archive, highlighting how these models can provide differential diagnoses based on visual input. For example, in one case involving a zoomed-out photo of the back, the model suggested potential diagnoses based on minimal context, prompting the need for advanced queries to extract more detailed and relevant information. Some pitfalls encountered included the model's inconsistency in identifying complex conditions and the challenge of performing quantitative analyses like counting lesions. The speaker also noted that altering prompts to provide clearer context could improve accuracy significantly, showcasing the necessity of careful interaction with these systems. Overall, while vision language models hold promise, their current limitations and the need for further validation were underscored.