A convolutional neural network (CNN) architecture from 2015 decisively outperformed modern frontier large language models (LLMs) on a skin cancer diagnostic task, according to a study flagged on Hacker News. The research, conducted by Stanford University's Lynch Lab, tested GPT-4o, Gemini 1.5 Pro, and Claude 3 Opus against a decade-old ResNet-50 model fine-tuned on dermatology images.
The results were stark. The specialized CNN achieved a diagnostic accuracy of 81.5%, while the best-performing LLM, GPT-4o, reached only 60.3%. The study involved 1,500 histopathological images across 15 skin disease categories.
How the Benchmark Was Structured
The researchers presented each AI system with a high-resolution image of a skin lesion alongside a text prompt asking for a diagnosis from a list of 15 possibilities. The ResNet-50 model, first published in 2015, was fine-tuned on a dataset of 17,000 dermatoscopic images. The LLMs were tested in their standard, publicly available configurations via API, without any special fine-tuning for this task.
Performance was measured by top-1 accuracy—whether the model's first-choice diagnosis was correct. The CNN's 21.2 percentage point lead is statistically significant and clinically meaningful in a field where diagnostic precision directly impacts patient outcomes.
| Model | Type | Top-1 Accuracy | Training Data Specificity |
|---|---|---|---|
| ResNet-50 (Fine-tuned) | CNN (2015) | 81.5% | 17k dermatology images |
| GPT-4o | Multimodal LLM | 60.3% | General web-scale data |
| Gemini 1.5 Pro | Multimodal LLM | 58.7% | General web-scale data |
| Claude 3 Opus | Multimodal LLM | 55.9% | General web-scale data |
Why a Specialized CNN Still Dominates
The core finding underscores a fundamental principle in applied machine learning: domain-specific tuning beats general-purpose breadth for narrow, high-stakes tasks. The ResNet-50's entire architecture is designed for image feature extraction, and its parameters were optimized exclusively for recognizing patterns in skin lesions.
LLMs, despite their multimodal capabilities, are primarily text engines. Their image understanding is a learned add-on, not a native strength. The study notes that LLMs often produced confidently written but incorrect reasoning, hallucinating details not present in the image to justify a wrong diagnosis.
Bottom line: For visual pattern recognition, a decade of architectural progress in general AI hasn't closed the gap with a model trained for one specific job.
What the Hacker News Community Noted
The Hacker News discussion garnered 95 points and 47 comments. Key insights from practitioners included:
- Confirmation of prior intuition: Many commenters were not surprised, noting that LLMs are "world-class bullshitters" when pushed outside their core competency.
- Discussion on fine-tuning: Several pointed out that the comparison wasn't entirely fair, as the LLMs were not fine-tuned on the medical dataset, highlighting the cost and complexity of doing so.
- Clinical workflow integration: Comments debated whether an LLM's explanatory text, even if sometimes wrong, could still aid a human doctor more than a CNN's silent classification.
Practical Alternatives for Medical Imaging
For developers or researchers building diagnostic tools, this study is a clear signal to choose the right tool for the job.
| Tool | Best For | Considerations |
|---|---|---|
| Fine-tuned CNNs (ResNet, DenseNet) | High-accuracy, production-grade diagnostic classifiers | Requires labeled medical datasets; excels at pure classification. |
| Vision Transformers (ViTs) | Cutting-edge image classification with large datasets | Often needs more data than CNNs; computational cost is higher. |
| Multimodal LLMs (GPT-4V, Gemini) | Prototyping or systems requiring narrative explanation | Accuracy is lower; cost per query is high; risk of confident hallucinations. |
| Specialized Medical AI (e.g., DermaDetect) | Turnkey clinical applications | Often proprietary and expensive; validated for specific use cases. |
For those exploring this space, starting with a pre-trained CNN from platforms like PyTorch Image Models (timm) or TensorFlow Hub and fine-tuning on a relevant dataset remains the most reliable path.
"How to Reproduce or Build Upon This Study"
The original study code and data methodology are crucial for replication.
Who Should Pay Attention to This Research
Medical AI developers should see this as validation that proven, specialized architectures are not obsolete. The hype around general-purpose multimodal LLMs does not automatically translate to superior performance in specialized verticals.
Clinical practitioners evaluating AI tools should prioritize validation studies on medical tasks over brand recognition. A tool based on a fine-tuned CNN may be more robust than one powered by a famous LLM.
Researchers benchmarking LLMs must include specialized, non-text tasks to accurately represent capabilities. This study is a template for creating rigorous, apples-to-apples comparisons in other domains like radiology or pathology.
Conversely, teams needing natural language interaction around images—such as generating patient-friendly reports or answering follow-up questions—might still leverage LLMs, but ideally in a hybrid system where a CNN handles the primary classification.
The Bottom Line on LLMs vs. Specialized Models
The Lynch Lab study provides a critical, data-driven correction to the narrative that frontier LLMs are the ultimate solution for every AI problem. In high-stakes, vision-dominated fields like medicine, accuracy, reliability, and specificity cannot be traded for generative fluency. The 10-year-old CNN wins because it was built and trained for one thing, and it does that one thing exceptionally well. This doesn't render LLMs useless, but it firmly places them as tools for augmentation and explanation, not replacement, for core diagnostic functions in their current form.
The path forward likely involves ensemble approaches, where a specialized model like a CNN provides the primary diagnostic signal, and an LLM, constrained by that diagnosis, generates explanatory text for clinicians or patients. For now, the old guard still holds the high ground on accuracy.
Top comments (0)