MENLO PARK — In the world of healthcare, having timely and accurate information can be a matter of life and death. Recognizing this need, Meditron, a suite of open-source large multimodal foundation models designed to aid in clinical decision-making and diagnosis, has been developed. Built on Meta’s Llama 2 platform and continually refined with input from clinicians and humanitarian experts, Meditron is specifically tailored for use in low-resource medical settings.

This project is the result of a collaboration between researchers at EPFL’s School of Computer and Communication Sciences and Yale School of Medicine, working closely with humanitarian organizations such as the International Committee of the Red Cross (ICRC). Meditron has quickly gained traction, with over 30,000 downloads within the first few months of its release, filling a critical gap in low-resource medical innovation. Following Meta’s release of Llama 3, the team quickly fine-tuned the new 8B model to create Llama-3[8B]-MeditronV1.0, which outperforms state-of-the-art models in its class on standard medical benchmarks such as MedQA and MedMCQA.

“Foundation models are becoming vital intellectual and cultural tools,” said Yale Professor Mary-Anne Hartley, who co-leads the project. “When applied to medicine, these models can offer life-saving advice. The greatest beneficiaries could be low-resource settings, yet they remain underrepresented.”

Meditron takes Meta’s Llama 2 model and adapts it to better align with evidence-based medical practices, delivering context-aware recommendations. It is designed to meet the needs of emergency situations, assisting healthcare workers in diagnosing and treating patients where resources are limited.

Hartley emphasized that releasing Meditron as fully open-source—complete with data, weights, and comprehensive documentation—can empower innovation in low-resource settings. “No region should have to reinvent the wheel to ensure their populations are represented in this critical technology,” she noted.

Focus on Quality in Development

One of the major hurdles for such initiatives is funding, especially in humanitarian and low-resource settings. Hartley explained that the team decided not to commercialize Meditron to preserve the neutrality necessary for validation. They started small, focusing on quality and using Llama 2 7B for initial experiments to determine optimal data mixtures before scaling up to the 70B model.

Meditron’s 7B model is highly effective despite being less performant than its larger counterpart. The suite integrates image interpretation, surpassing models like Medpalm M in medical image performance, and shows promise in scaling to higher capacities.

The team prioritized curating medically validated data, focusing on evidence-based guidelines in both high- and low-resource environments. Continued pretraining was used to ensure that the model retained its medical knowledge while minimizing bias and contamination from general datasets.

To tackle the technical challenges of distributed pretraining, the team integrated Llama’s architecture into the Megatron-LM high-performing distributed trainer. This adapted version of Megatron was also open-sourced to help others facing similar technical barriers.

Global Collaboration and Evaluation

A particularly exciting outcome of the Meditron project is the global response from medical professionals and humanitarian organizations interested in validating and evaluating the model. The Meditron MOOVE (Massive Online Open Validation and Evaluation) initiative has attracted doctors worldwide, particularly in low-resource settings, who are testing the model with challenging questions and providing critical feedback.

“Seeing time-constrained professionals volunteer to independently validate Meditron speaks volumes about its value,” Hartley said. “This feedback allows us to continue refining the model, and we hope that funders recognize the social and commercial importance of supporting this academic, open-source initiative.”

Hartley stressed that open-source technology has a long history of empowering innovation, particularly in underserved regions. “While open source isn’t new, the scale and cost of this contribution are unprecedented. We need to be bold in seeking neutral philanthropic support for efforts like these.”