When a medical study publishes its results, the underlying code can reveal exactly how researchers reached their conclusions — how they selected patients, handled missing data, and evaluated their models. But a new investigation led by MIT and the Dana-Farber Cancer Institute shows that this kind of transparency is still the exception, not the rule.
Using large language models (LLMs) to screen nearly 4,000 clinical prediction studies, the team found that only about one in eight papers reported sharing any code at all. And even when code was shared, the quality varied enormously — some repositories were meticulously documented, while others offered almost no guidance on what the code did or how to run it.
The findings, published in the journal Nature Medicine, are now informing a new reporting guideline called TRIPOD-Code, which aims to give researchers and journals a clear standard for what constitutes useful code sharing.
Why code matters in medical research
Clinical prediction studies develop or evaluate models that estimate patient outcomes — for example, the risk of developing a particular disease. Because these models can directly influence decisions about patient care, other researchers need to be able to scrutinise exactly how they were built and tested.
“Studies involve many choices about how to prepare data, develop models, and evaluate performance. These are sometimes called ‘researcher degrees of freedom,’” said Tom Pollard, a research scientist at MIT’s Laboratory for Computational Physiology and Computer Science and Artificial Intelligence Laboratory, and a senior author of the study. “The choices we make affect the results, and a methods section in a paper can only capture so much detail.”
Access to the code allows researchers to examine those choices, verify the results, and build on the work. But without clear documentation and structure, a shared code repository can be nearly as opaque as no code at all.
How the AI-assisted review worked
The team set out to understand current code-sharing practices by identifying studies that cited either TRIPOD or TRIPOD+AI — existing checklists that help researchers report how clinical prediction models were developed and evaluated. The review covered nearly 4,000 studies that developed, updated, or validated prediction models and whose full texts were available through PubMed Central’s open-access collection.
Because the information about code sharing was scattered and inconsistent, the researchers turned to LLMs. “The challenge was that the information was highly inconsistent. First, we had to identify whether or not a paper shared code,” said Thomas Sounack, the study’s first author. He noted that authors “discuss and link to code from a paper in many different ways.”
The LLM helped screen papers, identify statements about code availability, and extract links. Software then retrieved the linked repositories for further assessment, checking not just whether a link existed but also evaluating documentation completeness, software requirements, and terms of reuse.
“The real benefit of LLMs is their ability to process unstructured information such as text. These models helped us work with information that was scattered across papers and software repositories, rather than neatly organized in a table,” Pollard said. “That allowed us to examine how code was being shared in much more detail across a large collection of studies.”
Two researchers separately examined 500 articles and an additional set of code archives, settling differences prior to contrasting their evaluations against the machine-generated findings. Concordance was higher for tangible attributes, like the existence of a license file, compared to assessments regarding the sufficiency of documentation. Since misclassifications can occur, the researchers highlighted overarching trends instead of regarding each figure as precise.
Two key gaps
The review revealed two main problems. First, code sharing remains uncommon — only about 12.5% of studies reported sharing code. Second, making code public does not consistently mean providing the documentation and supporting information needed for reuse. Some repositories were “extremely well-organized and -documented, while others gave almost no guidance at all,” Sounack said. Good code sharing, he explained, means “giving enough information to understand what the code does and be able to run it to reproduce the results reported in the paper.”
Practices also varied substantially across journals and countries. Among journals, the proportion of studies sharing code ranged from zero to more than 70%. Some journals with low sharing rates nevertheless had written code-sharing policies, suggesting that policies alone may not be enough to motivate code sharing.
The review also identified examples of effective practice. Some researchers, for example, provided synthetic datasets to assist with code reuse where the original patient records could not be shared publicly.
What this means for students
For university students — particularly those studying medicine, statistics, data science, or computer science — the findings highlight a growing expectation that research should be reproducible. As TRIPOD-Code takes shape, students learning to conduct clinical prediction research will likely be expected to share well-documented code as part of their work. The guideline aims to provide practical guidance on which elements of shared analytical code should be considered essential, giving students and early-career researchers a clear benchmark for what good practice looks like.
Setting a standard
TRIPOD-Code builds on TRIPOD and TRIPOD+AI, the existing checklists for reporting clinical prediction models. Through an expert review process, the team is developing guidance on which elements of shared analytical code should be considered essential.
“We want to give researchers practical guidance on how to share code, and give journals clearer expectations to communicate to authors,” Pollard said. “The review gives us evidence to inform those discussions.”
The open-access findings were published in Nature Medicine on November 19, 2026.