A Case Study from Synthesa’s Research Team
At Synthesa, Inc., our mission is to accelerate and enhance evidence synthesis using large language models (LLMs) and AI-powered workflows. But how reliable are these AI-based processes when compared to traditional human-led systematic reviews?
Validating ChatGPT’s role in systematic reviews and meta-analyses: A case study on GLP-1 receptor agonists and all-cause mortality.
To answer this question, our co-founder Lefteris Teperikidis and colleagues conducted a validation study — applying ChatGPT to perform every step of a systematic review and meta-analysis on GLP-1 receptor agonists and all-cause mortality. This is a well-studied topic with several high-quality meta-analyses already published, making it an ideal test case for benchmarking AI performance.
What did ChatGPT do?
– Generated comprehensive search strategies for PubMed and Embase.
– Screened over 1500 abstracts, with decisions compared to two independent human reviewers.
– Extracted data from 12 eligible randomized trials with no discrepancies found after human review.
– Performed risk of bias assessments using Cochrane’s RoB 2.0 framework.
– Generated code to conduct the meta-analysis in R.
The results:
– ChatGPT and human reviewers reached near-perfect agreement during screening.
– ChatGPT’s meta-analysis result — Hazard Ratio for all-cause mortality of 0.86 (95% CI: 0.79 – 0.93) — closely matched the human-calculated result of 0.87 (95% CI: 0.82 – 0.91).
– Time savings were substantial, especially for screening and risk of bias assessments.
What does this mean for evidence synthesis?
This study confirms that AI-assisted systematic reviews can achieve human-level quality, while significantly reducing the time burden. By combining the efficiency of AI with the clinical judgment of human experts, we can build faster, scalable, and more transparent evidence synthesis workflows — exactly what Synthesa’s tools are designed to deliver.
The future of evidence-based medicine depends on scaling evidence synthesis to match the exponential growth of scientific literature. AI is no longer just an idea — it is already proving its value in real-world systematic reviews and meta-analyses.
Read the full publication here: https://pubmed.ncbi.nlm.nih.gov/39613702/
This is the third post in our “Research Behind Synthesa” series, where we showcase the research foundations driving our AI-powered tools for evidence synthesis. More to come soon.
