Does grounding a consumer health AI product in medical research make it more reliable?

My Doctor Friend members have asked if answers could be more linked directly to medical research, oftentimes pointing to clinician tools like OpenEvidence or research-focused tools that are patient-facing, like Inciteful Med. At least four times in our app's history, we've considered adding a more formal literature search or grounding feature. But each time, even after getting quite far in the development process, we've decided to not launch this feature. There are lots of reasons for this. One is that My Doctor Friend's built-in web search already draws on medical research and other useful sources and cites them.

But after quite a long time, we're now planning to release a more formal literature search. But as we get ready to make this a more deliberate part of the experience, and as our existing members are considering making it a requirement for its answers, I thought it a good time to explain why we've hesitated to build this approach into the app. This post explores what grounding a consumer health product in research means for its reliability. Specifically, it explores when this grounding may simply make the answers look more reliable, when they're really not necessarily more reliable or easier to use.

What do we mean by an AI health companion?

An AI health companion is something a person can turn to repeatedly and often as their health and circumstances change, whether they are using it to log symptoms, understand test results, prepare for appointments, or even just get emotional support during a difficult night. Over time, these interactions form a relationship with the technology. The hope of products like this is that the accumulated context makes the support they can provide a lot more useful.

For many of our members, this relationship now stretches back well over a year. We have some members who have used it nearly every day, sometimes several times a day during an acute illness or hospitalization. We have some members that have conversation threads with more than a thousand turns, while other members spread their conversations across several threads and use My Doctor Friend more intermittently.

When someone asks a question about a treatment today, the answer may depend on a side effect reported last week, six months ago, or a concern raised even further back in the past. Now, connecting the answer to a paper can certainly help, but it likely only helps if it can connect that paper to the person and their particular concern.

But I want to ground this post as much as possible in the evidence. And so we have to talk about the state of the evidence. Unfortunately, much of the evidence that's used to assess medical AI, whether it's clinician-facing or consumer-facing, comes from benchmarks, which are standardized sets of cases or conversations that allow researchers to compare models or products. Now, they can test whether an answer is accurate or well explained, but most all of the benchmarks that we're going to discuss and most of them that exist primarily evaluate individual responses, or clinical tasks, or at most a few clinical encounters.

There is still a massive gap around the ongoing work that people like our members are doing to manage their health with AI tools. But to understand what citations might add to a health companion, we need to explore this gap a little more deeply. And then we can look at the studies that have directly tested searching for evidence and what it means to attach the evidence and references to an answer. ^{1, 2, 3, 4}

The companions are here, but the outcomes research is nowhere to be found.

OpenAI has reported that hundreds of millions of people worldwide are using ChatGPT for health and wellness questions each week. That's their estimate of a broad category, not a specific count of people who are in continuing health conversations. But I think most people recognize that adoption has moved incredibly quickly. ⁵

If we take another area with high standards in research, the pharmaceutical industry, the difference in the maturity of a science is quite stark. It's well known that a treatment can kill cancer cells in a petri dish without helping a person with cancer, or that an experiment that extends a mouse's life does not necessarily establish that we've cured aging in humans. In the realm of pharmaceutical research, we well understand the distance between in vitro and animal studies and some kind of claim we might want to make for humans.

Alas, the vast majority of evidence used to promote health AI, whether for use by clinicians or by patients, is much closer to that early laboratory stage (in vitro or animal study data) than to actual clinical outcome studies for patients who are using this for their health. We may have benchmarks that are slightly better designed than the ones we had one or two years ago. But in most material ways, the central questions about these tools are still unanswered. What happens to people using these tools? What does it mean for their health, for their health outcomes?

When we use a benchmark score to imply a real-world benefit that hasn't been measured, we are conflating research theater with real clinical research.

Now, I don't mean to be uncharitable to the very smart researchers who are studying the science. My Doctor Friend has a data use agreement with a couple of academic medical centers who are interested in studying AI. There are likely a lot of reasons why the research we're still seeing is so preliminary, and not to get into a digression, but scientific journals are so hungry to publish anything in this domain that there's a much stronger incentive to publish quick, rapid studies than careful, methodical research that takes time and a lot more money. But that's maybe a topic for a separate essay.

Now, a brief aside here: I'm a strong believer that people in glass houses shouldn't throw stones. And I'll be the first to admit that My Doctor Friend does not have the rigorous clinical evidence that I wish we had either. We're a startup with a very small team and a limited budget. We have data use and collaboration agreements with these academic medical centers because we hope they can more rigorously study the outcomes that we don't have the funding yet to study.

How closely do our benchmarks reflect real use of health AI companions?

Here's my assessment of how relevant or rigorous several prominent studies are with regard to how this technology is being used by patients. To be clear, these are my coverage ratings, not product rankings, not validated grades of overall research quality. Red reflects a major gap in coverage. It does not mean anything about AI failing in the study.

On a small screen, scroll the table sideways to see every column.

Green / Good: direct coverage. Yellow / Limited: partial, narrow, or short-term coverage. Red / Not tested: absent from the design. In the time column, red means a single response, workup, or encounter; yellow means repeated visits or a few weeks of use. None establishes reliability over many months.

Real life test means participants using their own health concerns. Long-term use is limited when the study covers only a few weeks or scripted visits. Health outcomes means measured health changes, not answer scores. Therabot's answer checks were safety reviews. Its outcome rating is yellow because it studied mental-health symptoms only, with follow-up through eight weeks.

Answering a few questions correctly ≠ a reliable health companion.

HealthBench originally studied a relatively small number of messages with an average of 2.6, a median of 1, and a maximum of 19. It grades a single response to a supplied prompt or history. The authors did explicitly acknowledge that it doesn't measure health outcomes. This benchmark will not get better if we make the questions harder or we make the grading more rigorous. ¹

Now, HealthBench Professional used 525 clinician-authored tasks and multiple rounds of review by physicians, but it still grades the response to a fixed clinician history. Its makers do acknowledge that in the future more work will need to be done to study tasks that unfold over hours or days. But of course, long-term clinical relationships also unfold over weeks, months, and even years. We are still a considerable distance from real patient relationships. ²

Microsoft's recent Science paper in October 2026 did move beyond a completed vignette, asking the AI to come up with the right questions and request the right tests. But again, its cases came from published reports and hospital records. This does not reflect how actual care unfolds with real patients. And again, this was a clinician-facing measurement, not a measurement of a tool meant to interact with patients as a health companion. ⁶

AMIE may be the best we have so far

AMIE is one of the most serious attempts we've seen to move beyond in vitro testing. They enrolled 114 adults at Beth Israel Deaconess Medical Center in Boston, and then had people complete chats with AI and actual urgent primary care appointments. Patients would describe their own symptoms before a real visit, up to five days before the visit. Physicians did watch the chat live through a video call with screen sharing and had rules for intervening, and then a treating clinician would receive the transcript and summary before their appointment. AMIE did ask follow-up questions, checked its understanding, and allowed questions until the patient chose to finish and move on.

The public author manuscript doesn't report an average or range of conversational turns. Another sign of strength was that researchers did review medical records eight weeks later to assess the diagnoses. Granted, this was follow-up on a single episode and didn't reflect eight weeks of ongoing use. Reassuringly, none of the chats required a formal safety stop, although one hallucination was noted. ^{4, 11}

AMIE had a separate disease management study which followed a hundred simulated cases across three visits, but again, this relied on patient actors, not real use. ³

Alphabet, Google's parent company, funded these clinical studies and I really do think they deserve credit for putting real money into research with actual patients and actual care. We should all be asking for the same and greater level of rigor from other frontier model makers. Nudge, nudge. ^{3, 4}

Prospective studies outside the clinic…but only for a few weeks.

The rest of the literature is pretty sparse. We could talk about a 2026 diary study that followed 25 patients in China for four weeks. A very small study. A selected sample and not really a strong measure of or strong study of clinical benefit. ⁹

There was a randomized controlled trial by Therabot, which looked at how people fared using a chatbot for four weeks versus being on a wait list. This focused exclusively on mental health symptoms. While it's laudable that they studied outcomes and had safety monitoring, for our specific question about whether literature search is beneficial, or whether a health companion relationship can work over many months, there's not much here we can rest on or rely on. There are encouraging signs that people are trying to study these models and this technology in actual clinical use when it's patient-facing, but obviously none of them closes the enormous gap around the ongoing broad use for a range of health issues over much longer periods of time. ¹⁰

Now back to references…so why do people want them?

With so much concern about AI confabulation, it's easy to understand the emotional appeal of references. People want responses that are grounded in science. They want them to be grounded in trustworthy medical sources and they want to be able to check the work. Doctors want the same thing. It's not an accident that OpenEvidence is as popular as it is with clinicians. ¹²

As someone who has used OpenEvidence and many other tools, I can tell you the difficult part is telling when you're getting that trustworthy support or something that just looks like it.

For those not familiar, OpenEvidence offers clinicians quick answers linked to medical sources, including licensed full-text content from publishers such as the JAMA Network and New England Journal of Medicine. These are some of the premier medical journals, and some of their content is behind expensive paywalls. ^{13, 14}

Inciteful Med offers similar service to patients and they have published a description of how it finds research for patients and caregivers. It says it expands a question into 20 or more related clinical questions and searches a daily refreshed PubMed index. Its FAQ describes using connections between papers and the medical context you provide to find relevant research. ^{15, 16}

They've also reported developmental testing with nearly a hundred physicians and specialists, but this public account doesn't really give us an accurate number of cases, any kind of formal testing protocol or comparison group. Similar to OpenAI's claims of working or collaborating with hundreds of physicians, it gives us a description of the work being done without actual published evidence showing how much this helps or if it reliably helps patients over time. ^{1, 17}

In this regard, I think patients and clinicians often want the same thing, which is a well-cited answer, and ideally one they can get quickly. They want to feel that a recommendation has something solid behind it. Partnerships or pulling from respected medical journals can be a strong signal of credibility.

And for a long time I've thought that part of the appeal of these tools, whether it's OpenEvidence or Inciteful Med, is emotional. Seeing the links and the citations looks reassuring, even when it doesn't necessarily establish clinical efficacy.

Well, all of this seems like a slam dunk for including references, no? Well, not so fast.

Why answers with references may not be as good as you think

A good paper can be the wrong paper for you.

Imagine asking whether a treatment is likely to help you. The AI finds a large, well-conducted trial with an encouraging result. Then we learn that the trial excluded people with your kidney disease, your other medications, or your previous treatment history.

The paper can be good. Its relevance to you can still be uncertain.

Citing a paper doesn't mean the whole paper was accessed or read

Many of the tools out there that do reference the literature don't necessarily read the entire study. They may just be reading the abstract alone. This is especially the case for papers that are behind a paywall. So oftentimes the details that we may need about the population studied, eligibility criteria, etc., are behind a paywall that the model doesn't have access to. This is not to mention the already well-documented bias of our medical literature.

There are also gaps in who gets studied. A 2026 review of US Alzheimer drug trials found that nearly half of the 71 trials with publicly available results didn't report race or ethnicity. Among trials reporting White enrollment, the median was 91.3%. That finding concerns a particular disease and trial population, but it illustrates why a prestigious citation doesn't settle applicability. ¹⁸

Not all research gets published

A literature search can only find what has been made available. A 2025 study followed 347 randomized trials approved by ethics committees in four countries in 2016. By follow-up, 71 trials, about one in five, had no results available in either a publication or a trial registry. This was one defined group of trials, not an estimate for every field of medicine. ¹⁹

There is a known bias in the literature to publish positive results. And so in general, our corpus of medical literature gives a distorted picture of the science. Even a more thorough search or more complete access to full literature doesn't make up for this.

Bogus papers and outdated science stay online

Kind of obvious, but it still needs to be said. Many published papers turn out to be wrong. Some are formally retracted, but many more are just weakened by later evidence, while some just remain technically accurate while becoming much, much less useful for current care. There's no universal label attached to every outdated paper or conclusion saying something like, "You shouldn't rely on this anymore."

Even a formal warning can be missed. A 2026 study tested free versions of AI tools on 15 retracted papers selected from Retraction Watch: the ten most cited and five recent retractions. ChatGPT 5 handled all five research tasks correctly for only eight of them. That's a small test, not an error rate for ordinary medical conversations. It does show that using a newer model doesn't automatically solve evidence-status checking. ²⁰

And that's not to say that the newest paper is automatically the best paper either. Sometimes they may seem to overturn older findings only to themselves be overturned shortly thereafter when the results are not reproducible. One other way to look at all of this is that there's just a lot of uncertainty in the scientific literature. In so many ways, it's the map of what we've learned, the mistakes we've made along the way. It is not, without very careful curation, a foundation of scientific truth.

Conversations, especially longer ones, can pull search off course.

In a 2026 paper, Jung and colleagues described a simple experiment. If you add more conversation before a medical question, does it change how well guidelines and references are chosen? ²¹

Using GPT-5-mini, they tried searching a cardiology guideline before answering a question about blood-thinning medications after someone gets bypass surgery. In the first arm, there's no prior conversation and they tested it a hundred times. In the second arm, they generated ten question-and-answer pairs about related topics and then had it search the guideline again. Now, without the conversation, only five of a hundred answers included claims that were not supported by the passages that the system found, but with the added question-and-answer pairs, in other words, conversations, 40 out of 100 included claims that were not supported by the passages the system found. ²¹

Now, some of this may have come down to how their system was accessing references, but I think it's a good demonstration of a failure mode that I myself have seen across the board with any AI systems that search for references, which is that they can get distracted by conversation and many of the existing search systems for citations still rely on either semantic search or other techniques, technologies that can be slightly brittle. I've seen OpenEvidence even get off course after long conversations and start to bring in suboptimal references or just in general seen reasoning break down.

HealthBench Professional gives mixed results

Another test gives us perhaps a more mixed answer. OpenAI's 2026 HealthBench Professional tested GPT-5.4 alone, with basic web browsing, and inside its complete clinician product. On medical-research tasks, its scores were 58.1 without browsing and 66.5 with browsing. Across the whole benchmark, they were 48.1 and 45.8, respectively. These are adjusted grading scores, not percentages of correct diagnoses. The complete clinician system scored 59 overall, but added more than browsing. On medical-research tasks, it scored 67, with no statistically significant difference from browsing alone, which suggests that maybe the task and surrounding software do shape the result. It does not establish a benefit for literature search in an ongoing health companion. ²

The model may already know something useful

One of my biggest sources of hesitation is that I believe models may bring more to a question than what they're able to pull from papers. Granted, this is an unstudied hypothesis, but I believe that the training and post-training processes for models give them access to pattern recognition across many kinds of clinical problems. And I often worry that forcing every answer into a narrow set of retrieved sources may be getting in the way.

There's a parallel that supports this thinking, at least with human cognition. Human clinicians have been described as using different systems, especially around diagnostic thinking. On the one hand, there's the lightning-fast pattern recognition that expert and experienced clinicians use right away. And then there's the System 2 thinking, which is slower, more deliberative, more scientific, maybe more hypothesis-driven, that isn't always as accurate, but which often serves as a backup system when that first system fails. We can give credit here to Daniel Kahneman for helping popularize this distinction. ²²

Now, whether this is due to these models pulling in data from online forums or other non-traditional sources of health data, I'm not sure. I haven't found evidence that isolates this contribution. I do think that this is a question we should keep open, that while the literature may correct the model, it's not a foregone conclusion that it's going to improve its performance across all the things we may want it to do.

Do citations make us think less?

Imagine you receive an answer to a health question with 30 citations attached. Do you think it makes you more likely to check the reasoning, or does it make you feel like the checking has already been done?

My worry is that the same research theater we're seeing above to justify broad deployment of these models can also happen within a given AI's answer.

References can make output seem more dependable without necessarily helping someone judge whether it really is.

There's a practical burden here too. If you get 13 references and you decide you want to do the work, that's a lot more work. I have to wonder what the burden is of having to verify all those extra citations and references. Now, if citations make it easier for someone to challenge an answer, they of course could be extremely helpful. But just to state it again, if citations make an answer more persuasive without making it more dependable, they could leave someone worse off.

What can we actually say about OpenEvidence

I keep coming back to OpenEvidence because it's a widely used, citation-linked generator of answers to health questions. ¹² To be clear, I know that OpenEvidence has not been designed to be a health companion, and I've told many people, many times, that OpenEvidence is ill-equipped for this purpose. In my experience, it is not good with long conversational threads. I am not alone in identifying this failure mode with this admittedly wonderful tool that I enjoy using in my clinical practice.

There are many published evaluations of OpenEvidence, but they absolutely should not be confused with evidence that it would improve patients' health if patients could use it directly. ^{7, 8} Frankly, I haven't found a published study showing that its use by clinicians has improved patients' health. Of course, I'm aware that this isn't evidence that the product lacks benefit or would lack benefit. It just means that there again is a massive gap in our knowledge here. And back to our glass houses, the same distinction or the same critique would apply to My Doctor Friend. Daily use, long conversations, and people who value a product do not establish clinical benefit.

So why are we going to try a more formal literature search?

We listen closely to what our members are asking for. People are clearly asking for more direct access to the literature. My hesitation shouldn't settle this choice for everyone.

We're planning to test this option with the hopes that access to the right paper, guideline, or drug information may provide better support. And it may give members a better way to examine the evidence behind an answer, especially when it comes to consequential decisions.

This is why we're approaching this choice as transparently as we can. That's why I'm writing this essay to make some of these limitations visible. We also plan to link this feature to an extensive FAQ.

We plan as best as possible to keep the search connected to the person making it, to what they're trying to figure out, to what has changed, to what they've already tried, and what they need to solve their particular problem.

My hope is that our academic collaborators will help study what happens when literature search becomes part of the ongoing experience of using My Doctor Friend, including how citations affect people's judgment. We desperately need research that follows the work that patients are doing, most importantly, the work that they're doing outside of clinic, which, as I've said many times before, is most of the work to be done.

We're looking forward to hearing what our members and others think about this new feature. If it doesn't work, we're not afraid to remove it.

How this was written

This piece was written with ChatGPT Astra on the Pro setting. I also asked Claude to check the references. Given the criticism of "AI slop," I want to be clear about how I write this and other pieces: I dictate my thoughts directly to the AI. Much of this starts as voice dictation, with AI helping me find references when needed and phrase certain ideas more clearly. The ideas, and many of the sentences, phrases, and words, are my own.

Sources and scope

This post focuses on research published in 2025 or later, reviewed through October 30, 2026. MDF use patterns in the opening are founder observations, not audited usage statistics or outcome findings. The scorecard is an editorial assessment of study coverage, not a validated quality scale. Current-model performance claims use newer systems; Therabot illustrates an outcome-study design rather than current model performance. The main AMIE actor study also used an earlier model. Preprints are labeled; the patient-diary study was read in its preprint version. This is a targeted review, not a systematic review, and the gap concerns ongoing general medical support rather than every application of AI in healthcare. The AMIE clinic description draws on the final published abstract, the full author preprint (v3) and appendix, and the authors' research blog; the final journal full text was unavailable. SDBench's design is described from the authors' documentation. Product descriptions and adoption figures come from the companies or their publishing partners. Inciteful Med's methodology and development testing are described from its public documentation, not an independent product evaluation.

References

1. Arora RK, Wei J, Hicks RS, et al. HealthBench: Evaluating Large Language Models Towards Improved Human Health. arXiv. 2025;2505.08775. Original technical report. doi:10.48550/arXiv.2505.08775.

2. Hicks RS, Trofimov M, Lim D, et al. HealthBench Professional: Evaluating Large Language Models on Real Clinician Chats. arXiv. 2026;2604.27470. Research report and preprint. doi:10.48550/arXiv.2604.27470.

3. Liévin V, Palepu A, Weng WH, et al. Towards conversational artificial intelligence for disease management. Nature. 2026;655:1292–1299. Final published article and supplementary methods and results.

4. Brodeur PG, Koshy JM, Palepu A, et al. Conversational diagnostic artificial intelligence in ambulatory primary care: a prospective feasibility study. The Lancet. 2026;408(10565):1732–1744. Published online October 8, 2026. Final published abstract and full author preprint with appendix consulted; final journal full text unavailable. Final author abstract. Author preprint v3 (March 15, 2026).

5. OpenAI. Introducing ChatGPT Health. Company announcement. January 7, 2026. Accessed October 26, 2026. Adoption estimate reported by the company.

6. Nori H, Daswani M, Kelly C, et al. Sequential diagnosis with language models. Science. 2026;394(6822):423–428. Published October 22, 2026. Study design described from the authors' SDBench documentation.

7. Vishwanath K, Alyakin A, Ghosh M, et al. General-purpose large language models outperform specialized clinical AI tools on medical benchmarks. Nature Medicine. 2026;32:2405–2409.

8. Feng J, Patel V, Heagerty P, et al. Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries. arXiv. 2026;2606.28960. Preprint.

9. Cao Y, Ji Y, Fu Y, et al. More than Decision Support: Exploring Patients' Longitudinal Usage of Large Language Models in Real-World Healthcare-Seeking Journeys. Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems. April 13, 2026. Preprint consulted: arXiv v1, posted February 16, 2026.

10. Heinz MV, Mackin DM, Trudeau BM, et al. Randomized Trial of a Generative AI Chatbot for Mental Health Treatment. NEJM AI. 2025;2(4). Published March 27, 2025.

11. Schaekermann M, Karthikesalingam A. Exploring the feasibility of conversational diagnostic AI in a real-world clinical study. Google Research. March 11, 2026; updated October 8, 2026.

12. Patel VR, Finlayson SG, Liu-Hass M, et al. Clinician Use of an Artificial Intelligence-Enabled Clinical Information Platform. JAMA. Published online October 15, 2026.

13. JAMA Network. OpenEvidence and the JAMA Network sign strategic content agreement. Publisher announcement. June 5, 2025.

14. Springer Nature. Springer Nature and OpenEvidence announce agreement to maximise exposure of trusted findings on OpenEvidence platform. Publisher announcement. August 4, 2026.

15. Inciteful Med. How Inciteful Med Works. Company description. Accessed October 26, 2026.

16. Inciteful Med. How is Inciteful Med different than ChatGPT or an LLM? Company methodology FAQ. Accessed October 26, 2026.

17. Inciteful Med. Do doctors validate the results? Is the tool trustworthy? Company development-testing FAQ. Accessed October 26, 2026.

18. Lin Z, Sun R, Ross JS, Lau K, Stumpf S, Chen X. Racial and Ethnic Reporting and Representation in US Alzheimer Clinical Trials: A Systematic Review. JAMA Network Open. 2026;9(3):e262427.

19. Speich B, Taji Heravi A, Schönenberger CM, et al. Nonregistration, Discontinuation, and Nonpublication of Randomized Trials: A Systematic Review. JAMA Network Open. 2025;8(9):e2524440.

20. Labenbacher S, Niederer M, Hammer S, Bader M, Schreiber N, Bornemann-Cimenti H. Performance of AI Tools in Citing Retracted Literature: Content Analysis. Journal of Medical Internet Research. 2026;28:e88766.

21. Jung HJ, Cho K, Jun TJ, Kim YH. RAG in clinical practice: a cautionary tale of AI 'Truthfulness'. npj Health Systems. 2026;3:57.

22. De Neys W. Defining deliberation for dual-process models of reasoning. Nature Reviews Psychology. 2025;4:544–552. Author manuscript consulted. This is a theoretical perspective on human reasoning. Author manuscript.