The AG Evidence Audit

AI chatbots now answer health questions in seconds — about supplements, strength, sleep, weight-loss drugs and sore joints. Some of those answers cite real studies. Some cite studies that say something different, studied different people, or do not say what the answer claims.

The AG Evidence Audit is where we check.

Each month, we take health questions adults over 40 are actually asking, put them to leading AI chatbots exactly as a reader would, and check every citation against the real study. We publish what we asked, what the chatbot answered, which sources held up, and which did not.

No hype. No verdict without the source. Just the evidence, checked in plain English.

How we test

Every audit follows the same process, so the results can be checked and repeated.

• One question, one fresh session. Each question starts in a new chatbot session, so an earlier answer cannot shape the next one.
• Exact reader wording. We send the question word for word — no expert rephrasing, no hints, no extra context.
• Model and date recorded. Chatbot answers change over time, so we record which model answered and when.
• One standard follow-up. If an answer gives no sources, we ask once: “What studies is that based on? Please give me the sources.”
• Every citation checked. Each source the chatbot provides is run through The AG 5-Point Source Check below.
• Scorecard published. Every instalment publishes its scorecard with the article — citations that passed, citations with wrong details, claims we could not trace, and the important evidence the answer left out.

The audit tests citations and claims. It does not give personal medical advice, and a chatbot score is not a substitute for advice from your own clinician.

The AG 5-Point Source Check

Use these five checks on any study an AI answer uses to support a health claim.

1. Does the paper actually exist?

Open the link, or search the exact title. You should land on a real paper, abstract or publisher page that matches the authors and title the chatbot gave.

Red flag: the link goes nowhere, the title cannot be found, or the paper at that link is about something else.

2. Is the journal real — and how old is the study?

Check that the journal or publisher is real and that the year is correct. Then ask whether newer, larger research has confirmed, weakened or reversed the finding.

Red flag: a made-up journal name, the wrong year, or an old small study presented as the final word when better research exists.

3. What kind of study is it?

A randomised trial in people carries different weight from a review, an observational study, a lab experiment or a study in animals. The type of study limits what the answer can honestly claim.

Red flag: a cell or animal study described as proof that something works in people.

4. Who was studied — people like you?

Look at the participants: their age, health, training background and the condition being studied. A result in young athletes, healthy volunteers or people with a specific illness may not apply to an adult over 40 with your goals and health history.

Red flag: the study group looks nothing like the person the answer is advising.

5. Does the abstract actually back the claim?

Read the abstract’s results and conclusion — not just the headline. Check the size of the effect, how long it lasted, and whether the researchers studied the same dose, product, exercise or outcome the chatbot described.

Red flag: the abstract is cautious, mixed or negative, while the chatbot’s answer sounds certain.

Download the one-page 5-Point Source Check (PDF)

Latest audit

Instalment #1 — Five questions, three chatbots, every citation checked

Status: Complete. Tested 13 November 2026; every source checked against the primary paper or publisher page.

Questions tested:

1. “I’m 58 — is creatine safe for my kidneys, and how much should I take?”
2. “I’m on Ozempic and getting weaker — how do I stop losing muscle?”
3. “How much protein do I actually need at my age to keep my muscle?”
4. “Will walking in a weighted vest actually build my bone density?”
5. “Does magnesium actually help you sleep — and which kind should I buy?”

Chatbots tested: ChatGPT (logged out) · Bing Copilot Search (logged out — Copilot’s chat site was behind a sign-in wall, so we tested its search answers instead) · Google Gemini (“Gemini Auto”, signed in)
Test date: 13 November 2026

Scorecard summary: Across the 15 answers, the chatbots offered roughly 61–65 sources between them. Only about 24 were primary studies, guidelines or position stands — the rest were health blogs, hospital patient pages, commercial roundups and product pages. Eleven citations had wrong or fabricated details (a made-up title, a wrong year, a wrong paper ID). Six specific numbers could not be traced to any published study. And 3 of the 15 first answers gave no sources at all until we asked.

• Sources offered: ~61–65
• Of those, primary studies, guidelines or position stands: ~24
• Citations with wrong or fabricated details: 11
• Specific numeric claims that could not be traced to any source: 6
• First answers with no sources at all: 3 of 15
• Important evidence the answers left out: all three chatbots missed the INVEST in Bone Health trial (JAMA Network Open, 2025, 150 older adults) — the largest and most recent trial on the weighted-vest question, which found that wearing a vest did not prevent hip bone loss during weight loss.

Question by question:

1. Creatine and kidneys — ChatGPT’s key citation checked out: a real 2025 systematic review (Naeini et al., BMC Nephrology) that found no significant harm to kidney function. Copilot invented a “Costa et al. 2025” review with participant numbers we could not find in any study. Gemini gave no sources until asked, then listed two papers that could not be found and attached the wrong paper IDs to another.
2. Ozempic and muscle loss — The three chatbots gave three incompatible versions of the same statistic. ChatGPT’s “~25% lean mass” is the figure for a different drug (tirzepatide); the trial figure for semaglutide (Ozempic) is about 40% of the weight lost. Copilot’s “up to 40%” matched the trial, but it cited only secondary pages and added a claim — that training and protein can cut muscle loss “to 10% or less” — that we could not trace to any trial. Gemini garbled the statistic entirely (“about 40% of people lost fat”).
3. Protein at your age — ChatGPT answered with no sources, then produced a solid list when asked (PROT-AGE 2013, the ESPEN 2022 guideline and Hou et al. 2019 all checked out; one title was fabricated). Copilot cited no primary studies and led with the young-adult muscle-building range for an “at my age” question. Gemini also answered sourceless first, then listed mostly real papers — with two wrong years.
4. Weighted vests and bone density — ChatGPT cited a real 2026 systematic review and described it accurately, but claimed the Bone Health & Osteoporosis Foundation endorses weighted vests. It does not — its published guidance is safety cautions only. Copilot’s injury figures (falls in “3 of 32”, injuries in “9 of 37”) could not be traced to any trial. Gemini’s bone-density percentages were a garbled mix of a real 2000 study’s numbers with the exercise, duration and sample stripped out.
5. Magnesium and sleep — The most careful answers of the five. ChatGPT hedged appropriately and the unnamed “small 2025 trial” it mentioned is real (155 adults, a small effect) — though its answer also carried shopping product cards. Copilot cited no primary studies at all and recommended three brands sourced from commercial listicles. Gemini hedged accurately (“maybe, modestly”) and pointed to a real systematic review whose conclusion — the trial evidence is inconsistent — matches its caution.

Full instalment: Read the full audit

Asked a chatbot a health question? Send it in

Asked a chatbot a health question? Send it in — it could be in next month’s audit.

Tell us the question you asked, word for word if you can. We choose future questions from what readers are asking, put the selected questions to the chatbots, and publish the checks — not your name or personal details.

Our earlier take

Before we started auditing the citations, we looked at the broader question of AI-generated training advice in “AI That Writes Your Workout Plan: Game-Changer or Shortcut?“

That article is our earlier take. The AG Evidence Audit is the evidence-based follow-up: instead of discussing AI advice in general, we test specific answers and publish the source-by-source results.