He asked AI to count carbs 27000 times. It couldn't give the same answer twice

Ask ChatGPT to estimate the carbs in your lunch. Now ask it again. And again. Five hundred times. You’d expect the same answer each time. It’s the same photo, the same model, the same question. But you won’t get the same answer. Not even close — and the differences are large enough to cause a hypoglycaemic emergency. That’s the central finding of a study I’ve just published as a preprint, and it has direct implications for anyone using AI-powered carb counting in a diabetes app. I submitted 13 food photographs — real meals, photographed on a phone, the way you’d actually use them — to four leading AI models: OpenAI GPT-5.4, Anthropic Claude Sonnet 4.6, Google Gemini 2.5 Pro and Google Gemini 3.1 Pro Preview. Each photo was sent over 500 times to each model. Same prompt every time. Same photo. Same settings. 26,904 queries in total. All at the lowest randomness setting these models offer. The prompt was adapted from the one used in the iAPS open-source automated insulin delivery system — it’s a real production prompt, not a toy example. Every model returned different carbohydrate estimates for the same photo across repeated queries. But the degree of disagreement varies enormously. Each dot is one of the 13 test images. The violin shape shows the spread. Claude’s variation clusters below 5% for most images; the Gemini models regularly exceed 10-20%. The worst case? The paella photo. Here’s what happened when I sent it to each model 500+ times: Every dot is one query. Same photo. Same prompt. Same model. Gemini 2.5 Pro’s estimates span from 55g to 484g — a 429g range, equivalent to 42.9 units of insulin at a 1:10 ICR. Claude’s estimates cluster tightly by comparison. 42.9 units of insulin from a single photo. That’s not a rounding error. That’s a potential fatality. When you take a photo in a diabetes app, you get one number back. You have absolutely no way to know whether you received a typical estimate or a tail-end outlier from a distribution you can’t see. For Claude, that single number is probably close to the model’s consensus. For Gemini 2.5 Pro, you could be anywhere on the map. Here’s one that should be easy. Two slices of thick white bread (carbs on the packet: 20g per slice) plus cheddar cheese (negligible carbs). Reference value: 40g. Simple, unambiguous, packet-label accuracy. Left: Three models independently converge on ~28g — consistently wrong by 12g. Right: GPT-5.4 estimates ~74g with high variability — wrong in the other direction by 34g. The red dashed line is the actual value. Three of four models — Claude, Gemini 2.5 Pro and Gemini 3.1 Pro — independently converge on approximately 28g for a 40g meal. 510 queries from Claude, CV of 0.3%, and every single one is 12g below the actual value. The bread is right there in the photo. The carb value is on the packet. This is the “precisely wrong” problem: high consistency doesn’t guarantee accuracy. A diabetes app user getting 28g every time would consistently underdose by ~1.2 units. GPT-5.4 goes the other way: mean estimate 74g, nearly double the reference, and highly variable on top of it. I found food identification errors in 8 of the 13 test images: Some of these misidentifications have modest nutritional impact. Others could change the carbohydrate estimate substantially. On the five images where I had the strongest reference values (packet labels and weighed portions), here’s how often each model’s individual queries would have pushed insulin doses into clinically dangerous territory: Green is safe ( 5U). Claude is the only model with no queries in the orange or red zones. Claude: 100% of queries in the safe or moderate zone. No single query would have caused more than a 2-unit insulin error. GPT-5.4: 37% of queries would cause a clinically significant insulin error (>2U). That’s more than one in three queries landing in the danger zone. Gemini 3.1 Pro Preview: 12% of queries would cause a clinically significant insulin error (>2U). Better than ChatGPT-5.4. Gemini 2.5 Pro: 12% of queries would cause a >5U error — the threshold associated with severe hypoglycaemia requiring third-party assistance. The study identifies two distinct failure modes: Systematic bias (chronic risk). All four models overestimate carbs on average, meaning the dominant direction of error is toward too much insulin and hypoglycaemia. GPT-5.4 averages +1.2 units overdose per meal on strong-reference foods. Three meals a day, that’s 3.6 units of extra insulin per day. Stochastic variability (acute risk). The within-image variation means a single unlucky query could produce a catastrophic outlier. Gemini 2.5 Pro’s worst single query on strong-reference data would have caused an 11.3 unit insulin overdose for a 34g meal. That’s a potential severe hypo. The prompt I used asks each model to return a confidence score (0 to 1) for every food ite

He asked AI to count carbs 27000 times. It couldn't give the same answer twice

Key takeaways

Also covered by

He asked AI to count carbs 27000 times. It couldn't give the same answer twice

Key takeaways

Also covered by

More in computer-science