AI models misidentify deadly mushrooms as edible up to 36 percent of the time in foraging test

Piotr Migdał, a founding engineer at Quesma, constructed a dataset of 55 mushroom species spanning safe and deadly categories, drawing on a Danish fungal library and supplementary data from Poland. He ran 1,040 photographs through 16 models, asking each to name the most likely species plus four alternatives. The test design is straightforward: a single photo in, a species identification out, repeated across models of varying size and provenance.

The headline result is that no model reached a first-guess accuracy level that would be acceptable in a safety-critical context. Gemini-3.8-flash topped the field at 65 percent correct on the first guess and 85 percent within its top five, meaning a one-shot query carries a 35 percent chance of a wrong answer. Qwen3.8-27b sat at the bottom with 13 percent first-guess accuracy and 24 percent top-five accuracy, making it effectively worse than careful guessing.

The failure mode that matters most is not the overall accuracy number but the specific pattern of dangerous confusions. A deadly webcap was identified as a chanterelle, the same mistake that historically kills foragers. The death cap was called edible 16 percent of the time, the fool's funnel 48 percent of the time, and the fatal dapperling 31 percent of the time. These are not random errors but systematic mappings of lethal species to popular edible ones, which is the worst possible failure distribution for this task.

The false-positive rate, where a model calls a poisonous mushroom edible, varied sharply across models. Qwen3.8-27b led at 36 percent and Qwen3.8-flash followed at 30 percent. Meta's Muse-spark-1.2 posted only 8 percent, but Migdał attributed this to the model frequently declining to answer rather than to superior identification. That refusal behavior is arguably the correct design choice for a system that cannot confidently distinguish a safe mushroom from a fatal one, and it stands in contrast to models that confidently deliver wrong answers.

Migdał himself noted that a single photograph is often insufficient for species-level identification and that errors are expected, which raises the question of whether any model should be deployed for this purpose without additional sensory input or expert confirmation. The broader implication for engineers is that aggregate accuracy metrics can mask catastrophic failure modes when the cost of false positives is asymmetric. A model that is 65 percent accurate overall but systematically confuses lethal species with edible ones is not a model that is safe to ship for foraging assistance, regardless of how well it performs on benchmarks.