TECH Signal 488
Why Large Language Models Fail at Tabular Prediction
↗
A recent study shows that large language models lose predictive accuracy on tabular data as the number of features grows, while traditional models remain stable or improve.
Engineers often reach for LLMs as a one-size-fits-all solution, but this work demonstrates a fundamental limitation for high-dimensional tabular workloads. Switching to or augmenting with classical tabular models may be necessary to maintain performance, especially when feature sets cannot be drastically reduced.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Four common suspects, noise handling, CSV formatting, numeric tokenisation, and batch size, were experimentally disproved as causes of LLM failure on tables.
The decisive factor is input dimensionality: the LLM’s accuracy declines as the number of columns increases, unlike nine baseline methods that stay flat or get better.
In low-dimensional settings the LLM behaves like a local distance-based predictor, but its high-dimensional predictions cannot be replicated by any tuned classical model.
THE READ
What elseif makes of it.
The authors evaluated a cutting-edge language model by feeding it an entire training and test set in a single prompt, without any fine-tuning, external tools, or agentic scaffolding. This pure inference regime isolates the model's inherent ability to learn from tabular data. By comparing its performance across thirty-one benchmark datasets, they could directly measure how well the LLM handles typical predictive analytics tasks. Five plausible explanations for the poor results were put to the test. Experiments showed that the model can cope with noisy and non-linearly separable data, that the flat CSV representation does not hide column relationships, that tokenising numbers does not cripple learning, and that the number of test points per query is not the limiting factor. Each of these hypotheses was falsified, narrowing the search to the remaining variable. Dimensionality emerged as the critical issue. When the authors applied random linear projections to increase feature count, the LLM was the only method among nine competitors whose accuracy systematically fell. Classical baselines either maintained their performance or improved, indicating that the LLM’s internal repr
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER