AI Signal 480
Exploring Claude/GPT Knowledge Cutoffs and Pre-Training Timelines
↗
The post describes how probing language models with targeted queries can reveal estimates of their knowledge cutoffs, model size, training data mixture, and pre-training timeline.
Engineers who rely on LLMs need to know how current the model’s knowledge is and what data shaped it, because this affects reliability for time-sensitive tasks. Understanding the split between pre-training, capability tuning and persona alignment helps anticipate where a model will excel or require additional fine-tuning. The probing approach also shows the limits of public disclosures, highlighting when internal training dates may differ from published cutoffs.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
Incompressible knowledge probes and data mixture inference are used to approximate model parameters and infer the composition of pre-training data.
Error-rate analysis on a daily-fact quiz yields an estimated knowledge cutoff that aligns with published dates for some models.
The analysis suggests Opus 4.7+ models share a late-2025 pre-training checkpoint while GPT-5.6 stems from a separate early-2026 checkpoint, though uncertainties remain.
THE READ
What the cluster adds up to.
The post outlines a method for probing large language models with carefully curated requests to infer hidden details about their training. By using incompressible knowledge probes, the author attempts to estimate model size from performance on niche facts. Data mixture inference is applied by analysing how models break down tokens to guess the composition of their pre-training data. Date-oriented or self-identification queries are used to approximate when the model’s training data stopped being ingested.
The author first reviews the converged three-stage pipeline for building frontier LLMs. Stage one is a massive pre-training run on scraped internet data to produce a base checkpoint that often corresponds to a major version bump. Stage two uses domain-specific, textbook-quality data to improve capabilities such as long-text understanding. Stage three turns the base model into an assistant persona, refining personality, reasoning and tool-calling behaviour.
To locate the pre-training checkpoint date, the author built a daily-fact quiz drawn from Wikipedia and gave each model an eight-way multiple-choice version. By tracking the error rate across dates, the point where performance drops indicates when the model loses signal from its training set. The midpoint of the resulting curve is taken as the estimated knowledge cutoff, avoiding bias from over- or under-sampling of recent facts. For GPT-5.4 this estimate matches the publicly disclosed cutoff from OpenAI.
Applying the same analysis across models leads to the speculation that the Opus 4.7 family and later share a common training run that ends around late December 2025, suggesting a unified pre-training checkpoint. In contrast, the GPT-5.6 line appears to stem from a separate checkpoint that finishes around late February 2026, giving it a distinct effective knowledge cutoff. The author notes that some models, like Luna, show a high error rate even at low reasoning effort, which can be mistaken for future-prediction ability. The discussion ends with an incomplete remark about Opus 5, indicating the analysis remains tentative.
Written by elseif from the cluster below · checked for specifics the sources never containedTHE CLUSTER