AI Signal 447
AI Observatory aggregates 24,521 conversations to expose blind spots in company usage reports
↗
A multi-institution research project called the AI Observatory analyzed consent-collected AI conversations across seven datasets and found that company-published usage reports filter out nearly half of real-world interactions, particularly personal and sensitive ones.
Engineers and product teams relying on vendor-published usage reports to understand how people interact with AI models are working with a skewed picture. The AI Observatory's independent analysis shows that nearly half of real conversations fall outside the categories companies like Anthropic choose to report, meaning product and safety decisions may be built on incomplete data.
Written by elseif from the cluster below · every claim links back to a sourceThe three things worth knowing
The AI Observatory aggregated 24,521 conversations from seven existing datasets collected with user consent, covering models from Anthropic, OpenAI, Google, and xAI between 2023 and 2025.
When researchers applied Anthropic's own filtering methods to their dataset, 48% of conversations would have been excluded, with the filtered-out content disproportionately involving health, relationships, harassment, hate, and sexual content.
Usage patterns varied significantly by model and version, with Grok concentrating misinformation, Anthropic used more for coding, Gemini for social and roleplay, and ChatGPT conversations growing longer and more iterative with GPT-4o compared to GPT-3.5.
THE CLUSTER