OpenWALDO project launches open-source AI training dataset with 167B transparent tokens

OpenWALDO is an attempt to replicate the open-source software model for AI training data. The project, led by CentOS and Rocky Linux founder Gregory Kurtzer, offers a dataset of 167.3 billion tokens sourced from government records, academic papers, and public domain literature. This contrasts with proprietary datasets, which often rely on undisclosed, copyrighted, or user-generated content. The goal is to create a transparent, auditable foundation for AI training that anyone can contribute to or build upon.

The dataset’s scale is its biggest limitation. At 167.3 billion tokens, OpenWALDO is dwarfed by the tens of trillions of tokens used to train frontier AI models. This gap means the project is unlikely to replace proprietary datasets in the near term. However, its value lies in its collaborative potential: improvements to the dataset could benefit all future models trained on it, reducing redundant effort across the industry. The project’s success hinges on whether the AI community embraces this model of shared development.

Transparency is the core promise of OpenWALDO. Proprietary training data often lacks clear licensing, consent, or provenance, which can introduce legal and ethical risks for users. OpenWALDO addresses this by providing a verifiable bill of materials for its dataset, allowing developers to trace the origins of the data they use. This could mitigate risks like copyright infringement or biased outputs, but it also requires users to trust the project’s curation process and the quality of its sources.

Adoption of OpenWALDO faces significant hurdles. The AI industry has already invested heavily in closed training pipelines, and shifting to an open model would require cultural and technical adjustments. Companies may be reluctant to rely on a shared dataset for competitive reasons, even if it reduces costs. Additionally, the project’s long-term sustainability depends on community contributions, which are not guaranteed. If OpenWALDO gains traction, it could pressure proprietary providers to improve transparency, but its immediate impact is likely limited to niche use cases.