Startups train large action models on videogames and simulations to control robots

Startups like General Intuition are exploring large action models, also called world models, to train robots using synthetic data from videogames and simulations. This method avoids the need for real-world data collection, which is often slow, costly, and risky. By simulating environments, engineers can test robotic behaviors in edge cases without physical prototypes or field trials. The approach mirrors techniques used in autonomous vehicle development, where virtual testing complements real-world validation.

The shift to synthetic training data could lower barriers for robotics adoption in industries like logistics, manufacturing, and disaster response. Physical training requires expensive hardware, controlled environments, and repeated iterations, all of which add time and cost. Simulations, however, allow for parallel testing of thousands of scenarios at once. This scalability may enable faster iteration cycles, but the fidelity of synthetic data remains a limitation, real-world physics and unpredictability are hard to replicate perfectly.

While the concept is promising, its effectiveness depends on the quality of the simulation and the model’s ability to generalize to real-world conditions. Videogames and simulations often simplify physics or omit variables like friction, lighting, or material properties, which can lead to gaps between virtual and physical performance. Startups will need to validate their models in controlled real-world settings before deployment, adding a layer of complexity to the development process.

The broader implication is a potential decoupling of robotic training from physical hardware, at least in early stages. If large action models prove reliable, they could democratize robotics development by reducing the need for specialized facilities or large datasets. However, the approach is still experimental, and its success will hinge on whether synthetic training can match or exceed the robustness of traditional methods.