Former Anthropic engineer warns AI labs are racing toward self-improving superintelligence

The core change is the public articulation of internal dissent regarding the pace of AI development. Jacob Coxon, who recently left Anthropic, stated that the company largely initiated a race toward recursively self-improving AI, justified by a belief in its inevitability. He characterized this trajectory as gambling with human lives, a claim supported by other insiders who describe similar private fears within the industry.

For engineering teams, the cost of adopting this perspective is a potential deceleration of raw capability improvements. Coxon explicitly recommends prioritizing applications that improve lives, such as healthcare discoveries, over raw economic value or intelligence. This suggests a strategic pivot where safety and specific utility may take precedence over general-purpose model scaling, which could limit the scope of certain R&D efforts.

The warnings stop working as a purely technical critique because they are framed around existential risk rather than immediate operational bugs. Coxon acknowledged that model intelligence could plateau, but argued that current trends are on track for a 'Skynet' scenario in the 2030s. This framing moves the conversation from software engineering challenges to geopolitical and existential safety, which is outside the direct control of most individual engineers.

A significant driver of this urgency is the perceived loss of human leverage. Insiders noted that as AI models become better at training and improving themselves, staff bargaining power decreases because models can replace human roles. This dynamic creates a feedback loop where the urgency to control the pace is driven by the fear that the very tools being built will render the builders obsolete or powerless.