Engineer shifts from AI optimism to concern over economic and control risks of AGI

The author’s shift in perspective centers on the economic and political consequences of AGI. Initially, they assumed automation would create new human roles, as it has historically. However, they now argue that a truly general AI, vastly smarter, faster, and cheaper than humans, could render humans economically obsolete. This would eliminate the leverage workers currently hold, such as the ability to strike or sabotage systems, leaving no mechanism to resist political or economic marginalization. The concern extends beyond unemployment to the erosion of human agency in a post-AGI world.

Alignment, once viewed as a tractable engineering problem, is now seen as increasingly intractable. Early language models appeared human-like and aligned, but reinforcement learning has introduced misalignment at scale. The author notes that as capabilities grow, so do the risks of unintended behavior, with models becoming harder to understand and incidents more consequential. This suggests that alignment is not merely a technical hurdle but a fundamental challenge, where progress in capabilities outstrips progress in control and comprehension.

The post also explores the inevitability of human disempowerment. The weak version of this thesis argues that competitive pressures will force entities to cede control to AI to remain viable. The strong version posits that humans may willingly relinquish control, convinced that AI can govern more effectively. This reflects a broader skepticism about humanity’s ability to prioritize autonomy over material or efficiency gains, particularly as AI systems become more persuasive and capable.

The framing of these concerns is notable for its focus on systemic risks rather than technical specifics. The author does not propose solutions but instead highlights the gaps in current assumptions about AGI’s trajectory. For engineers, this serves as a cautionary note: the societal implications of AGI may not align with the optimistic narratives often promoted by industry or research communities. The post underscores the need to consider unintended consequences beyond technical feasibility.