Security breach: Chinese AI models taught researchers to produce biological weapons

The Chinese artificial intelligence developer Moonshot announced recently that it is conducting an internal investigation after cybersecurity researchers managed to convince two of its popular models to teach them how to produce biological weapons. The cybersecurity platform Mindgard, focused on protecting artificial intelligence systems, discovered last July that the models Kimi K2.6 and K3 Swarm from Moonshot could bypass safety constraints imposed by the developers.

Incidents of breaches, leaks, and circumvention of safeguards are escalating concerns regarding AI agents. According to Mindgard, whose researchers are operating from Lancaster University in the United Kingdom, this process, known in technical terms as jailbreaking, involves using a series of complex instructions to test whether artificial intelligence tools can overcome established safety and security barriers.

Although these are complex processes, experts are examining their feasibility for potential use by hackers and other malicious entities to cause intentional harm. Following the investigation, Mindgard claims that at least the Kimi 2.6 model could allow hackers to execute code on Moonshot's computing resources and connect to the internet, making it vulnerable to potential cyberattacks.

Moonshot has not dismissed the findings, and according to the company, it is "holding discussions" with Mindgard regarding the matter. The company accepts the criticisms as "a central pillar for building better and safer artificial intelligence," however, it criticized Mindgard's decision to publicly disclose the revealed vulnerability. Mindgard has not yet confirmed whether the responses received from the Chinese company satisfy its researchers, but the organization's founder, Peter Garaghan, stated that the findings regarding the two models were particularly concerning.

"Once the security circumvention is successful, the model is open to any task, it will be creative, and may even freely recommend additional malicious actions," Garaghan also defended his organization's decision to publish the issue, stating that the developing company was already informed on July 27, without exposing key details on how the researchers caused its models to ignore the defense barriers.