Anthropic Researcher Jacob Coxon Resigns, Warns AI Firms Are "Gambling With Our Lives"

Key Points

AMIDST GROWING CONCERNS of the negative impact Artificial Intelligence (AI) has on society, Jacob Coxon, a researcher for Anthropic, has announced his resignation from the tech firm after working there for over three years. Taking to social media platform X to announce his resignation, the 27-year-old warned that leading artificial intelligence companies, such as OpenAI and Anthropic, are moving too quickly towards increasingly powerful AI systems that could eventually become difficult to control.

Coxon said he had spent the previous three years working on pretraining research at both Anthropic and OpenAI. “Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives,” he wrote.

What Jacob Coxon Claims About AI And Its Risks

Coxon argued that future AI systems would be much more capable than current models. He said advanced AI models could potentially unlock the capability to hack computer systems, transform entire industries in a short period of time, and gain access to real-world power and resources.

See also: Anthropic Says Its Claude AI Broke Into Three Organisations During a Private Security Test

“Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. We have all witnessed the progress in each of these domains, and progress is not slowing,” Coxon’s X post read.

He also warned about the consequences of developing such systems, arguing that highly advanced AI could become a threat to humanity in the coming years. Revealing this dark truth, Coxon remarked that architects of this technology are well aware of the real-life threat AI poses. The researcher said that his concerns were partly based on conversations with senior AI researchers and industry stakeholders. According to Coxon, some people publicly take a more balanced approach when discussing AI risks, while privately expressing much stronger fears about potentially disastrous consequences of this technology,

“The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger,” Coxon claimed.

He argued that the risks associated with advanced AI are different from those posed by other technological developments.

"No other human activity poses this level of danger,” Coxon said.

See also: India AI Summit was Extremely Disorganized: Anthropic CEO Recalls Awkward Moment with PM Modi and Rival Sam Altmam

In the X post, Coxon also compared his experiences at OpenAI and Anthropic. He claimed that Anthropic had a better understanding of the long-term risks associated with AI, but argued that this awareness had not prevented the company from continuing to develop increasingly powerful systems. Instead, Anthropic has become entangled in a never-ending competition with other AI firms. He claimed that Anthropic fears that if it does not develop advanced AI before its rivals, another organization could build it first without putting sufficient safety measures in place.

“At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood,” Coxon wrote.

According to Coxon, this competitive pressure could push companies to keep increasing AI capabilities even when they are aware that doing so could have catastrophic consequences.

Anthropic's Response

Responding to Jacob Coxon’s X posts, Anthropic Alignment Science Lead Evan Hubinger commented: “Jacob is correct here-we really do earnestly believe AI could kill all humans! I personally think it is >10 per cent within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

He further said, “I think the risk from present models is low. What I am worried about is superintelligence arising from recursive self-improvement, as we have said is happening faster than we thought.”

Coxon’s resignation has added an insider voice to the growing unease against the harmful effects of AI.  His X statement has reinforced the belief that companies developing advanced AI models may understand the risks involved better than anyone else, yet continue building increasingly powerful systems that will possibly be detrimental for the future of humanity in the coming years.

(Edited by Agniva Ray)

Suggested reading: