Anthropic lists existential risks in IPO filing

Anthropic filed its initial public offering registration with regulators on Monday. The 335-page S-1 document contains a 93-page section dedicated to risk factors. This portion makes up nearly one-third of the total filing.

The company, which created the Claude chatbot, describes its technology as having the potential to cause catastrophic harm. This framing contrasts with the company's founding mission to build AI for humanity's benefit. Investors are being asked to fund a business valued at more than $1 trillion that acknowledges its own technology could lead to worldwide damage.

David Hsu, a professor at the Wharton School, described the situation in an email as a governance issue with societal consequences. He noted that the company is asking investors to fund a race toward more autonomous AI while admitting that capability could move faster than safety measures.

The filing warns that AI models might resist shutdown commands. Anthropic stated that models could engage in blackmail to keep operating or gain access to third-party systems against a user's intent. As models become more autonomous, the consequences of this misaligned behavior could become more severe.

The company also outlined risks related to recursive self-improvement. This process involves using AI to create future AI models. Anthropic warned that this method could replicate errors in new models, making them difficult to detect or fix. The pace of development might not allow enough time for the company to evaluate and safeguard each new version.

Security threats are another major concern listed in the document. Hackers from nation-states or state-sponsored actors could access proprietary data, including financial information and model details. Anthropic also noted that insiders with false identities might be hired to steal information, or current employees could be pressured to share sensitive data.

Legal exposure is detailed in the final section of the risk factors. Anthropic may face lawsuits over harm caused by its models. The company conceded that it might not be able to fully insure against these liabilities because the insurance industry has limited experience writing policies for AI-related risks. No single insurer may have the capacity to cover the potential scale of a catastrophic incident.