State attorneys general warns Microsoft, OpenAI, Google and other AI giants to correct ‘delusional’ output

Following a series of disturbing mental health incidents related to AI chatbots, a group of state attorneys general has written to top companies in the AI ​​industry, warning them that they risk violating state law if they don’t fix their “delusional output.”

The letter, signed by dozens of auditors from U.S. states and territories with the National Association of Attorneys General, calls on companies, including Microsoft, OpenAI, Google and 10 other large AI companies, to implement a variety of new internal safeguards to protect users. The letter also included Anthropic, Apple, Chai AI, Character Technologies, Luka, Meta, Nomi AI, Perplexity AI, Replika, and xAI.

The letter comes as a battle erupts between state and federal governments over AI regulation.

These safeguards include transparent third-party audits of extensive language models that look for signs of delusions or flattering thoughts, and new incident reporting procedures designed to notify users when chatbots produce psychologically harmful output. These third parties, including academic and civil society groups, should be allowed to “evaluate the system before release without retaliation and publish their findings without prior approval from the company,” the letter says.

“GenAI has the potential to change the way the world works in positive ways. But it also causes, and can cause, serious harm, especially to vulnerable populations,” the letter said, pointing to a number of well-known incidents in the past year where violence has been linked to excessive AI use, including suicides and murders. “In many of these incidents, the GenAI products produced sycophantic or delusional output that encouraged the user’s delusions or assured the user that they were not delusional.”

The AG also suggests that companies treat mental health incidents in the same way that technology companies treat cybersecurity incidents, with clear and transparent incident reporting policies and procedures.

Companies should develop and publish “a timeline for detecting and responding to sycophantic or delusional output,” the letter said. Similar to how they currently respond to data breaches, companies should “promptly, clearly, and directly notify users if they have been exposed to potentially harmful flattery or delusional output,” the letter says.

Another question is that companies develop “reasonable and appropriate safety tests” for GenAI models to “ensure that the models do not produce potentially harmful flattery or delusional output.” These tests should be conducted before the model is made available to the public, it added.

TechCrunch was unable to reach Google, Microsoft, and OpenAI for comment before publication. This article will be updated if we receive a response from the company.

Tech companies developing AI are far more well received at the federal level.

The Trump administration has been unashamedly pro-AI, with multiple attempts over the past year to pass a nationwide moratorium on state-level AI regulations. So far, these efforts have failed, in part because of pressure from state authorities.

Undeterred, President Trump announced on Monday that he plans to pass an executive order next week that would limit states’ ability to regulate AI. In a post on Truth Social, the president said he wants the CEO to stop AI from being “destroyed in its infancy.”