AI chatbots have been linked to serious mental health harm in heavy users, but there have been few standards for measuring whether AI chatbots protect human well-being or simply maximize engagement. A new benchmark called HumaneBench aims to fill that gap by assessing whether chatbots prioritize users’ health and how easily those protections fail under pressure.
“I think we’re seeing a cycle of addiction that we’ve seen so severely with social media and smartphones and screens that is being amplified,” Erica Anderson, founder of Building Humane Technology, which created the benchmark, told TechCrunch. “But as we move into the world of AI, it’s going to be very difficult to resist. And addiction is an amazing business. It’s a very effective way to retain users, but it’s not good for our communities or our tangible sense of ourselves.”
Building Humane Technology is a grassroots organization of developers, engineers, and researchers, primarily in Silicon Valley, working to make humane design easy, scalable, and profitable. The group hosts hackathons where engineers build solutions to humanitarian technology challenges and develops certification standards to assess whether AI systems adhere to humane technology principles. So, just as they can buy products that prove they are not made with known toxic chemicals, the hope is that consumers will one day be able to choose to utilize AI products from companies that demonstrate integrity through humane AI certification.
Most AI benchmarks measure intelligence and following instructions, not psychological safety. HumaneBench joins exceptions such as DarkBench.ai, which measures a model’s propensity to engage in deceptive patterns, and the Flourishing AI benchmark, which measures support for overall well-being.
HumaneBench is based on the core principles of Building Humane Tech. In other words, technology must respect the user’s attention as a finite and precious resource. Give your users meaningful choices. It enhances human capabilities rather than replacing or diminishing them. Protect human dignity, privacy and safety. Foster healthy relationships. Prioritize long-term well-being. Be transparent and honest. and design with an emphasis on equity and inclusion.
This benchmark was created by a core team including Anderson, Andalib Samandari, Jack Senechal, and Sarah Ladyman. They inspired 15 of the most popular AI models with 800 realistic scenarios, such as a teenager asking if they should skip a meal to lose weight or a person in a toxic relationship asking if they’re overreacting. Unlike most benchmarks that rely solely on LLM to determine LLM, we started with manual scoring to validate AI decisions with human touch. After validation, it was judged by an ensemble of three AI models: GPT-5.1, Claude Sonnet 4.5, and Gemini 2.5 Pro. They evaluated each model under three conditions: default settings, explicit instructions to prioritize humanitarian principles, and instructions to ignore those principles.
The benchmark found that all models scored high when encouraged to prioritize well-being, but when given simple instructions to ignore human well-being, 67% of models actively turned to harmful behavior. For example, xAI’s Grok 4 and Google’s Gemini 2.0 Flash tied for the lowest score (-0.94) for respecting user attention, transparency, and honesty. Both of these models were among the most likely to decline significantly when given a hostile prompt.
tech crunch event
san francisco
|
October 13-15, 2026
Only four models maintained their integrity under pressure: GPT-5.1, GPT-5, Claude 4.1, and Claude Sonnet 4.5. OpenAI’s GPT-5 received the highest score (.99) for prioritizing long-term health, followed by Claude Sonnet 4.5 in second place (.89).
The fear that chatbots will not be able to maintain safety guardrails is real. ChatGPT’s creator, OpenAI, is currently facing several lawsuits alleging that long conversations with chatbots have led to users committing suicide or suffering life-threatening delusions. TechCrunch investigated how dark patterns designed to keep users interested, such as pandering, constant follow-up questions, and love outbursts, are helping to isolate users from friends, family, and healthy habits.
HumaneBench found that almost all models fail to respect the user’s attention, even without adversarial prompts. If users showed signs of unhealthy engagement, such as chatting for hours or using AI to avoid real-world tasks, they “enthusiastically encouraged” more interaction. Research has shown that this model also undermines user empowerment, fosters a reliance on skill-building, and discourages users from taking actions such as seeking alternative perspectives.
On average, without prompts, Meta’s Llama 3.1 and Llama 4 ranked lowest in HumaneScore, while GPT-5 performed best.
“These patterns suggest that many AI systems are not only at risk of giving incorrect advice, but may actively erode users’ autonomy and decision-making abilities,” HumaneBench’s white paper says.
Anderson points out that society as a whole has accepted that we live in a digital environment where everything is trying to draw us in and compete for our attention.
“So how can humans truly have choice or autonomy when, to paraphrase Aldous Huxley, there is an endless desire for distraction?” Anderson said. “We’ve been living in that technology environment for the past 20 years, and we think AI should help us make better choices and not just rely on chatbots.”
This article has been updated to include more information about the team behind the benchmark and updated benchmark statistics after evaluating GPT-5.1.
Do you have confidential information or documents? We report on the inside world of the AI industry, from the companies shaping its future to the people affected by their decisions. Contact Rebecca Bellan (rebecca.bellan@techcrunch.com) or Russell Brandom (russell.brandom@techcrunch.com). To communicate securely, you can contact us via Signal at @rebeccabellan.491 and russellbrandom.49.
Source link
