One of the new flagship AI model meta released on Saturday, Maverick ranks second in the LM Arena. This is a test in which a human evaluator compares the output of the model and selects preferences. However, the version of Maverick that Meta deployed in LM Arena appears to be different from the version widely available to developers.
As some AI researchers pointed out in X, Meta said that Maverick of LM Arena has announced that it is an “experimental chat version.” Meanwhile, the chart on the official Llama website reveals that Meta’s LM Arena test was conducted using “Llama 4 Maverick optimized for conversation.”
As I wrote before, for a variety of reasons, LM arena was not the most reliable measure of AI models’ performance. However, AI companies generally do not customize or tweak their models, or at least allow them to do so, in order to score better at LM Arena.
The problem with adjusting the model to its benchmark, withholding it, then releasing a “vanilla” variant of the same model is that it becomes difficult for developers to accurately predict the performance of the model in a given context. That’s also misleading. Ideally, the benchmark is as badly insufficient as it is – providing a snapshot of the advantages and disadvantages of a single model across a variety of tasks.
In fact, X researchers have observed significant differences in the behavior of publicly available Mavericks compared to models hosted at LM Arena. The LM Arena version seems to use a lot of emojis and provide a very long answer.
OK llama4 is a lol with def cooked.
– Nathan Lambert (@natolambert) April 6, 2025
For some reason, the Arena Lama 4 model uses more emojis
together. ai, it seems better: pic.twitter.com/f74odx4ztt
– Tech Dev Notes (@techdevnotes) April 6, 2025
For comments, we contacted Chatbot Arena with Meta, the organization that maintains LM Arena.
