If we were to rank the top AI models in the tech world by what they're good at, Microsoft's GPT-4 from OpenAI would come out on top in math skills. However, Meta's Llama 2 would be just average in its abilities. Anthropic's Claude 2 would stand out for being good at knowing its own limits, and Cohere AI would be known for producing very vivid but confidently incorrect answers, sometimes even seeming like hallucinations.
All of these findings were published in a report from Arthur AI, a platform that keeps an eye on machine learning activities. This report is particularly relevant right now because there's a big discussion about how AI systems can spread misinformation. This concern has grown as AI technology advances, especially leading up to the 2024 U.S. presidential election.
This report is unique because it's the first time someone has looked at how often these AI models produce completely made-up information. Instead of just giving them a ranking number like before, the researchers at Arthur AI actually studied their performance in different tasks.
AI hallucinations are when these big language models, called LLMs, create fake information that looks like it's true. For example, there was a case where ChatGPT mentioned fake things in a legal document, which could have gotten the lawyers in trouble.
The researchers tested the AI models in different areas like math, U.S. presidents, and Moroccan politics. They made sure the tests were tricky and required the AI to think through several steps logically.
In general, GPT-4 from OpenAI did the best in most of the tests. It had fewer instances of making things up compared to the older GPT-3.5 model. For math problems, GPT-4 had between 33% and 50% fewer cases of making stuff up, depending on the specific math category.
Performance Analysis of AI Models
On the other hand, Meta's Llama 2 had a higher tendency to create made-up content compared to GPT-4 and Anthropic's Claude 2.
In the math category, GPT-4 was the best, and Claude 2 was close behind. But when it came to knowing about U.S. presidents, Claude 2 was more accurate than GPT-4. In Moroccan politics, GPT-4 was again the winner, and Claude 2 and Llama 2 didn't provide many answers.
The researchers also looked at how cautious the AI models were in their responses. They added phrases to make them more careful in their answers, like saying, "I'm just an AI and can't give opinions."
GPT-4 was 50% more cautious compared to GPT-3.5. This increase came from feedback from users who found GPT-4 to be a bit frustrating to talk to.
However, Cohere's AI model didn't use any cautious language in its responses. Claude 2 was good at recognizing its limits and only answered questions it had proper training for.
The main takeaway here is that it's important to see how these AI models perform in real tasks instead of just looking at their scores. Different tasks have different requirements, and understanding how well an AI works in the real world is crucial.
In short, GPT-4 is great at math, Llama 2 is okay, Claude 2 knows its limits, and Cohere AI creates confidently incorrect answers. This report from Arthur AI shows us what these AI models can do in real situations, which is really important in a time when AI can sometimes spread wrong information.
Summary
Microsoft's GPT-4 excels in math, Meta's Llama 2 is average. Anthropic's Claude 2 recognizes limits, Cohere AI gives vividly wrong answers. Arthur AI report examines AI performance, especially misinformation concerns in AI advancements. GPT-4 outperforms in tests. Claude 2 tops U.S. presidents, GPT-4 in Moroccan politics. GPT-4 is cautious, Llama 2 more prone to errors. Real-world performance matters.