Why Anthropic calls the new Claude 3 its ‘most intelligent’ AI model yet
Context- Anthropic, an AI start-up founded by former OpenAI members, has launched a new family of AI models named Claude 3, which they claim sets new industry standards for a variety of cognitive tasks. The Claude 3 family consists of three models: Haiku, Sonnet, and Opus, each offering a balance of intelligence, speed, and cost tailored to their specific use cases. According to co-founder and president Daniela Amodei, these models are twice as likely to answer questions correctly compared to similar AI chatbots. Anthropic is also focusing on addressing the challenges businesses encounter when integrating AI into their workflows.
What is Claude 3?
- Claude is a series of large language models (LLMs) developed by Anthropic, capable of processing text, voice messages, and documents. The models are known for their fast, context-aware responses.
- The latest releases include Claude 3 Opus, the most powerful model, Claude 3 Sonnet, a capable and cost-effective model, and Claude 3 Haiku, designed for use cases requiring instant responses.
- Currently, Claude Sonnet powers the free Claude.ai chatbot, accessible with an email sign-in. Claude 3 Opus is available for $20 a month through Anthropic’s web chat interface for Claude Pro service subscribers.
- All models have a 200,000-token window, indicating potential for improved performance, accuracy, and the ability to handle more information in a user prompt.
How did Claude 3 perform?
- Anthropic’s Claude 3 models appear to be competitive with OpenAI’s models, even surpassing many with the launch of GPT-4 Turbo. However, this comparison is based solely on benchmark scores shared by Anthropic, and some experts suggest such benchmarks may be selectively presented.
- Claude 3 reportedly excels in cognitive tasks like reasoning, expert knowledge, mathematics, and language fluency, despite ongoing debates about whether large language models (LLMs) can truly “know” or “reason”.
- The company claims that the Opus model demonstrates near-human comprehension and fluency in complex tasks. While the scores indicate near-human performance on certain benchmarks, this does not imply that Opus possesses human-like general intelligence.
Claude 3 vs GPT-4
- Claude 3 Opus, developed by Anthropic, has outperformed GPT-4 on 10 AI benchmarks, including MMLU (undergraduate level knowledge), HumanEval (Coding), HellaSwag (common knowledge), and GSM8K (grade school maths).
- In the five-shot MMLU trial, Claude 3 scored 86.8%, slightly higher than GPT-4’s 86.4%. In Multilingual Maths (MGSM), Claude 3 significantly outperformed GPT-4, scoring 90.7% compared to GPT-4’s 74.5%.
- While these scores are impressive, their real-world implications for users are uncertain, and experts advise caution when interpreting LLM benchmarks. Claude 3 has demonstrated improvements in analysis, forecasting, content creation, multilingual conversations, code generation, and more. It also reportedly has enhanced vision capabilities, allowing it to process photos, charts, and diagrams, similar to GPT-4V.
Limitations of Claude 3
- Early users of Anthropic’s Claude 3 have reported that the model performs well in tasks such as answering factual questions and optical character recognition (OCR). It is also adept at following instructions and completing tasks like writing Shakespearean sonnets.
- However, it sometimes struggles with complex reasoning and mathematical problems and has shown biases in its responses. Similar issues have been observed in other AI models, such as Google’s AI chatbot Gemini. Anthropic has highlighted the safety features of Claude 3, including its refusal to generate harmful or illegal content.
- The company has also pioneered Constitutional AI, implementing a set of values for the system to ensure politically and socially responsible actions. Currently, Claude 3 is the most expensive model on the market, but more affordable versions are planned.
- Based on early reports, benchmarks, and confidence from the AI community, Claude 3 appears to be a significant advancement in the development of large language models (LLMs).
Conclusion- Anthropic’s Claude 3, a series of large language models, marks a significant advancement in the field of AI. Despite some challenges with complex reasoning and biases, the models have demonstrated impressive performance in various tasks, including factual question answering and optical character recognition. The company’s commitment to safety and ethical considerations, as evidenced by its refusal to generate harmful content and its implementation of Constitutional AI, is commendable. While currently the most expensive model on the market, the promise of more affordable versions indicates Anthropic’s commitment to accessibility.
However, it’s important to remember that while these models show near-human performance on certain benchmarks, they do not possess human-like general intelligence.
