Ready to build trustworthy AI systems?

AI21 Labs' task-specific Summarize API was evaluated against OpenAI's general-purpose models, Davinci-003 and GPT-3.5-Turbo. The methodology included automated metrics for faithfulness and compression, alongside blind human evaluations measuring the pass rate for real-world scenarios. Results indicated that the Summarize API performed better or on par, achieving a higher pass rate, better faithfulness, and a greater compression rate. OpenAI's models were significantly more prone to producing unreliable summaries with hallucinations, even with extensive prompt engineering.