I Tested Four Ways to Give an AI Assistant a Memory. The Obvious One Was the Worst Value.

Four different methods for providing memory to an AI assistant were tested to evaluate their effectiveness and cost. The experiment used a custom benchmark involving a simulated chat history to measure how well models like GPT-4o mini and Claude Haiku could complete tasks, not just answer questions. The results demonstrated that retrieval-based memory was more effective and up to 31 times cheaper than providing the full chat history in the prompt. Surprisingly, the full history method led to poor task performance despite high accuracy on direct questions, making it the worst value option.