How Morgan Stanley deploys AI that actually works (hint: it's evals) | Human in the Loop: Episode 13

Morgan Stanley began experimenting with Generative AI in early 2022 to solve a significant knowledge management challenge for its financial advisors. The firm developed what is now known as a Retrieval-Augmented Generation (RAG) system to provide answers from internal procedure and research documents. A crucial component of their successful deployment was the creation of a rigorous evaluation framework that involved subject matter experts rating AI outputs for accuracy and completeness. This human-in-the-loop process helped identify issues like hallucinations and search problems, allowing the team to iterate and improve the system. This early focus on evaluation proved the AI's value and built the necessary organizational confidence for enterprise-wide adoption.