Prompt Fidelity: Measuring How Much of Your Intent an AI Agent Actually Executes

A new metric called 'Prompt Fidelity' is proposed to measure how much of a user's intent an AI agent actually executes versus how much it infers or hallucinates. Using Spotify's AI playlist feature as an example, it demonstrates that agents often confidently guess to fulfill complex constraints without informing the user, leading to partially incorrect results. Prompt Fidelity is calculated as the ratio of information from verifiable tool calls against information inferred by the LLM, providing a score of trustworthiness rather than simple accuracy. This metric helps users understand how much of an agent's output is provably grounded in data, which is critical for building trust in agentic systems.