Keyword Search Tied an 8B Embedding Model at Picking Agent Tools

A comparison of tool retrieval methods for AI agents found that the classic keyword search algorithm BM25 performed as well as a large 8B parameter embedding model. The experiment, conducted on the ToolRet benchmark with up to 3,794 tools, also revealed that BM25 was approximately 40 times faster on a CPU. Performance for all methods, including embedding models and rerankers, degraded as the number of irrelevant tools increased. Providing full tool documentation was found to be more beneficial for retrieval accuracy than simply using a larger model.