RAG On-Premise

An on-premise Retrieval-Augmented Generation (RAG) system is presented, designed to run without a GPU using Ollama, Qdrant, and ASP.NET Core. The system processes internal regulatory documents, converting them and user queries into embeddings to find and generate relevant answers. An attempt was made to improve performance by adding a semantic cache, which would reuse answers for questions with high semantic similarity. However, this caching mechanism was found to be unsafe, as it failed to distinguish between questions with subtle but critical differences, leading to potentially incorrect answers. The cache was ultimately disabled by default because a reliable similarity score threshold for safely reusing answers could not be established.