0
Local LLMs Handle Textbook RAG but Fail on New Research Papers
https://towardsdatascience.com/local-llms-handle-textbook-rag-but-fail-on-new-research-papers/(towardsdatascience.com)An experiment tested a local Retrieval-Augmented Generation (RAG) system's performance on textbooks versus a new research paper. Small, local LLMs performed well on textbook questions but their performance collapsed on questions about the new paper, even when provided with the correct passages. For example, a local gpt-oss model scored 12 out of 20 on textbook questions but dropped to 0 out of 15 on the paper's questions. The analysis suggests that a model's familiarity with the subject matter is a critical factor, as smaller models struggle to synthesize answers solely from new information. Additionally, the study found that most failures were due to poor retrieval, and significant improvements were made by adding figure descriptions and document summaries to the context.
0 points•by hdt•18 hours ago
Comments (0)
No comments yet. Be the first to comment!
Have an account? Log in to join the discussion.