Building a Multimodal RAG That Responds with Text, Images, and Tables from Sources

Building a reliable multimodal Retrieval-Augmented Generation (RAG) system that returns images and tables from complex documents is challenging because standard image summarization lacks surrounding textual context. A proposed improved pipeline addresses this by creating context-aware image summaries using the text immediately before and after a figure. This method ensures captions capture the author's narrative and differentiate between similar-looking figures. The system first generates a textual response based on retrieved text chunks and then uses this response to select the most relevant images, improving the contextual accuracy of the final multimodal output.