title: "Glean : lire efficacement MySQL avec Apache Beam en streaming"

Glean required an efficient method to index a vast corpus of documents from MySQL databases for its enterprise AI search platform. The initial batch processing approach using Apache Beam proved inefficient, causing high data latency and significant resource consumption. To overcome these challenges, the team transitioned to a streaming pipeline utilizing Apache Beam's `Watch` transform. This new system monitors MySQL tables for changes in near real-time, using sharding to parallelize the reading process and a custom function to detect new and updated rows, which drastically improved efficiency and reduced the load on the databases.