Scaling Indic Pretraining: 2.8 Trillion Tokens and an Open Data Generation Pipeline

Developed by NVIDIA and AI4Bharat, a groundbreaking new resource provides 2.8 trillion tokens of pretraining data across 22 official Indian languages. More than just a dataset, the project also releases the open, GPU-accelerated generation pipeline used to create it, empowering developers to produce more data for their own needs. The synthetic corpus was created by translating curated English material, including mathematics and code, into both native-script and romanized text formats. This initiative directly addresses the critical lack of data that has historically limited AI development for Indian languages, aiming to build more capable and accessible models for the region.