PRX Part 4: Our Data Strategy

The data strategy for the PRX model involves assembling a large, diverse dataset from public and internal sources for pre-training. A key step is re-captioning all images with a Vision-Language Model (VLM) to ensure accurate and descriptive text. The guiding philosophy is to prioritize breadth and coverage during pre-training, leaving aesthetic curation for later fine-tuning stages. The pipeline utilizes the Lance data format for building and exploring the dataset and Mosaic Data Shards (MDS) for efficient streaming during distributed training. This approach allows for flexibility, such as computing text latents on the fly, which simplifies the process at a minimal throughput cost.