Why Package Installs Are Slow (And How to Fix It)

Slow package installations are often caused by metadata bloat, where package managers use a single, large index file for all available packages. This forces clients to download and parse the entire index for every operation, leading to slow downloads and high memory consumption. A solution to this scaling problem is sharding, which splits the monolithic index into individual, per-package metadata files. Using this method, a client downloads a small manifest and then fetches only the necessary metadata shards, drastically improving efficiency. This approach, implemented in the conda ecosystem, has resulted in 10x speed improvements, a 35x reduction in network transfer, and a 15x decrease in memory usage.