Choosing the Best Model Size and Dataset Size under a Fixed Budget for LLMs

When training large language models under a fixed compute budget, a trade-off exists between increasing the model's parameter count or the size of the training dataset. This analysis explores the relationship by running experiments with 'tiny transformers' of varying sizes on different amounts of data. The results indicate that for any given compute budget, an optimal combination exists, and as the budget grows, allocating more resources to data rather than model size becomes more efficient. For smaller budgets, this means a medium-sized model trained on more data can outperform a larger model trained on less data.