Context Payload Optimization for ICL-Based Tabular Foundation Models

Tabular foundation models use in-context learning (ICL) to adapt to new tasks on the fly, creating a crucial trade-off between prediction accuracy and response latency. While larger context payloads with more examples can improve model performance, they also increase processing time and operational costs, especially for centrally hosted services. Context payload optimization aims to solve this by distilling the most valuable information into a smaller, more efficient package. Strategies range from simple, task-agnostic methods like random sampling to more complex, task-aware approaches like K-nearest neighbors (KNN) that select the most relevant data for a given query. These optimizations can be pre-computed offline or performed dynamically at inference time to strike the right balance between response quality, speed, and cost.