I Built a Tiny Computer Inside a Transformer

A transformer can be engineered to function as a programmable machine by compiling a program directly into its weights. This method treats the residual stream as working memory, uses attention for lookups and routing, and employs feed-forward networks for local computations. The resulting vanilla transformer executes a deterministic program without requiring any training through gradient descent. This approach offers an alternative to external tool use, enabling precise computations to be performed directly inside the model.