Dreaming in Cubes

A new generative AI pipeline can "dream" of Minecraft worlds by learning the game's fundamental building blocks and spatial grammar. The process first uses a Vector Quantized Variational Autoencoder (VQ-VAE) to compress complex 3D chunks into a vocabulary of 512 unique structural "codewords," much like creating a set of custom LEGO bricks. Next, a transformer model learns how to arrange these codewords in a sequence, understanding the logic of how terrain features like mountains and rivers should connect across large areas. This two-stage approach successfully generates novel grids of terrain containing recognizable features like caves, coastlines, and snow-capped peaks, demonstrating a grasp of the game's underlying world structure.