How Much Does It Actually Cost to Run a Local LLM? (Euros per Million Tokens, Measured)

The actual electricity cost of running local large language models can be precisely measured by tracking a GPU's real-time power consumption. An experiment using an RTX 3090 calculated the cost in Euros per million tokens for eight different models, comparing them to commercial cloud APIs. The results showed that five of the eight local models were cheaper to run than paying for a service like Gemini Flash. Surprisingly, cost was not directly correlated with model size, as a 32.8B parameter model was the most expensive to run, costing even more than a 106B model.