Should We Use LLMs As If They Were Swiss Knives?

An experiment was conducted to compare the performance of popular LLMs against a custom-built algorithm in solving a logic game similar to Wordle. Initial tests using base models from ChatGPT, Gemini, and Llama revealed poor and inconsistent performance, with the LLMs often making logical errors, whereas the specialized algorithm won every game. When an LLM with enhanced reasoning capabilities was introduced, its performance improved dramatically, becoming much more consistent and achieving results nearly as good as the custom algorithm. The results suggest that while general-purpose LLMs may struggle with specific logic-heavy tasks, models with explicit reasoning functions are far more capable, though purpose-built solutions can still hold an edge in performance.