Skip to content
sprite.solutions
← All notes

zebra_simple: Zebra Puzzle Test for LLMs

Research ReportJune 21, 20253 min readby Sergey Arkhipov

Last week I published zebra_simple on GitHub — a simple LLM reasoning tester. It measures how often LLMs solve Zebra Puzzles from the ZebraLogic dataset.

We chose Zebra Puzzle as a good example of pure logic reasoning task for our research. But actually, I had one more reason — as a child I loved playing Sherlock, so I wanted to recapture that feeling and treat the models to this game 😉

Sherlock the game

Sherlock 2.0 game description from Mobygames:

The game screen is divided in several areas. The top-left area is the puzzle solve board. The top-right area contains the horizontal or multi-column clues. The bottom row contains the vertical or single-column clues.

A vertical clue tells the player that two squares are or are not in the same column. There are five types of horizontal clues. The "is next to" and "is not next to" clue says that two tiles are or are not adjacent to each other, but it doesn't say which one is on the left and on the right. The "is left of" clue (middle tile is yellow with 3 dots) says that one tile is to the left of another tile, but it doesn't say how far or close to the left.

The "is between" clue (three tiles with two sided arrow) says that three tiles are adjacent but this can be from left to right or from right to left. The "is not between" says the same except that the middle tile is not between the other two.

zebra_simple is pretty simple

zebra_simple compares the success rates of N selected models on the same M Zebra Puzzle tasks and extracts their chains of thought for further analysis.

All models, from all providers, are accessed through the same OpenAI Completions API — the de-facto standard for querying LLMs. For now we have tested models from Google, Groq, NVidia, OpenAI and DeepSeek but have also connectors to xAI, Anthropic and any custom providing compatible Completions API.

By default all models are queried with temperature=0.6 and top_p=0.95 (except o3 and o1, which do not support these parameters) — this is the most recommended combination for logic reasoning, it could be changed in code now or added as command line arguments in future.

This tool is pretty simple to use and to modify for your own needs. Prompt template lives in zebra_template.py. This command runs 2 Zebra Puzzle tests of size 3x3 (#0 and #1 from the dataset) on llama-3.3-70b model by Groq with tokens limit 4k and writes results to the json log:

python -u zebra.py -m llama-3.3-70b-versatile@groq -s 3*3 -i 0,1 -t 4096

This is a summary of a six-model comparison (the number after each model name is the success rate in %):

{
    "duration": "0:13:22.876821",
    "size": "3*3",
    "count": 5,
    "max_tokens": 16000,
    "models_summary": {
        "gemini-2.5-flash-preview-05-20-none@google": 100,
        "gemini-2.5-flash-preview-05-20-default@google": 100,
        "nvidia/llama-3.1-nemotron-70b-instruct@nvidia": 40,
        "nvidia/llama-3.1-nemotron-ultra-253b-v1@nvidia": 100,
        "qwen/qwen3-32b-none@groq": 60,
        "qwen/qwen3-32b-default@groq": 100
    }
}

This is an example of a JSON log for one model; skipped parts are listed as '...':

{
    "model": "nvidia/llama-3.1-nemotron-ultra-253b-v1@nvidia",
    "size": "3*3",
    "max_tokens": 16000,
    "total_tokens": 13804,
    "started": "2025-06-17T15:40:48.655072",
    "duration": "0:03:28.259337",
    "count": 5,
    "successed": 5,
    "rate": 100,
    "items": [
        {
            "id": "lgp-test-3x3-0",
            "success": true,
            "finished": "2025-06-17T15:42:05.356933",
            "prompt": "\n\nTHE PUZZLE: ...",
            "solution": { "House 1": ... },
            "answer": { "House 1": ... },
            "text": "",
            "thought": "<think>\nOkay, let's tackle this logic puzzle step by step. ... the correct solution emerges.\n</think>",
            "usage": {
                "completion_tokens": 4037,
                "prompt_tokens": 511,
                "total_tokens": 4548,
                "prompt_tokens_details": null
            }
        },
        ...
    ]
}

Why a new tool

The original ZebraLogic bench by ZevoEval is great for models benchmarking but was too complicated for our research task and had several problems in results analysis. We need:

  • to compare different models from different providers on the same list of tasks
  • to compare different prompts on exactly same tasks in a simple repeatable way
  • to store results in a structured json per each model allowing to process them further
  • to extract internal reasoning tokens separately from the answer for reasoning models
  • to experiment with different levels of reasoning for different models and providers
  • to extract and compare results as stable as possible (try to avoid json parse problems)
  • to ignore small differences in answers as 'ford f150' vs 'Ford F-150' but ensure logic
  • to run on any desktop OS (Windows, Linux, MacOS) with minimal dependencies
  • to keep your model keys in your own environment instead of handing them to third parties
  • and to have common space to work with shared code for our experiments

The problem of reasoning tokens disappearing from the completion response in structured output mode prompted me to use text mode with specific prompt clauses for easy json extraction.

The problem of usually broken json in answer led me to move reasoning description from the final json to preceding text and leave the final json as simple as possible.

The harder problem — providers differ both in how the reasoning level is controlled and in how the reasoning comes back in the response — needs a post of its own.

References

Open the source ↗

Topics

LLM Testing · Logic Reasoning · AI Research