ExamX, part 1: a practice-exam generator that marks its own papers
One evening, five past papers and a 4 GB GPU. What worked, what the local models got wrong, and why the code ended up doing most of the marking.
On this page
My FIT2014 exam (Theory of Computation at Monash) was coming up, and I had five past papers but only two sets of solutions. I wanted more practice than that, so I wrote a brief for a system and handed it to Claude Code, running on my homelab server: build a tool that generates new practice papers with worked solutions and marks my attempts, using only free tools and a local model.
This post covers that first evening. Part 2 covers cloud models, putting it online, and why I paused it.
The brief
- Data: the 2013 sample exam plus the 2013, 2014, 2015 and 2017 papers. Only two of them came with solutions.
- Hardware: the server has 8 CPU cores, 31 GB of RAM and a GTX 1650 with 4 GB of VRAM. That's a small budget for running an LLM.
- Rules: free tools only, a Python virtual environment inside the repo, small commits pushed often, and no credentials stored anywhere.
Step 1: getting the papers in
All the PDFs had real text in them, so no OCR was needed. Ingestion used PyMuPDF to split each paper into questions and parts, tag each one with a topic, and match it to its official solution where one existed.
One check made me trust the parser: every paper's marks added up to exactly 120. That caught several splitting bugs early.
Two details mattered more than I expected:
- Superscripts. Expressions like ambn lose their meaning if the exponent becomes plain text. The parser reads font sizes and positions so they survive as
a^{m}b^{n}. - Colour. In the solution PDFs, the answers and the marker's comments are in different colours, so the parser separates them by colour.
The result was 101 structured questions in JSON, with the diagrams saved as images.
Step 2: let code mark what code can mark
Much of FIT2014 has answers you can check exactly. Two regular expressions either describe the same language or they don't, and a Turing machine either computes the right output or it doesn't. So before any LLM was involved, the project got a set of exact checkers:
| Checker | What it decides |
|---|---|
| Regex / automata | Language equivalence, with the shortest string where two answers differ |
| Subset construction and minimisation | Whether an NFA→DFA conversion or a minimised DFA is correct |
| Context-free grammars | Membership (Earley parsing), derivations, bounded language comparison |
| Turing machines and PDAs | Simulation on test inputs |
| Propositional logic | Truth tables and CNF |
Because the checkers name a concrete counterexample, the feedback is specific: "your regex accepts abba, the language doesn't".
One bug was a good lesson. Some reference Turing machines kept crashing off the left end of the tape. FIT2014 uses one-way infinite tapes, so a machine can't detect the left end unless it marks it first. The fix belonged in the machines, not in the simulator.
Step 3: generating new papers
The generator samples a blueprint from the past papers (which topics, how many marks) and fills each slot in one of two ways:
- Procedural questions. Code generates the automata, grammar, logic and decidability questions, so their solutions are correct by construction. Each model solution is fed back through the grader and must score full marks.
- LLM questions. For proofs and explanations, a local model writes new questions using past-paper questions as examples.
Papers come out as Markdown, PDF and JSON. The PDF path is Markdown → HTML → PyMuPDF, so no LaTeX install was needed.
Where the local model fell over
I benchmarked the Qwen3 models through Ollama. The 8B model was the largest that ran at a usable speed (about 8 tokens/s, split between the 4 GB GPU and system RAM), and the only one that wrote a correct proof in the benchmark.
A full paper took 19 minutes, and about half of the LLM-written questions were poor:
- near-duplicate questions;
- a "prove that…" of a statement that is false;
- wrong answers on decidability;
- an NP-completeness question that didn't make sense.
So we added quality controls: near-duplicate drafts got rejected, a self-review pass checked each question, and LLM-written questions got flagged in the solutions. The second run took 39 minutes and was worse. The self-review changed a correct answer ("is this grammar property decidable? No") into a wrong one. Five questions fixated on the same grammar, and two of them asked students to prove something it doesn't generate.
An 8B model on a 4 GB card can't be trusted to invent exam questions and their solutions. The checkers can't save a question that's wrong to begin with.
The fix was to give the LLM less to do. Decidability tables and language-class questions moved to code. Proof and explanation slots now use real past questions with their official solutions by default. LLM-written questions became an opt-in --prose llm experiment.
Marking proofs
Proofs can't be checked by code, so an LLM marks them against a rubric. The 8B model gave a correct proof 1–3 out of 7 and claimed a true step was false. Moving proof marking to qwen3:14b fixed most of that:
- the correct proof got 5/7, with accurate feedback;
- a deliberately bogus proof got 0/7, so the 14B model wasn't just being lenient;
- each proof took about 2 minutes to mark.
Proof marks are still labelled advisory in the report.
Where it ended up after one evening
- A CLI (
ingest,generate,grade,check,serveand more) and a small web UI built on Python's standard-library HTTP server. - 108 passing tests, many built from real past-paper answers.
- An example paper, a deliberately mixed sample attempt, and its marking report in the repo.
- A README that records both LLM runs and their failures as well as the successes.
examx generate --no-llm makes a fresh paper in about a second, and the exact checkers mark it instantly. That alone was worth building. The open question was whether a better model would fix the question writing, which is where part 2 picks up.