ExamX, part 2: cloud models, a public site, and pressing pause

Part 1 ended with a working generator and grader, held back by a local model that was too small to write good questions. Around midnight I asked the obvious question: would a free cloud model fix it?

Would Gemini solve the problem?

Mostly, yes, with a few catches that were worth spelling out before building anything:

• It's no longer local. That was fine: I dropped the "runs locally" requirement.
• The free tier may use your prompts for training. That would include past-paper text and my answers.
• It needs an API key, which has to live in the environment and never in the repo.
• The exact checkers stay in charge of anything they can check. A better model makes fewer mistakes, but not zero.

Three back ends and a model switcher

The LLM client got three interchangeable back ends, chosen with a provider:model spec:

examx generate --model qwen3:8b                  # local, via Ollama
examx generate --model gemini                    # Google's default model
examx generate --model claude:claude-sonnet-5-5  # Anthropic
examx models                                     # which back ends are ready

The web UI got a matching AI model page. It shows whether each provider is ready and how to set it up if not, and a chip in the header always shows the current model. The UI also gained dark mode, a phone layout, past-paper diagrams inside questions, and answers saved in the browser as you type.

The free tier is 20 requests a day

I set up a Gemini key (in a file only my user can read, loaded by ~/.bashrc, nowhere near the repo). The first surprise: gemini-2.5-flash was no longer offered to new keys, so the default moved to gemini-3.5-flash. It answered a test question about decidability correctly in under 4 seconds.

Then a full paper ran for 22 minutes and produced almost nothing from Gemini:

• the free tier allows 20 requests per day for that model, and one AI-written paper needs 15–25;
• several calls failed earlier anyway with "model experiencing high demand";
• my retry logic waited and retried for minutes, even after the daily quota was gone.

The fix was to treat "quota used up for hours" differently from "slow down for a minute". ExamX now stops at once, skips that model for the rest of the run, and says when the quota resets. In practice the free tier gives you about one AI-written paper or one marked attempt a day.

Putting it online

I decided to try Claude Sonnet instead and asked for the app at exam-x.vrp2206.com. The server already ran Caddy and a Cloudflare Tunnel for my hackathon demo sites, so ExamX went in as a Docker container behind its own tunnel.

Two design decisions from this part are worth keeping:

1. API keys live in the visitor's browser, not on the server. You paste your key into the AI model page. It's stored as an HttpOnly cookie, checked once against the provider's free model-list endpoint, used only while your job runs, and never written to disk or logged. By default it's forgotten when you close the browser.
2. Fail closed. The app first refused every request that hadn't passed Cloudflare Access.

Cloudflare Access then asked for a credit card, so I asked to drop it. Before switching the guard off, Claude added the protections a public site needs:

• visitors can only use their own Claude or Gemini key, never my local GPU models or a server-side key;
• each browser gets an anonymous ID and only sees its own papers and reports;
• the model choice is per browser, and each visitor has daily job limits;
• the owner still has full access from the server itself.

Pressing pause

That's where I stopped. The Claude API is billed separately from a Claude Pro subscription. The free Gemini tier is too small to rely on. The local models are too weak to write new questions. And the real goal was passing the exam, not shipping a product the night before.

So ExamX was shut down cleanly:

• both containers (the app and its tunnel) removed, and the Caddy route removed and reloaded, with the other sites checked afterwards;
• a "Project status: paused" section at the top of the README, covering what's done, what I learned and the next steps;
• the code tagged v0.1-paused.

I kept the Ollama models (they're useful for other ML work) and left the GitHub repo public rather than archived, in case someone else wants to pick it up.

What I took away

• Let code mark what code can mark. The exact checkers were the most reliable part of the whole project, and they need no model at all.
• Small local models are good at following a template, not at inventing correct maths. Using real past questions with official solutions beat every LLM-written alternative I tried.
• Free tiers have shapes. "20 requests a day" changes the whole design of a tool that needs 25 requests per run.
• Don't paste secrets into a chat. I pasted a tunnel token into the conversation to save time, which meant treating it as leaked and replacing it. The final version of the app asks for keys through a form instead.
• Stopping is a valid ending. The tool still works offline for practice: examx generate --no-llm and exact marking, no AI needed.