Skip to content
Tutorials Center
My learning

Topics

Lesson 68 of 77 ยท Projects

Project: A Mini Online Judge

Build a tiny online judge in Python: run submissions against test cases, return AC, WA, RE, TLE and CE verdicts, enforce a step budget and rank a scoreboard.

Coding-interview platforms and programming contests all rest on the same machine: an online judge. It takes a submission, runs it against hidden test cases, and answers with a short verdict such as Accepted or Time Limit Exceeded. Building a small one is a great project because it forces you to think about the things interviewers care about: edge cases, failure modes, limits and how to report results clearly.

In this project you will build a mini judge in about 80 lines of Python. It runs in your browser, so everything here is real and runnable. The honest limit: a production judge runs untrusted code in a locked-down operating-system sandbox (separate process, no network, memory and CPU limits). We cannot do that in a browser tab, so our judge uses a deterministic step budget as a stand-in for a CPU time limit, and it is for learning, not for running hostile code. See Python: Classic Coding Questions for the kind of problems you would judge.

What a judge does

Receive Source code and a problem id.
Run Every test case, one at a time.
Compare Output against the expected answer.
Verdict One short, precise result.
1 / 2

Step 1: Problems and test cases

A problem is a function name plus a list of test cases. Each case is a pair: the arguments and the expected result. Good judges include small cases, edge cases (empty input, one element, duplicates) and one large case that punishes slow solutions. Keep them in data, not in code, so adding a case never changes the judge.

Python Step 1: one case, one verdict

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The trick is sys.settrace. Python calls our tracer for every executed line of the submission, so counting lines gives a deterministic limit: the same code always takes the same number of steps, on any machine. A real judge measures seconds, which is noisy. The price is that a step is not a real unit: a single line such as sum(range(10**9)) is one step but takes seconds. Keep that in mind when you explain the design.

Step 2: Compile errors and the full judge

Before running anything, the judge must turn source text into a function. exec runs the text and fills a dictionary with the names it defines. A syntax error means the program never starts, which is a CE. A missing function is also a CE, because the submission does not match the problem.

The judge then loops over the cases and stops at the first failure, reporting which case failed. Revealing the case number helps contestants; revealing the input or expected output would let them hard-code answers, so real judges keep those hidden.

Python Step 2: judging source code

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the results. The "forgot duplicates" submission passes the first case, then fails the second, where [7, 7, 2] has a repeated maximum. That is exactly what a hidden edge case is for. The bubble-sort submission is correct but burns its budget on the 100-element case, so it gets TLE: the large case separates an O(n log n) idea from an O(nยฒ) one.

Why stop at the first failure? Speed and honesty. Contest judges often run every case to give partial scores, but interview-style judges stop early, because one failure is enough to say "not accepted".

Step 3: A scoreboard

Contests rank people with a scoreboard. The classic rule: rank by problems solved (more is better), then by penalty (lower is better). A solved problem costs the minute of the accepted submission plus 20 minutes for every earlier wrong submission on that problem. Wrong attempts on a problem you never solve cost nothing. Submissions arrive as a log of (team, problem, minute, verdict).

Python Step 3: a contest scoreboard

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check the numbers by hand: ana solved A at minute 20 with one earlier wrong try (20 + 20 = 40) and B at 55, so her penalty is 95. Ben solved A at 8 and B at 70 with two wrong tries (70 + 40 = 110), for 118. Both solved two problems, so the lower penalty wins. Cleo solved one problem. The sort key (-solved, penalty, team) is a classic trick: negate what should sort descending, and add the name as a final tie-break so the order is stable.

Where this is simplified

Be ready to talk about the gap between this project and the real thing. That conversation is often more valuable than the code.

  • Sandboxing. Real judges run each submission in its own process with limits on CPU time, wall time, memory, output size, file access and network. Containers, seccomp filters or virtual machines enforce them. Our tracer cannot stop import os or a single huge allocation.
  • Output comparison. We compare return values. Real judges compare text output from standard input, usually ignoring trailing whitespace, and use a special checker when many answers are valid (for example "print any valid path").
  • Concurrency. A real judge is a queue of submissions with a pool of workers, so a burst of submissions waits instead of overloading the machine. See System Design: The Building Blocks for queues and workers.
  • Hidden tests and cheating. Test data is secret, submissions are rate-limited and plagiarism is checked.

Extensions to try

  1. Add a verdict PE (presentation error) for answers equal after stripping whitespace.
  2. Report the total steps used, to rank accepted solutions by speed.
  3. Support several problems per contest by making the log reference problem ids that map to the PROBLEM dictionaries.
  4. Freeze the scoreboard for the last hour of a contest, the way real contests do.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does the judge count executed lines instead of measuring seconds?

    Wall-clock limits vary with machine load. A step budget gives the same verdict every run, at the cost of steps not being a real time unit.
  2. 2. A submission fails to parse because of a missing colon. Which verdict is right?

    The program never starts, so it is a compile (syntax) error. RE is for exceptions raised while running.
  3. 3. Why does a judge keep the failing input and the expected output hidden?

    If the judge revealed the case, a submission could simply return the expected value for it. The case number is enough to debug.
  4. 4. In the scoreboard, a team solves problem A at minute 30 after two wrong tries. What is the penalty for A?

    Minute 30 plus 20 minutes for each of the two wrong submissions gives 70.
  5. 5. What is the main reason a browser-only judge like this one is not safe for hostile code?

    A trace hook only counts lines. Real judges isolate each submission in a separate process or container with hard resource limits.

Summary

  • A judge is a loop: build the function from source, run each case, compare, return one verdict.
  • Verdicts: AC, WA, RE, TLE and CE. Report the failing case number but hide the data.
  • A step budget through sys.settrace gives a deterministic stand-in for a time limit.
  • Large cases separate fast algorithms from slow ones; edge cases catch careless ones.
  • Scoreboards rank by problems solved, then by penalty minutes, with a deterministic tie-break.
  • Real judges add OS sandboxes, queues, secret data and special checkers.

Next: Project: Build a Rate Limiter and LRU Cache.