Skip to content

Design LeetCode (online judge)

TL;DR

  • An online judge is a job system that runs hostile code: ~100 submissions/s, each costing a few CPU-seconds, so the fleet is sized in cores (~500 busy cores) rather than in QPS.
  • The cruxes an interviewer probes: (1) the submission queue and worker pool, (2) the sandbox (container, seccomp, cgroups, no network), (3) test-case storage, streamed results and deterministic verdicts, (4) contest mode with a live leaderboard and plagiarism detection.
  • The verdict taxonomy is part of the design, not a detail: Accepted, Wrong Answer, Time Limit Exceeded, Memory Limit Exceeded, Runtime Error, Compile Error — and the precedence between them has to be pinned down.

Problem statement and clarifying questions

"Design a site where users write code against a problem, submit it, and get a verdict in a few seconds — including during contests where tens of thousands submit at once." Every hard part comes from one fact: you are executing arbitrary code written by strangers, and you promised them a repeatable answer within seconds.

Question Assumption taken
Scale? 10M registered users, 2M DAU, ~5 submissions per active user per day.
Languages? About ten, compiled and interpreted, each with its own time-limit multiplier.
How are answers checked? Stdin/stdout against stored expected output, plus special judges for multi-answer problems.
How fast must a verdict be? p50 under 2 s, p95 under 10 s outside contests.
Do users see progress? Yes: per-test-case results streamed as they complete.
Contests? Yes: scheduled, live leaderboard, scoreboard frozen for the last hour.
Can a submission see the network or the filesystem? No network at all, a read-only root and a small writable scratch area.
Are test cases secret? Yes, except the samples on the problem page.
What happens if a worker dies mid-judge? The submission is re-judged; a verdict must never be silently lost.

Requirements

Functional

  • Submit source for a problem and language; run against the sample cases ("Run") or the full set ("Submit").
  • Stream per-case results and return one final verdict with peak runtime and memory.
  • Store submission history per user and per problem, with the failing case revealed only for samples.
  • Contest mode: scheduled window, live ranking with penalties, frozen scoreboard, post-contest rejudge.
  • Flag suspiciously similar submissions for review.

Non-functional

  • Isolation is the top requirement: untrusted code must not reach the network, other submissions, the host, or the test data of another problem.
  • Determinism: the same source, language and test set produce the same verdict. Borderline timing is the enemy of that.
  • Latency: p50 verdict < 2 s, p95 < 10 s; contest queue wait p95 < 5 s at ten times normal load.
  • Throughput: ~100 submissions/s average, ~300/s peak, ~1,000/s in the first minute of a large contest.
  • Durability: a submission accepted with a 202 is judged exactly once effectively — at-least-once execution plus an idempotent verdict write.

Out of scope

The code editor itself, discussion and editorial content, payments, the special-judge authoring tools, and interview-simulation products.

Estimation

Using the latency and estimation tables (a day is ~10^5 s, peak is 3x average, an event can be 10x). A submission is ~2 KB of source and runs ~50 test cases.

Quantity Arithmetic Result
Submission write QPS 2M DAU x 5 = 10M/day / 10^5 ~100/s average, ~300/s peak, ~1,000/s at a contest start
Judging cost 100/s x 50 cases x 100 ms = 500 CPU-s per second ~500 busy cores; at 16 cores a node, ~32 nodes, ~64 with headroom
Problem-page read QPS 2M x 30 views = 60M/day / 10^5 ~600/s average, ~1.8k/s peak, ~95% served by the CDN
Test-case storage 3,000 problems x 50 cases x 200 KB ~30 GB total — small enough to mirror onto every worker
Submission storage/year 10M/day x 2 KB x 365 ~7 TB/year, x3 replicas ~21 TB
Result bandwidth 300/s x 1 KB streamed updates ~300 KB/s — the streaming is free, the CPU is not
Problem-page cache 20% of 60M daily reads x 10 KB ~120 GB behind the CDN

The number that decides the architecture is 500 busy cores. A judge is a CPU-bound batch system wearing a web front end: the API tier is trivial, and every design choice — queueing, autoscaling, pinning, warm sandboxes — exists to keep those cores fed and predictable. The second number is 30 GB of test data: the whole corpus fits on a worker's local NVMe, so no submission ever waits on object storage.

API design

Endpoint Request Response Notes
POST /v1/submissions {problem_id, language, source, mode} + header Idempotency-Key 202 {submission_id, status: "Queued"} Never blocks on judging. mode is run (samples only) or submit (full set). Per-user rate limit of a few per minute.
GET /v1/submissions/{id} — 200 {status, verdict, cases_passed, runtime_ms, memory_kb, failed_case?} The fallback when the WebSocket drops; failed_case is only populated for sample cases.
WS /v1/submissions/{id}/stream — case_result events, then verdict One message per test case as it completes; the client renders a progress bar.
GET /v1/problems/{slug} — 200 {statement, samples, limits, languages} Immutable per version and CDN-cached; secret cases are never in the payload.
GET /v1/contests/{id}/leaderboard?cursor=... — 200 {rows, my_rank, next_cursor} Cursor over (score desc, penalty asc, user_id); served from a sorted set and frozen in the final hour.

Idempotency-Key matters more than usual: a user who double-clicks Submit during a contest must not consume two judging slots or two penalty points.

Data model

Sources and verdicts are rows; test cases are blobs; the leaderboard is a sorted set.

erDiagram
    USER ||--o{ SUBMISSION : makes
    PROBLEM ||--o{ SUBMISSION : receives
    PROBLEM ||--|{ PROBLEM_CASE : defines
    PROBLEM ||--o{ LANGUAGE_PROFILE : limits
    SUBMISSION ||--o{ CASE_RESULT : produces
    CONTEST ||--o{ CONTEST_ENTRY : ranks
    USER ||--o{ CONTEST_ENTRY : competes_in
    CONTEST ||--o{ PROBLEM : includes
    USER {
        bigint user_id PK
        string handle
    }
    PROBLEM {
        bigint problem_id PK
        string slug
        int base_time_ms
        int memory_kb
        int version "bumped by a rejudge"
    }
    PROBLEM_CASE {
        bigint problem_id PK
        int case_no PK
        string input_key
        string expected_key
        bool is_sample
    }
    LANGUAGE_PROFILE {
        bigint problem_id PK
        string language PK
        int time_multiplier
        int memory_extra_kb
    }
    SUBMISSION {
        bigint submission_id PK
        bigint user_id
        bigint problem_id
        string language
        string source_key
        string status
        string verdict
        int runtime_ms
        int memory_kb
        int attempt
    }
    CASE_RESULT {
        bigint submission_id PK
        int case_no PK
        string verdict
        int runtime_ms
        int memory_kb
    }
    CONTEST {
        bigint contest_id PK
        timestamp starts_at
        timestamp freeze_at
    }
    CONTEST_ENTRY {
        bigint contest_id PK
        bigint user_id PK
        int score
        int penalty_minutes
    }

A submission's lifecycle: the states a worker crash has to be safe in.

stateDiagram-v2
    [*] --> Queued : accepted by the API
    Queued --> Judging : a worker claims the lease
    Judging --> Queued : lease expired, the worker died
    Judging --> InternalError : attempts exhausted
    Judging --> Compiling : sandbox ready
    Compiling --> CompileError : the toolchain rejects the source
    Compiling --> Running : binary produced
    Running --> Accepted : every case passes
    Running --> Failed : first failing case decides the verdict
    Accepted --> [*]
    Failed --> [*]
    CompileError --> [*]
    InternalError --> [*]

Store choices: SUBMISSION and CASE_RESULT go in a relational store sharded by user_id (every user-facing query is "my submissions"); sources and test cases are blobs in object storage keyed by content hash, with the test data mirrored to each worker's local disk; CONTEST_ENTRY is materialized into a Redis sorted set for ranking, exactly as in the leaderboard design.

High-level design

v1: a thin API, a leased queue, a pool of judge workers, one throwaway sandbox per run.

flowchart LR
    subgraph clients["Clients"]
        c_web["Browser editor"]
    end
    subgraph edge["Edge"]
        e_cdn["CDN (statements, assets)"]
        e_api["API gateway (auth, per-user rate limit)"]
        e_ws["WebSocket gateway"]
    end
    subgraph services["Services"]
        s_sub["Submission service"]
        s_prob["Problem service"]
        s_board["Leaderboard service"]
    end
    subgraph async["Async"]
        a_q[["Submission queue (leased)"]]
        a_pool["Judge worker pool"]
        a_sbx["Sandbox per run"]
        a_plag["Plagiarism scanner"]
    end
    subgraph data["Data"]
        d_sub[("Submission store")]
        d_cases[("Test cases in object storage")]
        d_local[("Worker-local test mirror")]
        d_board[("Leaderboard sorted set")]
    end
    c_web --> e_cdn
    c_web --> e_api --> s_sub --> a_q --> a_pool --> a_sbx
    e_api --> s_prob --> d_cases
    s_sub --> d_sub
    a_pool --> d_local
    a_pool --> d_sub
    a_pool --> e_ws --> c_web
    a_pool --> s_board --> d_board
    d_sub --> a_plag

Write path: accept fast, judge asynchronously, stream each case as it finishes.

sequenceDiagram
    autonumber
    actor U as User
    participant API as API gateway
    participant SS as Submission service
    participant Q as Submission queue
    participant W as Judge worker
    participant SB as Sandbox
    participant WS as WebSocket gateway
    U->>API: POST /v1/submissions (problem_id, language, source)
    API->>SS: create submission
    SS->>SS: store the source, status Queued
    SS-)Q: enqueue submission_id
    SS-->>U: 202 {submission_id}
    Q-->>W: claim: a lease and its fencing token
    W->>SB: compile inside a throwaway container
    loop each test case until one fails
        W->>SB: run with time, memory and output limits
        SB-->>W: exit code, stdout, cpu ms, peak rss
        W-)WS: case result streamed to the browser
    end
    W->>SS: write the final verdict (idempotent on submission_id)
    W->>Q: complete under that token, releasing the lease

Read path: statements from the CDN, history from a sharded store, ranking from a sorted set.

sequenceDiagram
    autonumber
    actor U as User
    participant CDN as CDN
    participant API as API gateway
    participant PS as Problem service
    participant SS as Submission service
    participant LB as Leaderboard service
    U->>CDN: GET /problems/two-sum
    CDN-->>U: cached statement and sample cases
    U->>API: GET /v1/submissions?problem_id=...&cursor=...
    API->>SS: list my submissions
    SS-->>API: verdicts, runtime, memory
    API-->>U: 200
    U->>API: GET /v1/contests/{id}/leaderboard
    API->>LB: top 100 plus my rank
    LB-->>API: ranked slice of the sorted set
    API-->>U: 200 (frozen in the final hour)

Walk-through: the API tier never runs code and never blocks — it writes a row and enqueues an id. Everything expensive happens in the pool, where the unit of work is one submission and the unit of isolation is one sandbox per run. Results reach the browser twice over: streamed on a WebSocket for responsiveness, and durably in the submission store for the reload.

Deep dive: the submission queue and the worker pool

The probing question is "a judge worker is OOM-killed halfway through a submission. What does the user see?" The answer must be "a verdict, a few seconds later" — which rules out a fire-and-forget queue.

Queue model Crash behaviour Notes
In-process thread pool Submission lost with the process Fine for a prototype, indefensible in an interview
Fire-and-forget topic Message acknowledged on receipt, then lost The user's submission is stuck in "Judging" forever
Leased queue with fencing tokens Lease expires, another worker claims it under a new token Chosen: at-least-once with a bounded attempt count

The queue hands out leases: a worker claims a submission, gets exclusive ownership for a lease window, and completes it. If the worker dies, the lease expires and the submission returns to the ready list. After a bounded number of attempts it becomes a dead letter with the Internal Error verdict and a page for an engineer — because the alternative, a submission that a poisonous input crashes forever, silently consumes the fleet.

Every lease carries a fencing token, minted fresh on each claim, and only the token that currently holds a lease may complete it. Skip that check and the lease is stealable: a worker that was reclaimed and then woke up would release its replacement's lease, and the live worker would fail to record a verdict it had actually produced. It is the same rule the job scheduler enforces with a worker id — a lease without a fencing token is a suggestion, not a lock.

code/hld/judge_runner.py — the leased submission queue
@dataclass(frozen=True, slots=True)
class Lease:
    """Proof that this worker, and nobody else, currently owns ``submission_id``.

    ``token`` is a **fencing token**: a fresh one is minted on every claim, so the lease a
    reclaimed worker is still holding in memory is detectably older than the live one.
    """

    submission_id: str
    token: str
    expires_at: float
    attempt: int


class SubmissionQueue:
    """FIFO queue with leases, so a worker that dies does not lose the submission.

    ``_lock`` guards ``_ready``, ``_leases``, ``_attempts`` and ``_next_token`` together; every
    public method takes it, which is why hundreds of workers can claim concurrently without
    double-judging.
    """

    def __init__(
        self,
        clock: Clock | None = None,
        lease_seconds: float = 30.0,
        max_attempts: int = 3,
    ) -> None:
        self._clock = clock or SystemClock()
        self._lease_seconds = lease_seconds
        self._max_attempts = max_attempts
        self._ready: list[str] = []
        self._leases: dict[str, Lease] = {}
        self._attempts: dict[str, int] = {}
        self._dead: list[str] = []
        self._next_token = 0  # one counter for the queue: every grant ever made is ordered
        self._lock = threading.Lock()

    def submit(self, submission_id: str) -> None:
        with self._lock:
            if submission_id in self._attempts:
                raise ValidationError(f"submission {submission_id!r} was already queued")
            self._attempts[submission_id] = 0
            self._ready.append(submission_id)

    def claim(self) -> Lease | None:
        """Take the next submission and lease it. Returns None when the queue is empty.

        Every claim -- the first one and every re-claim after an expiry -- mints a **new**
        token, which is exactly what makes the previous holder's lease stale.
        """
        with self._lock:
            if not self._ready:
                return None
            submission_id = self._ready.pop(0)
            self._attempts[submission_id] += 1
            self._next_token += 1
            lease = Lease(
                submission_id,
                f"lt-{self._next_token}",
                self._clock.now() + self._lease_seconds,
                self._attempts[submission_id],
            )
            self._leases[submission_id] = lease
            return lease

    def complete(self, submission_id: str, token: str) -> None:
        """Release the lease. Only its current holder may, which is what stops a stolen lease.

        Without the token, a worker whose lease expired, was reclaimed, and was re-claimed by
        somebody else would pop the **replacement's** lease when it finally woke up -- and the
        live worker would then fail to complete work it had actually done. Same check as
        :meth:`hld.cron_scheduler.Scheduler.complete`, with a fencing token in place of a
        worker id. The other half of the contract still holds: the verdict write is idempotent
        on ``submission_id``, so a late finisher rewrites the same answer rather than a second.
        """
        with self._lock:
            lease = self._require_current(submission_id, token)
            del self._leases[lease.submission_id]

    def _require_current(self, submission_id: str, token: str) -> Lease:
        """The lease must exist, carry this token, and not have run out."""
        lease = self._leases.get(submission_id)
        if lease is None:
            raise NotFoundError(f"submission {submission_id!r} is not leased")
        if lease.token != token or lease.expires_at <= self._clock.now():
            raise ConflictError(f"token {token!r} no longer holds the lease on {submission_id!r}")
        return lease

    def reclaim_expired(self) -> list[str]:
        """Re-queue leases that outlived a crashed worker; give up after ``max_attempts``."""
        now = self._clock.now()
        with self._lock:
            expired = [lease for lease in self._leases.values() if lease.expires_at <= now]
            for lease in expired:
                del self._leases[lease.submission_id]
                if lease.attempt >= self._max_attempts:
                    self._dead.append(lease.submission_id)
                else:
                    self._ready.append(lease.submission_id)
            return [lease.submission_id for lease in expired]

    def dead_letters(self) -> list[str]:
        """Submissions that failed every attempt: they get INTERNAL_ERROR and a page."""
        with self._lock:
            return list(self._dead)

    def depth(self) -> tuple[int, int]:
        """(ready, in flight) -- the two numbers a judging queue is autoscaled on."""
        with self._lock:
            return len(self._ready), len(self._leases)

The token settles ownership; idempotency handles what is left. At-least-once execution means the verdict write must be idempotent on submission_id: judging the same source twice is wasteful but harmless, writing two verdicts is a bug. Two operational details complete the picture. Autoscale on queue depth and oldest-message age, not CPU: CPU is pinned at 100% by design, so it tells you nothing, while a rising oldest-message age is exactly the user-visible symptom. And run separate queues for contests and practice, so a contest's 1,000/s burst does not sit behind an hour of practice submissions — the same priority argument as in the job scheduler design.

Deep dive: the sandbox

The probing question is "what stops a submission from reading /etc/passwd, mining crypto, or calling the internal metadata service?" A container alone is not an answer; the answer is a stack of independent limits.

  • No network. The container gets an empty network namespace: no interface but loopback. This single control removes data exfiltration, coordinated cheating, and calls to a cloud metadata endpoint at once.
  • cgroups cap memory, CPU shares and process count. The memory cap is enforced by the kernel's out-of-memory killer, so a runaway allocation dies without touching its neighbours; a pid limit stops a fork bomb.
  • seccomp-bpf allows only the syscalls a submission legitimately needs and kills anything else. ptrace, mount, unshare, raw sockets and keyctl do not belong in a solution to Two Sum.
  • A read-only root filesystem plus a small writable tmpfs scratch, dropped capabilities, no_new_privs, a non-root uid unique per run, and a wall-clock watchdog above the CPU limit so a sleeping process cannot hold a slot.
  • One sandbox per run, never reused. Reuse leaks state between submissions and turns a wrong answer into a security incident.
code/hld/judge_runner.py — the limits a sandbox enforces
@dataclass(frozen=True, slots=True)
class Program:
    """A stand-in for a compiled binary: what it prints and what it costs on each input."""

    outputs: Mapping[str, str] = field(default_factory=dict)
    time_ms: int = 1
    memory_kb: int = 1_024
    crashes_on: frozenset[str] = frozenset()
    compile_error: str | None = None
    default_stdout: str = ""


class Sandbox(Protocol):
    def run(self, program: Program, stdin: str, limits: Limits) -> RunResult: ...


class SimulatedSandbox:
    """A container with no network, a read-only root, a seccomp profile and cgroup limits.

    The real thing kills the process; here we compute the same outcome deterministically.
    Limits are enforced *before* the output is trusted, because a program killed at the limit
    has produced partial output that must never be compared against the expected answer.
    """

    def run(self, program: Program, stdin: str, limits: Limits) -> RunResult:
        if program.memory_kb > limits.memory_kb:
            # The cgroup out-of-memory killer stops the process: no usable output, non-zero exit.
            return RunResult(137, "", min(program.time_ms, limits.time_ms), program.memory_kb, 0)
        if program.time_ms > limits.time_ms:
            # The watchdog sends SIGKILL at the wall-clock limit.
            return RunResult(137, "", limits.time_ms, program.memory_kb, 0)
        if stdin in program.crashes_on:
            return RunResult(1, "", program.time_ms, program.memory_kb, 0)
        stdout = program.outputs.get(stdin, program.default_stdout)
        output_kb = len(stdout.encode()) // 1024
        if output_kb > limits.output_kb:
            return RunResult(0, stdout[: limits.output_kb * 1024], program.time_ms,
                             program.memory_kb, output_kb)
        return RunResult(0, stdout, program.time_ms, program.memory_kb, output_kb)

Two refinements worth naming. Container escapes are real, so high-risk deployments add a second boundary — gVisor's user-space kernel or a Firecracker microVM per run — accepting tens of milliseconds of extra start-up for a much smaller kernel attack surface. And because a cold container start would dominate a 100 ms test case, workers keep a pool of pre-warmed sandboxes, resetting the writable layer between runs rather than creating a namespace from scratch. That is a performance trick, not a security compromise: the process, the uid and the scratch space are still new every time.

Deep dive: test cases, streaming results and deterministic verdicts

The probing question is "the same code got Accepted yesterday and Time Limit Exceeded today — why?" Non-determinism in a judge is a product bug, and it has three usual causes: noisy neighbours, output comparison, and verdict precedence.

Noisy neighbours. Two submissions sharing a core make timing a lottery. Pin one run to one core, disable bursting, set the limit generously (2–3x a reference solution) and apply a per-language multiplier, since an interpreted language is legitimately several times slower. Measure CPU time, not wall-clock time, for the limit itself, and keep a wall-clock watchdog only as a backstop against sleeping.

Output comparison. Byte equality fails correct submissions on a trailing newline; token comparison is the sane default, and problems with several valid answers ship a special judge that is itself a sandboxed program reading the input and the contestant's output.

Verdict precedence. This is the part candidates skip. A process killed by the out-of-memory killer also exits non-zero, and so does one killed by the watchdog — so if you check the exit code first, every memory-limit failure is reported as a runtime error. Check resource limits first, then the exit code, then the answer:

code/hld/judge_runner.py — one verdict from many signals
class Checker(Protocol):
    def matches(self, expected: str, actual: str) -> bool: ...


class TokenChecker:
    """Default checker: whitespace-insensitive token comparison.

    Judging on raw bytes fails half the correct submissions on a trailing newline, so
    normalise; problems that accept several answers ship their own special judge instead.
    """

    def matches(self, expected: str, actual: str) -> bool:
        return expected.split() == actual.split()


class Judge:
    """Runs one submission against the test cases and turns runs into a single verdict."""

    def __init__(self, sandbox: Sandbox | None = None, checker: Checker | None = None) -> None:
        self._sandbox = sandbox or SimulatedSandbox()
        self._checker = checker or TokenChecker()

    def judge(
        self,
        program: Program,
        cases: Sequence[ProblemCase],
        limits: Limits | None = None,
        on_case: Callable[[CaseResult], None] | None = None,
    ) -> Judgement:
        if not cases:
            raise ValidationError("a problem needs at least one test case")
        limits = limits or Limits()
        if program.compile_error is not None:
            return Judgement(Verdict.COMPILE_ERROR, 0, len(cases), detail=program.compile_error)
        passed = max_time = max_memory = 0
        for case in cases:
            run = self._sandbox.run(program, case.stdin, limits)
            verdict = self._verdict_for(run, case, limits)
            max_time = max(max_time, run.time_ms)
            max_memory = max(max_memory, run.memory_kb)
            if on_case is not None:
                on_case(CaseResult(case.case_id, verdict, run.time_ms, run.memory_kb))
            if verdict is not Verdict.ACCEPTED:
                # Stop at the first failure: the verdict is decided and the worker is freed.
                return Judgement(verdict, passed, len(cases), case.case_id, max_time, max_memory)
            passed += 1
        return Judgement(Verdict.ACCEPTED, passed, len(cases), None, max_time, max_memory)

    def _verdict_for(self, run: RunResult, case: ProblemCase, limits: Limits) -> Verdict:
        """Precedence is the whole game: the resource limits are checked before the exit code,
        because a process killed by the out-of-memory killer or the watchdog also exits non-zero.
        """
        if run.memory_kb > limits.memory_kb:
            return Verdict.MEMORY_LIMIT_EXCEEDED
        if run.time_ms >= limits.time_ms:
            return Verdict.TIME_LIMIT_EXCEEDED
        if run.output_kb > limits.output_kb:
            return Verdict.OUTPUT_LIMIT_EXCEEDED
        if run.exit_code != 0:
            return Verdict.RUNTIME_ERROR
        if not self._checker.matches(case.expected_stdout, run.stdout):
            return Verdict.WRONG_ANSWER
        return Verdict.ACCEPTED

Judging stops at the first failing case, which saves most of the fleet's work (most failures are early) and keeps secret test data from leaking through timing. Cases stream to the browser as they complete over a WebSocket, while the durable record goes to the submission store — so a reload shows the same thing the stream did. Running the demo shows every verdict in the taxonomy, what the browser received, and a stale worker being fenced off the lease that replaced it:

sub-correct  Accepted              3/3  streamed=['t1', 't2', 't3']
sub-wrong    Wrong Answer          1/3 at t2  streamed=['t1', 't2']
sub-slow     Time Limit Exceeded   0/3 at t1  streamed=['t1']
sub-hungry   Memory Limit Exceeded 0/3 at t1  streamed=['t1']
sub-crash    Runtime Error         1/3 at t2  streamed=['t1', 't2']
sub-broken   Compile Error         0/3  streamed=[]
worker A claims sub-correct        -> token lt-1, depth (ready, in flight) (1, 1)
31 s later, the reaper runs        -> requeued ['sub-correct']
worker B judges sub-wrong          -> token lt-2, completed and released
worker C re-claims sub-correct     -> token lt-3, attempt 2
worker A wakes up, completes late  -> rejected: token 'lt-1' no longer holds the lease on 'sub-correct'
worker C completes it              -> depth (0, 0), dead letters []

Deep dive: contest mode

The probing question is "what changes when 50,000 people submit in the same ten minutes?" Three things: load shape, ranking, and trust.

Load shape. A contest is a scheduled 10x spike, so pre-scale rather than autoscale — the fleet is warm before the start bell, because scaling out after the spike arrives means the first five minutes are the worst five minutes. Contest submissions ride their own high-priority queue, and practice judging is throttled for the window.

Ranking. Score and penalty land in a Redis sorted set keyed by (score desc, penalty asc), updated by the worker on each accepted verdict; the top-100 read is one ZREVRANGE and a user's own rank one ZREVRANK. The scoreboard freezes for the final hour: the service keeps updating the true set and serves a snapshot taken at freeze time, which preserves the drama and, more practically, stops the last hour becoming a scoreboard-scraping arms race.

Trust. Two mechanisms. A rejudge re-runs every submission of a problem when a test set was wrong or a limit too tight; because judging is deterministic and idempotent, a rejudge is just the same journal of submissions replayed through a bumped PROBLEM.version. And plagiarism detection runs after the contest: normalize each source (strip comments, rename identifiers, canonicalize whitespace), hash overlapping k-grams and keep a deterministic sample of them (the winnowing scheme behind MOSS), then compare fingerprint sets pairwise within the contest. It flags for humans; it never auto-bans.

Scaling, bottlenecks and failure modes

v2: priority queues, a pre-warmed pinned pool plus autoscaled overflow, and test data on local NVMe.

flowchart LR
    subgraph edge["Edge"]
        e_cdn["CDN"]
        e_api["API gateway"]
    end
    subgraph queues["Queues"]
        q_contest[["Contest queue (high priority)"]]
        q_normal[["Practice queue"]]
    end
    subgraph fleet["Judge fleet"]
        f_pool_a["Pinned pool with warm sandboxes"]
        f_pool_b["Overflow pool (autoscaled on queue age)"]
    end
    subgraph data["Data"]
        d_sub[("Submission store, sharded by user_id")]
        d_cases[("Test cases in object storage")]
        d_mirror[("Local NVMe test mirror")]
        d_board[("Redis sorted set per contest")]
        d_arch[("Cold archive of sources")]
    end
    e_cdn --> e_api
    e_api --> q_contest --> f_pool_a
    e_api --> q_normal --> f_pool_a
    q_normal --> f_pool_b
    d_cases --> d_mirror
    f_pool_a --> d_mirror
    f_pool_b --> d_mirror
    f_pool_a --> d_sub
    f_pool_a --> d_board
    d_sub --> d_arch

What breaks first, and what you do about it:

  • The contest start: 1,000 submissions/s against ~500 cores. The queue absorbs it, the high-priority lane keeps contest latency bounded, and the honest degradation is a visible queue position rather than a timeout.
  • A poison submission that reliably crashes its worker: bounded attempts, then a dead letter and Internal Error. Without the cap it takes down workers indefinitely.
  • A problem with a huge test set: cases stream from local NVMe, and per-case results let a worker abandon early. Cap total judging time per submission so one problem cannot monopolize a core.
  • Timing flakiness near the limit: pin cores, use CPU time, keep limits generous, and re-run once at the boundary before reporting Time Limit Exceeded — flaky verdicts destroy trust faster than slow ones.
  • A sandbox escape: assume it will happen. Workers hold no credentials worth stealing, run in their own account and VPC with no route to production data, and are recycled after a bounded number of runs.
  • Losing the leaderboard cache: rebuild it from CONTEST_ENTRY rows, which are the source of truth. The sorted set is a materialized view, never the record.

Trade-offs summary

Decision Chosen Alternatives Why
Queue semantics Leased, at-least-once, bounded attempts Fire-and-forget; exactly-once A crashed worker must not lose a verdict; idempotent writes make duplicates harmless
Isolation Container + seccomp + cgroups + no network, one per run Process-level limits; a shared VM Independent controls; a single bypass is not a breach
Sandbox start-up Pre-warmed pool, reset between runs Fresh container per run A cold start would dominate a 100 ms test case
Test data Object storage, mirrored to worker NVMe Fetch per submission 30 GB total; a per-run fetch adds latency and a dependency
Verdict on failure Stop at the first failing case Run every case Saves most of the fleet's work and leaks less about secret cases
Autoscaling signal Queue depth and oldest-message age CPU utilisation CPU is pinned at 100% by design and says nothing
Leaderboard Redis sorted set, rows as truth Query the submission store live O(log n) rank updates at contest write rates

Interviewer follow-ups

Why not exactly-once judging?

Because it does not exist across a crash boundary. You get at-least-once execution plus an idempotent verdict write keyed by submission_id, which is effectively-once. Re-judging costs CPU; writing two conflicting verdicts costs trust.

How do you keep test cases secret?

They never leave the worker network, never appear in an API response, and a failing verdict names the case only for samples. Timing side channels are limited by stopping at the first failure and by reporting bucketed runtimes.

A user submits a fork bomb. What happens?

The cgroup pid limit refuses the extra processes, the memory cap bounds the damage, and the watchdog kills the run at the wall-clock limit. The verdict is Runtime Error and no other submission on the host notices.

How would you support a problem with multiple valid answers?

A special judge: a checker program supplied by the problem author that reads the input and the contestant's output and returns a verdict. Run it in the same sandbox with its own limits — it is untrusted code too, just written by someone you trust more.

How do you rejudge a problem whose test data was wrong?

Bump PROBLEM.version, enqueue every affected submission on a low-priority lane, and write new verdicts against the new version so history stays auditable. Determinism is what makes this safe: the same source and the same tests give the same answer.

Where does the 100 ms per test case go?

Almost entirely into the submission's own compute. Sandbox setup is amortised by the warm pool, test data comes off local NVMe at ~50 µs per MB, and the result write is one row. If setup were on the critical path it would be the dominant cost, which is exactly why it is not.

Interview tip

Convert the load into cores in the first five minutes: "100 submissions/s x 50 cases x 100 ms is about 500 busy cores." That reframes the problem from a web service into a batch fleet, and every later answer — priority queues, pinning, warm sandboxes, autoscaling on queue age — follows from it.

Common mistake

Saying "run it in a Docker container" and moving on. A container is a namespace, not a security boundary: without seccomp, cgroups, dropped capabilities and an empty network namespace it is a convenient way to run a crypto miner. Name the specific controls, and name what each one stops.

45-minute pacing

Minutes What to say and draw
0–5 Clarify: ten languages, stdin/stdout judging, secret cases, contests, verdict in seconds.
5–9 Estimation: 100 submissions/s, 500 busy cores, 30 GB of test data. State that this is a CPU-bound batch system.
9–14 API (submit, stream, history, leaderboard) and the data model; draw the submission state machine.
14–22 v1 diagram; narrate accept-then-queue and the worker loop with streamed per-case results.
22–38 Deep dives: the leased queue and worker pool, the sandbox controls one by one, verdict precedence and determinism.
38–45 Contest mode (priority lane, frozen scoreboard, rejudge, plagiarism) and the trade-offs table.