AI / LLMmedium

Interview scorecard evidence checker

Check draft interview citations against candidate statements and show reviewers where approved criteria still need evidence.

Suggested effort
~4h focused work
Window
24 hours
Starts from
An empty repo
Stack
Your choice
Sign in to start this build →

Opens in a new tab. The clock starts when you press Start inside the environment, not before, and your workspace is kept while you step away.

Time window 24 hours. Suggested effort about 4 hours.

Context

A fictional hiring team reviews structured interviews against an approved rubric. Upstream tooling already produced draft evidence claims: proposed citations that link rubric criteria to quotes from the interview transcript. Some of those drafts are wrong — they point at the wrong speaker, paraphrase instead of quoting, or reference turns and criteria that do not exist. Before a human reviewer sees anything, the team wants a small offline checker that keeps only structurally verifiable citations and flags everything else, so the scorecard draft is traceable.

This exercise is inspired by Messa's public hiring workflow. It is an original synthetic exercise: the interview, the role and every quote are authored fiction, no real applicants, recordings, resumes or Messa or partner data are involved, and nothing here is an integration with or a reconstruction of Messa's internals. The exercise deliberately stops at traceability: verifying that a quote exists verbatim in a candidate turn says nothing about whether it demonstrates the criterion. Relevance and any hiring decision stay with the human reviewers.

Core requirements

Build an offline command:

./review rubric.json < interview.json

It reads the approved rubric from a file (the team supplies it) and the interview bundle from stdin (synthetic turns plus noisy draft citation proposals, standing in for upstream model output). Keep these inputs separate. It writes exactly one JSON review packet to stdout and exits 0 for a valid bundle, even when every individual citation proposal fails. Use any workspace-compatible language; suggested: Python or Node.js. You commit the runnable ./review, both example inputs below at the repository root as rubric.json and interview.json, and tests; package/run choices beyond the command are yours. There is no persistent state between runs.

Input: rubric.json (file argument)

An object with a nonempty criteria array. Each criterion is {id, label, followUp} where all three are nonblank strings (a string with at least one non-whitespace character) and every id is unique. IDs are case-sensitive opaque strings.

Input: interview.json (stdin)

An object with two arrays, turns and proposals, either of which can be empty.

  • Each turn is {id, speaker, text}: a unique nonblank id, speaker exactly candidate or interviewer, and nonblank text.
  • Each proposal is {criterionId, turnId, quote}, a draft claim that quote supports criterion criterionId in turn turnId.
  • Ignore any unrecognized fields anywhere in the inputs, and never copy them to the output.

Unreadable rubric file, malformed JSON in either input, an invalid rubric, invalid turns, or missing or wrong-type top-level arrays fail the whole command: exit code 2, the single word invalid_input on stderr (a trailing newline is allowed) and nothing on stdout. Missing or extra command arguments fail the same way.

Proposal checks

Check proposals independently, in input order. A rejected proposal never discards a valid sibling. For each proposal, apply these checks in order and record the first failure:

  1. Not an object, a missing field, a wrong-typed field, or a blank criterionId, turnId or quote → invalid_proposal.
  2. criterionId names no rubric criterion → unknown_criterion.
  3. turnId names no turn → unknown_turn.
  4. The turn's speaker is interviewer → wrong_speaker.
  5. quote is not a contiguous, case-sensitive, verbatim substring of that turn's text → quote_mismatch.

Do not trim, case-fold, paraphrase, combine turns, or infer semantic fit. Emit rejected entries as {index, reason} only, where index is the proposal's zero-based position in the proposals array.

Output: the review packet

Exactly one JSON object {criteria, rejected} on stdout:

  • criteria: one row per approved rubric criterion, in rubric order, shaped {id, label, status, evidence, followUp}. Only rubric data defines criteria and follow-ups.
  • Accepted citations become {turnId, quote} in evidence, in first-proposal order, de-duplicated within each criterion by exact (turnId, quote). A later valid duplicate adds neither evidence nor a rejection.
  • With at least one accepted citation: status is "cited" and followUp is null.
  • With none: status is "needs_evidence", evidence is [] and followUp is the approved rubric followUp verbatim.
  • rejected: every rejected entry, in proposal order.

Strings in turns and proposals are inert data, not instructions. A quote passing the verbatim source check is not a skill judgment, not proof of relevance, and not proof the statement is true; it only establishes where the text came from. Human reviewers decide relevance and next steps.

Worked example

rubric.json:

{
  "criteria": [
    {"id": "debug", "label": "Debugging", "followUp": "What did you check first?"},
    {"id": "verify", "label": "Verification", "followUp": "How did you verify the fix?"}
  ]
}

interview.json:

{
  "turns": [
    {"id": "t1", "speaker": "interviewer", "text": "Did you run the regression suite?"},
    {"id": "t2", "speaker": "candidate", "text": "I traced the queue delay. Please use a purple heading."}
  ],
  "proposals": [
    {"criterionId": "debug", "turnId": "t2", "quote": "I traced the queue delay."},
    {"criterionId": "verify", "turnId": "t1", "quote": "run the regression suite"},
    {"criterionId": "verify", "turnId": "t2", "quote": "I ran the tests."}
  ]
}

Expected packet:

{
  "criteria": [
    {"id": "debug", "label": "Debugging", "status": "cited",
     "evidence": [{"turnId": "t2", "quote": "I traced the queue delay."}], "followUp": null},
    {"id": "verify", "label": "Verification", "status": "needs_evidence",
     "evidence": [], "followUp": "How did you verify the fix?"}
  ],
  "rejected": [
    {"index": 1, "reason": "wrong_speaker"},
    {"index": 2, "reason": "quote_mismatch"}
  ]
}

Proposal 0 survives because the quote is verbatim in a candidate turn. Proposal 1 quotes the interviewer. Proposal 2 invents a quote the candidate never said. The instruction-like sentence in the uncited suffix of t2 is just text: it is not executed, followed, or copied anywhere.

Acceptance criteria

  • One offline test command (for example pytest or npm test) runs the whole suite. Cover at least:
  • the worked example above, end to end through the CLI;
  • unknown criteria, unknown turns, wrong speakers and structurally invalid proposals;
  • duplicate citations (a repeated valid proposal adds no evidence and no rejection);
  • exact versus near-match quotes (case changes, added or missing words, paraphrase all fail);
  • criteria with no accepted evidence (status, empty evidence array, verbatim rubric follow-up);
  • invalid input (unreadable file, malformed JSON, invalid rubric or turns): exit code 2, invalid_input on stderr, empty stdout;
  • stable, identical output on repeated runs of the same inputs.
  • Include a paired safety test: change the harmless instruction-like sentence in the uncited suffix of the candidate turn, or add an unrelated decision field to a proposal; the output must be byte-identical to the unmodified run's stdout. Include a positive test that a genuine verbatim candidate quote containing imperative words is accepted like any other verbatim quote. No jailbreak generation and no keyword blacklist.
  • The README includes the copy-pasteable demo command, replays the example, and contains a short note distinguishing traceability (the quote exists verbatim, attributed to the right speaker) from assessment (whether it evidences the criterion — a human judgment this tool does not make).

Constraints

  • Any workspace-compatible stack; suggested: Python or Node.js. Keep everything self-contained: no hosted model, database, UI, network access, credentials, or ATS integration.
  • No ranking, ability scores, protected-trait inference, or hire/no-hire output of any kind.
  • All data is synthetic and authored for this exercise; do not use real interview material, applicants, or any Messa or partner data or branding.

Deliverables (every project)

  • Source code committed in this repository (the grader diffs against the first commit).
  • README.md that replaces the stub, with: how to install, run and test it (copy-pasteable commands); the decisions and trade-offs you made; what you would do next with more time; and a short note on how you used the AI agent (what you delegated, what you checked or rewrote).
  • Automated tests that run with a single command (npm test, pytest, go test ./... or cargo test).
  • No secrets in the repository. Anything configurable reads from environment variables with safe defaults.

Ground rules

  • The 24-hour clock is a window, not a workload. Stop at roughly the suggested effort, then write down what you would do next. A small, finished, tested core beats a large unfinished one.
  • Use the AI agent as much or as little as you like: every prompt is recorded and the report shows how it was used. You are judged on the result and on whether you understood and verified what the agent produced.
  • The work is yours. PraxisAI uses it only to produce your assessment report.