Interview panel calibration agenda: a 60-minute session plan

A calibration session is the hour a panel spends agreeing on what good looks like before it scores a single real candidate. It exists because of how a structured interview is built. The U.S. Office of Personnel Management (OPM) defines a structured interview as an assessment method that measures job-related competencies by asking about behavior in past experiences, or proposed behavior in hypothetical situations. Every candidate is asked the same predetermined questions in the same order, and every response is evaluated using the same rating scale and the same standards for acceptable answers. OPM also describes how panel scoring is supposed to work: each panel member rates an applicant's responses individually, and the panel then holds a group discussion to resolve significant scoring discrepancies. Calibration is where the panel rehearses that second step before it counts, so the discussion after a real interview is about evidence rather than taste.
Run the session once per panel, before the first interview for a role, and again whenever a new member joins. The facilitator is normally the recruiter or HR partner who owns the process; the hiring manager sits as a panel member, not as the chair. Bring the question set, the rating scale with its behavioral benchmarks, two sample answers (one strong, one mixed), and the blank calibration record described at the end. Sixty minutes is enough if the talking stays inside the blocks below.
0:00-0:05 — State the job of the hour
Who speaks: the facilitator, uninterrupted. What the panel does: listens, and writes the role title and the competencies being assessed at the top of their note sheets. The competencies are not chosen in this meeting. OPM's guidance is that interview questions should be based on the specific job competencies identified through job analysis as critical to job success, so the facilitator simply reads the competency list the job analysis produced and checks that every panel member has the same version. What gets written down: the session header on the calibration record — role, date, facilitator, panel members present.
Facilitator line: "By 60 minutes from now, we will have scored the same two answers and argued about them, so that next week we argue about candidates less. Nothing in this hour is about any real applicant."
0:05-0:12 — Walk the rating scale out loud
Who speaks: the facilitator reads; panel members interrupt only to ask what a level means. What the panel does: follows the rating scale benchmark by benchmark. In OPM's approach, the most structured interviews score responses according to benchmarks of proficiency, and the rating scale is developed with subject matter experts, with behavioral examples that differentiate one proficiency level from the next. OPM's own competency scale runs across five levels, from Level 1 (Awareness) to Level 5 (Expert). Whatever scale the panel uses, this block is where each level gets read in plain words, with the behavioral example attached to it. What gets written down: any word in a benchmark that two members read differently gets a one-line shared gloss, written on the record in the panel's own wording.
Facilitator line: "If a benchmark word means two things in this room, it will mean five things across five candidates, so we fix the word now, not the score later."
0:12-0:20 — Map the questions to owners
Who speaks: each panel member in turn, naming the questions they will lead. What the panel does: walks the question set in interview order and assigns a lead asker and an agreed set of follow-up probes for every question. This matters because OPM notes that interviews with a high degree of structure ask all applicants the same exact set of pre-defined lead and probe questions, and that higher degrees of structure show higher levels of validity, rater reliability, rater agreement, and less adverse impact. The panel also confirms the two question types in the set: situational questions, which ask what a candidate would do in a job-like situation, and behavioral questions, which ask what the candidate did in the past, on the principle OPM calls behavioral consistency — past behavior as the predictor of future behavior. What gets written down: the question ownership list, with the probes that are allowed and a note that unscripted follow-ups are the exception, not the habit.
Facilitator line: "Same questions, same order, same probes for every candidate. If a question is not on this sheet, it does not get asked this round."
0:20-0:32 — First practice score, in silence
Who speaks: nobody. The facilitator reads the first sample answer aloud, or plays the recording, and then keeps time. What the panel does: scores the answer independently, in writing, exactly as OPM's panel model requires — individual ratings first, discussion second. Each member writes three things: the score, the benchmark phrase that justifies it, and one sentence of the answer quoted or closely paraphrased as evidence. No glancing at a neighbor's sheet, no early verdicts. What gets written down: one independent score per member, with its evidence note, on the practice scoring sheet that will be attached to the calibration record.
Facilitator line: "Pens only for the next ten minutes. If you finish early, reread the benchmark for the level above the one you picked and check why the answer is not there."
0:32-0:42 — Reveal the scores and resolve the gaps
Who speaks: the facilitator reads out all scores at once; then the members furthest apart speak first. What the panel does: compares scores for the same answer. Identical or adjacent scores pass without discussion. The time goes to the significant discrepancies, which is precisely the work OPM assigns to the panel's group discussion. Members defend a score only by pointing at the evidence note they wrote and the benchmark wording it matches; "I just felt it was stronger" is ruled out of order. Two outcomes are possible: someone moves because the evidence supports a different level, or the benchmark itself turns out to be ambiguous — in which case the benchmark wording is what gets changed. What gets written down: the final agreed reading of that answer, any benchmark wording the panel amended, and the reason, in one sentence each.
Facilitator line: "Show me the sentence in the answer and the sentence in the benchmark. If neither exists, the score is a guess, and guesses do not go on the record."
0:42-0:50 — Second practice score, and the note-taking standard
Who speaks: the facilitator, briefly, then the member who scored lowest on round one. What the panel does: scores the second sample answer independently, faster this time, then compares. The second answer is deliberately mixed — strong on one competency, thin on another — so the panel can practice scoring competency by competency instead of forming one overall impression. The facilitator then sets the note-taking standard for live interviews: notes capture what the candidate said and did in the answer, in the candidate's own terms, not adjectives about the candidate. What gets written down: the round-two scores, the spread between highest and lowest, and the panel's short list of note-taking rules, copied onto the calibration record.
Facilitator line: "Write what the candidate said, not what you concluded. Conclusions are what the discussion at the end is for."
0:50-0:56 — Agree the rules for the real interviews
Who speaks: the hiring manager confirms the decision process; the facilitator states the working rules. What the panel does: agrees, out loud and on the record, how the real round will run. Each member scores every answer independently before any discussion of a candidate begins. Scores are submitted before the panel debrief, and a score can be changed afterward only for a stated evidence reason, with the change and the reason both kept. Scheduling follows the same discipline OPM builds into its hiring process model, which treats hiring as a sequence of named steps, each with an owner and an expected time to complete; the panel leaves with interview dates, a debrief date, and a named owner for chasing missing score sheets. What gets written down: the working rules, the dates, and the owner, on the calibration record.
Facilitator line: "Independent first, discussion second, every time. The first voice in the debrief will never be the most senior one; it will be the evidence."
0:56-1:00 — Read the record back and close
Who speaks: the facilitator reads; every member confirms. What the panel does: hears the calibration record read back — benchmarks amended, glosses agreed, rules adopted — and corrects anything that does not match what they remember agreeing. What gets written down: each member's initials beside the record, and the date of the next calibration check, set for after the first real debrief so the panel can compare its practice spreads against its live ones.
Facilitator line: "If it is not on this page, we did not agree it. Tell me now, or it waits for the next session."

The one-page calibration record
The session produces exactly one document, kept with the interview materials for the role:
- Role, requisition reference, date of session, facilitator, and panel members present
- Competency list assessed, copied from the job analysis, with the rating scale version used
- Benchmark glosses: every term the panel had to define in its own words
- Benchmark amendments: original wording, replacement wording, and the one-sentence reason
- Question ownership list: lead asker and approved probes for each question
- Practice results: each member's independent scores for both sample answers, the spread, and how each significant discrepancy was resolved
- Note-taking rules adopted by the panel
- Working rules for the live round: independent scoring before discussion, evidence-based score changes only, debrief date and score-sheet owner
- Panel members' initials and the date set for the next calibration check
A record like this is what makes the panel's later decisions explainable. When a score is questioned — by a hiring manager, by HR, or by the panel itself a month later — the answer is not a memory of a meeting. It is a page that shows what the panel agreed a good answer looks like, before it met any of the people it was scoring.
Sources
- U.S. Office of Personnel Management, "Structured interviews" — definition of a structured interview (job-related competencies, past behavior and hypothetical situations), the same predetermined questions in the same order for all candidates, evaluation on the same rating scale and standards for acceptable answers, and panel scoring in which each member rates individually before a group discussion resolves significant scoring discrepancies. https://www.opm.gov/policy-data-oversight/assessment-and-selection/structured-interviews/
- U.S. Office of Personnel Management, "Structured interviews" (assessment methods guidance) — degrees of structure: pre-defined lead and probe questions scored according to benchmarks of proficiency; higher structure associated with higher validity, rater reliability, rater agreement and less adverse impact; questions based on job competencies identified through job analysis. https://www.opm.gov/policy-data-oversight/assessment-and-selection/other-assessment-methods/structured-interviews/
- U.S. Office of Personnel Management, "Leveraging assessments" (webinar slides) — rating scale development with subject matter experts, behavioral examples differentiating proficiency levels, OPM's five-level competency proficiency scale from Level 1 (Awareness) to Level 5 (Expert), and the situational and behavioral question types, including the behavioral consistency principle. https://www.opm.gov/policy-data-oversight/assessment-and-selection/assessment-strategy/leveraging-assessments.pdf
- U.S. Office of Personnel Management, Hiring Process Analysis Tool — OPM's hiring process model as a sequence of steps with owners and recommended completion times; the Request for Personnel Action (SF-52) step is performed by the manager, who creates and obtains approval of the request. https://www.opm.gov/policy-data-oversight/human-capital-management/hiring-reform/hiring-process-analysis-tool/
Run this on your own hiring data
A six-week recruiting pilot measures exactly these things — parser accuracy, reviewer agreement, time per application and compliance records — on your role family, with your reviewers, before you decide anything.