How to Build an AI Interviewer That Conducts Interviews and Scores Responses

Introduction

To build an AI interviewer that conducts structured technical or behavioural interviews and produces reliable, legally defensible candidate evaluations, you must replace open-ended LLM impressions with Behaviorally Anchored Rating Scales (BARS), bounded adaptive depth-probing, and deterministic code test-case execution. When an unstructured LLM conducts an interview, it often asks leading questions, penalises non-native speech disfluencies ("um", pauses, regional accents), or exhibits recency bias. By structuring the interview as a finite state machine—where each competency has a standardized root question, up to 2 adaptive follow-up probes triggered only when specific rubric evidence is missing, and a 1-to-5 BARS rubric anchored to explicit behavioural indicators—you achieve high inter-rater reliability and fairness.

Tip: Architecture Reference: Internal Link Suggestion: Connect this structured interview pipeline with Article 5: How to Build an AI Recruitment Agent That Screens Resumes and Shortlists Candidates and Article 17: How to Build an AI Phone Receptionist.

Structured Interview Workflow & BARS Evaluation Architecture

flowchart TD
    Consent([Candidate Informed Consent & Audio Check]) --> CompetencyLoop[Load Next Competency & Root Question]
    CompetencyLoop --> VoiceInteraction[LiveKit / Pipecat Voice Turn + Transcript Display]
    VoiceInteraction --> ProbeCheck{Evidence Completeness Analyzer: Situation, Action, Outcome?}
    ProbeCheck -- "Missing Detail & probes_used < 2" --> GenerateProbe[Ask Targeted Clarifying Probe]
    GenerateProbe --> VoiceInteraction
    ProbeCheck -- "Complete or probes_used == 2" --> CodeCheck{Includes Coding Task?}
    CodeCheck -- "Yes" --> CodeSandbox[Run Candidate Code Against Unit Test Suite]
    CodeCheck -- "No" --> BARSScorer[BARS 1-5 Rubric Evaluator with Verbatim Quotes]
    CodeSandbox --> BARSScorer
    BARSScorer --> Report([Human Hiring Manager Review Report + Audio/Code Artifacts])

Bias Mitigation, Candidate Consent & Evaluation Reliability

1. Mandatory Informed Consent & Accommodations: Before recording a single audio frame, display a clear disclosure explaining how the AI interviewer operates, what competencies are evaluated, that a human hiring manager reviews all transcripts and scores, and provide a one-click option to request a traditional human interview or accessibility accommodation (e.g., extended response time or text-only mode).
2. Acoustic & Accent Fairness: Evaluate strictly on the semantic content of the transcript and executed code test cases—never on speech rate, pitch, filler word frequency, or accent. Show the live transcript on screen so the candidate can see and correct any ASR misrecognition of technical terms ("idempotency", "PostgreSQL").
3. Measuring Evaluation Reliability (Cohen's Quadratic Weighted Kappa $\kappa_w$): Validate your BARS evaluator against a benchmark set of 50 human-graded interview transcripts and require Quadratic Weighted Kappa $\kappa_w \ge 0.75$ before production deployment.

Technology Comparison: Paid vs. Open-Source Stack

Recommended Implementation Options

Option A: Best Practical Implementation (Recommended): LiveKit Agents (https://github.com/livekit/agents) WebRTC + Deepgram Nova-3 with technical keyterms + LangGraph (https://github.com/langchain-ai/langgraph) competency state machine + OpenAI Structured BARS Evaluator + isolated pytest code runner.

Option B: Fully Open-Source Self-Hosted Implementation: Pipecat (https://github.com/pipecat-ai/pipecat) + Faster-Whisper (https://github.com/SYSTRAN/faster-whisper) + local Ollama (https://github.com/ollama/ollama) (qwen2.5:32b) + Dockerized code runner.

Option C: Enterprise Assessment Platform: Multi-region WebRTC rooms + Monaco code editor sync + Quadratic Weighted Kappa calibration dashboard.

Step-by-Step Implementation Guide

Step 1: Project Structure

ai-interviewer/
├── app/
│   ├── __init__.py
│   ├── bars_rubric.py
│   └── reliability_metrics.py
├── tests/
│   └── test_interviewer.py
└── requirements.txt

Step 2: Implement Adaptive Probe Logic & BARS Rubric Schemas (app/bars_rubric.py)

from typing import Dict, List, Optional
from pydantic import BaseModel, Field

class CompetencyBARSRubric(BaseModel):
    competency_id: str
    competency_name: str
    root_question: str
    max_probes: int = 2
    anchor_level_1_unsatisfactory: str
    anchor_level_3_proficient: str
    anchor_level_5_exceptional: str

class ProbeDecision(BaseModel):
    has_specific_example: bool
    has_technical_tradeoffs: bool
    has_measurable_outcome: bool
    should_ask_probe: bool
    next_probe_question: Optional[str] = None

def decide_deterministic_probe_gate(
    probes_already_asked: int,
    max_probes: int,
    has_specific_example: bool,
    has_technical_tradeoffs: bool,
    has_measurable_outcome: bool,
) -> Tuple[bool, str]:
    """Deterministically decides whether to ask a follow-up probe or advance to the next competency."""
    if probes_already_asked >= max_probes:
        return False, "MAX_PROBES_REACHED"
    if not has_specific_example:
        return True, "MISSING_CONCRETE_EXAMPLE"
    if not has_technical_tradeoffs:
        return True, "MISSING_TECHNICAL_TRADEOFFS"
    if not has_measurable_outcome:
        return True, "MISSING_MEASURABLE_OUTCOME"
    return False, "EVIDENCE_COMPLETE"

class CompetencyScoreCard(BaseModel):
    competency_id: str
    bars_score: int = Field(..., ge=1, le=5)
    verbatim_candidate_quotes: List[str] = Field(..., min_length=1)
    matched_anchor_rationale: str
    coding_tests_passed: Optional[int] = None
    coding_tests_total: Optional[int] = None

Step 3: Implement Quadratic Weighted Kappa ($\kappa_w$) for Evaluation Reliability (app/reliability_metrics.py)

To verify that your AI BARS evaluator agrees with human senior engineers across 1-to-5 ordinal scores, compute Quadratic Weighted Cohen's Kappa ($\kappa_w$):

from typing import List

def compute_quadratic_weighted_kappa(
    human_scores: List[int],
    ai_scores: List[int],
    min_rating: int = 1,
    max_rating: int = 5,
) -> float:
    """Computes Quadratic Weighted Cohen's Kappa between human and AI ordinal 1-5 rubric ratings."""
    if len(human_scores) != len(ai_scores) or len(human_scores) == 0:
        raise ValueError("Score lists must be non-empty and of equal length.")

    num_classes = max_rating - min_rating + 1
    n = float(len(human_scores))

    # Observed confusion matrix O and marginal histograms
    obs = [[0.0] * num_classes for _ in range(num_classes)]
    hist_h = [0.0] * num_classes
    hist_a = [0.0] * num_classes

    for h, a in zip(human_scores, ai_scores):
        i = h - min_rating
        j = a - min_rating
        obs[i][j] += 1.0
        hist_h[i] += 1.0
        hist_a[j] += 1.0

    num = 0.0
    den = 0.0
    for i in range(num_classes):
        for j in range(num_classes):
            weight = float((i - j) ** 2) / float((num_classes - 1) ** 2)
            expected_ij = (hist_h[i] * hist_a[j]) / n
            num += weight * (obs[i][j] / n)
            den += weight * (expected_ij / n)

    if den == 0.0:
        return 1.0
    return round(1.0 - (num / den), 4)

Testing the AI Interviewer (tests/test_interviewer.py)

from app.bars_rubric import decide_deterministic_probe_gate
from app.reliability_metrics import compute_quadratic_weighted_kappa

def test_adaptive_probe_gate_stops_at_max_probes_or_complete_evidence():
    should_probe, reason = decide_deterministic_probe_gate(
        probes_already_asked=0,
        max_probes=2,
        has_specific_example=True,
        has_technical_tradeoffs=False,
        has_measurable_outcome=True,
    )
    assert should_probe is True
    assert reason == "MISSING_TECHNICAL_TRADEOFFS"

    capped, cap_reason = decide_deterministic_probe_gate(
        probes_already_asked=2,
        max_probes=2,
        has_specific_example=False,
        has_technical_tradeoffs=False,
        has_measurable_outcome=False,
    )
    assert capped is False
    assert cap_reason == "MAX_PROBES_REACHED"

def test_quadratic_weighted_kappa_reliability():
    human = [5, 4, 3, 2, 1, 4, 5, 3]
    ai_good = [5, 4, 3, 2, 1, 4, 4, 3]
    kappa = compute_quadratic_weighted_kappa(human, ai_good)
    assert kappa > 0.90

Human Oversight & Cost Considerations

• Never Auto-Reject Without Human Oversight: Present the CompetencyScoreCard alongside the playable audio timestamps, verbatim candidate quotes, and unit-test execution logs to the human hiring panel.
• Cost Breakdown: A 30-minute voice + coding interview costs ~$0.55–$0.85 per interview all-in (WebRTC + Deepgram Nova-3 STT + LLM probe/rubric scoring + streaming TTS).

Useful Tools, Documentation and GitHub Repositories

Frequently Asked Questions

What if the speech-to-text model mishears a technical term like idempotency as item potency?

Pass a domain keyterm list to the STT engine and instruct the BARS evaluator prompt to account for phonetic ASR homophones of technical terms, while displaying the live transcript so the candidate can clarify.