Context Inspector: Measuring Context Quality Before the LLM Call

All examples in this article are synthetic — they illustrate engineering patterns only, not production or customer data.

Most Gen AI evaluation focuses on the generated response: whether it is accurate, grounded, and safe. Those evaluations are essential, but they cannot detect problems that originate before the model generates a single token.

In Retrieval-Augmented Generation, or RAG, the model is only as good as what it’s handed. If the retrieved context is stale, one-sided, missing a required source, or quietly contradicts itself, the model can still produce a fluent, confident-sounding answer — one that happens to be wrong. By the time you’re evaluating that answer, you’re debugging the wrong layer.

Context Inspector is a research prototype built to catch this type of issue earlier, at the moment after context is assembled but before it’s sent to the LLM. It asks a simple question: has the model been given a reliable basis to answer, before we spend the inference call finding out?

In this article, we’ll look at what makes context “good” before inference, how to measure common failure modes such as missing, stale, or conflicting evidence, and how those signals can be used to catch context-quality problems before the LLM call.

Pre-Inference Quality vs. Post-Inference Quality

Post-inference quality asks whether the model produced a good answer. This response-centric evaluation examines the generated output for accuracy, clarity, groundedness, completeness, tone, and policy alignment.

Pre-inference context quality asks a different question: did we give the model the right evidence to produce a good answer? This becomes an evidence-centric approach. It evaluates the assembled context before generation and asks whether the context is sufficient, current, balanced, consistent, and appropriate for the task.

User query
    -> Retrieval
    -> Context assembly
    -> Pre-inference context inspection
    -> LLM call
    -> Generated response
    -> Post-inference evaluation

Both stages matter. Post-inference evaluation validates the final answer, while pre-inference inspection validates the evidence that the model is about to use.

The distinction is important because many RAG failures are not caused by the model alone. They are caused by context assembly failures.

Why Pre-Inference Context Quality Matters

In a RAG-based Gen AI system, the LLM does not receive only the user’s question. It receives a carefully assembled context window that may include the system prompt, tool definitions, retrieved knowledge, conversation history, memory, agent scratchpad content, and the current user message. Each layer serves a different purpose, and each consumes part of the available token budget.

Figure 1: Representative LLM context-window anatomy. Context Inspector operates pre- inference on the RAG context layer, measuring coverage, utilization, source starvation, and contradiction risk before the LLM call.

The diagram shows that the RAG context layer is only one part of the full prompt, but it is often the layer that determines whether the model has the right evidence to answer. In the representative allocation, the RAG context occupies approximately 18,000 tokens out of a 128,000-token context window. The remaining budget is allocated across system instructions, tool definitions, conversation history, memory, scratchpad content, the current user message, and unused headroom.

A larger context window does not automatically create a better answer. A model can have plenty of remaining token capacity and still receive poor-quality evidence. The RAG layer may be incomplete, stale, contradictory, or dominated by a single source. In those cases, the model may generate an answer that appears confident but is not properly grounded.

This is where Context Inspector is positioned. Rather than inspecting every part of the prompt or waiting until after the model generates a response, Context Inspector fires pre-inference and focuses on the retrieved context layer. It evaluates whether the RAG context is fit for purpose before the LLM call happens.

From a privacy and governance perspective, this scoping is intentional. Context Inspector can operate on approved metadata and minimized signals such as source, version, timestamp, token count, and policy labels. The purpose is not to broaden access to user or system data. The purpose is to reduce risk by preventing low-quality or inappropriate context from entering the prompt.

What Does “Good Context” Even Mean?

Let the user query be represented as:

q

Let the retrieval system return a candidate set of chunks:

Represent each chunk as:

where x~i is chunk text, s~i is source, T~i is token count, R~i is retrieval relevance, t~i is timestamp, v~i is version, and m~i is metadata.

The final context sent to the model is a selected subset:

The selected context must satisfy a token budget:

Example

Suppose the token budget is B = 20,000. The selected context contains four chunks with token counts 2,200, 1,800, 6,000, and 1,200. The total token use is:

Because 11,200 ≤ 20,000, the selected context satisfies the token budget. However, satisfying the budget does not prove that the context is good. It only proves that it fits. The objective is not simply to maximize relevance. A high-quality context package may need to satisfy multiple constraints at once: source coverage, freshness, token efficiency, contradiction avoidance, and policy admissibility. This makes pre-inference context quality both a measurement problem and an optimization challenge.

The Context Quality Vector (Six Numbers Instead of One Score)

A practical representation is a context-quality vector:

In this vector, 𝜌 represents coverage, 𝜏 represents token utilization, 𝛿 represents source starvation, 𝜅 represents contradiction density, 𝜙 represents freshness, and 𝜋 represents policy admissibility.

Example

A context package may produce:

This means the context covers 75% of the required aspects, uses 56% of the token budget, has one-third of required sources starved, has contradiction density of 17%, has freshness score of 81%, and contains only policy-admissible chunks. The vector is useful because it explains why a context may be risky rather than hiding everything behind a single score.

Coverage: Is the Required Evidence Present?

Coverage measures whether the selected context contains the information needed to answer the query. Let the required answer aspects for query q be:

Let A(d~i) be the set of required aspects covered by chunk d~i:

Coverage is defined as:

A weighted version can account for aspect importance:

Here, w~j represents the importance of aspect a~j, and I(⋅) is an indicator function.

Example

Suppose a query requires four aspects: eligibility, effective date, steps, and availability. The selected context covers eligibility, effective date, and steps, but it does not cover availability. Unweighted coverage is:

For a weighted example, suppose the aspect weights are 0.40, 0.20, 0.25, and 0.15 respectively. Because the first three aspects are covered, weighted coverage is:

The weighted score shows that the selected context contains most of the high-value evidence, but the missing availability aspect still represents a concrete risk.

def coverage_score(selected_chunks, required_aspects):
    covered = set()

    for chunk in selected_chunks:
        covered.update(chunk.get("aspects", []))
    
    if not required_aspects:
        return 1.0

    return len(covered.intersection(required_aspects)) / len(required_aspects
)

Token Utilization: Is the Context Window Being Used Effectively?

Every LLM call operates under a token budget. Too little context can produce under-informed responses. Too much context can increase cost, latency, and noise.

Let B be the token budget and T~i be the token count of chunk d~i. Token utilization is:

A simple utilization risk function is:

Example

Suppose B = 20,000 and the selected context uses 11,200 tokens. Then:

If the configured healthy range is 0.10 ≤ 𝜏 ≤ 0.90, then r~𝜏(𝐶) = 0. If a different context used 19,400 tokens, then:

Because 0.97 > 0.90, the utilization risk would be 1. This does not mean the context is wrong, but it indicates token-budget pressure and a higher chance that important evidence was crowded out.

def token_utilization(selected_chunks, token_budget):
    used_tokens = sum(chunk["tokens"] for chunk in selected_chunks)

    if token_budget <= 0:
        raise ValueError("token_budget must be positive")

    return used_tokens / token_budget

Source Starvation: Are Required Sources Missing?

Many enterprise workflows require evidence from more than one source. A reliable answer may depend on policy, procedure, current system status, or approved reference material. If one source dominates the context, other required sources may be starved.

Let S~q be the set of required sources:

Let n~s(C) be the number of selected chunks from source s:

If each source has a minimum required count m~s, source starvation is:

Example

Suppose the required sources are policy, procedure, and system status. Assume the minimum required count is one chunk per source. The selected context has two policy chunks, one procedure chunk, and zero system-status chunks. Only system status is starved. Therefore:

A starvation score of 0.33 is actionable. It tells the team that the retriever or assembler must include system-status evidence before the context is ready for inference.

from collections import Counter

def source_starvation(selected_chunks, required_sources, min_chunks_by_source =None):
    min_chunks_by_source = min_chunks_by_source or {
        source: 1 for source in required_sources
    }

    counts = Counter(chunk["source"] for chunk in selected_chunks)

    starved_sources = [
        source
        for source in required_sources
        if counts[source] < min_chunks_by_source.get(source, 1)
    ]

    if not required_sources:
        return 0.0, []

    return len(starved_sources) / len(required_sources), starved_sources

Freshness: Is the Context Current Enough?

Relevance does not guarantee freshness. A document can be semantically relevant but operationally outdated. Freshness is especially important when content changes over time or has explicit effective dates.

Let a~i be the age of chunk d~i in days:

A simple freshness score can be modeled with exponential decay:

where 𝜆 controls how quickly older content loses value. For the full context, a relevance-weighted freshness score is:

For content with explicit validity windows, let V~i(t) be 1 when chunk d~i is valid at time t and 0 otherwise:

Then stale-context risk is:

Example

Suppose 𝜆 = 0.01 and three chunks have ages 10, 60, and 1 days. Their freshness scores are approximately:

If their relevance scores are 0.90, 0.80, and 0.70, the context freshness is:

For validity-window risk, if two out of three chunks are currently valid, then:

This indicates that one-third of the selected context is stale under the validity policy.

from datetime import datetime
from math import exp

def freshness_score(chunk, now=None, decay_lambda=0.01):
    now = now or datetime.utcnow()
    age_days = max((now - chunk["timestamp"]).days, 0)
    return exp(-decay_lambda * age_days)

def context_freshness(selected_chunks):
    if not selected_chunks:
        return 0.0

    weighted_sum = 0.0
    total_weight = 0.0

    for chunk in selected_chunks:
        relevance = chunk.get("relevance", 1.0)
        weighted_sum += relevance * freshness_score(chunk)
        total_weight += relevance

    return weighted_sum / total_weight if total_weight else 0.0

Contradiction Density: Does the Context Conflict With Itself?

Contradictions are difficult in RAG systems because every retrieved chunk may appear relevant in isolation. The problem emerges when those chunks are placed together. Let x~ij be 1 when chunks d~i and d~j conflict, and 0 otherwise:

Contradiction density is:

A weighted version can incorporate severity 𝜎~ij:

Example

Suppose the selected context contains four chunks. There are:

possible chunk pairs. If one pair conflicts, then:

If the conflicting pair has severity 0.80, weighted contradiction density is:

A contradiction score should be treated as a risk signal, not a final judgment. The system may apply source precedence rules, remove older chunks, retrieve a newer version, or route the case for additional validation.

def contradiction_density(selected_chunks, conflict_fields):
    if len(selected_chunks) < 2:
        return 0.0
    
    conflicts = 0
    total_pairs = 0

    for i in range(len(selected_chunks)):
        for j in range(i + 1, len(selected_chunks)):
            total_pairs += 1
            conflict_found = any(
                selected_chunks[i].get(field) != selected_chunks[j].get(field)
                for field in conflict_fields
                    if field in selected_chunks[i] and field in selected_chunks[j]
            )

        if conflict_found:
            conflicts += 1

    return conflicts / total_pairs if total_pairs else 0.0

Policy Admissibility: Should This Context Be Used?

Context quality is not only about relevance. It also includes whether the selected context is appropriate for the task and permitted under applicable access and governance rules.

Let P~i(q,u) be 1 when chunk d~i is allowed for query q and role u, and 0 otherwise:

Policy admissibility is:

For stricter workflows, every chunk must be admissible before inference:

Example

Suppose a context package has five chunks and four are admissible for the current task and role. Then:

If the configured policy requires 𝜋(C,q,u) = 1, the context fails the pre-inference gate even though 80% of the chunks are allowed. This strict behavior is intentional in workflows where inappropriate context should not enter the prompt.

def policy_admissibility(selected_chunks, allowed_scopes):
    if not selected_chunks:
        return 0.0

    allowed_count = 0

    for chunk in selected_chunks:
        if chunk.get("scope") in allowed_scopes:
            allowed_count += 1

return allowed_count / len(selected_chunks)

Combining Signals into a Pre-Inference Gate

The individual signals can be combined into a pre-inference quality gate:

Example

Suppose the thresholds are:

For a context package with:

all conditions pass, so:

For a different context with the same values except 𝛿 = 0.33, the gate fails because source starvation exceeds the threshold:

def context_quality_report(selected_chunks, config):
    rho = coverage_score(selected_chunks, set(config["required_aspects"]))
    tau = token_utilization(selected_chunks, config["token_budget"])

    delta, starved_sources = source_starvation(
        selected_chunks,
        set(config["required_sources"]),
        config.get("min_chunks_by_source")
    )

    kappa = contradiction_density(
        selected_chunks,
        config.get("conflict_fields", [])
    )

    phi = context_freshness(selected_chunks)
    pi = policy_admissibility(selected_chunks, set(config["allowed_scopes"]))

    passed = (
        rho >= config.get("min_coverage", 0.80)
        and config.get("min_token_utilization", 0.10)
            <= tau
            <= config.get("max_token_utilization", 0.90)
        and delta <= config.get("max_source_starvation", 0.0)
        and kappa <= config.get("max_contradiction_density", 0.0)
        and phi >= config.get("min_freshness", 0.70)
        and pi == 1.0
    )

    return {
        "passed": passed,
        "metrics": {
            "coverage": rho,
            "token_utilization": tau,
            "source_starvation": delta,
            "contradiction_density": kappa,
            "freshness": phi,
            "policy_admissibility": pi,
        },
        "diagnostics": {
        "starved_sources": starved_sources,
    },
}

Why This Becomes an Optimization Challenge

Pre-inference context quality is not only a measurement problem. Given a large candidate set D, the system must select a subset C that maximizes usefulness while satisfying token budget, source coverage, freshness, policy, and contradiction constraints.

A simplified objective can be written as:

This type of constrained selection problem can become hard in general, especially when the system must balance relevance, coverage, diversity, freshness, and token cost simultaneously. Context Inspector does not attempt to prove that every selected context is globally optimal. Instead, it provides practical, interpretable signals that help teams identify when the context is clearly not good enough.

Example

Suppose the scoring weights are:

For candidate context C~1:

The score is:

For candidate context C~2:

The score is:

Even though C~2 has higher raw coverage, C~1 receives the better score because it has stronger utilization, stronger freshness, and no source starvation. This illustrates why context quality must be evaluated across multiple dimensions rather than by relevance or coverage alone.

Using Context Inspector in Development and CI

One practical benefit of pre-inference inspection is that it can be moved into the development lifecycle. Teams can define representative queries, expected source requirements, minimum coverage thresholds, and policy gates. Those checks can run when retrieval logic, chunking strategy, metadata extraction, or ranking configuration changes.

def test_context_quality_for_process_query():
    selected_chunks = retrieve_context(
        query="What steps apply for the current process?"
    )

    report = context_quality_report(selected_chunks, config)

    assert report["passed"], report
    assert report["metrics"]["coverage"] >= 0.90
    assert report["metrics"]["source_starvation"] == 0.0
    assert report["metrics"]["contradiction_density"] == 0.0
    assert report["metrics"]["policy_admissibility"] == 1.0

This is not a replacement for response-level evaluation. It is a complementary guardrail. Response-level evaluation remains important, but context-quality checks can catch a class of failures earlier, when they are easier to diagnose and less expensive to correct.

Design Principles

1. Context quality should be inspectable. A score is useful only if developers can understand the reason behind it. Metrics such as coverage, starvation, freshness, and contradiction density provide actionable signals rather than opaque judgments.
2. Context quality should be measured before inference. Once the LLM has generated an answer, teams are debugging both model behavior and context assembly behavior at the same time. Pre-inference inspection separates those concerns.
3. Privacy and governance should be built into the design. A context inspector should operate within approved access patterns and should use metadata or minimized signals where possible. Its purpose is to reduce risk by preventing inappropriate or low-quality context from entering the prompt.
4. Context inspection should be configurable. Different applications have different risk profiles. A low-risk summarization workflow may tolerate broader context and softer thresholds. A policy-sensitive workflow may require stricter source coverage, version controls, and admissibility checks.

Conclusion

RAG systems are often judged by their final answers, but the quality of those answers begins earlier. It begins with the context.

Post-inference evaluation remains essential, but it is not sufficient on its own. It can tell us that an answer failed, but it may not immediately tell us whether the failure was caused by the model, the prompt, the retriever, the ranking strategy, stale content, missing sources, or conflicting evidence.

Context Inspector brings attention to the pre-inference layer. It evaluates whether the assembled context is complete, current, balanced, consistent, and admissible before the LLM call is made.

By inspecting context quality earlier, teams can reduce avoidable inference calls, improve debugging, strengthen observability, and build more reliable GenAI systems. The broader lesson is straightforward: better answers require better context, and better systems inspect that context before generation begins.