Evaluating AI research agents

AI agents can help apply a consistent set of research questions across an acquisition target list. Their usefulness depends on whether the answers are supported, comparable, and reviewable. Research on grounding, attribution, and evaluation offers a basis for assessing these properties. This guide explains the controls to look for and the limits that remain.

The shape of the problem

A screening mandate can be expressed as a set of questions applied to each company in a target list. Asking an agent to answer those questions from identified documents gives reviewers a defined task to evaluate. The relevant test is how accurately it answers your questions using the available evidence.

A defined screening task offers four useful controls:

• The questions are known in advance. You are not discovering what to ask mid-run.
• The answers are mostly categorical. Ownership structure, segment, certification status: small fixed sets, not free text.
• The evidence is retrievable and retainable. Pages can be fetched, stored and quoted.
• Answers can be reviewed. Source checks and later company conversations can identify errors.

These controls make it possible to evaluate specific outputs and identify where further review is needed.

Grounding is necessary and not sufficient

The basic pattern of retrieving documents and conditioning generation on them was named and evaluated in 2020 [1]. It supplies the model with source material to use when answering, reducing its reliance on information learned during training.

It does not eliminate it, and the specific residual failure is the dangerous one. Human evaluation of generative search systems found both that many generated sentences were not fully supported by the citations attached to them, and that many citations did not support their sentence [4]. A system can retrieve correctly, cite plausibly, and still state something the source does not say. Citation presence alone does not establish support.

This is why the measurement community moved from “did it cite” to attribution proper: is this statement supported by this identified source, judged by someone reading both [3]. This distinction should be part of any evaluation of a research system.

Common research errors

Lumping everything under “hallucination” hides that these are different bugs with different fixes [2]. Examples relevant to company research include:

Note how many are architectural rather than model-quality problems. Wrong entity, stale source and silent non-answer are not cured by a better model; they are cured by ordering the pipeline correctly and by refusing to accept an incomplete run.

Structured answers

For categorical questions, the agent can select an answer from a defined list. “Ownership structure: family owned” rather than “the business appears to have remained in family hands since the 1990s.”

Four things follow immediately:

1. The answer is comparable. You can sort, filter and score on it without anything re-reading and re-interpreting prose later.
2. The answer is checkable. A reviewer can compare the selected value with its definition and supporting evidence.
3. A missing answer is visible. An explicit unanswered state can distinguish a lack of evidence from incomplete research.
4. The answer stays linked to its interpretation. Reviewers can see the selected category alongside the source language that supports it.

A fixed list of answers needs maintenance. Review uses of “other” and record the evidence that did not fit an existing category. Repeated gaps can indicate that the definitions need to change; they can also reveal extraction errors. Inspect examples before deciding which.

Interpreting source language

Keyword matches can lose the meaning of a sentence. An illustrative example is a company described as “not a subsidiary”: matching the word “subsidiary” alone would reverse the meaning. Similarly, a company announcing an acquisition may be the buyer rather than the target.

Ask the reading agent to return the structured answer with its supporting excerpt, then review the relationship between them. Use format matching for identifiers and dates, where the task is to recognise a format. Ownership and business activity require interpretation of the full statement.

Selecting relevant evidence

A tempting shortcut is to put everything in the prompt: the whole site, all twenty questions, the full criteria document. Long context windows make this possible and it degrades quality in a measurable, position-dependent way. Retrieval accuracy is worse for material in the middle of a long context than at either end [7].

The practical consequences are mundane and worth stating:

• Retrieve the pages likely to answer this question rather than concatenating a website.
• Keep the answer schema near the evidence rather than a hundred thousand tokens away from it.
• Prefer several narrow runs over one enormous one, and accept the extra orchestration.
• Measure per-question accuracy, not per-run accuracy, because a long run hides which question it fumbled.

The general engineering guidance points the same way: the simplest structure that solves the task usually beats a more autonomous one, and tool use should be narrow and checkable rather than open-ended [10]. An agent free to decide what to do next is harder to evaluate and rarely better at a task whose steps you already know.

Evaluating a research agent

Evaluate the system against a sample reviewed by people familiar with the mandate. Four measurements help separate answer quality from research coverage.

Reference-free metrics for faithfulness and relevance let you iterate quickly without labelling every run [6], and atomic decomposition is the right granularity for long outputs [5]. But the held-out human-scored sample is the anchor, because everything else is measuring consistency with something rather than correctness.

On models grading models

Using a model as the evaluator is attractive and it has documented biases: position bias, verbosity bias, and self-enhancement bias where a model prefers its own outputs [8]. These are real and they are also manageable, if you know the specific shape of each.

Practical evaluation controls:

• Model grading is good for detecting disagreement, weak for settling it. Use it to surface cases where two readings differ, then have a different and more capable reader resolve.
• Randomise order and strip length cues when comparing candidate answers, to reduce the influence of position and verbosity on the comparison.
• Do not have a model grade its own output. Different model, different prompt, different information.
• Sampling several answers and looking at agreement is cheap and effective [9]. Disagreement across samples can identify answers for further review; agreement alone does not establish that an answer is correct.

Anchor all of it to a human-scored sample at a fixed cadence. Without that anchor, a model-graded pipeline can drift for months and look stable the whole time, because it is measuring itself.

A staged research process

One way to apply these findings is to separate the workflow into three stages. This is the structure Docket uses; each stage has a defined role and review criteria.

1. Resolve identity first, and gate on it. Confirm the website belongs to the company, confirm the profile page, establish what kind of business it is, and apply disqualifying questions before anything expensive runs. Wrong-entity research is the most damaging kind because it is indistinguishable from good research downstream.
2. Collect with typed answers and quoted evidence. Retrieve, read, answer from the fixed list, quote the span, retain the page. Refuse to close a run that left a required question unanswered: a wrong answer is invisible, a missing answer is a queue entry.
3. Review with the fetch tools taken away. A second reader that can only see what was retained checks rather than re-does. Cheap, bounded, and it catches the contradiction class of failures that grounding does not.

Two cross-cutting properties hold it together. Every claim records which run produced it, which is the provenance model the web standards community wrote down years ago [15]; and the sources are ranked, so that when two stages disagree the resolution is a rule rather than an argument [16].

Research progress and throughput

Research time includes retrieval, extraction, review, and any retries. A useful progress view shows completed companies, pending questions, failed attempts, and cost. This helps a team distinguish a completed answer from an unresolved task and assess when a shortlist will be ready.

Concurrent processing lets work continue while individual companies take longer to research. Evaluate throughput across a representative list, including difficult records and failed requests. The relevant measure is the number of reviewed records completed to the agreed standard.

Maintaining answer categories

Answer categories need to remain useful as the target list grows. Assign responsibility for reviewing cases that do not fit and for updating the definitions and scoring rules together.

A maintenance process should cover the following:

1. The collector never invents a value. It answers from the fixed list and writes what it could not name into a companion field. This keeps answers comparable across runs.
2. Fallback answers are reviewed.Track how often the agent uses “other” and inspect the underlying evidence. Decide whether the category definitions or the extraction need improvement.
3. Category changes have an owner.Someone reads the accumulated “could not name it” text, adds the values it names, and writes a short brief for each so two readers apply it the same way. Record the reason for each change.
4. A new value and its scoring treatment ship together. A precise new category that no scoring rule places will score as unremarkable and quietly pass — which is more dangerous than the vague fallback it replaced, because the fallback at least capped the score. Have a test that fails when a value exists which the policy does not place.
5. Re-measure after each widening. It is a loop, not a task. The rate at which the escape hatch is used is the health metric for the whole vocabulary.

When adding a question, apply it to existing companies as well as new ones. Until that work is complete, show which records have been assessed under the revised criteria. Agree the scope and cost of the additional research before running it.

The economics

The AI Index documents falling inference costs for a given capability level [11]. That can make broader research coverage affordable, but model usage is only one part of the cost.

Assess the cost per reviewed company, including source access, retrieval, model usage, retries, and human review. A pilot on representative companies can establish a useful budget for your mandate. Research coverage, contact availability, and the capacity to follow up all affect the value of a larger list.

Where this does not work

Four limits to account for:

• Facts that are not written down anywhere. Most private company revenue, real margin, customer concentration and owner intent are simply not public. No agent finds them, and a system that produces confident values for these is producing estimates with the styling of facts. The correct output is a recorded not-found.
• Judgements that need a relationship. Whether a founder would sell to you, whether a management team is any good, whether the numbers in a data room are real. Research gets you to a conversation; it does not replace one.
• Fast-moving situations. A research file is a snapshot with an expiry. In a live process measured in weeks, what it knows is a starting point and no more.
• Anything the vocabulary cannot express. Typed answers are the source of the leverage and also the ceiling. A genuinely novel property of a business that has no field will not be captured, which is why the escape-hatch rate has to be watched and why extending the vocabulary is standing work rather than a project.

There is also a broader point about how this work should be described. Mechanical combination of evidence has beaten holistic expert judgement in head-to-head comparisons for decades, across a wide range of prediction tasks [14], and that is the actual claim here: not that a machine judges better than an investor, but that applying the same questions to every candidate consistently beats applying different questions to whichever candidates somebody had time for. Governance frameworks for these systems ask for exactly that kind of specificity about what a system does and how well [12], and the regulatory direction of travel in Europe makes transparency about machine-generated output an obligation rather than a courtesy [13].

Sources

References for the research and standards discussed in this guide. Some publications require a subscription or institutional access.

1. [1]
   Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

   arXiv · 2020

   The paper that named the pattern of grounding a language model's output in retrieved documents.

2. [2]
   Survey of Hallucination in Natural Language Generation

   arXiv · 2022

   Taxonomy of how generated text departs from its source, and of the metrics that try to catch it.

3. [3]
   Measuring Attribution in Natural Language Generation Models

   arXiv · 2021

   Defines Attributable to Identified Sources (AIS): whether a statement is actually supported by the document cited for it.

4. [4]
   Evaluating Verifiability in Generative Search Engines

   arXiv · 2023

   Human evaluation finding that a large share of citations in generative search output do not support the sentence they are attached to.

5. [5]
   FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation

   arXiv · 2023

   Scores generated text one atomic claim at a time, which is the only granularity at which 'is this true' is answerable.

6. [6]
   RAGAS: Automated Evaluation of Retrieval Augmented Generation

   arXiv · 2023

   Reference-free metrics for faithfulness and answer relevance in retrieval pipelines.

7. [7]
   Lost in the Middle: How Language Models Use Long Contexts

   arXiv · 2023

   Retrieval accuracy degrades for material in the middle of a long context. Relevant to anyone dumping a whole website into a prompt.

8. [8]
   Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

   arXiv · 2023

   Documents position, verbosity and self-enhancement bias in model-graded evaluation.

9. [9]
   Self-Consistency Improves Chain of Thought Reasoning in Language Models

   arXiv · 2022

   Sampling several answers and taking the agreement beats taking one. The cheap version of a second reader.

10. [10]
    Building Effective Agents

    Anthropic

    Engineering guidance on when a workflow beats an agent, and on keeping tool use narrow and checkable.

11. [11]
    AI Index Report

    Stanford Institute for Human-Centered AI

    Annual, heavily sourced measurement of model capability, cost and adoption.

12. [12]
    AI Risk Management Framework (AI RMF 1.0)

    U.S. National Institute of Standards and Technology

    The vocabulary regulators and enterprise risk committees are converging on: valid and reliable, accountable and transparent.

13. [13]
    Regulation (EU) 2024/1689 (Artificial Intelligence Act)

    EUR-Lex · 2024

    The consolidated legal text, including transparency obligations that reach general-purpose systems.

14. [14]
    Clinical versus mechanical prediction: a meta-analysis

    Psychological Assessment · 2000

    136 studies. Mechanical combination of cues equals or beats expert holistic judgement roughly nine times out of ten.

15. [15]
    PROV-DM: The PROV Data Model

    World Wide Web Consortium (W3C)

    A standard vocabulary for saying which entity was derived from what, by which activity, and when.

16. [16]
    AS 1105: Audit Evidence

    Public Company Accounting Oversight Board

    Sufficiency and appropriateness of evidence, and why evidence from an independent external source ranks above management's assertion.

Keep reading