DDocket
Talk to us

The origination glossary

Deduplication

Identifying and consolidating repeated records while preserving evidence, history, and genuinely distinct entities.

Last reviewed · report an error

Definition

Deduplication removes redundant representations of the same subject from a working collection. In origination, it prevents the same business from being counted, researched, or approached repeatedly under alternate spellings, trading names, or imported identifiers. It depends on identity and entity resolution: apparent similarity should be evaluated before records are consolidated.

The goal is a coherent record, not deletion of inconvenient history. Sources can disagree, past names can matter, and previously contacted identities can carry suppression instructions. A useful consolidation preserves aliases, provenance, dated claims, and relevant relationship events. Two subsidiaries with a shared domain may be related without being duplicates; two websites may represent the same company even when their branding differs.

In origination work

Deduplicate the target list before detailed collection and again when importing new sources. Keep a clear link from redundant rows to the surviving company identity. Check that exclusions and communication restrictions survive consolidation, and avoid treating the absence of an email match as proof that records differ. Docket's Triage handles duplicate and wrong-site records before the research stages, helping prevent repeated work while retaining the basis for a reviewable company assessment. Ambiguous matches should remain visible for resolution.

Where duplicates come from

In target research, duplicates arise from several sources: the same company collected from two data providers; a company listed under its brand and its legal name; a business with several locations listed once per location; companies registered in more than one state; an entity renamed after a transaction; and spelling variation in names and addresses. Each source needs a different test, so grouping duplicates by cause helps choose the right rule.

Multi-state registration is a particular trap. A company formed in one state and doing business in another is usually registered as a foreign entity in the second, and both records describe the same business. Statutory frameworks for registered organisations give a basis for deciding where an entity is located for filing purposes, which is why registry searches for a company begin in the state where it was formed.

Rules and review

Begin with exact keys: the same registry number in the same jurisdiction, the same federal identifier, the same domain. Then apply normalised matching on name and address, with rules for suffixes such as LLC and Inc. Finally, queue ambiguous pairs for human or second-pass review. A layered approach keeps the cheap, safe rules first and spends effort only where it is needed.

A deduplication step should never destroy information. Keep both source records, link them to a single resolved entity, and store the reason. If a later review reverses a merge, the original records are still available. This is the same principle used in probabilistic record linkage, where linkage is a decision made on top of the source data, not a replacement for it.

Measuring the result

Report the duplicate rate and how it was estimated. A sample audit gives the best estimate: draw a random sample of resolved entities, check each against the original sources and count errors of both kinds. Track the rate over time, because it signals whether new sources or changed rules are degrading quality.

Duplicates affect decisions. If a market looks to contain 300 independent firms, and 20 percent are duplicates or affiliates of others, the opportunity is smaller and the map is different. Therefore present counts after resolution, state what was merged, and be ready to show examples.

Related terms

  • Entity resolution — Determining whether records describe the same real entity or different entities with a relationship between them.
  • Target list — A defined collection of candidate businesses to research or approach against a stated acquisition mandate.
  • Suppression list — A maintained set of records that prevents contact for specified recipients, companies, or communication scopes.

Further reading

Sources and further reading

To apply these distinctions to your own acquisition criteria, discuss your research brief or inspect the demo platform.

Send us ten targets. We research them for free.

A target list and your acquisition criteria are enough. You get back company records with scores, the evidence behind every field, and the questions the public record could not answer.

Start an enquiry →Open the platform