Record Linkage for Company Names: Fellegi-Sunter Explained

The problem: same company, many records

A target list assembled from more than one source will contain the same company several times. A state registry lists "Harbor Valve & Fitting, Inc." A trade directory lists "Harbor Valve and Fitting." A commercial database lists "Harbour Valve & Fittings Co" at an address the company left two years ago. No identifier is shared, so a join on exact name returns nothing and a join on address returns the wrong neighbors.

This is the record linkage problem. Jared Murray describes probabilistic record linkage as "the task of merging two or more databases in the absence of a unique identifier" and notes that it is closely related to deduplicating a single database ^[1]. In deal research the same work is usually called entity resolution, one of several names for the field listed in a Census Bureau overview ^[2].

Company records are a hard case. Winkler writes that business lists "are typically more difficult to match" than person or agriculture lists "because of the variants of names and the variants of addresses" ^[2]. Exact comparison fails even on person data: in one major census application, Winkler reports that more than 25% of matches would not have been found by exact character-by-character comparison.

This article explains the Fellegi-Sunter model with company-name examples. For applied procedures, see the guides on identifying the same company across state registrations and finding the legal entity behind a brand.

Deterministic vs probabilistic linkage

Deterministic linkage applies rules: "same normalized name and same ZIP code," or "same state registration number." The documentation for Splink, which describes itself as "a Python package for probabilistic record linkage" ^[3], calls this "a rules-based approach for joining records together." It lists the strengths as computational cheapness and high precision, and the weakness as low recall, meaning missed true matches ^[4]. A Census Bureau working paper on linking employers adds that, because deterministic matching keeps only records with exactly one match, the matched file can end up as "a small and potentially non-random subset of the original file" ^[5].

Probabilistic linkage weighs evidence from each field comparison against thresholds. Splink calls it "a evidence-based approach for joining records together" in which the user chooses the evidence threshold for a match ^[4].

The formal model is Ivan Fellegi and Alan Sunter's "A Theory for Record Linkage," published in the Journal of the American Statistical Association in December 1969 ^[6]. Winkler describes it as a formal mathematical model for ideas introduced by Howard Newcombe in 1959 and 1962 ^[2].

Blocking: deciding which pairs to compare

Comparing every record with every other is usually infeasible: Splink notes that comparisons grow with approximately the square of the number of records, and that 1 million records "would generate around 500 billion pairwise record comparisons" ^[7].

Blocking reduces that set. Winkler describes Newcombe's approach as "only considering pairs that agreed on a characteristic such as surname or date-of-birth," and adds: "Such reduction in the number of pairs is called blocking" ^[2]. Pairs that share no blocking key are never scored; they are nonmatches by construction. The CES employer study states this for its own design, which blocked on 3-digit ZIP code, 10-digit phone number, telephone area code and city-state: a record sharing none of those values "is assumed to have 0 probability of being a true match" ^[5].

The trade-off: looser rules catch more true matches but generate more comparisons, and a name-based rule "would miss a true match where there was a typo in the name." Splink therefore accepts several rules and keeps every comparison that satisfies at least one ^[7]. Winkler gives a similar multi-pass example (ZIP plus the start of last name; ZIP plus the start of street name; name plus street address), notes that blocking strategies have historically been "determined using trial and error," and warns that a set of passes generally will not bring together every pair that would score above the cutoff ^[2].

Illustrative example (fictional). A team deduplicates 40,000 company records. All pairs: 40,000 × 39,999 / 2 = 799,980,000. The team blocks on the first three letters of the normalized name plus state, and suppose the result is 2,000 blocks of 20 records. Each block has 20 × 19 / 2 = 190 pairs, so the total is 2,000 × 190 = 380,000. A second pass on phone number would recover pairs whose names differ in the first three letters.

Comparison vectors for company records

Inside a block, each field comparison is recorded as a level; the levels form the pair's comparison vector, which Winkler calls an agreement pattern. His simple example is a space of eight patterns built from agreement or disagreement on the largest name component, street name and street number ^[2].

A workable starting set for company records:

Standardization. Winkler describes replacing variant spellings with one, for example variants of "Incorporated" with "Inc." and "Street" with "St," then parsing names and addresses into comparable components; his name-parsing examples include business-name components ^[2].

String comparators. The "close" level needs a similarity score. Winkler explains that linkage needs "a function that represents approximate agreement," scored 1 for exact agreement and between 0 and 1 for partial agreement. He describes the Jaro comparator, which accounts for insertions, deletions and transpositions, and a variant he reports dramatically improves matching compared with not using string comparators ^[2]. The CES study used Jaro-Winkler scores for employer name and establishment address as its first two predictors ^[5].

m and u probabilities and match weights

The model attaches two probabilities to each comparison level. Splink defines them ^[8]:

• m probability: "the probability of a given observation given the records are a match."
• u probability: "the probability of a given observation given the records are not a match."
• lambda (λ): "the prior probability that any two records match."

The ratio m/u is the Bayes factor; its base-2 logarithm is the partial match weight. For a two-level field, disagreement has probabilities (1 - m) and (1 - u), so:

• agreement weight = log2(m / u)
• disagreement weight = log2((1 - m) / (1 - u))

Splink's key assumption is that comparisons on different columns are independent, which makes weights additive. The total match weight is the prior weight, log2(λ / (1 - λ)), plus the partial weights, and converts to a probability as 2^M / (1 + 2^M) ^[8]. Winkler writes the core quantity as the ratio R = P(γ | M) / P(γ | U) and calls R, or any monotonically increasing function of it such as a log, the matching weight ^[2].

Worked example (fictional numbers)

Illustrative example (fictional). Every m and u value below is invented. Weights are rounded to two decimals; totals and probabilities were computed before rounding.

The three name levels sum to 1.00 across m and across u. For binary fields, disagreement is the complement: for address, (1 - 0.80) / (1 - 0.01) = 0.20 / 0.99 = 0.202, and log2(0.202) = -2.31.

Assume a fictional prior of 1 true match per 1,000 pairs within blocks: λ = 0.001, prior weight log2(0.001 / 0.999) = -9.96. A missing field contributes 0.

Four fictional candidate pairs for "Harbor Valve & Fitting, Inc.":

Pair B as arithmetic: 4.32 - 2.31 - 1.32 + 9.97 = 10.66; 10.66 - 9.96 = 0.70; 2^0.70 / (1 + 2^0.70) = 0.62. Pair D's displayed values sum to -9.26; the unrounded total is -9.27.

Two points carry over to real data. The shared domain in pair B outweighs a moved address and a changed phone, because in this parameter set non-matching pairs almost never share a domain. And pair D shares an address yet scores low: companies share buildings and registered-agent addresses, so a real model's u for address agreement should reflect that.

Independence is the weak assumption for company data: address and phone often change together when a business moves, so their disagreements may be double-counted. The CES authors, discussing their employer setting, call the independence assumption "untenable" ^[5].

Real m and u values come from labeled pairs or from estimation without labels. Winkler cites methods for optimal linkage both with and without training data ^[2]; the CES paper notes approaches that use the expectation-maximization (EM) algorithm "without a training dataset," while its own model was trained on about 1,000 hand-reviewed pairs ^[5]. Winkler also describes frequency-based weighting, in which agreement on a rare value counts for more than agreement on a common one. Agreement on a distinctive company name is stronger evidence than agreement on "ABC Plumbing."

Thresholds and clerical review

The decision rule has two cutoffs. As Winkler states it ^[2]:

• if R is above the upper cutoff, designate the pair a match;
• if R is between the cutoffs, designate it a possible match "and hold for clerical review";
• if R is below the lower cutoff, designate it a nonmatch.

The cutoffs "are determined by a priori error bounds on false matches and false nonmatches." Raising the upper cutoff lowers false matches; lowering the lower cutoff lowers false nonmatches. Both send more pairs to review.

Illustrative example (fictional), continued. Set the upper cutoff at a match probability of 0.95 (total weight 4.25) and the lower at 0.05 (weight -4.25). Pair A is a match, pair D a nonmatch. Pairs B and C go to a reviewer, who checks evidence outside the four fields, such as a registry filing or the company's website, and records the decision with that evidence.

At scale the savings can be large: Winkler reports that in the 1992 U.S. Census of Agriculture, "the computerized procedures were able to reduce the clerical review from 75 clerks over 3 months to 6500 hours of review." The listed entities were farms corresponding to individuals, partnerships and corporations. Based on a large validation follow-up, the computerized procedures identified more duplicates automatically than the clerks had in the previous census, and duplication in the final file was 2%, against 10% five years earlier ^[2].

Clerical review has weaknesses of its own. The CES paper summarizes Belin and Rubin's (1995) criticism: the error properties of manual review are unknown, standards may vary across reviewers, and review may not yield many unequivocal matches. The CES team had each training pair reviewed by two different human experts ^[5]. For a deal team: a written rubric, a second reviewer on a sample and a disagreement log.

Pairwise decisions also need clustering: if A links to B and B links to C, the three become one entity even if A and C scored low against each other. Winkler cites graph-based methods that partition linkage relationships "into clusters representing individual entities" ^[2]. Write the clustering rule down; one false link can merge two companies.

Measuring error rates

A linkage run makes two kinds of error:

• False match: two companies linked as one, merging their ownership, revenue or contact facts.
• False nonmatch: one company left as two records, counted and contacted twice.

Precision is the share of linked pairs that are true matches; recall is the share of true matches that were linked. Splink frames the deterministic trade-off in these terms ^[4].

Measuring them takes labeled pairs:

1. Sample randomly from each band: auto-match, clerical review, and auto-nonmatch above a floor score.
2. Label each sampled pair using evidence beyond the compared fields; have a second reviewer label a subset.
3. Estimate the false match rate from the auto-match sample and the false nonmatch rate from the nonmatch sample.
4. Treat pairs that blocking never compared as a separate risk. Winkler describes a capture-recapture method for estimating matches missed by blocking and cautions that it can be inaccurate ^[2].
5. Adjust cutoffs and re-run.

The audit sampling worksheet covers sample sizes. Rates measured on a labeled sample are verified for that sample; misses outside the blocks remain open.

Open implementations exist. Enamorado, Fifield and Imai (American Political Science Review, 2019) describe "a fast and scalable algorithm to implement a canonical model of probabilistic record linkage," and their abstract states that an open-source software package is available ^[9]. Splink is another implementation of the Fellegi-Sunter model.

Practical limits for company data

The model answers one question: do two records describe the same entity? Company data raises others.

• Name changes and successors. Whether a renamed or merged company is "the same" is a definitional choice first.
• Brands vs legal entities. A trade name, its operating LLC and a holding company can share phone, domain and address while being distinct entities. The model will link them unless the design separates them.
• Shared addresses. Office parks, registered agents and virtual offices raise u for address agreement.
• Many small candidates. The CES study found matching blocks populated with many small employers, each receiving a trivial match probability, a problem made worse when EINs could not be used for blocking ^[5].

Identifiers beat probabilities when they exist. The CES paper describes Census Bureau longitudinal business data as starting with exact identifier linkage (Census File Number to EIN) and using Fellegi-Sunter methods "when EINs produced many-to-many linkages," or none, as with births, deaths and changes in legal identifiers ^[5].

Decision rule:

• Link on a shared, reliable identifier first (registry number, tax ID, license number).
• Use probabilistic linkage where no identifier is shared, or where identifiers conflict.
• Send the middle band to clerical review with a written rubric.
• Measure false match and false nonmatch rates on a labeled sample; record blocking misses as open.
• Keep the evidence for every reviewed decision with the record.

Further reading: Fellegi and Sunter's 1969 paper is the original statement of the model; Winkler's 2006 overview is an open survey from the Census Bureau; Murray's 2016 paper reviews how indexing, blocking and filtering interact with the Fellegi-Sunter framework; and Peter Christen's Data Matching: Concepts and Techniques for Record Linkage, Entity Resolution, and Duplicate Detection (Springer, 2012) is a book-length treatment ^[10].

In Docket, the Triage agent handles this step for a client's acquisition targets: it resolves identity and removes duplicates.