Skip to main content

Clearing a Large Initial Backlog of Duplicates

Most organizations arrive with years of accumulated duplicates and no clear sense of what finishing looks like. This page sets the expectation and gives you a working order.

How long it takes​

Days or weeks, for a large backlog of duplicates. Once you factor in maintenance-window constraints and the human validation that low-confidence sets require, that is the honest range.

Nothing about that is a limitation of the service. It's the arithmetic of merging tens or hundreds of thousands of records while a human stays in the loop where it matters.

Split the work by confidence​

The single most useful decision you make is where to draw the line between what merges in bulk and what gets looked at.

Above the line: bulk merge​

Start with your highest-precision sets and work down to a threshold you're confident carries no false positives. Where that threshold sits depends on your data and your rules. You find it by reviewing output, not by guessing.

Everything above it can be merged with minimal review, using scheduled AutoMerge requests from the Management App.

Below the line: validate first​

Lower-precision sets need real human judgment. These get merged directly in your CRM: select a page of duplicate sets in the dupes list views, review, then bulk-merge that page.

Slower per operation, and unavoidable, because it's the part where a person decides whether two records are the same human being.

Parallelize the manual work

Page-by-page merging in the CRM parallelizes well. Open several browser windows against different pages of duplicate sets and work them simultaneously rather than waiting for one page to finish before starting the next.

Suggested order of operations​

  1. Analyze/Tag with your tuned rules against Production.
  2. Review the precision distribution on the dashboard.
  3. Choose your bulk-merge threshold.
  4. Queue scheduled AutoMerge requests at that threshold, one per evening, each stopping before your maintenance window.
  5. While those run overnight, work the low-precision tail by hand in the CRM, several windows at a time.
  6. Re-run Analyze/Tag periodically. Sets split by Maximum Duplicate Set Size, and groups split across matching rules, resolve themselves across passes.
  7. When the backlog of duplicates is down to a manageable trickle, ask support to set up a recurring monthly analysis so it doesn't rebuild.

Signs you should adjust rather than continue​

  • A lot of low-precision sets. Usually a matching rule that is too loose, not a data problem.
  • Consistently wrong winners. Your ranking rules don't reflect how your business decides which record matters.
  • Timeouts. See Timeout errors. Lower the Maximum Duplicate Set Size before anything else.

Merging faster is rarely the answer. Better rules mean less validation, and less validation is what actually shortens the project.