Deduplication Survey
Deduping is a litigation cost control technique that could be applied to virtually all litigation regardless of the litigation support or document review platform used by the law firms or corporate counsel.
In May 2009, Anne Kershaw and Joe Howie conducted a survey of leading e-discovery providers to determine what their practices were regarding deduping e-discovery or ESI, specifically regarding deduping only within the records of individual custodians (sometimes called “vertical” deduping) compared to deduping across the records of multiple custodians (sometimes called “horizontal” or project-level deduping).
Responses were received from: 
- ACT Litigation Services
- Business Intelligence Associates (BIA)
- CaseCentral
- Clearwell Systems
- Daticon EED
- Encore Discovery Solutions
- Fios
- FTI Consulting
- Gallivan, Gallivan & O’Melia (GGO)
- Iris Data Services
- Kroll Ontrack
- Legal Document Management (LDM Global)
- Legal Document Services International (LDSI)
- Rational Retention
- Recommind
- StoredIQ
- Trilantic
- Valora Technologies
The survey results showed that, on average, deduping within the records of single custodians resulted in a 21% reduction in the volume to be reviewed whereas deduping across the records of all the custodians resulted in a 38% reduction in volume. In rare cases dedupuing across custodians did not achieve additional volume reductions compared to just deduping within the records of individual custodians. However, many of the survey respondents reported reductions in volume approaching 60-80% in some cases with across-custodian deduping.
Considering that reviewing productions for responsiveness and privilege constitutes a significant percentage of the cost of litigation, it is surprising that parties did not use across-custodian deduping for all their cases. However, a little over half of all projects received across-custodian deduping, roughly 40% received single-custodian deduping and about 7% received no deduping.