Why “Useful” Differential Privacy Has No Single Score

Last updated: October 15, 2026

Differential privacy promises to release data while protecting individual records—but how "useful" is the resulting output? A guest post on the NIST Privacy Engineering blog, written by privacy researcher Claire McKay Bowen of the Urban Institute, argues that no universal utility index exists. Instead, the field should define usefulness against the specific analyses and decisions the data will support.

The argument targets a common expectation: that one tidy score can rate any privatized dataset for quality. That score does not exist in general, and chasing it can mislead both data providers and data users.

—

The Problem with One Number

The post reframes the question. Instead of asking "how good is this dataset?", ask "which users will touch the data, which analyses will they run, and which decisions hang on the result?" Useful is a property of a dataset-plus-analysis pair, not a property of a dataset alone.

This distinction matters because summary statistics can look fine while a downstream model quietly breaks. Two datasets can match means and correlations and still produce different policy calls.

—

How Utility Is Actually Measured

Two metric families are spelled out.

Summary statistics** cover counts, means, and correlations for single variables and combinations. Distance measures—bias and root-mean-square error—compare noisy output to the original. The 2020 U.S. Census disclosure avoidance system used a defined list: mean absolute error, count error, intentional numeric error, RMSE, absolute error rate, variance coefficient, stock total error, and percentage differences at fixed thresholds. These were averaged across geographies because most users want values for tracts and block groups.

Outcome-specific metrics** go further. The idea is that a policy microsimulation should produce the same go or no-go answer whether it runs on confidential data or the privatized version. Urban Institute researchers applied this logic in 2020, collaborating with the IRS Statistics Division to generate synthetic tax data and run a microsimulation model. They compared adjusted gross income estimates, deductions, and tax liability figures across original and synthetic files—and found close agreement on the outcomes that mattered for policy analysis. In 2021, the team extended this work into a broader feasibility study on differentially private methods for tabular statistics, regression analyses, and quantile statistics, drawing on interviews with tax experts to identify which analyses actually drive decisions.

—

Our Analysis

The strongest contribution here is reframing utility as a decision-matching problem rather than a number-matching problem. Teams that only check means and correlations can ship a release that looks clean and quietly breaks a downstream model.

The trade-off is real.** Per-use-case metrics mean you cannot certify a release as universally "good enough." You must enumerate realistic uses, agree on thresholds with data practitioners, and audit against them. That is harder than publishing one score. It is also closer to how the data will actually be used.

Who wins:** agencies with well-defined decision rules and established relationships with privacy researchers. They can build outcome-specific tests and iterate.

Who loses:** anyone expecting a quick, universal quality badge. Regulators and data providers who ask for a single metric will get vague answers—and should press harder for use-case documentation instead.

What to watch:** whether federal statistical agencies publish per-release utility reports alongside privacy budgets. The 2020 Census set a precedent. The next major release will show whether the practice spreads or stalls.

—

Decision Checklist: Before You Release

Use this when evaluating a privatized dataset:

• List your data users.** Researchers, agencies, and contractors may run different analyses.
• Map each user to their key analysis.** What regression, microsimulation, or flagging model do they run?
• Define the decision that hinges on the output.** A policy call? A funding threshold? A public report?
• Build an outcome-specific test.** Run the analysis on confidential data and privatized data. Compare the decisions, not just the numbers.
• Set tolerance thresholds with practitioners.** How many divergent counties, incorrect flags, or reversed policy calls is acceptable?
• Report per-use-case, not per-dataset.** Pair every release with a utility memo that documents which tests passed and which failed.

—

Bottom Line

Stop hunting for one utility score. Pick the analyses your users actually run, agree on metrics with them, and test whether decisions survive privatization.

—

Key takeaways

• Utility is a property of a dataset plus its intended analyses, not of the dataset alone.
• Treat summary statistics as a smoke test, then build outcome-specific checks.
• Use the Census Bureau's published error-metric menu as a template for your own reports.
• Loop data practitioners into metric selection; they know the decision rules.
• Pair every release with a per-use-case utility report, not a single number.