Predictive Test Selection for change-based regression testing

Build-dependency selection would exercise all tests that transitively depend on modified files — on average as much as approximately one-quarter of all available mobile-code tests per change; treated as 100% of the transitively-impacted set for pre-submit runs.→The deployed system runs 'just a third of all tests that transitively depend on modified code' per change (Engineering at Meta); the ICSE 2019 industry paper reports 'fewer than a third of the tests that would be selected on the basis of build dependencies'.

Deployed at Facebook for more than a year at 2018-11-21; also stated as 'operational at Facebook for several months' in the ICSE 2019 industry-paper text. · Facebook's monolithic mobile codebase and continuous-integration system — exact monthly change or test counts are not disclosed in either source.

The one-third figure is a comparison against the build-dependency selection strategy — not a comparison against literally all tests in the repository. Meta does not publish per-change absolute test counts or infrastructure-cost figures.

The prior build-dependency strategy is treated as the reliability floor; catch rate is not separately quantified in Meta's public disclosure.→'Catches more than 99.9 percent of all regressions before they are visible to other engineers in the trunk code' (blog); a required production floor of 'catch at least one failing test for more than 99.9 percent of problematic changes' (blog) and 'over 99.9% of faulty changes are still reported back to developers' (ICSE paper).

Deployed at Facebook for more than a year at 2018-11-21. · All pre-submit changes to Facebook's monolithic mobile codebase during the deployed period; the absolute change and regression counts are not disclosed.

The 99.9% is a production correctness threshold enforced by Meta on the deployed model, not a third-party audit. The catch-rate metric is Meta-defined ('regressions visible in trunk') and Meta-measured.

Under the prior transitive-dependency selection strategy, treated as the baseline cost of change-based testing (index = 1x).→'Doubled the efficiency of our testing infrastructure' (blog); the ICSE 2019 paper phrases the same result as reducing 'the total infrastructure cost of testing code changes by a factor of two'.

Reported for the more-than-a-year Facebook production deployment as of 2018-11-21. · Facebook's change-based testing infrastructure for the mobile codebase; absolute compute or dollar figures are not disclosed.

'Efficiency' is defined by Meta as change-based testing infrastructure cost; the blog phrasing is operator-narrated, but the ICSE 2019 paper restates it as 'reduces the total infrastructure cost of testing code changes by a factor of two' in the industry-refereed abstract, so the two-times figure is corroborated by a peer/industry-reviewed source. No independent audit.