Test Flakiness Across Programming Languages

Abstract

Regression Testing (RT) is a quality-assurance practice commonly adopted in the software industry to check if functionality remains intact after code changes. Test flakiness is a serious problem for RT. A test is said to be flaky when it non-deterministically passes or fails on a fixed environment. Prior work studied test flakiness primarily on Java programs. It is unclear, however, how problematic is test flakiness for software written in other programming languages. This paper reports on a study focusing on three central aspects of test flakiness: concentration, similarity, and cost. Considering concentration, our results show that, for any given programming language that we studied (C, Go, Java, JS, and Python), most issues could be explained by a small fraction of root causes (5/13 root causes cover 78.07% of the issues) and could be fixed by a relatively small fraction of fix strategies (10/23 fix strategies cover 85.20% of the issues). Considering similarity, although there were commonalities in root causes and fixes across languages (e.g., concurrency and async wait are common causes of flakiness in most languages), we also found important differences (e.g., flakiness due to improper release of resources are more common in C), suggesting that there is opportunity to fine tuning analysis tools. Considering cost, we found that issues related to flaky tests are resolved either very early once they are posted ( < 10 days), suggesting relevance, or very late ( > 100 days), suggesting irrelevance.

Authors

Keila Barbosa, Federal University of Pernambuco, Recife, PE, Brazil

Ronivaldo Ferreira, Federal University of Pará, Belém, PA, Brazil

Gustavo Pinto, Federal University of Pará, Belém, PA, Brazil

Marcelo d’Amorim, Federal University of Pernambuco, Recife, PE, Brazil

Breno Miranda, Federal University of Pernambuco, Recife, PE, Brazil