Known Unknowns: Testing in the Presence of Uncertainty

Sebastian Elbaum and David S. Rosenblum

Wednesday, June 03

The following Great Innovative Idea is from Sebastian Elbaum, Professor of Computer Science and Engineeringat the University of Nebraska-Lincoln andDavid S. Rosenblum, Dean of the School of Computing at theNational University of Singapore.

 Sebastian Elbaum and David S. Rosenblum

Their paper Known Unknowns: Testing in the Presence of Uncertainty won second place at the Computing Community Consortium (CCC) sponsored Blue Sky Ideas Conference Track series at the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE), November 16-22, 2014 in Hong Kong.

The Innovative Idea

Uncertainty is present in most systems we build today, whether introduced by human decisions, machine learning algorithms, external libraries, or sensing variability. This uncertainty leads to occasional misbehavior or incorrect output that is deemed to be acceptable. In the context of software testing, this uncertainty makes it difficult to distinguish acceptable from unacceptable misbehaviors, and to determine when there are faults in the system that are being masked by acceptable misbehaviors. Existing approaches to deal with uncertainty in testing have been partial and of limited scope, leaving the systematic treatment of uncertainty in testing still open.  In the paper, we explore the use of Hidden Markov Models and statistical reasoning about behaviors observed during testing in order to distinguish between acceptable misbehavior and behavioral errors that are due to latent faults.


The idea will allow us to deal better with uncertainty that arises in modern complex software systems in many different forms, such as imprecision in readings from hardware sensors or imprecision in machine learning-based classifiers, and correspondingly in a broad range of applications from human activity recognition to quadcopter stabilization.

Other Research

We both work in testing and analysis of complex software systems.  Sebastian has interests in the development and assessment of automated techniques to improve software dependability. David has interests in the design, verification and testing of distributed systems and context-aware ubiquitous computing systems.

Researcher’s Background

Sebastian is a Professor of Computer Science and Engineering at the University of Nebraska – Lincoln (UNL). He is a co-founder of the E2 Software Engineering Group at UNL, and the Nimbus UAV Lab at UNL. He is the Program Co-Chair for the 2015 International Conference in Software Engineering, and a member of the ACM Transactions on Software Engineering and Methodology Editorial Board.

David is Professor of Computer Science and Dean of the School of Computing at the National University of Singapore (NUS).  He previously held academic positions at University College London and the University of California, Irvine, and he also was a research scientist at AT&T Bell Laboratories (Murray Hill) and CTO at a technology startup called PreCache, Inc.  He is the Editor-in-Chief of the ACM Transactions on Software Engineering and Methodology (ACM TOSEM) and is Past Chair of the ACM Special Interest Group on Software Engineering (ACM SIGSOFT).


Sebastian’s homepage is at

David’s homepage is at