Research Highlight: CRA Board Member Brent Hailpern

In: , /

By Brent Hailpern, IBM Research
brent-hailpern

My research involves understanding and facilitating the life cycle of cognitive software, which is substantially different than the life cycle of conventional software. This difference has profound implications for the methodology and tools required to build such software. Cognitive software possesses at least one “cognitive” or “intelligent” component, such as a component implemented using machine learning, neural networks, or rules. Multiple cognitive components will often be involved in a cognitive application or service, but even just one component is enough to impart special and challenging complications.

Three primary characteristics distinguish cognitive components from conventional software components:

1) They depend on data as well as code. The data sets used to train machine learning components–the models produced during training and the rules that drive rule-based components–embody much of the semantics of the components. Acquisition, curation, versioning, and other operations on the data have an essential place in the life cycle.

2) Their traditional semantics are unpredictable; they work “right” even if their answer is “wrong.” Developers can, in principle, understand exactly what conventional code does, and the process of producing code that meets requirements is essentially a design process. This code can be tested and even verified, assuming appropriate formal specification exists. What exactly a trained model or complex rule set will do is much less predictable, and the process of evolution to meet requirements in largely experimental. Even the best training procedures may produce models that are accurate only a certain percent of the time, or accurate for some types of input data but not others. Furthermore, the algorithms used for training models are often non-deterministic, which adds another layer of non-reproducibility related complexity for programmers to manage.

3) Software developers usually do not have the skills needed to debug cognitive software. All developers know how to debug code and have sophisticated debugging tools. But how do you “debug” a machine-learning component that is delivering disappointing results? Is your training set insufficient? Are you using the wrong modelling algorithm? Or using the wrong parameters to the right algorithm? This level of diagnosis currently takes machine-learning experts, and even for them, it involves considerable experimentation.

These key differences have a profound effect on the software engineering methodology and on the tools and development environments that support traditional DevOps. Tools supporting big data management and manipulation (beyond simple editing and file versioning) and experimental processes have never been part of the design or implementation of conventional software development environments. Similarly, existing development environments were never designed to handle versioning of big data sets. Ethnographic studies of software developers in a machine-learning context indicate that they perform a great deal of manual work and recordkeeping far beyond what is normal for a traditional developer. This work is both tedious and error prone. The developers also frequently consult machine-learning experts, which is a development bottleneck, since these experts are a scarce resource.

At IBM Research, I am working with colleagues to create a comprehensive software development environment providing end-to-end support for the cognitive software development life cycle.

About the author

Brent Hailpern received a Ph.D. degree in computer science from Stanford University in 1980. His thesis was entitled ‘‘Verifying Concurrent Processes Using Temporal Logic.’’ Hailpern joined IBM’s Thomas J. Watson Research Center as a research staff member in 1980, where he worked on and managed various projects relating to issues of concurrency and programming languages. From 1999 to 2004, he was the associate director of computer science for IBM Research. From 2004 to 2011, he managed departments at Watson Research covering programming languages, software engineering, and human-computer interaction where he was the director of programming models and tools, with a worldwide responsibility for IBM Research’s strategy and research agenda in software technology.  From 2011 to 2013, he was the director of the computer science department, IBM Research – Almaden.  In 2014, he became a distinguished research staff member, head of computer science for IBM Research, and co-lead for the IBM research cognitive platform and tool strategy.

Hailpern is a Fellow of the ACM and the IEEE.  He is a past secretary of the ACM, a past member of ACM Council, a past chair of the ACM Special Interest Group on Programming Languages (SIGPLAN). He has chaired or co-chaired several SIGPLAN conferences and was a co-chair of the 2014 CRA Conference at Snowbird. Hailpern is currently a member of the CRA Board of Directors (2011-present) and the NSF Directorate for Computer and Information Science and Engineering (CISE) Advisory Committee (2012-present).

This article is the first in a series highlighting the research of CRA board members.