About the Impact of Software Engineering Research

This blog post is in response to a Twitter thread and then a blog post by Ian Sommerville.

In short, Ian's main point is that the impact of software engineering research is underwhelming and that it has been so for over 40 years.

I have made similar statements in the past, though perhaps not as severe as qualifying 40 years of research as mostly unsubstantive.

Ian's main arguments are centered around three points but this blog post will mostly focus on the first one, referred to as "short-termism".

Ian claims that researchers focus on short-term problems and interim results, thus leading to research that lacks ambition and long-term, significant impact. Several causes are discussed, including the "publish and perish" culture in academia and the pressure to collaborate with industry, for example in European funding programs. I don't question the former, which I will get back to, but I take issue with the latter. It is true that the time horizon of innovation in industry is much shorter than that of academic research. It is also true that addressing many of the hard problems in software engineering takes time and persistent effort, especially in academic environments relying on students and postdoctoral fellows.

However, as I have previously argued, software engineering challenges need to be addressed in the context where they occur, based on clear, realistic working assumptions and well-defined problems. The software industry is indeed highly diverse and the same high-level challenges cannot be tackled the same way across the board. Such context-driven research can, however, only be achieved through collaboration with industry (or public institutions when relevant). How to resolve such a dilemma then? This is exactly where dedicated public research funding comes into play. For example, both Canada and Luxembourg have generous programs to support such collaborative research where industry investments are matched with public funding. This enables industry to support research at a reasonable level of risk, though some degree of commitment is expected, as it should be. But suggesting, as Ian more or less indirectly does, that we should encourage a linear model of research, where basic research feeds applied research, which is then expected to support industrial innovation, is a mistake. As far as engineering disciplines are concerned, it has failed in the past and I argue that it is an obsolete model of research, as thoroughly discussed by Ben Schneidermann with his Twin-Win research model.

Where is the problem then? In many countries government funding programs are not to blame. It is relatively easy to get access to such collaborative research funding in Canada for example, through the NSERC and Mitacs programs. I would rather point to the way many universities are organised and managed. In many institutions, when professors engage in such intense collaborations, they are provided with very little support, finding themselves in the position of quasi-entrepreuneurs without administative help and senior employees. Such activites are often perceived by academic institutions as being slightly more than an ancillary responsibility, failing to recognize the effort that they entail and their importance. Similar to teaching or purely academic research, collaboration with industry and, more in general, technology transfer, should be recognized as a primary role of academia, at least in applied fields of research, thus creating the necessary conditions for a true synergy between public research and private innovation. Additionally, and here I fully agree with Ian, the quantitative evaluation of research, for example in the context of tenure or promotion evaluations, severly discourages researchers from working on difficult and important challenges.

The arguments above are actually related to the second point made by Ian, referred to as "Reductionist thinking". Or, in other words, the failure to account for scale in research solutions. This is indeed a problem I have also addressed in the past, which can, however, only be tackled through collaboration with industry. Indeed, researchers don't have direct access to real systems and engineers (except for open source, which raises other issues outside of the scope of this blog post). Blaming collaborative research with industry for the lack of research impact is therefore contradicting the objective of facilitating research that accounts for scale when devising solutions to software engineering problems. Indeed, scalability, as well as more generally practicality, cannot be an afterthought in impactful software engineering research. And such considerations can only result from collaborative research.

To conclude, it is indeed important to recognize that software engineering research could be much more impactful on society and industry. However, it is also paramount to accurately identify the causes for such a situation and potential avenues of improvement. This, in addition to appropriate funding mechanisms, requires adequate incentives and support for academic researchers in their institutions. Finally, strategies to enable long-term collaborations, establishing clear roles and expectations among stakeholders, are needed and have been discussed by Basili et al.

Lionel C. Briand is professor of software engineering and has shared appointments between (1) The University of Ottawa, Canada and (2) The SnT centre for Security, Reliability, and Trust, University of Luxembourg. In collaboration with colleagues, over 25 years, he has run many collaborative research projects with companies in the automotive, satellite, aeropsace, energy, financial, and legal domains. Lionel has held various engineering, academic, and leading positions in six countries. He was one of the founders of the ICST conference (IEEE Int. Conf. on Software Testing, Verification, and Validation, a CORE A event) and its first general chair. He was also EiC of Empirical Software Engineering (Springer) for 13 years and led, in collaboration with first Victor Basili and then Tom Zimmermann, the journal to the top tier of the very best publication venues in software engineering.

Lionel was elevated to the grades of IEEE Fellow and ACM Fellow for his work on software testing and verification. He was granted the IEEE Computer Society Harlan Mills award, the ACM SIGSOFT outstanding research award, and the IEEE Reliability Society engineer-of-the-year award, respectively in 2012, 2022, and 2013. He received an ERC Advanced grant in 2016 — on the topic of modelling and testing cyber-physical systems — which is the most prestigious individual research award in the European Union. He currently holds a Canada Research Chair (Tier 1) on "Intelligent Software Dependability and Compliance". His research interests include: software testing and verification, applications of AI in software engineering, model-driven software development, requirements engineering, and empirical software engineering.