Research Methods and Software Maintenance

Jurgen Vinju is senior researcher in CWI’s Software Analysis and Transformation (SWAT) group, part-time full professor at TU Eindhoven in the Model Driven Software Engineering (MSDE) group and co-owner and senior language engineer at Swat.engineering BV. His research is focused on metaprogramming and applications thereof in (domain-specific) language engineering and software reverse engineering. His goal is learning how to control software complexity; in order to be able to better understand software such that we can maintain and extend it more efficiently and with better quality guarantees.

What do you think the status of software maintenance research at ETAPS is?

First, I must say I am an ETAPS child. My first paper was at ETAPS, and I stayed in the community; you know how these things go. But I noticed, as many others, around the change of the millennium, that ETAPS had quite a clear school of thought: The ETAPS thinking is that we can or must invent or create the future. We are building elegant theories and beautiful software, based on principles and provably correct by deduction. The main way of reasoning is deduction and the way of the attitude towards the material is I want to create something new that is beautiful.

Then the industry in the Netherlands around the year 2000 asked whether we can apply them to existing software. We said yes, of course, and already after a couple of weeks, we realized that the stance of platonic ideals is not the reality of existing software. There is no concept of correctness, the scale is beyond what you can conceive—as a human investigator, you will never understand the whole system.

For me, there was an important point: We did not actually know what we were studying at the moment and what questions we were asking. The new perspective came with an empirical stance looking at the world where you go and accepting that the COBOL code that you are going to parse is in principle undefined, and you do not know what it is, but you are still going to do something.

You could not publish something about an unsound tool, but that was exactly what was needed at the time. But there was the Conference on Program Comprehension and the Conference on Software Engineering and Maintenance and others. So there our papers would flow automatically, but the same papers would get rejected at ETAPS. I think this happened to more people at ETAPS.

Since then, ETAPS has also migrated a bit with it; that’s nice, specifically the people that build the tools. I think one reason is that, for example, as a PhD student, you get a compiler from your supervisor, and have to deal with a compiler that exists. In the moment you have to maintain something, you become an empirical scientist.

Did this change the ETAPS school of thought?

You have to pick a school of thought to be able to judge a paper, and you cannot say “I’m doing all induction and deduction and abduction” in the same paper. Everybody would get lost. And so deduction at the software maintenance conferences just does not fly. On the other side, at ETAPS abduction, choosing the explanation that fits your observed facts best, is frowned upon. You have to choose, because otherwise everybody gets confused.

One of the best papers I ever wrote was on interpreting the soundiness manifesto for the Java real world at that time. The question was if we take all of the soundy assumptions that are made by the current static analyzers and apply them to real Java code, where do we go wrong? Where do we go right? How many times? When is it really relevant to make a certain soundy assumption? When does it make any sense? And what other soundy assumptions are still missing? It was really hard work: two years, two people full-time. Because every static analysis tool makes arbitrary assumptions based on the context of the author, depending on whether they were writing for POPL or making a type checker for IBM, for example. You cannot just compare these tools without considering all the different contexts that they were in.

How does this relate to, for example, the artifact evaluations that are introduced at ETAPS conferences now?

An artifact evaluation is not a different approach to thinking about what your system is for. It’s just a different way to evaluate it. But it’s more about the research methods. And in every conference, of course, there are the favorite methods. For a while, we did ground truth evaluation from social engineering in these empirical studies. That went out of vogue because it was too much work. But if you did not do it back then, you would not fit in.

All these things are part of a school of thought, and help the reviewing become more efficient. Because if you have to come up with a new methodology for every paper, the reviewers make many more mistakes. There is a benefit in these schools, but I like venues where more forward-looking papers can have a less stringent evaluation because otherwise, you close off the community for innovation. That is a real danger of empirical methods. They are very hard, and you should not invent them all the time. But then, on the other hand, if you do not invent them, you do not go forward. And in a theoretical conference, it is the other way around. If you keep inventing new methodologies, your relevance becomes very low—you basically grow your own ivory towers all the time. You have to find a balance, otherwise there is a danger of navel staring on both sides. That is why it is nice to switch once in a time, to go from one conference to the other, but it is not such a smart idea to think that you have to change cultures and incorporate everything that can be thought of, because that’s where you lose both your identity and your efficiency.

How should we handle these different schools of thought when training junior researchers?

First, I think that every master’s thesis or first paper from a PhD student should contain a reflection on what school of thought they are in and why they are arguing in a certain way and not another. And they should make the difference between abduction, deduction, and induction. If you do not know how you’re thinking, then you’re not a master of your own mind. At least one must know in what school they are in and what other schools exist. I think that should be required, but sadly it is not and I do not see it happen often.

Differences are fun. For example, in my working group, you can meet Asian, African, American, South American, and European people, 35 in total. Wow, when they meet, and if you’re open, you learn so much. That’s great, once in a while. But if I have five PhD students under me, and they all have these existential discussions every day, that is not really productive. The whole PhD experience is already an existential crisis, you do not want to put the existential crisis of the entire field on to.

I think that a PhD candidate should be able to live in a school by choice and know what school of thought they are in. And I do not think it’s fair to ask them to cross the cultural boundaries of the whole world while they’re doing their PhD. In a master’s thesis, it is expected that you can reflect on different schools. In the PhD, you show that you can survive in one, and then afterward you can think about crossing boundaries.

This is not true for everybody, but I guess that would work for most people. Of course, there are these rare people who cannot be held and cannot be captured in one school. They make up their own minds, and often these people are those that become professors. I’m not forcing anybody into a school, but I think it’s a service for many. It’s a home to be in for years, where you do not have to convince everybody of your research methods while you’re publishing. That would not be workable.

I published quite a few interdisciplinary papers, and sometimes you have to write to unconvinced people about things that you’ve just assumed since your master study’s days. It’s a cool thing to do, but I do not think it’s fair to put that on every piece of paper. But if you have the tenure, and you have the time, of course, it’s very valuable, because you can take bigger risks.

What I find hard is very successful people with closed minds. So you sometimes get these people with hundreds of publications and an incredible h-index, and they think that what they do is the only road to success. And of course, it’s not true. The reason for their success is also the community that supports them, the culture and the way of thinking that helps them grow further. And if you get crushed by someone like Dijkstra, who was one of those people that could crush somebody when they came in with a different way of thinking, that could really destroy your career. Sometimes that makes me angry. If you are a 50-year-old or 60-year-old successful researcher, you should have an open mind to other ways of thinking.

Is there something you would like to see in the ETAPS community?

Before I say anything, I have to admit that I don’t know what or how ETAPS is today. I have been away for a long time. So if I’m proposing things that have already happened, it’s because of that. I think that authors should clearly signal if a paper is from an empirical standpoint and then a reviewer should be selected that can review such papers. It is wrong to think that because you were a reviewer for decades, you can review an empirical paper. To review empirical papers, there are so many ifs and buts, so many details always, and any logical fallacies that can happen that I think that it requires the same level of attention that we give to the other papers, the same quality of attention. So get more reviewers for that, and then you also get more papers for that, which are appreciated by the right people.