Software Engineers as Research Infrastructure

A year ago, Reiner Hähnle wrote in this blog about the importance of software as infrastructure—the laboratory equipment of the Computer Scientist. Reiner observed that “Scientists, even Computer Scientists are not necessarily the best choice for building and maintaining software, even if they were actually paid for it” and urged investment in software development positions that are permanent rather than being supported on an ad hoc basis by individual funded projects. This echoes our own experience at Newcastle University where we have created a Research Software Engineering (RSE) team along such lines. So here is our experience, in which, to extend Reiner’s observation, we see not only software, but software engineers as key research infrastructure…

From Short-term Projects to Long-term Services

In the late 2010s at Newcastle University, a series of projects had led to our having a group of colleagues who were not academics but had a wealth of expertise in data science and cloud computing. Rather than existing “hand to mouth” on short-term research projects, over three years we were able to create open-ended RSE positions under Mark Turner’s leadership. This was a small team whose time was charged directly to research grants, with the University providing some underwriting to bridge short gaps between projects. When the team was small, charging directly to grants worked. However, as it grew, the process became cumbersome to a point there was enough political will to create a new “facility”, charging a daily rate to simplify cost recovery, before finally moving off fixed-term contracts to open-ended positions. The RSE positions are at the level of the whole faculty rather than in one school, but offer a service across the whole University and externally. This helps to ensure sufficient varied work to justify the posts and allows integrated access to career development and other services.

The Newcastle University Research Software Engineering team now has about 25 members working on projects across the university, but mostly in science, engineering, and medicine. The primary service (around 80% of activity) is software engineering in research or innovation projects. Around 15% of time is spent helping academic investigators to integrate software engineering into their research proposals, as well as project planning. The remaining 5% is spent training colleagues and PhD students in Software Carpentry with the goal of reducing the need for specialist RSEs to undertake more routine software development.

Though it initially had to invest in outreach to raise its profile, the facility is in high demand. In 2023, the team contributed to projects as diverse as: processing environmental data from low-powered IoT sensors; agent-based software for simulating epidermal cell dynamics; deep learning for real-time spectroscopy; and science and humanities research on bees! RSEs have worked on at least one project with every school and institute in the University apart from Fine Art, and for external clients such as the Alan Turing Institute, the UK National Health Service, and other universities.

Covering the Costs

A crucial aspect of this model has been creating mechanisms to cost RSE time into projects as a facility, rather than specially recruited (relatively short-term) research staff. Through this approach, careful monitoring (and chasing invoices!), around 90% of the team’s cost is currently recovered by chargeable activities. Arguably, this is too high, and time needs to be left clear for RSEs to undertake non-chargeable training and development.

The “facility” mechanism underscores the notion of imagining Research Software Engineers as infrastructure. Most other facilities are centred on physical equipment—think of microscopes, spectroscopy, and imaging devices. In this case, however, the people, and their varied skills and experiences, are central. The facility approach has financial benefits in the UK’s funding context, and within the institution. First, facilities are well understood by funders and easy to cost onto grants. Second, adjusting expected utilisation rates allows for job security when individual RSEs are between projects. Third, the projects that use the service pay for it, via their awarded grant funding. Two services are free at the point of use—grant writing support and training. The grant writing is done at a point where the researcher does not yet have funding, so can’t easily pay. However, this is an investment in future funded projects—in business terms it is the cost of sales. Software Carpentry training is the way RSEs give back to the community, empowering the next generation of researchers so that they can help themselves in the first instance, and only the more challenging tasks come to the team.

RSE Careers

The Newcastle RSEs mostly hold graduate or doctoral degrees, and come from many disciplines—programmers with unusual backgrounds make good RSEs. Some have computing or engineering degrees; others come from the arts, digital humanities, and biodiversity to name a few. Some join from academia, often PhD graduates or postdocs who are motivated more by producing software than papers and grant applications. However, we have also attracted staff from industry, often those who are searching for the variety, interest, and sense of purpose that academia can provide. These colleagues bring commercial engineering rigour that complements the wider team and institution. Any of them might gain more highly paid positions in industry, so the RSE role aims to provide a supportive, flexible, and stimulating work environment.

The RSE role has grown in significance in the UK. 2022 was the 10-year anniversary of the term “Research Software Engineer” and the community has met several milestones in that time. In 2015, the Engineering & Physical Sciences Research Council—one of the leading research funding bodies—invested in RSE development fellowships. In 2016, the community ran its first conference, now an annual week-long event. In 2019, MT was a founding trustee of the Society of Research Software Engineering, a national network for those in the profession, advocating for greater recognition for RSEs at their home institutions and with national bodies. Much remains to be done—there’s no standard formal role definition, and organisations are still deciding on the optimum size, shape, and funding streams for their RSE teams. Other nations are catching up, with RSE positions and teams growing in Europe and the rest of the world.

What have we learned so far?

• Our goal was to treat the engineers as our critical resource, and so most of our lessons have focussed on the skilled professionals in these roles.
• One line of research will not generate enough software engineering work to maintain an RSE team with critical mass, so it is vital to create a team able to undertake varied work across a range of subjects—RSEs are actively looking for this variety in their role.
• Investment is needed initially to generate profile, interest and confidence in the service, but it’s important that demand isn’t the sole measure of success. Healthy demand is part of the picture, but the danger is that time is filled entirely with relatively routine project work, reducing the opportunity to grow software skills more widely in the research community, and to allow RSEs to develop their own careers.
• Consequently, it is critical to develop a clear mission for an RSE team and use this to influence strategic decisions about the team’s shape and size, and the level of cost recovery appropriate to its mission.
• In many organisations, it will be critical to have a strong working relationship with the often-centralised IT services so that there’s minimal duplication of expertise, especially in research-specific IT such as Linux systems and High-Performance Computing.
• Develop career paths for RSEs. These are highly skilled professionals well able to win lucrative employment elsewhere, so the offer needs to be engaging and supportive, making the most of technical strengths, offering projects in areas not seen commercially, and allowing them to develop. Recruitment processes need to take account of the job market.

Reiner ended his blog post on a positive note, and we will do the same. It is possible, given the right environment in both the institution and the national picture, to create an RSE facility that is sustainable and that lifts the quality of research and its impact in computer science and beyond.

In some ways, it’s like starting a software business. You shape your offer and understand your customer base—in our case, the research that needs support, and the researchers who are motivated by producing excellent outputs and winning funded projects. You market the facility, establish the infrastructure and processes for dependable operations, from version control to project management and workload tracking. Most critically, you need to attract and retain great software engineers!

John Fitzgerald is Dean for Strategic Projects in Science, Agriculture and Engineering at Newcastle University, and a full professor in Computing. Having initially worked on semantics and tools for model-oriented formal methods, he went on to lead research on formal approaches to cyber-physical systems and systems-of-systems. His career later turned towards university leadership following a period as Research Director and then Head of the School of Computing at Newcastle. Having eventually stepped outside Computer Science, he is now responsible for Newcastle University’s infrastructure and strategic development in a range of disciplines from farming to quantum physics!

Mark Turner has led the Newcastle University Research Software Engineering team since its creation in 2018, growing it organically from a team of two to now twenty-five. During that time he has delivered multiple software projects across a wide range of disciplines, and now works on team management, policy & strategy, and research engagement. He graduated with a BSc in Computing from Northumbria University in 2008 followed by an MSc from Newcastle University in 2012. In 2016 he was elected as a trustee for the UK Research Software Engineering Association, contributing to the transformation of the association into a registered charity in 2018. In 2022 he was the programme chair for the UK RSE Conference.