BerandaComputers and TechnologyConverting legacy code to a modern language

Converting legacy code to a modern language

A question Sabine Hossenfelder asked on
Twitter prompted this essay:

If you had a ten million lines, decades-in-the-making fortran code,
how’d you convert it to a modern language, how long would it take, and
how much would it cost?

There are no short or easy answers to this question. Inevitably, it
depends on a lot of factors:

We need a perfectly clear understanding of what the program is, what
it’s used for, and by whom.

We need to know what we’re trying to achieve by changing it. If this
is not a clear and significant benefit, and not clearly articulated
and shared by the project team and sponsors, the project will flounder
and ultimately fail.

We need to know if the program still works after we’ve changed it.
This sounds basic, but is a surprisingly complex topic, especially for
large, old systems.

And finally we need to understand the organisational politics that
will govern our project.

Let’s try to unpack some of these questions.

Domain

What is this thing even? 10 millions lines of code is a lot.

Some systems are figuratively speaking responsible for everything from
the moon landing to the cafeteria menu, an “everything but the kitchen
sink” system. This is especially common in older systems, conceived at
a time when the overhead of building and integrating new systems made
just adding an extra feature to an existing system a very tantalising
proposition, even when (modern) software architecture principles
suggest it should be separate.

Some systems are figuratively speaking responsible for everything from
the moon landing to the cafeteria menu, an “everything but the kitchen
sink” system
   

These systems are comparatively easy to reengineer, as we can often
peel off these features in an iterative manner. Indeed, this will
often have happened: We may find that the cafeteria menu module was
last used in 2002 when that particular usage was migrated to another,
new system.

At the other end of this scale, we find the single, contained model,
responsible for converting a tightly bounded set of inputs into a
similarly tightly bounded set of outputs. It is easier to reason about
a system like this, but the stakes are higher: This system likely
exists because what it does is very important to someone powerful.

Conversion? Or The Big 2.0 Rewrite

Why is it important to convert this program? Fortran works fine, so
why not keep it in Fortran? Presumably, there are changes that need to
be made, and it’s easier to find programmers in more modern languages.
However, the hardest part of programming isn’t the language, it’s what
we’re trying to do with it. Next to the challenge of understanding
what those millions of lines are doing, learning Fortran will be a
rounding error.

The hardest part of programming isn’t the language, it’s what we’re
trying to do with it. Next to the challenge of understanding what
those millions of lines are doing, learning Fortran will be a rounding
error
   

One could imagine that one of the reasons it’s difficult to make
changes to the existing codebase is that it’s poorly structured.
Getting the structure just right is difficult for the best
programmers, and especially under the pressure of deadlines and other
constraints, shortcuts are often taken. These shortcuts are known as
tech debt, which optimistically suggests the intention to pay it down
in the future. In reality, this is often neglected. These decisions
compound over the years, and in our case, the problem is quite likely
to be exacerbated by the code being written by people who are academic
domain specialists, not expert software engineers. Computers have a
nasty habit of not caring how well engineered software is, only
whether it runs, so “works” trumps “good”.

These shortcuts are known as tech debt, which optimistically suggests
the intention to pay it down in the future. In reality, this is often
neglected
   

This mess, much more than the language, is why making changes to the
codebase is so difficult. But untangling it will complicate any
rewriting effort to an even larger degree.

Things you should never do

I would be remiss if I didn’t take the opportunity to link one of the
seminal software engineering blogs here: Joel on Software: Things You
Should Never
Do
.
Spoiler alert: Exactly this.

What we can do, however, is to progressively develop the program in a
certain direction that allows us to achieve some of the same outcomes.
A favourite text on this process is Growing Object-Oriented Software,
Guided by
Tests
.
Your Fortran program probably isn’t object oriented, but many of the
principles still apply. The idea is to identify and carve out
independent pieces of code, iteratively change and improve the code,
continuously adding smaller and smaller tests as we work through it.
This requires a “ground truth”, which we will return to.

The reason we want to do this, as opposed to starting from a blank
slate, is to enjoy the disciplining constraint of running our code
against a realistic workload (ideally production, depending on the
exact situation) as soon as possible after writing it. Plenty of
issues big and small don’t become apparent until this happens, so
modern software engineering preaches this practice with rather some
zealotry.

Ground truth: What does good look like?

In order to test, we need to know what correct looks like across a
wide range of inputs. Assuming the code isn’t well documented and
tested, the fall back plan is to have a substantial corpus of inputs
which we can run through the existing program to provide expected
results for an end-to-end test suite.

For each segment of the program we want to work on, we run the suite,
recording what goes into the segment and what comes out: This is now
our test suite for the segment, and we can set to work on it in
isolation.

However, this only works if we can be sure the test suite is fully
representative across the full input domain. Depending on the program,
this can be a very difficult exercise. It seems likely that a very
large part of this code was written to deal with various
idiosyncrasies in the input data – of these, few will be easy to
understand: They will have unstated assumptions, and if we don’t have
input data exhibiting these idiosyncrasies, there are some deep rabbit
holes to crawl down here, trying to understand what’s going on, or if
its even needed anymore. Remember, if we just blindly copy the
existing logic, we will just be carrying over the complexity, not
achieving the (presumed) goal of the project.

A very large part of this code was written to deal with various
idiosyncrasies in the input data – of these, few will be easy to
understand: They will have unstated assumptions, there are some deep
rabbit holes to crawl down here
   

The Vetocracy

Last, but in many ways the most important issue: Assuming we get all
our technical ducks in a row, who will we need to convince to actually
undertake the project? One of the biggest and most underrated
challenges of large software projects in non-software organisations is
to get all the stakeholders aligned . For more sprawling systems, it
can be difficult to even identify these, much less convince them of
the value of the project.

Assuming we get all our technical ducks in a row, who will we need to
convince? One of the most underrated challenges of large software
projects is to get all the stakeholders aligned
   

It’s easy enough to argue the benefits of a reengineered system, it’s
even comparatively easy to secure a budget for the project. What’s
almost impossible is to get through the vetocracy.

Our rewritten program almost certainly won’t produce the exact same
results, especially if it deals with numerical computation. There will
be minor variations on how floating point numbers are rounded or
truncated and how certain data types are represented. The idiomatic
way to do some things in the target language will be slightly
different from the Fortran way, with results being slightly off in
some decimal, or presented in a slightly different way. How many
people will we need to convince that this is actually OK?

Inevitably, we will find errors in the Fortran code and want to fix
them. Someone somewhere more powerful than us got their current job on
the basis of a report that (may have) included an incorrect figure due
to the error we found. They will be obliged to go back and report
this, but not before they sucked all the oxygen out of our team for
three months exhausting all possible theoretical reasons that the
error we found is not, in fact, an error.

That cafeteria menu subsystem? Because it hasn’t been used for 20
years, nobody knows who owns it. Signing off on its removal is a risk,
punting on the decision isn’t, so expect this decision to get punted.

And this is all before the lawyers get involved.

One of the reasons so many of the large and important organisations
running big, old systems don’t set out to do something like this, or
fail miserably if they try, is that they underestimate this dynamic.

One of the reasons so many of the large and important organisations
running big, old systems don’t set out to do something like this, or
fail miserably if they try, is that they underestimate this dynamic
   

Mountain lake

Conclusion

Doing this is literally possible, but just barely. In this essay, I’ve
set out some of the minimal groundwork required to even contemplate
such a project: We need clarity on what this program does, why we want
to change it, whether we can provide a compelling argument for the
correctness of our changes and whether the bureaucratic context even
allows us to.

Doing this is literally possible, but just barely
   

Good luck, you will need it.

Follow @mseebach

Read More

RELATED ARTICLES

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Most Popular

Recent Comments