Volatile Infrastructure is worse than volatile applications

design a new module hierarchy, introduce aliases for module bindings, and still supply the old module hierarchy during a few years for backward compatibility.

Please do not do this. It is a recipe for disaster.

Do you remember when Lilypond broke with Guile 2.0? How long it took to get it working with modern Guile again? This plan would cause similar problems — but much, much worse.

Lilypond is the one tool using Guile — the single tool — which actually reigns supreme in its domain. Nothing else comes even close in quality compared to competitors.

The tools broken by breaking things “with sufficient warning” are usually the most advanced ones. Specialized tools. The ones which rule in their domain. That people depend on.

There’s something many useful things have in common: they work and need little changes.

When the infrastructure these tools use intentionally breaks the tools and requires constant upkeep just to keep working, this makes the infrastructure volatile and unreliable.

Such large changes promise to make the system better, but they leave it in a state of eternal semi-brokenness, because before the first breakage is fully resolved, the next part gets rewritten and it causes more breakage. And so the next breakage comes. Because doing such “let’s just change it all” steps changes the culture.

Python tools had just kept working for years and years before Python 3. After the release of Python 3, they broke every few years. It seems like the culture had changed to one that accepts being volatile.

Making our tool volatile would make it a dumb idea to depend on it for infrastructure. And I very much want to use it for my infrastructure.

I reduced my reliance on Python after having to spend time again and again when my existing tools broke with different Python 3 releases. Other people did the same.

Please let us be a reliable foundation for infrastructure.

About breaking backward compatibility , i understand it could be a disaster… but if Python made this choice

The experience of Python should not be encouragement. It was a disaster.

Let’s look at what the one responsible for Python 3 said about changing code:

No one wants their code to break, but they always want everyone elses code to break by adding the keyword they want.
— Guido van Rossum, 2018

He has a slide of what went wrong.

• underrated Python’s popularity: people using every trick in the book of what worked.
• Underestimated the importance of 3rd party packages: dependencies on lots of tiny little modules that solve one little problem.
• clumsy migration: only mostly working auto-translation.
• no runtime compatibility: a single non-converted dependency blocked compatibility.
• supporting different versions in the same code is hard.

Despite those lessons learned, many tools are still broken, especially specialist tools. Many of these are dying. More than one and a half decades after Python 3 got introduced. The developers continued to maintain Python 2 for 12 years — until 2020.

And even worse: some Python 3 point releases broke existing code again. The big breakage seems to have caused a cultural change. Breaking compatibility just a bit now seems to be deemed OK.

Python 3 still got takeup. I think it was because the AI craze hit and Python was in a good position to provide readable (yet brittle) APIs to C++ code. It was in a growing field. But I personally lost many weekends fixing existing tools. And while I can still use Python, I usually use different languages for new tools. And I am sure that I’m not the only one.

If you have to invest lots of time anyway, you can just rewrite in another language instead, which is much more enjoyable than spending days over days searching for the source of some remaining bytes-to-unicode breakage.

And I am certain that Python 3 caused lots of damage to Mercurial. Do you remember when Mercurial showed that Python can compete in performance with C while only using a minimal set of performance critical tooling in C?

Mercurial competed on roughly equal terms with Git.

One distinguishing factor were thirdparty extensions which added specialized capabilities. Many of these got broken with Python 3. I personally gave up maintenance of two extensions because I wrote them when I had free time and enthusiasm for version tracking workflows, but as most other people, when they were broken as Mercurial finished the Python 3 transition, I didn’t have that time anymore.

These extensions were useful, but many where not maintained. Before Python 3 they did not need maintenance: they solved a problem and just kept working. After Python 3 that was no longer the case.

Mercurial has become a niche tool now, even though it is still much easier to use than git at comparable power.

I do not want that fate for our tools.

Sacha Chua writes how she tinkers incrementally, finding time between tasks, asking “What's the smallest step I can take? What can I fit in 15-30 minutes?” — Choosing what to hack on: https://sachachua.com/blog/2024/01/choosing-what-to-hack-on/

This approach improves the work environment step by step to become better than any other. In a similar way, I now ended up using exwm, and while not perfect, it just works better for me than all other systems.

But since this depends on doing small steps and moving forwards in little steps, there’s no time for large-scale maintenance. You define what’s the right step, and then you take it.

If something breaks, that takes up at least a full improvement slot. Often just searching for a solution takes longer than you have.

So this approach — which can lead to the best personal work environment possible — depends critically on API stability. Even a deprecation that affects multiple of your modifications can put a stop on tinkering for a long time, because fixing something that broke due to changes from someone else is a very different kind of working than letting your curiosity lead to even out a rough edge in your workflows.

Someone would surely come in and quote xkcd 1172: https://xkcd.com/1172/ I consider that a harmful strip — it has a point, but it got weaponized to brush away concerns about stability and muddies up the understanding that those with the most complex or most advanced setup are the ones most likely hit by API breakages.

If you see 1172, remember that Lilypond had almost ended up ditching Guile because of breakage that hit them with the 2.0 release. The one Guile-using tool that is absolutely dominant in its domain (the most beautiful music scribe) had almost stopped using Guile.

For us, the impact is even bigger, because far more people tinker to optimize their setup.

So I want to plead with you to remember the risk of volatile software¹, volatile infrastructure², and soft trauma³, when taking decisions about backwards compatibility.

Breaking backwards compatibility has much wider-ranging implications than it seems while working on code, and it hits the most most advanced specialist tooling and the most enthusiastic tinkerers the worst.

¹ Volatile Software — do not be the tool which breaks itself or other tools on update.

² Volatile Infrastructure is worse than volatile applications.

³ Software developers should avoid traumatic changes — two kinds of trauma: everything needs work to get working again or to get idiomatic again.

Keep in mind that a new structure we define is not guaranteed to actually stay better. There would be mistakes, but different ones.

We can only expect that the new one will be significantly better in case where either the computing environment changed substantially or where our knowledge and skill of how to define such a structure improved substantially.

Your argument would be more compelling for me if we were talking about updates which occur without user intervention or control.

In my case it is updated automatically. I use rolling release distros. There’s never a time when I say “now I get all the new versions, let’s look for breakage”.

Change is inevitable and such changes will from time to time, break things.

Is breakage truly inevitable? Do fundamentals have to change?

If some small detail changes in a convoluted setup I have, that’s something I can cope with.

That there are packages that almost never break and packages that almost always break on update, but little in-between sounds like most breakage isn’t inevitable.

Here’s the source for the statement: stevelosh.com/blog/2012/04/volatile-software

It makes my point much more succintly:

and a second point:

With respect to the current issue, I think the key question here is was communication of this change sufficient and if not, what can be done to make such communication more effective.

It cannot be made effective enough. If you make it effective enough, the other gazillion packages on the system will use the same mechanism and that will make it ineffective again.

The only way that works is to avoid breakage on update — except for the few cases where it is truly unavoidable.

Things you should never do — painfully learned — are things which prevent flagship programs from updating.

That Python made it a huge task for Mercurial to switch to Python 3, one of the programs that did everything the Pythonic way and achieved speed competitive with tools written in C, was a big mistake. Mercurial was the poster child of creating an application that uses Python to the best of its abilities: building everything in Python except for the few critical hot paths. And I can hardly overstate how much damage the forced Python 3 transition did to Mercurial.

I have the impression that this disrupted the whole Python environment. With Mercurial broken pretty badly by the change and taking years to adapt to Python 3, the example of using Python to wrap a tiny core of C seems to have been missing, because the new Machine Learning tools are written very differently: with a huge C++ backend that’s controlled by a pretty thin layer of Python. Insulated against volatility of Python, but delegating the language to second place.

That Guile 2 broke Lilypond — the one Guile-using tool which reigns supreme in its domain — costed Guile dearly and blocked adoption of Guile 2 for a long time.

Part of this block was a performance regression that got resolved with Guile 3. Other parts were painfully resolved over several years, and the development team of Guile got lucky that dedicated Lilypond maintainers decided in Guile’s favor in the end.

Other things you should not do is to break user-scripts which have to be touched by everyone who bought into your system.

Any time people have to invest work to keep using your system, you lower the bar of moving to something else. It creates a breaking point where your existing users might wander off. Like I wandered off to Scheme.

Keep in mind that adding a version number to libraries does not solve backwards compatibility. It moves the required work to source maintenance instead of package maintenance — and then adds more complexity in keeping your own tools moving parts compatible with different versions of themselves.

You’ll then have to test every version defined by your tool with every other version, else you get breakage the moment one library that uses your tool updates to a newer version while another is still using the old version and they are both imported by a third.

When a library updates to a new dependency version and another does not, I have two options: Either this is allowed, then I just doubled my testing area, or this is forbidden, then programs break on update of a library and need to update all their libraries to the new dependency version.

Also this increases the amount of source code you need to keep around — or if you use versions on package level, the number of tools that have to be installed for specific programs to work.

To minimize this, you can define different versions as just aspects of an internal implementation which fulfills all their requirements and can do conversions between the different versions. Just interfaces to something more abstract.

So you can use versioning to shift the work of backwards compatibility to different groups of people, maybe ones where it is easier to ensure, so its cost might be reduced, but you cannot erase this work. So take the time to check where versions really bring more benefit than cost. There’s still no silver bullet.

IOW, if you don't want changes in your dependencies, then just don't update them.

This does not work.

You often have to update dependencies for security reasons. Got a new gnutls or openssl or openssh with new cyphers you need to have a working program — will Guile 3 get updated to support them or will you be forced to migrate to Guile 4 to keep your tool working?

Even where it’s not security, you will need to interact with new formats or changed parts of the system that need up to date modules which depend on new library versions.

That’s why it does not work that way. Freezing libraries is the path to automatically turn working software into legacy software by creating a constant upkeep cost to avoid becoming stale and unusable.

There are sometimes actually good reasons to break backwards compatibility, but they are very, very few, and if you have an issue that you think is a good reason to break backwards compatibility, it most likely is not.

Do you want to create tools that people have to restrict to internal networks a decade from now (anyone else thinking of companies still bound to Windows XP?)?

Or do you want tools to forge a reliable path into the future where we can consistently build upon existing work and actually stand on the shoulders of giants?

If so, we must avoid regularly breaking the giants’ knees.

If you manage to freeze best practices without blocking ways into the future, then you found part of the holy grail of software development: You managed to find one fragment that’s so good that it never needs to change again and everything new you do fits to it.

Typically reality isn’t quite as beautiful and changed requirements can break your model. They say about Lisp that it’s a snowball: You can keep adding stuff to it and it always stays a snowball. That’s close to this beauty. But Lisp is also full of car/cdr-namings and legacy you cannot shed, even though you might want to.

You cannot reach-and-keep perfection in a changing world, you can only try to limit the pain for users and stay close to something which feels right.

Volatile projects do not work to limit the pain.

Stale projects do not try to stay close to ways that feel right in a changing reality.

A good project needs to get as close as possible to a consensus (I’m not saying compromise here, because that’s not what I mean — the goal is something which unites both) of not being volatile and not becoming stale. A consensus of being stable and being up to date.