When considering the utility of a programming language, to decide what to learn next, I mostly care about how well its concepts help me perform generalisation and specialisation. I question myself, to what extent does that concept (e.g. structural typing, constrained generics) help me to express abstract, general ideas just as well as actual jobs correctly and efficiently?

Most concepts of programming languages can be evaluated that way, regardless of the nature of the language. That is, I’m inclined to say:

Programming is all about generalisation and specialisation.

The notion is quite pointless in its abstract glory. You might view this assertion as a neat allegory to the subject of this post – useless abstractions and over-generalisation.

For the purpose of this article, generalisation is the act of refining existing abstractions and generalisations. I am not talking about abstracting the real world to make it computationally tractable, but about the later stages in program evolution in which abstraction and generalisation tend to aid program-internal organisation.

To further clarify, abstraction is leaving out detail while representing things or concepts meaningfully. Generalisation is giving a name (and scope) to detail that cannot be omitted. Specialisation means filling out the details. The modes of thinking behind them are not subject to this post.

The issue

Much of a software’s success depends on the delicate balance between the two aforementioned modes of program construction. Likewise, many of its ailings can be described as an imbalance in the application of generalisation vs. specialisation. The famous C2 Wiki describes this phenomenon very well as premature generalisation.

The actual problem is that building generalisations requires understanding. An existing premature generalisation makes it much harder to build that understanding later on while creating artificial complexity and inhibiting progress on the software. Often, the root cause is hard to see and hard to fix. Complexity kills.

Once I fell into that trap myself – to err on the side of generalisation – and after learning from my mistakes, I’d like to rephrase:

Programming is all about generalisation and specialisation, each at the right time.

This is begging the question “What is the right time then?”. To me, it depends on the kinds of information available. Perhaps counter to intuition the right time does not, in my opinion, depend very much on the task at hand.

Whether I work as a software developer or architect certainly influences the concreteness of my output. A system design or an application interface is much less concrete than a component implementation or even an abstract class. But the task type does not typically change the balance of generalisation and specialisation inherent in the work. Overall, I attribute about 10-20% of the core work activities to generalisation and the rest goes to specialisation.

Gauging information

This leaves us with available information to ponder. The idea is to generalise when you have the right information and actual or foreseeable needs. Otherwise, go for specialisation.

The kinds of information you need to perform specialisation on software may sound familiar to any software practitioner:

  • Requirements
  • Use cases
  • Bug reports
  • Designs and Mocks
  • Data models
  • Protocol descriptions
  • Network layouts

The kinds of information required to guide generalisation are quite different:

  • Business goals
  • The expected range of requirements
  • Variability and similarity of artefacts
  • Architectural constraints and goals
  • Observed (!) rate of change
  • Likelihood of future change
  • Unimplemented or hardly implementable features
  • Known bugs and their root causes
  • Artificial complexity in the system
  • Well-established similar system designs

I do hope the above lists will help you be more critical of your evidence when deciding whether to generalise some of your code.

Let us reflect on these kinds of information. It is easy to see that information supporting specialisation is often consumable up front, whereas most information required to underpin generalisations is only available as a project unfolds. Also, the relation of input information to a given output (e.g. a commit) is much clearer for specialisation than it is clear for generalisation.

It should not be hard, then, to decide about generalisation issues day by day. Most of this challenge is nicely captured in the mantra “Simplicity Before Generality, Use Before Reuse” populated by Kevlin Henney. Or just YAGNI! – You Ain’t Gonna Need It!

But there seems some more to it.

Information vs. Experience

I was recently challenged on a proposed separation of messages, alleging they could be merged and thus simplified. That was true, but it was settling things on an important business goal whose implementation strategy was not even decided. Unifying the message would have limited available options far too early. Citing this resolved the issue, but I had never really clarified this dependency for myself until I got challenged.

I can only blame experience for this outcome, but it shouldn’t have been experience – it should have been knowledge of business drivers. There was uncertainty instead, and the teams needed to proceed.

The above anecdote contains two lessons about our subject. First, uncertainty may force some flexibility, together with its abstractness and complexity toll. Being agile may mean generalisation just as well as specialisation. Second, it highlights a glaring omission above. Did you notice? “Experience” is not on any of the lists above. It was the first thing I wanted to include, but I took a second to reflect and then refrained.

Experience is a great differentiator. However, back when an earlier me was an easy victim to over-generalisation, I told myself that my experience allowed me to do so (and I might otherwise not learn enough).

By an amazing coincidence, developers I consider prone to early generalisation also cite their experience. Less commonly, they admit that their curiosity, or worse, fear of losing out is driving them. How experience can become a pretext is staggering once you realize the pattern. Experience as a pretext paves the way for future-proof, smart-as-smart-can premature generalisations. It would just be silly to keep experience enlisted.

Yet these excuses have a point – the right kind of experience still helps. To learn how to abstract and generalise properly, you pretty much have to do it prematurely, too late, too much, and too little. You don’t know until you do, but prefer to experiment with pet projects of your own. The people you work with will thank you.

Generalisation vs. Quality

Ineffective teams sometimes display a strong tendency towards early generalisation, especially when they view generalisation as the posterchild of quality code.

In the worst case, this notion becomes groupthink, and people may be shamed into premature generalisation that way. In extreme cases, such behaviour may create a split between ownership and authorship, i. e. people rejecting responsibility for code they merely wrote to comply with peer pressure. This characterises an important turning point; a team’s tendency towards early or over-generalisation may cause a constant churn of purportedly sensible improvements (thinly veiled as refactorings) that undermines quality and delivery.

Although a few highly disciplined teams may not benefit from going through a less generalised code phase some of the time, what differentiates them is their deep domain knowledge, not their programming skills (which will be good on their own standing). E.g. in scientific computing or well-planned projects, people may be able to read up on their subject and come up with good generalisations up front. But don’t mistake the exception for the rule.

Bad code is often devoid of abstraction – but that doesn’t make the mere presence of abstractions and generalisations a sign of quality. It is typically hard to read back the information underpinning a generalisation, or even to know whether it’s working well. On top of that, under-abstract code yields better information about the missing generalisation than prematurely generalised code.

Summary

Generalisations are never as good as the mechanisms used to implement them are clever. They are as good as the information used to code them was a good predictor of future variability. Being careful to have enough of the right information first-hand will help ourselves and the people we work with.

In his mentioned article, Kevlin Henney proposes “Simplicity through experience rather than generality through guesswork.” This hints at a useful application of experience towards generalisation. We should be careful to use our experience in generalisation, and when we do, do so in favour of simplicity. Likewise, we should not conflate experience with guesswork – I know I did. A single person’s experience should at most serve as a tiebreaker.

In any non-trivial generalisation endeavour, we should strive to have enough relevant information. Feel free to use the lists above to estimate. Especially when important information is still missing, you are most likely to just add noise to the process. Don’t be tempted to supplant first-hand information with (however relevant) experience, or arbitrary claims of future-proof reusability.

You can generalise less or later, and in the meantime, most people will find your code easier to work with. To wrap it up:

Good programmers know how to generalise, and great programmers know when to generalise.