Accidental Complexity in Software Development

[This post is a response to this tweet from @RSessions]

Software complexity can be sliced and measured in many different ways. One way that software developers and architects choose to talk about complexity is by dividing it into accidental and essential complexity.

Accidental complexity is unnecessary complexity that depends on the method used to solve the problem. If your software solution uses EJB 2.1 it has higher accidental complexity than if you would have used EJB 3. If your website uses Java Servlets, JSP, Struts and JDBC it has higher accidental complexity than if you would have used Ruby on Rails.

The software industry is always working to reduce accidental complexity but the last few years have seen an increased effort and adoption of tools and techniques with as little accidental complexity as possible. The Spring framework was a game changer for Java developers in the same way the relational database was for DBAs in the 1970s and these changes are for the better.

In 2000 when Java became an alternative to C++ it removed a lot of complexity. In 2005 the Spring framework removed a lot of complexity from EJB. In 2007 Sun decided to help the adoption of dynamic languages (such as Python and Ruby) on the JVM which provides alternatives that remove a lot of complexity. This reduction of complexity on the lowest level of IT projects should have an impact on the overall IT projects failure numbers.

However, I believe that it will take some time yet. The adoption of new technology for an organization takes time. There’s plenty of existing code that’s not easily migrated and there’s issues such as knowledge, competence and the general resistance to change to consider as well. My personal estimate is that the industry at large is somewhere around 2005-2006 on this curve.

Accidental complexity is not limited to describe technical implementation details such as programming languages. In Lean terms, I believe accidental complexity is called waste. Agile project methods such as Scrum can be described as “some other project method” (you choose) with accidental complexity removed. Removing accidental complexity is good, but it’s not enough in it self. Methods, such as Simple Iterative Partitions (SIP), need to be applied and incorporated in projects as well.

In 2002 Technology Review published an article called Why Software is so Bad. In it they refer to a multi-year study from Carnegie Mellon University where it is found that the average programmer makes 100-150 errors per 1000 lines of code. It’s not explicitly stated in that article how many of these errors are based on misunderstanding business requirements and how many are based on the accidental complexity of the programming language, but I think it’s safe to say that a fair amount are errors related to memory handling (this is also hinted at elsewhere in that article).

And while those numbers are probably still true for C/C++ developers today, I don’t believe that they describe the situation for Java/JEE programmers and certainly not for Python and Ruby programmers.

Unfortunately, I cannot find any research on this subject so I remain hopeful, but unsure.

The Architecture of Complex Systems

These are some notes and comments on the MIT Sloan School of Management Working Paper: The Architecture of Complex Systems: Do Core-periphery Structures Dominate?

The paper sets out to investigate common structural features of several open source applications and how these structures evolve over time. 1 286 releases of 19 different applications have been investigated and the most interesting conclusions drawn are:

1. … we find evidence that variations in system structure can be explained, in part, by the different models of development used to develop systems. That is, product structures “mirror” the structure of their parent organizations (Henderson and Clark, 1990). This result is consistent with work that argues designs (including Dominant Designs) are not necessarily optimal technical solutions to customer requirements, but rather are driven more by social and political processes operating within firms and across industries (Noble, 1984; David, 1985; Tushman and Rosenkopf, 1992; Tushman and Murmann,
1998; Garud et al, 2002).

2. Our findings highlight the cognitive difficulties that face a system architect. In particular, we find no discernible pattern of direct dependencies that can reliably predict whether a system has a core, and if it does, how large it is. In essence, system structure is driven to a large extent by the indirect dependencies between components, which are much harder for an analyst to understand.

It would be very interesting to extend on this work and to investigate business applications/systems rather than Operating Systems, Databases and Desktop applications.

It would also be interesting to see if the same results can be obtained when looking at Java/JEE software rather than applications written in C/C++. The reason I think we need updated numbers is because software development has taken a few important steps forward lately.

A huge problem with C/C++ (manually handling memory) has been removed in languages like Java and C#. However, early (2000-2005) Java/JEE applications are likely equally complex as earlier systems because of the accidental complexity of EJB 1.x and 2.x. Also, the rise of dynamic languages (2007-) such as Python and no-nonsense frameworks such as Ruby on Rails have further reduced the amount of code necessary to implement a system.

So, even though the study is very interesting it is sad that it doesn’t investigate the current state of art. If anyone knows of such a study, please let me know. It would be very interesting to see if we’ve moved forward since 2002.

On average, professional coders make 100 to 150 errors in every thousand lines of code they write, according to a multi year study of 13,000 programs by Humphrey of Carnegie Mellon.

              - Why Software is so Bad -Technology Review July/August 2002

The IT Complexity Crisis

These are some notes and comments on Roger Sessions’ white paper on complexity.

Whilst I agree with the gist of the paper I have several doubts about details. I will, however, not go into most of those here since they are discussed elsewhere. Those who remain unanswered are listed below and I’d be very interested to hear other people’s thoughts about these.

1. The paper talks a lot about reducing complexity in architecture but the only perspective discussed is business level functionality. I know that Roger has commented elsewhere that this composition of systems/subsystems and services also be used in implementation but that still leaves out many aspects which contain complexity.

As architects we need to look at a system from many different perspectives and produce a solution that solves, or at least addresses, all of our stakeholders’ concerns. In my experience, complexity builds up during this process. I’m not convinced that the simplest possible composition of business capabilities gives the simplest possible overall architecture.

2. The paper fails to talk about how complexity builds up in IT systems over time. I guess activities during maintenance/operations is slightly out of scope since the paper mostly talks about projects. But I wish they weren’t.

They represent one way of how complexity comes into being, and one that can’t be ignored. If everyone wants to work on new projects and all of your best, most skilled and talented resources are busy writing new code it’s no wonder a system usually ends up being a big ball of mud after a couple of years. Regardless of what it looked like to begin with.

To me, the paper gives the illusion that reducing complexity is, more or less, a one time effort and not a continuous battle that spans a system’s whole lifecycle. I’m hoping that this is not what Roger means but it would be nice to have this point clarified as much as possible.