Accidental Complexity in Software Development
[This post is a response to this tweet from @RSessions]
Software complexity can be sliced and measured in many different ways. One way that software developers and architects choose to talk about complexity is by dividing it into accidental and essential complexity.
Accidental complexity is unnecessary complexity that depends on the method used to solve the problem. If your software solution uses EJB 2.1 it has higher accidental complexity than if you would have used EJB 3. If your website uses Java Servlets, JSP, Struts and JDBC it has higher accidental complexity than if you would have used Ruby on Rails.
The software industry is always working to reduce accidental complexity but the last few years have seen an increased effort and adoption of tools and techniques with as little accidental complexity as possible. The Spring framework was a game changer for Java developers in the same way the relational database was for DBAs in the 1970s and these changes are for the better.
In 2000 when Java became an alternative to C++ it removed a lot of complexity. In 2005 the Spring framework removed a lot of complexity from EJB. In 2007 Sun decided to help the adoption of dynamic languages (such as Python and Ruby) on the JVM which provides alternatives that remove a lot of complexity. This reduction of complexity on the lowest level of IT projects should have an impact on the overall IT projects failure numbers.
However, I believe that it will take some time yet. The adoption of new technology for an organization takes time. There’s plenty of existing code that’s not easily migrated and there’s issues such as knowledge, competence and the general resistance to change to consider as well. My personal estimate is that the industry at large is somewhere around 2005-2006 on this curve.
Accidental complexity is not limited to describe technical implementation details such as programming languages. In Lean terms, I believe accidental complexity is called waste. Agile project methods such as Scrum can be described as “some other project method” (you choose) with accidental complexity removed. Removing accidental complexity is good, but it’s not enough in it self. Methods, such as Simple Iterative Partitions (SIP), need to be applied and incorporated in projects as well.
In 2002 Technology Review published an article called Why Software is so Bad. In it they refer to a multi-year study from Carnegie Mellon University where it is found that the average programmer makes 100-150 errors per 1000 lines of code. It’s not explicitly stated in that article how many of these errors are based on misunderstanding business requirements and how many are based on the accidental complexity of the programming language, but I think it’s safe to say that a fair amount are errors related to memory handling (this is also hinted at elsewhere in that article).
And while those numbers are probably still true for C/C++ developers today, I don’t believe that they describe the situation for Java/JEE programmers and certainly not for Python and Ruby programmers.
Unfortunately, I cannot find any research on this subject so I remain hopeful, but unsure.
