Why does it happen?

Technical debt occurs in large, complex systems that exceed the boundaries of human understanding. These codebases are simply too large and complicated for anyone to understand in their entirety.

Technical debt is also a function of the development process in which large groups of people work simultaneously on different parts of the code. Both CMMI and Agile can pose problems that lead to technical debt. Designers of today’s large systems struggle with the fact that these systems are both complicatedand complex.  Hundreds (or even thousands) of engineers now contribute to the design of single artifacts and systems are often interconnected in ways that produce unexpected interactions. The whole often does not behave in a way that one would expect from its parts. Time must be spent tracing through all ways in which a given change can impact the system as a whole, leading to frustration with “hairballs,” “spaghetti,” and “complexity catastrophes.” System architects often take great pains to keep complexity under control to avoid a range of risks and costs. Complex systems are more expensive to design, harder to maintain, and can be more prone to life-threatening failure. Technical debt corresponds to the cost of making and verifying future changes in a complex technical system. Even if the desired changes are not known in advance, it is often possible to predict change costs for the system as a whole and key parts of it. There are three main reasons that technical debt is incurred:

1. Tradeoffs The first reason is that a developer may make a tradeoff between the present and the future. “I will do something now that I know will be costly to fix later.” If a product never needs to change, then the technologists who created the product have no future obligations hence no debts. But software-intensive systems are different from many physical products in that they last a very long time and are capable of being modified by changing the code. Because software systems can change, both developers and users expect that they will change. But at the same time, much of the legacy code will continue exist for years and even decades. Changes are desirable, because users will come to see and demand new features. System designers, in turn, must deal with defects, errors, and ‘bugs.’ The software system evolves, but there is almost never a chance to start over from scratch with a blank slate. One of the contributing factors to technical debt is that the developer weighs short-term gain against long-term tradeoffs. Every developer makes tradeoffs based on his or her local knowledge, personal benefit, incentives and beliefs.

2. Agency Cost The second reason technical debt is incurred is that the person incurring the debt isn’t the person paying for it. In this case, a developer does something that he/she will not suffer from. Technical debt is related to the cost of making changes. The debt burden is borne by future developers (and users) who must work with the code as it evolves.

3. It’s Invisible The third reason is that there is no way to track technical debt. There is no double entry bookkeeping in software development, a measure that tracks errors in accounting. Architectural debt happens when developers don’t know they’re creating linkages that are incurring debt. As the system grows, the inability to measure and manage things that are meaningful makes the situation worse.

4. Architectural Patterns Many modern systems are so large that no one truly understands how they work.  It is well known in the engineering community that architectural patterns — hierarchies, modules, and abstraction layers – should be used in design because they play an important role in controlling complexity.  These patterns make a system easier to evolve and keep its separate portions within the bounds of human understanding so that distributed teams can operate independently while jointly fashioning a coherent whole. Design is not easy or straightforward.  Weighing the costs and benefits of alternative architectural choices is difficult.  Designers must choose between multiple competing ways to decompose a system into hierarchical structures and competing criteria for determining which functionality should be clustered in each module. They must determine how big each module should be and how interfaces between them should be structured. In addition, hierarchy and modularity are not free – they impose their own costs, may impact performance, and can limit the scope of future decision-making. A designer must trade performance requirements against complexity controlling features across the system being designed.  These choices will have a profound impact on how complicated and complex different portions of a system will be.  As a result, a system may have regions bearing widely varying costs and risks.