A Methodology for Operationalizing Enterprise Architecture and Evaluating Enterprise IT Flexibility

A Methodology for Operationalizing Enterprise Architecture and Evaluating Enterprise IT Flexibility by Alan MacCormack, Robert Lagerstrom, and Carliss Y. Baldwin

We propose a network-based methodology for operationalizing enterprise architecture. Our methodology is based upon using a “Design Structure Matrix” (DSM) to capture the coupling between different components in a firm’s architecture, including business and technology-related aspects. We apply our methodology to data gathered in a large pharmaceutical firm. We show that this methodology helps to identify layers in the firm’s architecture associated with different technologies (e.g., applications, servers and databases). We also show that it reveals the main “flow of control” within the architecture, as denoted by the classification of components into Core, Peripheral, Shared and Control elements. We analyze the cost of change for a subset of software applications within this architecture. We find that the cost of change is associated with the degree to which applications are highly coupled. We show the best measure of coupling that predicts the cost of change is one that captures all the direct and indirect connections between components. We believe our work constitutes an important step in making the concept of enterprise architecture more operational, improving a firm’s ability to analyze its architecture, understand its performance implications, and adapt and improve it in the future.

Read Paper

Contact the Authors:

Managing Technical Debt: Insights from Recent Empirical Evidence

Managing Technical Debt: insights from Recent Empirical Evidence by Narayan Ramasubbu, Chris F. Kemerer, and C. Jason Woodard

Technical debt refers to maintenance obligations that software teams accumulate as a result of their actions. Empirical research has led researchers to suggest three dimensions along which software development teams should map their technical-debt metrics: customer satisfaction needs, reliability needs, and the probability of technology disruption.

Read Paper

Contact the Authors:

Design Capital and Design Moves: The Logic of Digital Business Strategy

Design Capital and Design Moves: The Logic of Digital Business Strategy by C. Jason Woodward, Narayan Ramasubbu, F. Ted Tschang, and V. Sambamurthy

As information technology becomes integral to the products and services in a growing range of industries, there has been a corresponding surge of interest in understanding how firms can effectively formulate and execute digital business strategies. This fusion of IT within the business environment gives rise to a strategic tension between investing in digital artifacts for long-term value creation and exploiting them for short-term value ap- propriation. Further, relentless innovation and competitive pressures dictate that firms continually adapt these artifacts to changing market and technological conditions, but sustained profitability requires scalable archi- tectures that can serve a large customer base and stable interfaces that support integration across a diverse ecosystem of complementary offerings. The study of digital business strategy needs new concepts and methods to examine how these forces are managed in pursuit of competitive advantage. We conceptualize the logic of digital business strategy in terms of two constructs: design capital (i.e., the cumulative stock of designs owned or controlled by a firm) and design moves (i.e., the discrete strategic actions that enlarge, reduce, or modify a firm’s stock of designs). We also identify two salient dimensions of design capital, namely, option value and technical debt. Using embedded case studies of four firms, we develop a rich conceptual model and testable propositions to lay out a design-based logic of digital business strategy. This logic highlights the interplay be- tween design moves and design capital in the context of digital business strategy and contributes to a growing body of insights that link the design of digital artifacts to competitive strategy and firm-level performance.

Read Paper

Contact the Authors:

Technical Debt and the Reliability of Enterprise Software Systems: A Competing Risks Analysis

Technical Debt and the Reliability of Enterprise Software Systems: A Competing Risks Analysis by Narayan Ramasubbu and Chris F. Kemerer

Enterprise software systems are required to be highly reliable as they are central to the business operations of most firms. However, configuring and maintaining these systems can be highly complex, making it challenging to achieve high reliability. Resource constrained software teams facing business pressures can be tempted to take design shortcuts in order to deliver business functionality more quickly. These design shortcuts and other maintenance activities contribute to the accumulation of technical debt, that is, a buildup of software maintenance obligations that need to be addressed in the future. We model and empirically analyze the impact of technical debt on system reliability by utilizing a longitudinal dataset spanning the 10 year lifecycle of a commercial enterprise system deployed at 48 different client firms. We use a competing risks analysis approach to discern the interdependency between client and vendor maintenance activities. This allows us to assess the effect of both problematic client modifications (client errors) and software errors present in the vendor-supplied platform (vendor errors) on system failures.

We also examine the relative effects of modular and architectural maintenance activities undertaken by clients in order to analyze the dynamics of technical debt reduction. The results of our analysis first establish that technical debt decreases the reliability of enterprise systems. Second, modular maintenance targeted to reduce technical debt was about 53% more effective than architectural maintenance in reducing the probability of a system failure due to client errors, but had the side-effect of increasing the chance of a system failure due to vendor errors by about 83% more than did architectural maintenance activities. Using our empirical results we illustrate how firms could evaluate their business risk exposure due to technical debt accumulation in their enterprise systems, and assess the estimated net effects, both positive and negative, of a range of software maintenance practices. Finally, we discuss implications for research in measuring and managing technical debt in enterprise systems.

Read Paper – Forthcoming in Management Science 2015

Contact the Authors:

  • Narayan Ramasubbu (narayanr@pitt.edu)
  • Chris F. Kemerer (ckemerer@katz.pitt.edu)

MIT System Design and Management Presentation

How to Analyze and Visualize a Large, Interconnected Software System: A Study of Fedora 20 with Lessons for All by Dan Sturtevant, PhD., and Dave Allan

Presented on January 26th, 2015 – YouTube video of presentation available here, presentation slides here

When working inside a system of enormous scale, people often understand only the part with which they are involved. As long as the whole system is sufficiently modular, people tend to believe they can construct reasonably reliable mental models of their component and how it interfaces with others. However, this is not always safe; research shows that hidden structures can interconnect components of a complex system at higher levels, causing organizational problems that are difficult to see, understand, and address.

Fedora 20 is composed of more than 2,500 interconnected software packages developed and managed by globally distributed teams. Estimates have placed the number of software developers who have contributed at over 100,000. In this webinar, Daniel Sturtevant and David Allan will present research that addresses the architectural complexity of the Fedora Linux operating system and software collection. They discuss:

1) How to visualize the system at multiple levels (including the view from 60,000 feet) and gain meaningful insights about its hidden structure,

2) How to benchmark across the system to better understand its composition and variations in complexity and quality, and

3) How this approach might be applied to other software systems.

Contact the Authors:

  • Dan Sturtevant, CEO of Silverthread Inc. and Researcher at Harvard Business School
  • Dave Allan, Director of Software Engineering Silverthread Inc.

Technical Debt in Large Systems: Understanding the Cost of Software Complexity

Technical Debt in Large Systems: Understanding the Cost of Software Complexity by Daniel J. Sturtevant

Many modern systems are so large that no one truly understands how they work. Because these systems exceed the bounds of human understanding, different design teams must work on separate chunks, glue their work together, and hope that the whole thing behaves as expected. In this process, high-level architectural design patterns (such as hierarchies, modules, and abstraction layers) play an important role. By keeping complexity under control, they give systems the ability to scale, make them more able to evolve, and reduce the likelihood of unexpected side effects or integration problems.

View Presentation

Trade-offs Between Productivity and Quality in Selecting Software Development Practices

Trade-offs between Productivity and Quality in Selecting Software Development Practices by Alan MacCormack, Chris F. Kemerer, Michael A. Cusumano, and Bill Crandall 

Choosing appropriate practices for a project can be hard, given the various dimensions of performance each claims to optimize. Using data from 29 projects, this study examines the impact of eight development practices on both productivity and quality. Results suggest managers should choose a coherent system of practices on the basis of the specific objectives the software must meet.

Read Full Article

Faculty Member Outlines Values of Software System Architecture

Faculty Member Outlines Value of Software System Architecture by Alan MacCormack

Over the last 10 to 15 years, even the most traditional of industries have come to rely on software, for everything from inventory control to vehicle navigation. The average automobile today has more software than the first Apollo moon rocket. Your garden variety microwave may even have an algorithm for cooking popcorn to fit your specific tastes. Despite this dramatic increase in the pervasiveness and importance of software, however, many companies lack a fundamental understanding of the architecture underlying their code. This problem costs firms millions of dollars per year.

Read Full Article

Technical Debt: From Metaphor to Theory and Practice

Technical Debt: From Metaphor to Theory and Practice by Philippe Kruchten, Robert L. Nord, and  Ipek Ozkaya 

The metaphor of technical debt in software development was introduced two decades ago by Ward Cunningham to explain to nontechnical product stakeholders the need for what we call now “refactoring.” From the original description—”not quite right code which we postpone making it right” —various people have used the metaphor of technical “debt” to describe many other kinds of debts or ills of software development, encompassing broadly anything that stands in the way of deploying, selling, or evolving a software system or anything that adds to the friction from which software development endeavors suffer: test debt, people debt, architectural debt, requirement debt, documentation debt, or just an amorphous, all-encompassing software debt. Consequently, the concept of technical debt in software development has become somewhat diluted lately. Is a new requirement, function, or feature not yet implemented “requirement debt”? Do we call postponing the development of a new function “planning debt”? The metaphor is losing some of its strength.

To make some progress, we need to go beyond debt as a “rhetorical concept.” We need a better definition of what constitutes technical debt and some perspective or viewpoints that let us reason across a wide range of technical debt. In short, we need a theoretical foundation.

Read Full Article

More Than the Sum of Its Parts: The Impact of Modularity on the Computer Industry

More Than the Sum of Its Parts: The Impact of Modularity on the Computer Industry

The “power of modularity,” write HBS Dean Kim Clark and Professor Carliss Baldwin in their new book, rescued the computer industry from a problem of nightmarish proportions and made possible remarkable levels of innovation and growth in a relatively short period of time.

Read Full Article

Technical Debt in Firefox and Chromium

Technical debt in Firefox and Chromium by Ali Almossawi 

In his previous article How maintainable is the Firefox codebase, the author discussed how an important quality attribute, namely, maintainability, could be quantified. It looked at how Firefox was doing with respect to those measures and suggested some preliminary findings. The aim of this second, much shorter, article is to provide better context by comparing Firefox to another popular and complex software system.

Read Full Article

How Maintainable is the Firefox Codebase?

How maintainable is the Firefox codebase? by Ali Almossawi

This work explores a particular facet of quality in Firefox, namely, maintainability. By appealing to the explanatory powers of five practical measures of architectural complexity, and with the aid of static analysis and network manipulation tools, we arrive at some preliminary findings. We find that 21% of files in Firefox are highly interconnected, a value that went up significantly following version 3.0. We find that making any change to a randomly selected file can, on average, directly impact 10 files and indirectly impact over 2,300 files. We see that files’ internal complexity is on average going down. Their external complexity as measured by direct dependencies is also going down, whereas their external complexity as measured by propagation cost has remained steady. With respect to process, we find that the switch to the rapid release cycle from version 5.0 onwards had a positive impact on quality. Though the analysis reveals no imminently alarming red flags when it comes to maintainability, it does shine light on a number of issues that would be worth investigating. An upcoming article will take a closer look at how Firefox’s architectural measures of complexity compare to those of other open-source projects.

Read Full Article

Evolution of the Firefox Codebase

Evolution of the Firefox Codebase by Ali Almossawi 

Evolution of the Firefox Codebase presents a set of metrics for all releases of Firefox that are indicative of quality and allows one to inspect them through one of several views. By looking at changes in these metrics, one can see the evolution of the Firefox codebase over time. This work is also be useful as a retrospective, investigative tool to help infer when, say, architectural issues may be the cause for unfavorable user sentiment following a release. Metrics such as lines-of-code (LOC) and cyclomatic complexity are widely used in industry, whereas others like propagation cost are based on some of the more recent research to come out of academia.

Read Full Article

SDM Best Thesis Analyzes Costs of Software System Complexity

SDM Best Thesis analyzes costs of software system complexity by Andrei Akaikine

Maintaining a complex system at a “good enough” level  of functionality comes at a cost. In systems where stability is
paramount, personnel dealing with small emergencies traditionally rely on “Band-Aid” fixes that increase complexity and heighten the risk of destabilizing the whole system with each new modification, however minor.

Andrei Akaikine’s thesis focuses on the following three questions:
• Is there a relationship between engineering efforts required for a basic
maintenance task and the overall software system complexity?
• Are any components more susceptible than others to change during
system maintenance tasks?
• Can the redesign of a system be economically justified?

Read Full Article

Design Rule Spaces: A New Form of Architecture Insight

Design Rule Spaces: A New Form of Architecture Insight by Lu Xiao, Yuanfang Cai, and Rick Kazman

In this paper, we investigate software architecture as a set of overlapping design rule spaces, formed by one or more structural or evolutionary relationships and clustered using our design rule hierarchy algorithm. Considering evolutionary coupling as a special type of relationship, we investigated (1) whether design rule spaces can reveal structural relations among error-prone files; (2) whether design rule spaces can reveal structural problems contributing to error-proneness. We studied three large-scale open source projects and found that error-prone files can be captured by just a few design rule sub-spaces. Supported by our tool, Titan, we are able to flexibly visualize design rule spaces formed by different types of relationships, including evolutionary dependencies. This way, we are not only able to visualize which error-prone files belong to which design rule spaces, but also to visualize the structural problems that give insight into why these files are error prone. Design rule spaces provide valuable direction on which parts of the architecture are problematic, and on why, when, and how to refactor.

Read Paper

Understanding the Effects of Product Architecture on Technical Communication in Product Development Organizations

Understanding the effects of product architecture on technical communication in product development organizations by Manuel E. Sosa, Steven D. Eppinger, and Craig M. Rowles

This paper examines the impact of architectural decisions on the level of defects in a product. We view products as collections of components linked together to work as an integrated whole. Previous work has established modularity (how decoupled a component is from other product components) as a critical determinant of defects, and we confirm its importance. Yet our study also provides empirical evidence for a relation between product quality and cyclicality (the extent to which a component depends on itself via other product components). We find cyclicality to be a determinant of quality that is distinct from, and no less important than, modularity. Extending this main result, we show how the cyclicality–quality relation is affected by the centrality of a component in a cycle and the distribution of a cycle across product modules. These findings, which are based on analysis of open source software development projects, have implications for the study and design of complex systems.

Read Paper

Linking Cyclicality and Product Quality

Linking Cyclicality and Product Quality by Manuel Sosa, Jürgen Mihm, and Tyson Browning

Effective communication in product development organizations is widely recognized to be a key element of product development performance. Furthermore, management of product architecture knowledge by the development organization provides important competitive advantage for established firms facing architectural innovation. This research studies how the combination of product architecture and organizational structure determines technical communication in development teams. By documenting and analyzing both the design interfaces between the components that comprise a product and the technical interactions between the teams that design each of these components, we learn how the architecture of the product and the layout of the organization drive development team interactions. Several hypotheses are formulated to explain the unexpected cases when: 1) known design interfaces are not matched by team interactions, and 2) observed team interactions are not predicted by design interfaces. We test the hypothesized effects due to organizational and system boundaries, and design interface strength. Hypotheses are tested using both categorical data analysis and log-linear network analysis. The research is conducted using data collected describing a large commercial aircraft engine development process.

Read Paper

Factors that Influence Technical Communication in Distributed Product Development: An Empirical Study in the Telecommunications Industry

Factors that influence technical communication in distributed product development : an empirical study in the telecommunications industry by Manuel E. Sosa, Steven D. Eppinger, Michael Pich, David G. McKendrick, and Suzanne K. Stout

Understanding the communication process in product development organizations has been recognized as a key element to improve product development performance. It is particularly interesting to study information exchanges in geographically distributed product development teams because of the highly interdependent nature of design organizations. AAdditionally, the use of electronic-based communication media has changed how development teams communicate. By studying the way product development teams use various communication media (face-to-face, telephone, media), we assess how the process of exchanging technical information is influenced by factors such as geographic dispersion, organizational bonds, and degree of team interdependence. We develop a theoretical framework that allows us to formulate several hypotheses about how these factors influence both communication frequency and media choice. We use empirical evidence from the telecommunications industry to test our hypotheses. We confirm previous results about the obstructive influence of distance on technical communication. However, we found that such negative effects may be mitigated by other factors such as the recognizing of highly interdependent team members, the existence of strong organizational bonds, and the use of electronic communication media.

Read Paper

Degree Distribution and Quality in Complex Engineered Systems

Degree Distribution and Quality in Complex Engineered Systems by Manuel Sosa, Jürgen Mihm and Tyson Browning

Complex engineered systems tend to have architectures in which a small subset of components exhibits a disproportional number of linkages. Such components are known as hubs. This paper examines the degree distribution of systems to identify the presence of hubs and quantify the fraction of hub components. We examine how the presence and fraction of hubs relate to a system’s quality. We provide empirical evidence that the presence of hubs in a system’s architecture is associated with a low number of defects. Furthermore, we show that complex engineered systems may have an optimal fraction of hub components with respect to system quality. Our results suggest that architects and managers aiming to improve the quality of complex system designs must proactively identify and manage the use of hubs. Our paper provides a data-driven approach for identifying appropriate target levels of hub usage.

Read Paper

The Impact of Component Modularity on Design Evolution: Evidence from the Software Industry

The Impact of Component Modularity on Design Evolution: Evidence from the Software Industry by Alan MacCormack, John Rusnak and Carliss Y. Baldwin

Much academic work asserts a relationship between the design of a complex system and the manner in which this system evolves over time. In particular, designs which are modular in nature are argued to be more “evolvable,” in that these designs facilitate making future adaptations, the nature of which do not have to be specified in advance. In essence, modularity creates “option value” with respect to new and improved designs, which is particularly important when a system must meet uncertain future demands.

Despite the conceptual appeal of this research, empirical work exploring the relationship between modularity and evolution has had limited success. Three major challenges persist: first, it is difficult to measure modularity in a robust and repeatable fashion; second, modularity is a property of individual components, not systems as a whole, hence we must examine these dynamics at the microstructure level; and third, evolution is a temporalphenomenon, in that the conditions at time t affect the nature of the design at time t+1, hence exploring this phenomenon requires longitudinal data.

In this paper, we tackle these challenges by analyzing the evolution of a successful commercial software product over its entire lifetime, comprising six major “releases.” In particular, we develop measures of modularity at the component level, and use these to predict patterns of evolution between successive versions of the design. We find that modularity has a strong and unambiguous impact on design evolution. Specifically, we show that i) tightly-coupled components are “harder to kill,” in that they have a greater likelihood of survival in subsequent versions of a design; ii) tightly-coupled components are “harder to maintain,” in that they experience more surprise changes to their dependency relationships that are not associated with new functionality; and iii) tightly-coupled components are “harder to augment,” in that the mix of new components added in each version is significantly more modular than the legacy design.

Read Paper

Evolution Analysis of Large-Scale Software Systems Using Design Structures Matrices and Design Rule Theory

Evolution Analysis of Large-Scale Software Systems Using Design Structure Matrices and Design Rule Theory by Matthew J. LaMantia, Yuanfang Cai, Alan David MacCormack and John Rusnak

Designers often seek modular architectures to better accommodate expected changes and to enable parallel development. However, we lack a formal theory and model of modularity and software evolution, which can be used for description, prediction, and prescription. According to Baldwin and Clark’s  theory, modular architectures add value to system designs by creating options to improvethe system by substituting or experimenting on individual modules. In this paper, we evaluate their theory by looking at the design evolution of two software product platforms through the modeling lens of design structure matrices (DSMs) and design rule theory. Our analysis shows that DSM models and options theory can explain how real-world modularization activities in one case allowed for different rates of evolution in different software modules and in another case conferred distinct strategic advantages on a firm (by permitting substitution of an at-risk software module without substantial change to the rest of the system). The experiment supports our hypothesis that these formal models and theory can account for important aspects of software design evolution in large-scale systems.

Read Paper

Visualizing and Measuring Software Portfolio Architectures: A Flexibility Analysis

Visualizing and Measuring Software Portfolio Architectures: A Flexibility Analysis by Robert Lagerstrom, Carliss Y. Baldwin, Alan MacCormack, and David Dreyfus

In this paper, we test a method for visualizing and measuring software portfolio architectures and use our measures to predict the costs of architectural change. Our data is drawn from a biopharmaceutical company, comprising 407 architectural components with 1,157 dependencies between them. We show that the architecture of this system can be classified as a “core-periphery” system, meaning it contains a single large dominant cluster of interconnected components (the “Core”) representing 32% of the system. We find that the classification of software applications within this architecture, as being either Core or Peripheral, is a significant predictor of the costs of architectural change. Using OLS regression models, we show that this measure has greater predictive power than prior measures of coupling used in the literature.

Read Paper

Visualizing and Measuring Enterprise Architecture: An Exploratory BioPharma Case

Visualizing and Measuring Enterprise Architecture: An Exploratory BioPharma Case by Robert Lagerstrom, Carliss Baldwin, Alan MacCormack and David Dreyfus

We test a method that was designed and used previously to reveal the hidden internal architectural structure of software systems. The focus of this paper is to test if it can also uncover new facts about the components and their relationships in an enterprise architecture, i.e., if the method can reveal the hidden external structure between architectural components. Our test uses data from a biopharmaceutical company. In total, we analyzed 407 components and 1,157 dependencies. Results show that the enterprise structure can be classified as a core-periphery architecture with a propagation cost of 23%, core size of 32%, and architecture flow through of 67%. We also found that business components can be classified as control elements, infrastructure components as shared, and software applications as belonging to the core. These findings suggest that the method could be effective in uncovering the hidden structure of an enterprise architecture.

Read Paper

Designing Systems for Adaptability by Means of Architecture Options

Designing Systems for Adaptability by Means of Architecture Options by Avner Engel, and Tyson R. Browning

The value of a system usually diminishes over its lifetime, but some systems depreciate more slowly than others. Diminished value is due partly to the increasing needs and wants of the system’s stakeholders and partly to its decreasing capabilities relative to emerging alternatives. Thus, systems are replaced or upgraded at substantial cost and disruption. If a system is designed to be changed and upgraded easily, however, this adaptability may increase its lifetime value. How can adaptability be designed into a system so that it will provide increased value over its lifetime? This paper describes the problem and an approach to its mitigation, adopting the concept of real options from the field of economics, extending it to the field of systems architecture, and coining the term architecture options for this next-generation method and the associated tools for design for adaptability. Architecture options provide a quantitative means of optimizing a system architecture to maximize its lifetime value. This paper provides two quantitative models to assess the value of architecture adaptability. First, we define three metrics—component adaptability factors, component option values, and interface cost factors—which are used in a static model to evaluate architecture adaptability during the design of new systems. Second, we enhance a dynamic model to evaluate architecture adaptability over the maintenance and upgrade lifetime of a system, formulating a Design for Dynamic Value (DDV) optimization model. We illustrate both models with quantitative examples and also discuss how to obtain the socio-economic data required for each model.

Read Paper