Preface to the 2010 Edition
The following article was presented in 1999 to introduce the value of measurable software process improvement to commercial and publicly traded software development companies. These ideas are vital today, so I am publishing it again.
Executive Summary
The cost to "implement measurable software process improvement" is 11 percent of the total software development cost. Some argue that is too expensive or the following results are not accurate. The question is, how much do you pay for current improvements and do your results show a return on investment? In addition, are there any SEL ideas that are worth the investment and testing in your company?
The SEL reports the following results over the long term.
- 85-percent reduction in software errors
- 35-percent reduction in cost to develop a new line of code
- 60-percent reduction in development cost
- 340-percent increase in overall improvement reuse
- 25-percent reduction in cycle time
The goal of this paper is to prompt further investigation and study of the SEL resources and processes and their potential benefit to your organization. It provides a brief overview of the experience of the NASA's Software Engineering Laboratory (NASA SEL) and a basic understanding of its organization, process improvement approach, and its experimentation and operations -- including their "Keys to Software Development Success" and "Do's and Don'ts for Project Success" plus a list of some of their experiments and reports.
Foreword
The Software Engineering Laboratory (SEL) is an organization sponsored by the National Aeronautics and Space Administration/Goddard Space Flight Center (NASA/GSFC) for the purpose of studying the effectiveness of software engineering technologies and improving the overall software process and products that are developed within the GSFC Flight Dynamics Division (FDD). The SEL was created in 1976 and has three primary organizational members:
NASA Goddard Space Flight Center
* the user and manager of all of the relevant software systems
University of Maryland, Department of Computer Science
or Experimental Software Engineering Group (ESEG)
* with the focus of advanced concepts in software process and experimentation
Computer Sciences Corporation, Software Engineering Operation
* the major contractor responsible for building and maintaining the software used to support the NASA missions (note: the CSC recently won the milti-billion dollar IRS upgrade contract)
SEL Background
At the time the SEL was established, significant advances were being made in software engineering. However, very little empirical evidence existed for selecting and applying promising techniques and processes. In fact, little guidance was available regarding which approaches were of any value in software production. Additionally, there was very limited data available to qualify or quantify the existing software process and associated products, or to aid in understanding the impact of specific methods. Thus, the SEL staff developed a means by which the software process and products could be understood, measured, qualified, and objectively improved.
SEL Introduction
Since its inception, the SEL has conducted experiments on approximately 130 production software projects at NASA/GSFC, in which numerous software process changes have been applied, measured, and analyzed. Software engineering projects have approximately 250 personnel who, in roughly 1-2 years, develop and maintain complex products and systems with about 10K to 1.5 million software lines of code. The continual experimentation with software process and products has yielded an extensive set of empirical studies that has guided the evolution of standards, management practices, technologies, and training within the organization. Because of these studies, appropriate processes have been adopted and tailored within the environment, which has guided the SEL to improve the software products, significantly. Additionally, the SEL has produced over 200 reports that describe experiences from the experimentation process and its overall software process improvement approach.
Through experimentation and sustained study of software process and its resultant product, the SEL has been able to identify refinements to its software process and to improve product characteristics. This effort has been driven throughout by the goals of achieving significant overall improvement in three product measures:
- Reduction in defect rate of delivered software
- Reduction in cost of software to support similar missions
- Reduction in average cycle time to produce mission support software
Since a major GSFC reorganization in January 1998, GSFC sponsorship for the SEL has transferred from the FDD to the new Information Systems Center (ISC). The wider scope of the ISC's software responsibilities should lead to the SEL collecting data across more software domains and processes.
SEL Goals
- Understand the software development process in the GSFC environment
- Measure the effect of various methodologies and tools on the process and product during development
- Identify, publish, standardize, and apply successful development practices
SEL Organizational Structure
SEL Process Improvement Strategy - reuse of experience/tasks
Over the years, the SEL has selected from an array of evolving software technologies (e.g., new languages, tools, and processes) and applied the chosen ones in the FDD production environment during development, testing, and maintenance. It simultaneously measured the impact of these technologies on the products being created. In this way, the most beneficial approaches were identified through these empirical studies and then standardized, once improvements were identified.
SEL Bottom-Up Process Improvement Concept
The most significant changes to the SEL process improvement concept are being driven by experience at attempts to infuse process change and improvements within a "real" production organization. The SEL improvement concept can be described as a “bottom-up” software improvement approach, where the focus is process, defined and improved based on corporate knowledge that is extracted from the experiences of people at the lowest level or bottom of the project organization. This reuse of experience is the driving element for change and improvement in the software process. It is used with the more traditional top-down approach, or Capability Maturity Model, which focus is product changes. Each concept plays an important role in the goal of improving software development business.
SEL Reuse of Experience
Historically, a significant shortcoming in software development organizations has been their failure to capitalize on experience gained from similar completed projects. Most of the insight gained has been passively obtained instead of being aggressively pursued. One example, the SEL conducts post phase analysis compared with the more traditional post project analysis. Software developers and managers generally do not have the time or resources to focus on building corporate knowledge or planning organizational process improvements. They have projects to run and software to deliver. Thus, reuse of experience and collective learning must become a corporate priority and asset. Reuse of experience and collective learning must be supported by an organizational infrastructure dedicated to developing, updating, and supplying regular synthesized experiences and competencies. This organizational infrastructure emphasizes achieving continuous sustained improvement over identifying possible technology breakthroughs.
SEL Experimentation and Analysis
For each project, detailed measurements were provided toward the end goal of analyzing the impact that any change to software process had on the resultant software product. If a new process experiment did not prove beneficial to the product, it is not incorporated into the SEL standards.
SEL Analyzing Data
"One of the most important facts that the SEL has learned from its experience with analysis of software data is that the actual measurement data represent only one small element of experimental software engineering. Too often, data can be misinterpreted, used out of context, or weighted too heavily even when the quality of the information may be suspect." One of several key rules: "Software measures will be flawed, inconsistent, and incomplete; the analysis must take this into account. Do not place unfounded confidence in raw measurement data." Note: SEL data can be acquired for independent studies, research, and analyses.
SEL Data Collection Categories
- Project information high-level information about projects and personnel.
- Product information estimated and actual counts of components, applications, and source lines of code (SLOC).
- Process information effort estimates, test estimates, actual effort data, inspection data, change data, and testing data.
- Schedule information actual and estimated phase start and end dates, build start and end dates, and milestone dates.
- Metadata data about data or the database, form status, SEL terminology and database table descriptions.
- Application/Subsystem Information
- Component Name Change
- Component Origination
- Change Request
- Development Completion
- Development Estimates
- Development Status
- Inspection Data Collection
- Independent Testing Status
- Project Information
- Project Messages
- Project Startup
- Reuse Information
- Software Release Estimates
- Weekly Effort & Help
- Weekly Services
- Weekly Independent Testing Effort
- Instructions for the COTS & Tool Information
The SEL conducts three general types of analysis, all of which are active continually in the environment:
- Pilot Studies of specific techniques and technologies on a project(s)
- Studies of completed projects for development and refinement of local process and product models
- Trend analysis of completed projects to track impact of specific process changes on whole environment
- Each improvement cycle begins with setting improvement goals based on the current business needs and strategic direction of the organization.
- The next step is to identify software engineering technologies (processes, methods, and/or tools) that are likely to affect the leverage area.
- The third and longest step of the improvement cycle is to conduct experiments to understand the value and applicability of the new technology in the local organization.
- The final step in an improvement cycle is to deploy the beneficial process/technology throughout the organization.
"During the understanding phase of the SEL paradigm, the goal is to produce a baseline of development practices and products attributes against which change can be measured as process modifications are applied. . . . The most critical element of the SEL’s process improvement program is the understanding step -- where the only goal is to gain insight into the local software business."
The two most critical core measures that the SEL has found are:
- error data including numbers and five types of errors (Computational, Initialization, Logic/control, Interface, and Calculation); and
- effort data including total weekly hours by life-cycle phase and by activity.
Assessing / Experiments
Of all the models and relations that the SEL has developed during the understanding phase, the most useful for project planning and management and for observing change have been:
- Effort distribution (cost characteristics).
- Error characteristics (numbers, types, origins).
- Change and growth rates (of the source code during development).
The primary products of the packaging step are standards, tools, and training that give practical guidance on how to apply the new techniques in the context of the local process. See the following sections titled "Recent SEL Technology Case Studies" and "Short SEL Documentation List" for a list of public documents.
Although the technology reports are valuable, the full value of the process analysis is felt when modifications and enhancements are made to the instruments that actually guide the way the development/maintenance organization carries out its business. These include standards, tools, and training classes.
Probably the most important lesson that has been derived from the studies is that specific techniques can help the overall goals of process improvement when appropriately selected and tailored. However, the most effective element of the improvement paradigm is the continuous analysis of the software business and the continuous expansion of the understanding of the software process and product.
SEL Tools
An important packaging concept is the infusion of technology in the form of support tools for use by project personnel. The SEL developed a project management tool called the Software Management Environment (SME). SME provides project managers access to the SEL database of previous project data and access to the baseline set of SEL process models. The SEL also provides tools to automate parts of the software measurement process, working forms and a detailed training plan.
SEL Standards
Although the initial plan was to begin experimenting with various techniques, the SEL soon learned that without a firm, well-understood baseline of process and product characteristics, valid experimentation was impossible. However, most important is the process by which the SEL gathers the information and ensures that the standards reflect the actual process -- process that experiments, analyses and more importantly experience, proved measurably beneficial to the product.
The SEL development organization uses a standard set of policies that is updated on a periodic basis to reflect new experimentation results. It comprises a set of guidebooks that describe the SEL’s baseline methodology and several guidebooks that define major tailoring instances of the baseline process.
The SEL has evolved its approach to standards over the years. The SEL has found that the baseline process is best presented at a medium level of detail (between principles and detailed procedures); it is more important to communicate the rationale and guidance for applying the methods on projects rather than providing detailed procedures for them. This allows the detailed procedures to evolve as improvements are made and specific project needs change, without requiring waivers or continual updates to the formal standards. The SEL typically updates its baseline standards every 5 years.
SEL Baseline Standards
- Manager’s Handbook for Software Development
- Recommended Approach to Software Development
- Operational Software Maintenance Procedures
- Cost and Schedule Estimation Study Report
- Data Collection Procedures for the SEL Database
- Ada Developer’s Supplement to the Recommended Approach
- C Style Guide
- Cleanroom Process Model
The total investment that the SEL has made in the improvement of process and products has been approximately 11 percent of the total software development cost.
SEL Results on Process
Although specific techniques or methodologies may have measurable impact on a class of projects, significant improvement to the software development process occurs where the sustained, continuous incorporation of detailed techniques into higher-level organizational process effects an overall change in the environment. The most significant process attributes that distinguish the current SEL production environment from the environment of a decade ago include:
1. Process change has been infused as a standard business practice
All standards and training material now contain elements of the continuous improvement approach to experimentation that has been promoted by the SEL.
2. Measurement is now our way of doing business
The measurement activity is as common a part of the software standards as documentation.
3. Change is now driven by product and process, not merely process alone
A set of product goals is always defined before process change is infused, and because of this, measures of product are as important as (and probably more important than) those of process.
4. Change is now bottom-up
Direct input from developers as well as measures extracted from development activities are essential factors in change.
5. “People-oriented” technologies are emphasized rather than automation
Automation techniques have sometimes provided improvement, but people-driven approaches, leverage the thinking ability of the developers, have had farther-reaching effects.
6. Focus on a single goal for each process/technology change
Provide a clear definition of the expected change and measure its effect, rather than overload a single project with multiple changes hoping that at least one will work.
7. Deploy a subset of the changes as soon as the benefit is shown
Often it is clear that certain sub-processes or techniques, deployed early, are very beneficial even though the entire new process/technology may not yet be proven.
8. Allocate sufficient experimental time for iterative application/learning of new concepts
The SEL’s experience shows the benefit of taking a little more time to develop a more usable product rather than deploying the more abstract concepts first.
9. Set improvement time expectations appropriately
The more familiar the organization is with the process being changed, the faster it can be tuned and deployed, and its impact realized.
SEL Results on Products
The SEL process produces systems with about 50 bugs compared to 850 bugs without the SEL process, with approximately the same software lines of code, resources, cost, and schedule. The metrics collected have been critical for evaluating the strengths and weaknesses of new modifications to the software development process that are frequently incorporated within special experiments or standard projects. Such evaluations of new techniques have allowed the SEL to tailor the techniques to the best advantage of the environment. The SEL was able to document these achievements through its rigorous Data Collection and Reporting System (DC&RS).
The SEL reports the following results over the long term. The baseline measurements to evaluate improvement include total cost, total duration, development error rate, reuse percentage, cost per new line of code, and cost per delivered line of code.
- 85-percent reduction in software errors
- 35-percent reduction in cost to develop a new line of code
- 60-percent reduction in development cost
- 340-percent increase in overall improvement reuse
- 25-percent reduction in cycle time
- Also a measure of predictability, a subjective measure difficult to quantify
Because of its history of achievement, the Software Engineering Institute (SEI) and the Institute of Electrical and Electronic Engineers (IEEE) in 1994 recognized the NASA SEL as the inaugural recipient of the IEEE Computer Society Award for Software Process Achievement.
SEL Technology Case Studies
- Requirement Metrics for Risk Identification
- Baselining the New GSFC Information Systems Center
- The SEL Adapts to Meet Changing Times
- Packaged-Based Development Process in the FDD
- The Web Measurement Environment: A Tool for Combining and Modeling Distributed Data
- The Improvement Cycle: Analyzing Our Experience
- Evolving the Reuse Process
- Studies on Reading Techniques
- Risk Knowledge Capture in the Riskit Method
- Package-Based System Development
- Development Process for Generic Reusable Systems
- Perspective-Based Requirements Reading
- Recent Research Results for Software Maintenance
- Improving the Software Testing Process in the SEL
Publications are available to the public at no charge. The entire collection of SEL literature is described in the Annotated Bibliography of SEL Literature. See also the “Standard Bibliography of SEL Literature”, which lists all current SEL titles.
- Recommended Approach to Software Development
- Ada Developers' Supplement to the Recommended Approach
- Manager's Handbook for Software Development
- Cost and Schedule Estimation Study Report
- C Style Guide
- An Overview of the Software Engineering Laboratory
- Software Measurement Guidebook
- Impact of Ada and Object-Oriented Design in the Flight Dynamics Division at NASA/GSFC
- Software Process Improvement Guide
- SEL COTS Study, Phase 1 - Initial Characterization Study Report
The SEL has packaged its process improvement experience (methods) in the form of guidebooks, such as the Software Measurement Guidebook, that are designed to be used outside, as well as inside, the SEL. Many of these results are published in software engineering journals and presented at major international conferences.
SEL Workshops
To facilitate the sharing of software engineering experiences among practitioners, the SEL sponsors an annual Software Engineering Workshop, with paper sessions, panels, and tutorials -- which has been rated as the best conference for software practitioners.
Conclusion
The SEL’s success with incremental process change, as opposed to leading edge technology adoption, has led them to select the experimental approach to changing process gradually. The improvements in product characteristics and development process standards illustrate the impact of the SEL improvement program and it is value to other organizations. Today, software developers in this organization are building better software more efficiently using many techniques and methods considered experimental only a few years ago because of the SEL.
For a comprehensive overview see "The Improvement Cycle: Analyzing Our Experience" and An Overview of the Software Engineering Laboratory and other reports and forms go to The NASA Software Engineering Laboratory Database or The Goddard Library.
References, Updates, Forms, and Studies
Old links were updated and connected to archives where the original source was no longer available. There was simply not enough time to post all the original reference documents to my Scribd archive. So if you are interested in a referenced document that has no link, let me know. If I have a copy, I will post it.
Some reference links to forms and studies were removed because the original source was not maintained and no archive was available at the time of this publication. If you know of a link to an original reference or archive, please let me know and I will add a link. In addition, where links or documents are no long available, I can add copies to my Scribd archive, if you would like to share copies.
Additional References
Software Process Improvement In the NASA Software Engineering Laboratory
Copyright © 2010 Michael Rybin All Rights Reserved.
Originally published and Copyright © 1999 Michael Rybin All Rights Reserved.
Many direct and indirect references to NASA SEL information are included. Copyright information, documents, graphics, brand names, and trademarks from NASA and SEL affiliates are open-source, public domain, or the property of the respective owners. All other trademarks and copyrights are the property of their respective owners.
