Test strategies for reducing software dev't costs

The cost of fixing a bug grows exponentially as the distance from coding to fix increases. By reducing this distance, integrating testing tightly into development saves time and money. Yet ask an embedded developer about test-driven development or using test tools during development, and you'll often hear "it's too expensive", "we don't have time for that", or "it takes too long". When you hear any of these answers, make sure to ask if the developer regularly delivers on-time, on-budget, and with low bug-count.

Pulling testing forward into the development process reduces the cost of development to the first release (in addition to reducing life-cycle cost). Integrating testing into development is often a culture change; thus to be successful requires:
 • Recognising patterns of development difficulties easily remedied by integrating testing,
 • Explanation and communication so the entire team internalizes the benefits to everyone, and
 • Use of simple, appropriate, cost-effective tools and techniques.

Why costs increase exponentially with time from coding to fix
When a bug is fixed immediately after coding, normally the fix is implemented by the original developer, using the testing environment and tools used for development of the code containing the bug. This testing may be completely ad-hoc, stuffing variables in a debugger or worse, but at least the code and test are current in the developer's mind. Important reasons bug-fix costs escalate as time elapses include:

The test environment may not be easily available (especially if it was ad-hoc or requires a special hardware setup).
 • The code is no longer current in the developer's mind.
 • The original developer is often unavailable, adding learning time for a new developer.
 • The development team and/or tooling may no longer be available.
 • Details about design decisions, code structure, and pitfalls may no longer be available.
 • Ongoing development during the delay period means the problem may be hard to reproduce, and the fix may need to be applied to multiple source branches (ie last release and current development head).
 • If the product is released, customer support costs are added.
 • For some products, a recall may be required for updates.

What is "Integrated Testing"?
"Integrated Testing" means:
 • Testing is highly automated and repeatable.
 • Testing is tightly integrated into software development such that developers can run tests easily and often during the development process.
 • Most testing is run in a simulated target environment on a workstation, and does not require using a target embedded platform.
 • Ideally, most tests are created prior to system design, and certainly prior to coding.

Tests span unit tests to system tests, and always include system "smoke tests" (described below). As new problems are discovered, additional tests are added to ensure a fix and prevent regressions. Ideally, test runs are integrated into continuous build tools so regressions are caught immediately.

Because software is developed and tested primarily on workstations, software development can commence before target hardware is available, and development can continue without all developers requiring target hardware and associated tools.

"Integrated Testing" does not mean no separate QA or testing engineers ! It does mean that QA works tightly with engineering to ensure they have the test cases required, such as helping to define Test-Driven Development (TDD) test cases during detail design. Critically, developers work closely with QA to ensure that the code is readily testable on workstations as well as the target – software testability (discussed below). Integrated Testing mitigates problems with an isolated QA department and a "throw it over the wall to QA ghetto every once and a while" mentality.

Why costs to first release are lower with integrated testing
Without integrated testing, the delay between writing code and testing it adds cost as summarised above. Techniques to reduce this time (and to reduce bugs and cost) include:
 • TDD. Tests may be specified/implemented by the primary developer or a QA engineer—a QA engineer may do a better job of identifying boundary and exception conditions to test.
 • Code review. On average, the cost of code review (including fixing problems found during review) is less than the cost that would be incurred by finding and fixing these problems later, due to the cost of repair increasing over time. Performing code reviews thus reduces costs, and brings the additional benefits of improved code quality and additional developers familiar with the code.
 • Scripted tests, often using record-playback tools