Dealing with automotive software complexity (Part 3)

Code coverage measurement and fault injection testing are two highly important activities to boost the quality of tests. However, the application of both activities during the embedded software integration and testing phases is typically very limited. On the one hand, code coverage is mainly applied to on-host software unit testing.

Although useful, on-host testing at the unit test level does not exercise big parts of the embedded software running on the device and therefore it is not sufficient to credibly guarantee that faulty software is not being deployed. On the other hand, conventional fault injection techniques that can be applied during software integration and test phases have multiple limitations in terms of intrusiveness and controllability and offer only a limited set of injection points that could help to reach a more extensive coverage.

In the following subsections we will describe how virtual prototypes can overcome some of these limitations, help increase the quality of the tests, and hence the quality of the embedded software.

The ISO 26262 functional safety standard
Functional safety is the part of the overall safety of a system or piece of equipment that depends on the system or equipment operating correctly in response to its inputs, including the safe management of likely operator errors, hardware failures and environmental changes. Its objective is to eliminate the risk of physical injury or other health damage to the end-product users. Any assessment of functional safety must examine the function of any component or sub-system in the context of whole-system behaviour.

ISO 26262 [5] is a functional safety standard that replaces the older and more generic IEC 61508 for passenger vehicles. ISO 26262 addresses hazards caused by malfunctioning behaviour of electric and electronic safety related systems. The standard focuses on the electrical and electronic programmable systems (EEPS) but requires assurance that functional safety extends to the parts of the system that the EEPS activates, controls or monitors. The standard provides:

An automotive specific safety lifecycle: The safety lifecycle is composed of three phases: concept phase, product development phase and after start of production phase.

An automotive specific risk-based approach based on Automotive Safety Integrity Levels (or ASIL): Three aspects define the ranking of an item onto a specific Automotive Safety Integrity Level. Severity, which varies from light injuries to fatal injuries in case of failure of the item under investigation. Exposure, which indicates the probability that the item fails. And controllability, which indicates how difficult it is to control the effects of the hazard by the driver. ASIL D requires the highest safety integrity level for the item and ASIL A the lowest.

Requirements and recommended methods for the validation of the safety levels: The standard breaks down the requirements to verify and validate the item under consideration in different sub-parts (system-level, hardware and software). For each of these sub-parts a set of verification methods are described in the standard, with specific weights for the different ASIL levels: no recommendation, recommended and highly recommended. For example, section 4.7 (system design) [6] highly recommends simulation as a method to verify the system for ASIL C and D compliancy.

In general ISO 26262 highly recommends simulation and prototyping methods for system, hardware and software design and verification. When it comes to software integration and testing in particular, the standard highly recommends both fault injection testing and structural coverage metrics for the highest safety integrity levels (ASIL C/D). Fault-injection is more specifically mentioned for system, hardware and software integration and test activities. Fault-injection can be applied successfully to improve the test coverage of safety mechanisms at the system level, covering corner cases that are difficult to trigger during normal operation. It can also be applied whenever a hardware safety mechanism is defined to analyse its response to faults or where arbitrary faults that may corrupt software or hardware components must be injected to test the safety mechanisms.

Code coverage
Code coverage is a very important technique to help achieving higher software quality and cost effective testing, i.e. getting the same results with fewer tests. By using code coverage analysis, it is clear when a new set of tests increases the quality of the results and when not. Many of the added tests can be redundant and go exactly through the same path and conditions exercised by previous tests. Those tests are not adding anything but cost.