But most software cannot be exhaustively or "heroically" tested.
- The software may be too complicated for testing the requirements against all possible combinations of internal program logic.
- The range of software inputs or outputs may be continuous or unbounded.
- The software may model or simulate some process only approximately. (This is almost always true for physical processes.)
- The software may model some physical process for which the software's intended range of usage is not completely matched by experimental data.
Examples of an important class of software that cannot be heroically tested are the GCMs (both general circulation models and global climate models). Such software are used in support of potentially high consequence decisions (and the timing of those decisions.) This requires a correspondingly high level of SQA. How can this be accomplished if exhaustive testing is impossible?
A common approach for low or medium consequence software is selective (limited) testing by an accredited authority (person, group, organization, etc.). The SQA authority determines the scope and emphasis of software testing, performs the appropriate tests, and then simply avers the quality of the software. This approach is also sometimes attempted for high consequence software. For example, the GCMs.
The efficacy of this approach is dependent on the level of trust that can be placed in the SQA authority. Logically, the quality of software can only be assured to the level of trust in the SQA authority itself. Thus, for high consequence software, an even higher level of trust in the accredited authority is required. There are currently few examples of an SQA authority being able to establish and maintain a very high level of trust. Unfortunately, partly because of "climategate," the GCMs currently lack a such a highly trusted SQA authority.
But another approach is possible and it will lead us to the other key to software quality. This other approach is to control and document the organizational processes used to plan, develop, and maintain the software. All of the software engineering consensus standards acknowledge the importance of these software management processes to software quality assurance.
The key process in this other approach, and thus the other key to software quality besides testing, is an iterative process for continuous quality improvement. Regardless of the initial trust in the software organization, instituting a consensus acknowledged process for improvement will guarantee that ultimate software quality is assured. It seems likely that soon all the GCM development organizations will adopt such a process.