Is Improving IV&V for Scientific/Engineering Software Worth the Effort?

IMHO, a significant number of scientists/engineers would opine that to impose a lot of independent verification and validation (IV&V) methodology for scientific/engineering applications would not be worth the effort.

But this would not be the consensus view. In May of 2006, a National Science Foundation (NFS) Blue Ribbon Panel issued a report on its findings and recommendations for Simulation-Based Engineering Science. Section 3.2 of the document talks about the verification, validation, and uncertainty quantification of computer-based simulations. The section addresses the question: "What level of confidence can one assign [to] a predicted [simulation] outcome in light of what may be known about the physical system and the model used to describe it?"

To quote from the Panel's findings:
While verification and validation and uncertainty quantification have been subjects of concern for many years, their further development will have a profound impact on the reliability and utility of simulation methods in the future. New theory and methods are needed for handling stochastic models and for developing meaningful and efficient approaches to the quantification of uncertainties. As they stand now, verification, validation, and uncertainty quantification are challenging and necessary research areas that must be actively pursued.


About verification and validation (V&V), the report stated:
The entire field of V&V is in the early stage of development. Basic definitions and principles have been the subject of much debate in recent years, and many aspects of the V&V remain in the gray area between the philosophy of science, subjective decision theory, and hard mathematics and physics.


On the subject of validation, the report states:
The twentieth century philosopher of science Karl Popper asserted that a scientific theory could not be validated; it could only be invalidated. Inasmuch as the mathematical model of a physical event is an expression of a theory, such models can never actually be validated in the strictest sense; they can only be invalidated. To some degree, therefore, all validation processes rely on prescribed acceptance criteria and metrics. Accordingly, the analyst judges whether the model is invalid in light of physical observations, experiments, and criteria based on experience and judgement.


And about verification the report states:
Verification processes, on the other hand, are mathematical and computational enterprises. They involve software engineering protocols, bug detection and control, scientific programming methods, and, importantly, a posteriori error estimation.


A more recent consensus is the 2009 WTEC Report which also has a section on validation, verification, and uncertainty quantification. The WTEC Report contains a lot more detail than the NSF Blue Ribbon Panel Report. However, not much has changed. This later Report notes that: "There are currently no funded U.S. national initiatives for fostering collaboration between researchers who work on new mathematical algorithms for V&V/UQ frameworks and design guidelines for stochastic systems."
So the answer is -- yes. Improving IV&V for scientific/engineering software would be worth the effort.

Verification and Validation of Scientific Software

On his blog, Jon Pipitone recently commented on the validity and soundness of scientific software. Jon found it curious that the terms verification and validation are not in more common use. Jon also mentions that there does not seem to be a standard term or adjective that is applied to software that been verified or validated.

I commented that:
In the nuclear safety software area you will often encounter the term "qualified software." The adjective refers to software that has been subjected to a defined level of verification and validation appropriate to the software's level of usage. This is called "qualification" of the software. Qualification is a part of overall software risk management and helps to ensure the "quality" (e.g., "soundness", "validity", "reliability", "security", etc.) of the software.

I would like to expand on my comment here. The scope and emphasis of software risk management processes (of which software verification and validation are subprocesses) depend on the nature and use of the software. What may be adequate for one type of software may not be adequate for another. However, much scientific software as well as much engineering design and analysis software are similar in nature and use. They model some physical process, system, structure, or component about which information is needed. Thus, a uniform software verification and validation approach may be appropriate.

The goal of software verification and validation is to ensure the quality of the software. For scientific and engineering programs this often means ensuring the accuracy and precision of the output.



The above figure is from the Wikipedia article on accuracy and precision. Although all models of nature or engineered systems are simplified abstractions, they must be relevant. That is, the abstractions must be inherently able to provide adequate accuracy and precision for the intended use. The process that ensures this required applicability is validation.

Additionally, the software code must faithfully embody these simplified abstractions. That is, the code must be adequately bug free. The process that ensures this required reliability is verification.