Can Scientific/Engineering Code Be Validated?

I am starting to read Patrick J. Roache's book, The Fundamentals of Verification and Validation. I thought I knew the fundamentals of IV&V for scientific and engineering software already, but reading Roache seems to have me feeling a bit ignorant.

Be that as it may, I think I must disagree with the limited scope of Roache's definition of validation.

In Appendix B of the book, Validation -- What Does it Mean? Roache writes:
Before examining the definition of validation, we need to make a small distinction on what it is we are claiming to validate, i.e., between code and model. A model is incorporated into a code, and the same model (e.g. some RANS model) can exist in many codes. Strictly speaking, it is the model that is to be validated, whereas the codes need to be verified. But for a model to be validated, it must be embodied in a code before it can be run. It is thus common to speak loosely of "validating a code" when one means "validating the model in the code".

I think this distinction is too limiting. The embodied code must be validated too.

IMHO, Roache is using the word verification in the sense of formal verification. That's fine, except scientific and engineering software can rarely be heroically tested. Formal proof of such software's correctness is a practical impossibility. Does Roache really think verification is impossible?

Suppose I found a DVD case on the sidewalk and inside was a DVD with a label that said: "Models the Earth's Climate." I put the DVD in my computer and, sure enough, on the DVD is what appears to be a complete and sophisticated complex climate model. How would I go about verifying such a software's outputs? What amount of testing would be sufficient? What verification processes would I choose to use?

On the other hand, suppose I obtain funding to develop, from scratch, a new and complete Earth climate modeling software. What methodologies would I choose to develop and test the software? Would I think it important to verify the processes completed at various stages of the software's development?

And here is the rub. Suppose that it turns out each software actually has the same physics model. Nevertheless, would I need to validate that the different processes I used to verify the software on the DVD and to verify the software built from scratch were appropriate for each? Yes! The verification processes for each software would be different. These differences must be validated as appropriate and effective.

So if I feel a bit ignorant under a limited definition of validation, I now feel even more so under an expanded definition.

2 comments:

  1. George, when you say this:
    The embodied code must be validated too.

    Are you talking about validating the process of verification? Which was kind of the gist I got from this:
    Nevertheless, would I need to validate that the different processes I used to verify the software on the DVD and to verify the software built from scratch were appropriate for each

    I think that is a bit different than the way Roache uses those terms. What you're talking about seems like a sort of a mixture of the broader SQA view of things with the more narrow focus Roache has.

    Writing your reactions as you read through the book would be an interesting set of notes; if you take requests, this reader would like more.

    ReplyDelete
  2. Yes, I am taking a broader view. I do feel Roache's distinction is too limiting. Of course, it's best to judge for yourself.

    In Appendix B, Roache defines validation as follows:

    Validation: The process of determining the degree to which a model {and its associated data} is an accurate representation of the real world from the perspective of the intended uses of the model.

    Roache adds:

    Unfortunately, considerable disagreement exists on what this definition
    means, or should mean.

    This definition of validation has been cited extensively in CFD (Computational Fluid Dynamics) and other computational modeling fields, and is widely accepted.


    At the risk of contributing needlessly to the "considerable disagreement," I think the definition clearly implies that the result of validation is a value statement. A Bayesian (i.e., quantitative) estimate, from the authority accredited with performing the validation, of the degree to which the software is an accurate representation of the real world from the perspective of the intended uses of the software.

    Surely the scope and rigor of code verification has an effect on this probability estimate. Thus, to the extent that a set minimum level of validation is required, deciding on the appropriate scope and rigor of code verification is a part of the validation process. (Note: the required scope and rigor depends on the intended usage.)

    If nothing else, verification effects the validation priors. To continue with my example of the climate model DVDs. Suppose that instead of finding the software on the sidewalk, I found the software in a national lab's software configuration management system. With full documented traceability from the time the software was first conceived. What would be the scope and rigor of verification required to establish the embodied code as validated for my own usage at my own lab? A whole lot less than finding it on the sidewalk.

    ReplyDelete