Chapter 26A - Software Test and Evaluation: A View from the US

This addendum to Section 26 Software Test and Evaluation (T&E) will provide a brief overview of a software test and evaluation philosophy as being applied in the United States and the role that the Flight Test Engineer (FTE) must play in this scenario. The role of the FTE in the software T&E process has been changing rapidly and will continue to evolve as new software development and T&E techniques are developed. Currently, only few projects utilize the new techniques described in this paper, however, the techniques are becoming widely known through many different forums of information exchange and the techniques have been well received. Moreover, new instructions such as the replacement for DOD-STD- 2167A (i.e., MIL-STD-498) espouse these processes and techniques. It is important that the FTEs understand how testability is achieved. Because of the evolving nature of this process, the new FTE is well advised to carefully review and become intimately familiar with the procedures and techniques that are available and utilized at his home base. One of the highest paybacks to any development is assuring that a repeatable process is in place which is well managed. In the recent past the software T&E "process" was far less structured than it is, and will be, under evolving directives and standards. The software T&E process described in this section follows the Total Quality Leadership (TQL) paradigm of inspect, measure, and improve. By evolving the process, we will have considerably better testability than previously available. Software T&E, in the US, has historically been applied much as one would apply checks of quality to hardware, i.e., waiting until the product begins to appear, and then begin to make measurements on overall performance. This approach had some merit and could have positive results while software programs were only a few tens of thousands of Source Lines Of Code (SLOC) in length, and subsystems were black box oriented. However, with today's highly integrated architectures, the virtual disappearance of black boxes in new systems, and software sizes greater than 2 million SLOC in platforms, this approach is woefully inadequate. The current software T&E process, which consists of tests derived from black-box and white-box techniques based on requirements analysis as determined by the human mind are simply not sufficiently comprehensive to test the software to any depth. A complete restructuring of our thinking for software T&E is required to have any hope of actually determining if the software will meet any particular specification or set of requirements. This restructuring will require software tools and techniques; in particular, techniques for formal requirements elicitation, in addition to modeling and simulation tools which supply engineering rigor to the entire development process. From this standpoint, the test engineer in today's developments and updates is learning to become a stakeholder from the very beginning of the development process. (In this context, a "stakeholder" is one who has become involved in the software generation process and retains responsibility for his inputs as the process evolves). As a stakeholder, he is an integral part of the development team for the life cycle of the product. Albeit at the beginning of the process, his inputs and presence are small - but critical - to the entire development.