Analysis of faults in an N-version software experiment

The authors have conducted a large-scale experiment in N-version programming. A total of 27 versions of a program were prepared independently from the same specification at two universities. The results of executing the versions revealed that the versions were individually extremely reliable but that the number of input cases in which more than one failed was substantially more than would be expected if they were statistically independent. After the versions had been executed, the failures of each version were examined and the associated faults located. The analysis showed that in some cases the programmers made equivalent logical errors, indicating that some parts of the problem were simply more difficult than others. The authors also found cases in which apparently different logical errors yielded faults that caused statistically correlated failures, indicating that there are special cases in the input space that present difficulty in various parts of the solution. A formal model is presented to explain this phenomenon. It appears that minor differences in the software development environment would not have a major impact in reducing the incidence of faults that cause correlated failures.