Date: Tue, 22 Aug 1995 23:25:31 -0700
From: ee380 <ee380@shasta.stanford.edu>
Reply-To: ee380 <ee380@shasta.stanford.edu>
To: colloq@cs.stanford.edu
Subject: Bart Miller on Testing Real Programs.  EE380 Video Lecture 3:15 Fri



                EE380 Computer Systems Colloquium
                        VIDEO LECTURE

                      Summer Quarter 1994-1995

                          Lecture #9

Date:           Friday, Aug 25,1994

Time:           3:15-4:30 pm

Location:       Skilling 191

SITN:           Friday, Channel E3, 3:15-4:30 pm

Speaker:        Barton P. Miller
                Computer Sciences Department
                University of Wisconsin

Title:          Making Programs Explode: Using Simple Random
                Testing on Real Programs

Presented:      Wednesday, Apr 19,1994

                        Abstract

In 1990, we published the results of a study of the reliability of standard
UNIX utility programs. This study showed that by using simple (almost
simplistic) random testing techniques, we could crash or hang 25-33% of the
these utility programs. Five years later, we have repeated and significantly
extended this study using the same basic techniques: subjecting programs to
random input streams. A distressingly large number of UNIX utilities still
crash with our tests.

We tested a wide variety of utility programs on nine UNIX platforms.  The
programs were sent random input streams.  We used a conservative and crude
measure of reliability: a program is considered unreliable if it crashes
with a core dump or hangs (infinite loop). We used the random testing to
also test X-Window applications and servers, network servers, and system
library interfaces.

The major results of this study are: (1) In the last five years, all
previously-tested versions of UNIX made noticeable improvements in the
reliability of their utilities.  But ... the failure rate of these systems
is still distressingly high (from 18-23% in the 1995 study). (3) Even worse
is that many of the same bugs that we reported in 1990 are still present in
the code releases of 1995. (4) The failure rate of utilities on the
commercial versions of UNIX that we tested (from Sun, IBM, SGI, DEC, and
NEXT) ranged from 15 to 43%. (5) The failure rate of the utilities on the
freely-distributed Linux version of UNIX was second-lowest, at 11%. (6) The
failure rate of the public GNU utilities was the lowest in our study, at
only 7%. (7) We could not crash network services on any of the versions of
UNIX that we tested. (8) Almost a quarter of the X- Window applications that
we tested crash on purely random input data streams (random binary data).
More significant is that more than 40% of the applications crash given
random, but legal X-event streams. (9) We could not crash X server on the
versions of UNIX that we tested (i.e., sending random data streams to the
server).

                        Biography

Bart Miller joined the Computer Sciences Department at the University
of Wisconsin-Madison in 1984, where is he is currently a Professor.  He
received his M.S. and Ph.D. in Computer Science from Berkeley in 1980
and 1984.  His current research interests include parallel programming
tools, network name services, and mobile computing.

Contact Information:

   Barton P. Miller
   Computer Sciences Department
   University of Wisconsin
   1210 W. Dayton
   Madison, WI 53706-1685
   bart@cs.wisc.edu
   bart@wis.stanford.edu