Statistical Testing Framework
As software systems grow larger and
more complex, the demands on testing these systems grow quickly,
oftentimes exponentially with respect to size of the system.
Developers expend ever-increasing amounts of resources to ensure the
best software products are released for use, within various resource
requirements. New methods of testing are required to ensure
complete, correct, and reasonably reliable systems can be built, yet
allow the developers to remain in business.
Statistical testing is advancing
rapidly in response to these needs. By implementing the
collection of usage data into operational profiles, developers can
utilize well-known statistics to direct the application of testing,
thereby reducing redundant testing, focusing testing on portions of
the software with the biggest impact on the system, and reducing the
amount of testing required overall. These improvements can
significantly decrease the amount of resources required for software
testing. Statistical testing can also be used to determine
when it is time to stop testing a software product, through
reliability and entropy metrics. Strategically designed
application of statistical testing can improve reliability measures
and reduce the levels of uncertainty present in the testing.
The Critical Systems Research Group (CriSys),
in collaboration with industry organizations and governmental
agencies, is researching the capabilities of statistical testing.
Our intent is to create a formal methods development framework for
statistical testing with a threefold objective:
-
Build on the known and
well-developed mathematical foundation of statistics and
statistical testing,
-
Leverage the knowledge and
capabilities of specification-based software development by
integrating the knowledge of statistical testing into a
specification-based software development environment, and
-
Build a set of tools for software
development to specifically address statistical testing
requirements
The analysis of Markov chain models is
well-known but becomes prohibitively expensive for models with
greater complexity. Our framework allows us to produce
operational profiles of significantly more complex systems than what
can currently be tested. We are focusing on test suite generation
from these operational profiles and we can continue building test
suites when analysis of the model is impossible due to the size and
complexity of the systems under test. We are also continuing
research into how changes in the operational profile affect the
fault-detection and coverage capabilities of statistical testing.
We currently have added a statistical
test suite generator and a tool for executing statistical test
suites on a specification to our software development framework.
Future research will focus on the fidelity of operational profiles,
the development of statistical testing metrics, and the creation of
statistical analysis tools. These tools should be easy to use and
will complete an integrated package of modeling, testing, and
development tools to address statistical testing in all stages of
the software development lifecycle. |
