Apparatus and method for testing computer systems

Abstract

A white box testing method tests public interfaces within a component of a software system. A setup function is executed to configure a nested public interface within the component to raise an exception. Initialization code is executed that calls the setup function. Test code is executed that calls the component and that evaluates how the component handles the exception raised by the nested public interface. Code is executed within the nested public interface to determine whether the setup function has been called and to raise the exception in response to an affirmative determination.

Claims

What is claimed is:

1. A white box testing method for testing how code of a software system being executed by a computer reacts to an exception raised by a public interface that is nested within a component the method comprising:

providing a setup function for execution by the computer that, when called by code executing by the computer, is executed by the computer to configure the nested public interface within the component to raise an exception;

providing initialization code for execution by the computer that, when executed by the computer, calls the setup function; and

providing test code for execution by the computer that, when executed by the computer, calls the component so as to cause the public interface nested within the component to raise an exception, wherein the test code evaluates how the component handles the exception raised by the nested public interface.

2. The white box testing method of claim 1, and further including:

providing code within the nested public interface of the component for execution by the computer that, when executed by the computer, determines whether the setup function has been called to configure the nested public interface to raise an exception and that raises the exception in response to an affirmative determination.

3. The white box testing method of claim 1, wherein

the setup function, when executed by the computer, takes an input "nextOccurrence" argument; and

the setup function, when executed by the computer, configures the nested public interface to not raise the exception until the nested public interface has been called a number of times represented by the nextOccurrence argument.

4. The white box testing method of claim 1, wherein

the setup function, when executed by the computer, takes an input "nextOccurrence" argument; and

if the nextOccurrence argument so indicates, when the setup, function is executed by the computer, the setup function configures the nested public interface to not raise the exception until the nested public interface has been called a number of times as determined from a random function.

5. The white box testing method of claim 1, wherein

the exception is a first exception;

the setup function, when called, is executed by the computer to configure the nested public interface within the component to raise both the first exception and a second exception; and

the test code calls the component so as to cause the public interface nested within the component to raise both the first exception and the second exception, wherein the test code evaluates how the component handles the first exception and the second exception raised by the nested public interface.

6. The white box testing method of claim 1, wherein the test code, when executed by the computer, includes code that when executed starts a plurality of threads, wherein each thread executes code for a separate test case.

7. The white box testing method of claim 6, wherein the test code, when executed by the computer, schedules the plurality of threads responsive to an ordering member.

8. A white box testing method for testing how code of a software system being executed by a computer reacts to an exception raised by a public interface that is nested within a component, the method comprising:

executing by the computer a setup function to configure the nested public interface within the component, when executed by the computer, to raise an exception;

executing by the computer initialization code that calls the setup function; and

executing by the computer test code that calls the component so as to cause the public interface within the component to raise an exception and that evaluates how the component handles the exception raised by the nested public interface.

9. The white box testing method of claim 8, and further including:

executing code by the computer within the nested public interface of the component to determine whether the setup function has been called to configure the nested public interface to raise an exception and to raise the exception in response to an affirmative determination.

10. The white box testing method of claim 8, wherein

the setup function, when executed by the computer, takes an input "nextOccurrence" argument; and

the step of executing the setup function by the computer includes the computer configuring the nested public interface to not raise the exception until the nested public interface has been called a number of times represented by the nextOccurrence argument.

11. The white box testing method of claim 8, wherein

the setup function, when executed by the computer, takes an input "nextOccurrence" argument; and

the step of executing the setup function by the computer includes, if the nextOccurrence argument so indicates, configuring the nested public interface to not raise the exception until the nested public interface has been called a number of times as determined from a random function.

12. The white box testing method of claim 8, wherein

the exception is a first exception;

the step of executing the setup function by the computer includes configuring the nested public interface within the component to, when executed by the computer, raise both the first exception and a second exception; and

the step of executing the test code by the computer includes calling the component so as to cause the public interface nested within the component to raise both the first exception and the second exception and then evaluating how the component handles the first exception and the second exception raised by the nested public interface.

13. The white box testing method of claim 8, wherein the test code, when executed by the computer, starts a plurality of threads, where each thread executes code for a separate test case.

14. The white box testing method of claim 12, wherein the test code, when executed by the computer, schedules the plurality of threads responsive to an ordering member.

15. An article of manufacture comprising a computer useable medium having computer-readable white box testing program code embodied therein, the white box testing code for configuring a computer to test a public interface nested within a component of a software system being executed by the computer, the white box testing code embodied in the article of manufacture comprising:

setup function computer readable software means that, when called by code executing by the computer, is executed by the computer to configure the nested public interface within the component to raise an exception;

initialization code computer readable software means for execution by the computer that, when executed by the computer, calls the setup function; and

test code computer readable software means that, when executed by the computer, calls the component so as to cause the public interface nested within the component to raise an exception, wherein the test code evaluates how the component handles the exception raised by the nested public interface.

16. The article of manufacture of claim 15, and further including:

determining and raising computer readable code means within the nested public interface for execution by the computer that, when executed by the computer, determines whether the setup function has been called to configure the nested public interface to raise an exception and raises the exception in response to an affirmative determination.

17. The article of manufacture of claim 15, wherein

the setup function computer readable software code means, when executed by the computer, takes an input "nextOccurrence" argument; and

the setup function computer readable software code means, when executed by the computer, configures the nested public interface to not raise the exception until the nested public interface has been called a number of times represented by the nextOccurrence argument.

18. The article of manufacture of claim 15, wherein

the setup function computer readable software code means, when executed by the computer, takes an input "nextOccurrence" argument; and

if the nextOccurrence argument so indicates, the setup function computer readable code means, when executed by the computer, configures the nested public interface to not raise the exception until the nested public interface has been called a number of times as determined from a random function.

19. The article of manufacture of claim 15, wherein

the exception is a first exception;

the setup function computer readable code means, when called, is executed by the computer to configure the nested public interface within the component to, when executed by the computer, raise both the first exception and a second exception; and

the test code computer readable code means that, when executed by the computer, calls the component so as to cause the public interface nested within the component to raise both the first exception and the second exception and evaluates how the component handles the first exception and the second exception raised by the nested public interface.

20. The article of manufacture of claim 15, wherein the test code computer readable code means includes computer readable code means that, when executed by the computer, start a plurality of threads, wherein each thread executes code for a separate test case.

21. The article of manufacture of claim 20, wherein the test code computer readable code means, when executed by the computer, schedules the plurality of threads responsive to an ordering member.

Description

BACKGROUND OF THE INVENTION

The invention relates generally to the field of structural software testing tools. More specifically, the invention defines both an instrumentation method that increases software testability, as well as an implementation of a test driver that provides the mechanism for individually testing each software component as the software component is developed.

Quality assurance of software products is a specialized segment of the software industry. Software testing products that provide analyses and estimations of the quality of a software product are widely available. Within the set of such widely available products, many different test methods are defined and implemented to verify product quality. Each product or test implementation varies in the type and range of tests performed by it, from very broad to very specific. Test products also vary from one to another in their specific implementation of a test methodology.

In general, there are two high level categories of software testing products. These areas are functional, or black box, testing; and structural, or white box, testing. Functional testing compares test program behavior against a set of requirements. Structural testing compares test program behavior against the intended behavior of the source code.

A primary concern within the practice of structural testing, is the ability to test 100% of the product code. This test parameter is referred to as "code coverage". Code Coverage becomes increasingly more important as the role of the software product being developed comes closer to "mission critical".

Within the telecommunications industry, where systems are designed to operate non-stop, the ability to verify, and prove, the quality of a software product, prior to placing it in service, is critical. While software testing products are available that will report code coverage parameters, and software testing methods are available to improve those coverage parameters, what is needed is a product or methodology that guarantee absolute 100% code coverage as part of software quality assurance. Present methodologies rely on an ad hoc sequence of writing tests and measuring code coverage numbers, and then repeating the process until acceptable code coverage is obtained.

SUMMARY

In accordance with an embodiment of the invention, a white box testing method tests public interfaces within a component of a software system. A setup function is executed to configure a nested public interface within the component to raise an exception. Initialization code is executed that calls the setup function. Test code is executed that calls the component and that evaluates how the component handles the exception raised by the nested public interface. Code is executed within the nested public interface to determine whether the setup function has been called and to raise the exception in response to an affirmative determination.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 illustrates processing to be executed by test code to test a nested public interface.

FIG. 1A illustrates the step of executing test code of FIG. 1 in greater detail.

FIG. 2 illustrates processing to be executed by the nested public interface in support of the testing of FIG. 1.

DESCRIPTION OF AN EMBODIMENT

One embodiment of the invention includes a processor coupled to a memory. The memory stores data and instructions used by the processor to implement one embodiment of the invention. In one embodiment of the invention, the processor and memory are included in a Sparcstation.TM. workstation from Sun Microsystems, Inc., of Mountain View, Calif. Other embodiments run on other computer platforms such as an IBM PC compatible running the Windows NT.TM. operating system.

1.0 Instrumentation Architecture

Every public interface in the system contains a description of all of the exceptions that can be raised by the interface. The descriptions of the public interfaces and their associated exceptions are specified in Interface Definition Language (IDL). The instrumentation architecture provides a mechanism to configure an interface to raise each of the exceptions defined in the interface definition.

To provide a consistent interface for the test cases, and to support the internal instrumentation of the code under test, a single base type is defined from which all other pseudo objects defined in the system must inherit. The public interface of the base type is:

    ______________________________________
    interface DSERoot
     // Exception control
     void setException(in string operationName,
      in string exceptionName,
      in unsigned long nextOccurance,
      in unsigned long frequency);
     void setNestedException(
      in string operationName,
      in string exceptionName,
      in unsigned long nextOccurance,
      in unsigned long frequency);
     // Instrumentation operations
     boolean checkException(
      in string operationName,
      in string exceptionName);
     boolean checkNestedException(
      in string operationName,
      out string exceptionName,
      out unsigned long nextOccurance,
      out unsigned long frequency);
     // Operating system errors
     void setOSError(in string functionName,
      in long errorCode);
     boolean checkOSError(in string functionName,
       out long errorcode);
    };
    ______________________________________

    ______________________________________
    exception someException{};
    interface TestExample:DSERoot
     void someOperation() raises(someException);
    };
    ______________________________________

    ______________________________________
    TestExample   testExample;
    // Configure someOperation to raise someException
    // the next time it is called. The exception
    // should be raised only once
    testExample.setException("someOperation", "someException", 1, 1);
    // Now call someOperation. It will raise an
    // exception
    try
     testExample.someOperation();
    }
    catch(someException)
    {
     // Handle exception
    }
    ______________________________________

    ______________________________________
    TestExample::someOperation(void)
     // Check to see if an exception should be
     // raised this time through. If so
     // declare the exception and raise it using
     // something like the C++ throw operator.
     if (checkException("someOperation",
      "someException))
     {
      someException exception;
      throw (exception);
     }
     // Perform normal functionality of operation
    }
    ______________________________________

    ______________________________________
    #if       !defined(DEBUG)
    #define   CHECKOSERROR(F,E)
    #else
    #define   CHECKOSERROR(F,E)
                             checkOSError(F,E)
    #endif    // DEBUG
    ______________________________________

    ______________________________________
    exception someException{};
    exception nestedException{};
    interface TestComponent:DSERoot
     void someOperation() raises(someException);
    };
    interface NestedComponent:DSERoot
    {
     void nestedOperation()
      raises(nestedException);
    };
    ______________________________________

    ______________________________________
    TestComponent   testComponent;
    // Configure nestedOperation to raise
    // nestedException the next time it is called.
    // The exception should be raised only once
    testComponent.setNestedException("nestedOperation",
     "nestedException", 1, 1);
    // Now call someOperation which in turn calls
    // nestedOperation
    try
     testComponent.someOperation();
    }
    catch(nestedException)
    {
     // Handle exception
    }
    ______________________________________

    ______________________________________
    TestComponent::someOperation(void)
     NestedComponent   nestedComponent;
     // Normal processing for this operation.
     // Just before calling nestedOperation
     // the following code must exist
     // Check to see if we need to configure
     // the nested component to raise an
     // exception. The exception to configure
     // (if any) is returned in nestedException.
     while (checkNestedException(
      "nestedOperation", &nestedException,
      &nextOccurance, &frequency)
     {
      // configure the requested exception
      nestedComponent.setException(
       "nestedOperation,
       nestedException, nextOccurance,
       frequency);
     }
     // call the nested operation
     try
     {
      nestedComponent.nestedOperation();
     }
     catch(nestedException)
     {
      // Handle exception
     }
    }
    ______________________________________

    ______________________________________
    typedef struct
     unsigned long ordering;
     string        testCaseName;
     TestCase       testCase;
    } TestCaseDescription;
    typedef TestCaseDescription   testCaseList[];
    ______________________________________

                  TABLE 1
    ______________________________________
    Test Driver Configuration Variables
    Variable    Description
    ______________________________________
    Test.sub.-- LogFileRoot
                The root name of the log file. The actual log file
                name has the thread identifier added as a suffix.
                The default value of this configuration variable is
                "TestLog".
    Test.sub.-- TestCase.sub.-- List
                A colon description separated list of the test cases
                that should be run. If there are multiple instances
                of a particular named test case configured in the
                testCaseList array, all instances are run. The
                default value of this configuration variable is an
                empty string, indicating that all test cases in the
                testCaseList array are to be run.
    Test.sub.-- TestContinue
                Indicate whether the driver should continue to
                execute test cases following a test case failure.
                Valid values for this variable are "true" or
                "false". "true" indicates that the test driver
                should continue following a test case failure,
                "false" indicates that the test driver should stop.
                The default value for this configuration variable is
                "true".
    ______________________________________

    ______________________________________
             testCaseList[0].ordering = 0;
             testCaseList[1].ordering = 1;
             testCaseList[2].ordering = 2;
             testCaseList[3].ordering = 2;
             testCaseList[4].ordering = 3;
    ______________________________________

    ______________________________________
    testCaseList[0].testCase.runTestCase(.sub.--);
    testCaseList[1].testCase.runTestCase(.sub.--);
    // Create another thread since two test cases
    // need to be started at the saine time
    testCaseList[2].testCase.runTestCase(.sub.--);
    testCaseList[3].testCase.runTestCase(.sub.--);
    // When both of these test cases have finished,
    // run the final test case
    testCaseList[4].testCase.runTestCase(.sub.--);
    ______________________________________

    ______________________________________
              TestCase.initialize(.sub.--);
              TestCase.runTestCase(.sub.--);
              TestCase.terminate(.sub.--);
    ______________________________________

    ______________________________________
    module Testing
     // IDL for DSERoot interface
     interface DSERoot
     {
      // Exception control
      void setException(in string operationName,
       in string exceptionName,
       in unsigned long nextOccurance,
       in unsigned long frequency);
      void setNestedException(
       in string operationName,
       in string exceptionName,
       in unsigned long nextOccurance,
       in unsigned long frequency);
      // Component under test operations
      boolean checkException(in string operationName,
       in string exceptionName);
      boolean checkNestedException(
       in string operationName,
       out string exceptionName,
       out unsigned long nextOccurance,
       out unsigned long frequency);
      // Operating system errors
      void setOSError(in string functionName,
       in long errorCode,
       in unsigned long nextOccurance,
       in unsigned long frequency);
      boolean checkOSError(in string functionName,
       out long errorCode);
     };
    };
    ______________________________________

    ______________________________________
    // Exceptions raised by this interface
    exception TestCaseFailed
    {
    string reason;
    };
    exception TestInitFailed
    {
    string reason;
    };
    // IDL for TestCase interface
    interface TestCase
    {
    attribute readonly string  testCaseName;
    void initialize(in SA::ThreadAccess threadAccess,
            in SA::MessageLog messageLog)
            raises(TestInitFailed);
    void runTestCase(in SA::ThreadAccess threadAccess,
            in SA::MessageLog messageLog)
            raises(TestCaseFailed);
    void terminate(in SA::ThreadAccess threadAccess,
            in SA::MessageLog messageLog);
    };
    };
    ______________________________________

    __________________________________________________________________________
    Test Case Code
                Code Under Test
                             Operating System
    __________________________________________________________________________
    /*Normal I/O operation*/
    do.sub.-- normal.sub.-- read( ); (1)=>
                file I/O code checks if
                OSError is set. No OSError
                set, therefore file I/O
                request executed (2)=>
                             access file (3)
                             <= return information (4)
                <= return file information (5)
    Test case passed the normal
    read test? Yes (6)
    /* Force error on next read
    */
    setOSError(
    "codeUnderTest", -255, 1,
    1); (7)=>
                file I/O code sets up to
                             /* no operation in OS */
                error and returns control to
                test case (8)
    do.sub.-- normal.sub.-- read( ); (9)=>
                file I/O code checksOSError
                             /* no operation in OS */
                and returns value of
                OSError (10)
    Test case passed abnormal
    read test? Yes (11)
    __________________________________________________________________________

• Inventors
  Sifter, Daniel J.;
• Assignee
  ObjectSwitch Corporation (Larkspur, CA)
• Published
  Feb-8-2000
• Current US Classes:
  717/124
• Application #
  887958
• International Classes
  G06F 009/45
• Field of Search
  395/704
• Examiner
  Hafiz; Tariq R.
• Agent
  Limach & Limbach L.L.P.
• US Patent References:
  4819233
  5495571