Test executive system and method including distributed type storage and conflict resolution

Abstract

A test executive program which provides greatly improved configurability and modularity, thus simplifying the creation, modification and execution of test sequences. The test executive program includes process models for improved flexibility, modularity and configurability. Process models provide a modular and configurable entity for encapsulating operations and functionality associated with a class of test sequences. The process model thus encapsulates a "testing process" for a plurality of test sequences. The process model enables the user to write different test sequences without repeating standard testing operations in each sequence. The test executive program also includes step types for improved configurability. A step type is a modular, identifiable unit configured by the user which defines common properties and/or operations associated with a plurality of steps. Instances of the step type incorporate this common functionality and/or properties from the step type, and thus the user is not required to hard code that functionality and/or properties in each instance or step. The test executive program also provides distributed type storage and conflict resolution. When the user loads a file which includes a stored type, the method performs conflict resolution, including determining if the loaded type conflicts with one or more previously loaded/registered types. If the types do conflict, then the test executive or user selects a type, preferably using the version data associated with each type. The test executive system also performs automatic result collection to automatically collect the results of each step. The user can enable or disable result collection for a particular sequence or for the entire test station.

Claims

We claim:

1. A method for managing types in a test executive system, wherein the test executive system is used for creating test programs for testing one or more units under test (UUTs), the method comprising:

loading a file with at least one type, wherein the at least one type comprises one of: 1) a data type or 2) a step type, wherein the at least one type includes version data;

determining the type being loaded in response to said loading;

determining if the type conflicts with one or more previously loaded/registered types, wherein said determining includes comparing the type being loaded with the previously loaded/registered types in the system;

selecting a type based on the version data in response to determining a type conflict;

registering the at least one type in the system in response to determining that there is no type conflict.

2. The method of claim 1, wherein said determining the type being loaded and said determining if the type conflicts are automatically performed in response to said loading the file.

3. The method of claim 1, wherein said determining the type being loaded, said determining if the type conflicts, and said selecting a type are automatically performed in response to said loading the file.

4. The method of claim 1, wherein said determining if the type conflicts comprises determining if a name of the at least one type conflicts with a name of one of said previously loaded/registered types.

5. The method of claim 1, further comprising:

receiving user input regarding selection of a type in response to determining a type conflict;

wherein said selecting a type is performed in response to said user input.

6. The method of claim 1, wherein said selecting a type is automatically performed in response to determining the type conflict.

7. The method of claim 1, further comprising:

determining that the version data of the at least one type and the one or more previously loaded/registered types are the same after said determining a type conflict;

receiving user input regarding selection of a type in response to said determining that the version data are the same;

wherein said selecting a type is performed in response to said user input.

8. The method of claim 1, wherein the file stores a data type definition of the at least one type.

9. The method of claim 1, further comprising:

creating a step or data of a first type in the file in response to user input; and

automatically storing a type definition of the first type in the file in response to said creating.

10. The method of claim 1, further comprising:

creating the at least one type and assigning a name to the at least one type in response to user input; and

storing a step or data of the at least one type in the file;

wherein said creating and said storing are performed prior to said loading the file.

11. The method of claim 10, further comprising:

automatically storing a type definition of the at least one type in the file in response to said creating and said storing.

12. The method of claim 10,

wherein said creating and said storing are performed in a first computer; and

wherein said loading the file is performed in a second computer.

13. The method of claim 10,

wherein the at least one type has a first name;

wherein said determining if the type conflicts comprises determining if the first name of the at least one type conflicts with a name of one of said previously loaded/registered types;

wherein said creating and said storing are performed in a first computer;

wherein said loading the file is performed in a second computer, wherein the second computer includes a different type which includes said first name.

14. The method of claim 1, wherein said types comprise a data structure; the method further comprising:

conforming existing instances of one or more non-selected types with the selected type, wherein said conforming comprises modifying the data structure of the one or more non-selected types to conform to the data structure of the selected type.

15. The method of claim 1, further comprising:

conforming existing instances of one or more non-selected types with the selected type, wherein said conforming includes preserving at least a portion of old data of the one or more non-selected types.

16. The method of claim 1, wherein the version data comprises one or more of: user-entered version data, or date and time information.

17. A memory medium comprising program instructions for managing types in a test executive system, wherein the test executive system is used for creating test programs for testing one or more units under test (UUTs), wherein, in response to loading a file with at least one type, wherein the at least one type comprises one of: 1) a data type or 2) a step type and wherein the at least one type includes version data, the program instructions are executable to implement:

determining the type being loaded in response to said loading;

determining if the type conflicts with one or more previously loaded/registered types, wherein said determining includes comparing the type being loaded with the previously loaded/registered types in the system;

selecting a type based on the version data in response to determining a type conflict;

registering the at least one type in the system in response to determining that there is no type conflict.

18. The memory medium of claim 17, wherein said determining the type being loaded and said determining if the type conflicts are automatically performed in response to said loading the file.

19. The memory medium of claim 17, wherein said determining the type being loaded, said determining if the type conflicts, and said selecting a type are automatically performed in response to said loading the file.

20. The memory medium of claim 17, wherein said determining if the type conflicts comprises determining if a name of the at least one type conflicts with a name of one of said previously loaded/registered types.

21. The memory medium of claim 17, wherein the program instructions are further executable to implement:

receiving user input regarding selection of a type in response to determining a type conflict;

wherein said selecting a type is performed in response to said user input.

22. The memory medium of claim 17, wherein said selecting a type is automatically performed in response to determining the type conflict.

23. The memory medium of claim 17, wherein the program instructions are further executable to implement:

determining that the version data of the at least one type and the one or more previously loaded/registered types are the same after said determining a type conflict;

receiving user input regarding selection of a type in response to said determining that the version data are the same;

wherein said selecting a type is performed in response to said user input.

24. The memory medium of claim 17, wherein the file stores a data type definition of the at least one type.

25. The memory medium of claim 17, wherein the program instructions are further executable to implement:

creating a step or data of a first type in the file in response to user input; and

automatically storing a type definition of the first type in the file in response to said creating.

26. The memory medium of claim 17, wherein the program instructions are further executable to implement:

creating the at least one type and assigning a name to the at least one type in response to user input; and

storing a step or data of the at least one type in the file;

wherein said creating and said storing are performed prior to said loading the file; and

automatically storing a type definition of the at least one type in the file in response to said creating and said storing.

27. The memory medium of claim 26,

wherein said creating and said storing are performed in a first computer; and

wherein said loading the file is performed in a second computer.

28. The memory medium of claim 26,

wherein the at least one type has a first name;

wherein said determining if the type conflicts comprises determining if the first name of the at least one type conflicts with a name of one of said previously loaded/registered types;

wherein said creating and said storing are performed in a first computer;

wherein said loading the file is performed in a second computer, wherein the second computer includes a different type which includes said first name.

29. The memory medium of claim 17, wherein said types comprise a data structure; wherein the program instructions are further executable to implement:

conforming existing instances of one or more non-selected types with the selected type, wherein said conforming comprises modifying the data structure of the one or more non-selected types to conform to the data structure of the selected type.

30. The memory medium of claim 17, wherein the program instructions are further executable to implement:

conforming existing instances of one or more non-selected types with the selected type, wherein said conforming includes preserving at least a portion of old data of the one or more non-selected types.

31. The memory medium of claim 17, wherein the version data comprises one or more of: user-entered version data, or date and time information.

32. A method for managing types in a test executive system, wherein the test executive system is used for creating test programs for testing one or more units under test (UUTs), the method comprising:

loading a file with at least one type, wherein the at least one type includes version data;

determining the type being loaded in response to said loading;

determining if the type conflicts with one or more previously loaded/registered types, wherein said determining includes comparing the type being loaded with the previously loaded/registered types in the system;

determining that the version data of the at least one type and the one or more previously loaded/registered types are the same after determining a type conflict;

receiving user input regarding selection of a type in response to said determining that the version data are the same;

selecting a type in response to said user input;

registering the at least one type in the system in response to selecting the at least one type.

33. A method for managing types in a test executive system, wherein the test executive system is used for creating test programs for testing one or more units under test (UUTs), the method comprising:

creating at least one type and assigning a first name to the at least one type in response to user input;

storing a step or data of the at least one type in a file;

loading the file with the at least one type, wherein the at least one type includes version data;

wherein said creating and said storing are performed in a first computer;

wherein said loading the file is performed in a second computer, wherein the second computer includes a different type which includes said first name;

wherein the method further comprises:

determining the type being loaded in response to said loading;

determining that the type conflicts with the different type, wherein said determining includes determining that the first name of the at least one type conflicts with the name of the different type;

selecting a type based on the version data in response to determining the type conflict.

34. A method for managing types in a test executive system, wherein the test executive system is used for creating test programs for testing one or more units under test (UUTs), the method comprising:

loading a file with at least one type, wherein the at least one type includes version data;

determining the type being loaded in response to said loading;

determining that the type conflicts with one or more previously loaded/registered types, wherein said determining includes comparing the type being loaded with the previously loaded/registered types in the system;

selecting a type based on the version data in response to determining the type conflict; and

conforming existing instances of one or more non-selected types with the selected type, wherein said conforming comprises modifying a data structure of the one or more non-selected types to conform to a data structure of the selected type.

35. A memory medium comprising program instructions for managing types in a test executive system, wherein the test executive system is used for creating test programs for testing one or more units under test (UUTs), wherein, in response to loading a file with at least one type, wherein the at least one type includes version data, the program instructions are executable to implement:

determining the at least one type being loaded in response to said loading;

determining that the at least one type conflicts with one or more previously loaded/registered types, wherein said determining includes comparing the at least one type being loaded with the previously loaded/registered types in the system;

determining that the version data of the at least one type and the one or more previously loaded/registered types are the same after determining the type conflict;

receiving user input regarding selection of a type in response to said determining that the version data are the same;

selecting a type in response to said user input;

registering the at least one type in the system in response to selecting the at least one type.

36. A memory medium comprising program instructions for managing types in a test executive system, wherein the test executive system is used for creating test programs for testing one or more units under test (UUTs), wherein, in response to loading a file with at least one type, wherein the at least one type includes version data, the program instructions are executable to implement:

determining the type being loaded in response to said loading;

determining that the type conflicts with one or more previously loaded/registered types, wherein said determining includes comparing the type being loaded with the previously loaded/registered types in the system;

selecting a type based on the version data in response to determining the type conflict; and

conforming existing instances of one or more non-selected types with the selected type, wherein said conforming comprises modifying a data structure of the one or more non-selected types to conform to a data structure of the selected type.

37. A method for managing types in a test executive system, the method comprising:

loading a test executive sequence file including a first data type;

determining the first data type being loaded in response to said loading;

determining that the first data type conflicts with a previously registered data type in the system, wherein said determining includes comparing the first data type with the previously registered data type;

prompting for user input selecting either the first data type or the previously registered data type in response to said determining that the first data type conflicts with the previously registered data type; and

selecting either the first data type or the previously registered data type in response to said user input.

Description

FIELD OF THE INVENTION

The present invention relates to test executive software for organizing and executing test sequences to control instrumentation systems.

DESCRIPTION OF THE RELATED ART

A test executive is a program that allows a user to organize and execute sequences of reusable test modules to control a test involving one or more instruments. The test modules often have a standard interface and typically can be created in a variety of programming environments. The test executive software operates as the control center for the automated test system. More specifically, the test executive software allows the user to create, configure, and/or control test sequence execution for various test applications, such as production and manufacturing test applications. Text executive software typically includes various features, such as test sequencing based on pass/fail results, logging of test results, and report generation, among others.

Test executives include various general concepts. The following comprises a glossary of test executive nomenclature.

Code Module--A program module, such as a Windows Dynamic Link Library (.dll) or LabVIEW VI (.vi), that contains one or more functions that perform a specific test or other action.

Test Module--A code module that performs a test.

Step--Any action, such as calling a test module to perform a specific test, that the user can include within a sequence of other actions.

Step Module--The code module that a step calls.

Sequence--A series of steps that the user specifies for execution in a particular order. Whether and when a step is executed can depend on the results of previous steps.

Subsequence--A sequence that another sequence calls. The user specifies a subsequence call as a step in the calling sequence.

Sequence File--A file that contains the definition of one or more sequences.

Sequence Editor--A program that provides a graphical user interface for creating, editing, and debugging sequences.

Run-time Operator Interface--A program that provides a graphical user interface for executing sequences on a production station. A sequence editor and run-time operator interface can be separate application programs or different aspects of the same program.

Test Executive Engine--A module or set of modules that provide an API for creating, editing, executing, and debugging sequences. A sequence editor or run-time execution operator interface uses the services of a test executive engine.

Application Development Environment (ADE)--A programming environment such as LabVIEW, LabWindows/CVI, or Microsoft Visual C, in which the user can create test modules and run-time operator interfaces.

Unit Under Test (UUT)--The device or component that is being tested.

Prior art test executive programs are generally hard-coded programs with little flexibility, modularity or configurability. Prior art systems generally include executive code, sequence code, test modules, and then instrument drivers. The executive code typically includes functionality such as UUT identification, operator notification, test report generation, and logging results. The vendor may or may not provide source code for the executive code to the end-user. If the vendor did provide source code for the executive code, the above-mentioned functionality would be subsumed in that source code but it would not be in a format which is easily configurable. In other words, this functionality was not separated out as a single configurable entity. Therefore, if the user desired to modify this functionality, the user would have to hunt through the source code to find and edit the code, which is a tedious process. Thus, in prior art, the user would be required to enter the programming language to change this functionality. This was tedious and difficult.

Therefore, an improved test executive program is desired which provides improved flexibility, modularity and configurability.

Prior art test executive programs also do not easily allow user configuration of steps. In general, step functionality was hard-coded with little flexibility, modularity or configurability. Therefore, an improved test executive program is also desired which provides improved configurability of steps.

An improved test executive program is further desired which provides improved type conflict resolution and improved automatic result collection.

SUMMARY OF THE INVENTION

The present invention comprises an improved test executive program which provides process model and step type functionality for improved flexibility, modularity and configurability. The test executive program of the present invention also provides distributed type storage and conflict resolution, as well as automatic result collection.

The test executive of the present invention, referred to as the TestStand test executive software, is a flexible, powerful test executive framework which provides a number of novel features. In particular, the test executive software of the present invention provides greatly improved configurability and modularity, thus simplifying the creation, modification and execution of test sequences. The test executive software provides numerous ways for the user to modify the default configuration and components or to add new components. These extensibility mechanisms enable the user to create the test executive that meets the user's particular requirements without modifying the TestStand test execution engine.

The test executive software expands on the traditional test executive concepts and introduces many new ones. The test executive software of the present invention includes new concepts and features, including step types, step properties, sequence variables, sequence parameters, module adapters, and process models.

The TestStand test executive software architecture includes operator interface programs for interfacing to various software programs. The TestStand test executive software also includes a sequence editor for editing sequences. The sequence editor and the operator interface programs interface to the test executive engine, referred to as the TestStand Engine. One or more process models couple to the TestStand Engine. The TestStand Engine interfaces through an adapter interface to one or more adapters. The adapters in turn interface to code modules and sequence files.

The TestStand Engine plays a pivotal role in the TestStand architecture. The TestStand Engine executes sequences, wherein sequences contain steps that can call external code modules. By using module adapters that have the standard adapter interface, the TestStand Engine can load and execute different types of code modules. TestStand sequences can call sub-sequences through the common adapter interface. TestStand uses a special type of sequence called a process model to direct the high-level sequence flow. The TestStand Engine exports an ActiveX Automation API used by the TestStand sequence editor and run-time operator interfaces.

Types--Conflict Resolution

The TestStand test executive system of the present invention includes various types, including step types, custom named data types, and standard named data types. A type can thus be a data type or step type. For each type that a file uses, the TestStand system stores the definition of the type in the file. More specifically, when a user creates a type in the test executive system, the user creates the type using a graphical user interface (GUI) and assigns a name to the type. The user may also assign version data to the type and/or the system may apply a timestamp or other version data to the type. When the user later stores a step or data of the at least one type in a file, the TestStand Engine automatically stores a type definition of the type in the file in response. The user can also specify that a file always saves the definition for a type, even if the file does not currently use the type. Because many files can use the same type, many files can contain definitions for the same type.

The TestStand system allows only one definition for each type to exist in memory. If the user modifies the type in one view, the type is updated in all views. However, problems can arise if there is not a single definition of a type in the system. For example, when the user creates a step type, and then the user stores an instance of the step type in the sequence file, the type definition of the step type is stored in the file. If someone later changes the contents of the step type, and then reloads the sequence which is based on the original step type, there will be a conflict. This conflict can also occur when the user moves a sequence file from a first computer to a second computer, wherein the second computer includes a different type definition for the respective type name.

When the user loads a file which includes a stored type, the method of the present invention executes to perform conflict resolution. Here presume that the user loads a file with at least one type wherein the file preferably stores a type definition of the loaded type, e.g., a data type or step type. In response to the user loading the file, the TestStand Engine automatically determines the type being loaded, and then automatically determines if the loaded type conflicts with one or more previously loaded/registered types. This determination includes comparing the type being loaded with the previously loaded/registered types in the system. In the preferred embodiment, the determination of a type conflict comprises determining if the name of the loaded type conflicts with the name of any of the previously loaded/registered types.

If the types do not conflict, then the TestStand Engine registers the type in the system. Thus, since there is no type conflict, the type can be safely registered in the system. If the types do conflict, then the TestStand Engine selects a type in response to determining the type conflict, preferably using the version data associated with each type. In the preferred embodiment, the TestStand Engine selects the type with the version data indicating the most recent version, e.g., the one with the most recent version number.

In one embodiment, the TestStand Engine requests user input regarding selection of a type in response to determining a type conflict, and this user input is used in selecting the type. In other words, when the TestStand Engine detects a type conflict, the Engine displays a dialog informing the user of the conflict and requesting the user to decide which type should be used. The dialog also preferably displays the version data from each of the types to aid the user in the selection. The user then selects the type to be used.

After selection of the type, the TestStand Engine conforms existing instances of one or more non-selected types with the selected type. This conforming comprises modifying the data structure of the one or more non-selected types to conform to the data structure of the selected type. This conforming also includes preserving at least a portion of the old data of the one or more non-selected types.

The conflict resolution performed according to the present invention is especially important, for example, where the user creates and stores the type in a first computer, and then later loads the file in a second computer. For example, the second computer may include a different type which includes the same name as the type created and stored in the first computer, thereby creating a conflict.

Step Types

A sequence comprises a series of steps, wherein a step is typically a test performed on an instrument. The present invention includes step types which allow more modular and configurable steps while minimizing user coding and effort.

In a test sequence with a number of steps, in many instances the user will desire a number of steps that have some commonality of functionality and/or properties. A primary purpose of a step type is to define common properties and/or operations associated with a plurality of steps in a single location, referred to as the step type, thereby eliminating the need for the user to define these common properties and/or operations with each of the respective steps. Instances of the step type incorporate this common functionality and/or properties from the step type, and thus the user is not required to hard code that functionality and/or properties in each instance or step. The functionality and/or properties defined by a step type is generally peripheral or associated with the actual test or the step being performed.

A step type thus essentially comprises a custom set of properties and/or operations associated with a step. Stated another way, a step type defines common operations and/or data that are associated with a test module in a similar way that a process model defines functionality associated with calling the main sequence file of a test. The test module is hard coded, and the step type represents operations and/or properties associated with calling this test module. A step type is a modular, identifiable unit configured by the user, preferably through a graphical user interface, e.g., dialogs.

Step types define common functionality by defining an edit sub-step and pre and post sub-steps. The pre and post sub-steps execute before and after, respectively, a step. The edit sub-step is used to configure the peculiarities of a particular instance of a step type. For instance the edit sub-step can be configured to display a message or dialog to request user. This message is typically displayed at configuration time, not run time. Step types also include pre and post expressions and status expressions for configurability. These expressions are executed at run time.

A step type has similar functionality to a type definition, meaning that once the user has configured a step type and used it throughout different steps of the sequence, if the user later changes that step type, those changes propagate through all of the steps which are based on that step type. This type definition functionality is not performed through copying or propagating changes. Rather, the imformation is held within the step type, and during run time the sequence examines the step type to determine the step type parameters for execution of the step.

In prior art test executives, the functionality performed by the step type of the present invention was hard coded in the test executive itself and was not easily changeable. Further, this functionality was not in a modular form which could be reused or applied to a plurality of steps. The step type of the present invention embodies this functionality, e.g., the pre and post operations in a typical test, and places this in a configurable and reusable form. Step types of the present invention are modular and user configurable and provide tremendous savings in developer effort.

Step types are generally configured and used as follows. First, the user configures a first step type. As discussed above, the first step type defines common functionality and common data for steps (instances) of the first step type. The common functionality preferably includes one or more of a common editing interface, pre-step functionality and post-step functionality. The common functionality is preferably specified by the user as one or more sub-steps that call code modules written in any of various programming languages. The common data includes one or more property values organized in a user-defined hierarchy. Configuration of the first step type also optionally includes creating one or more code templates, wherein the code templates are useable in creating a code module called by instances of the first step type.

The user then creates a test sequence file for testing the unit under test. The test sequence file includes at least one sequence comprising a plurality of steps, wherein one or more of the steps are of the first step type (instances of the first step type).

The test sequence file is then executed to test the unit under test. Execution of the test sequence file includes executing each of the plurality of steps in the at least one sequence. When steps of the first step type are executed, execution includes executing the common functionality for the one or more of the steps which are of the first step type and also includes utilizing the common data for the one or more of the steps which are of the first step type. More specifically, for a step of the first step type, i.e., an instance of the first step type, execution of the step includes:

executing the pre-step functionality;

executing a code module referenced by the step after executing the pre-step functionality; and

executing the post-step functionality after executing the code module.

Process Models

The present invention includes process models which provide a modular and configurable entity for encapsulating operations and functionality associated with a class of test sequences. The process model thus encapsulates a "testing process" for a plurality of test sequences. The "testing process" includes common operations required to be performed before and after the test executive executes the sequence that performs the tests. Common operations include identifying the UUT, notifying the operator of pass/fail status, generating a test report, and logging results.

The process model enables the user to write different test sequences without repeating standard testing operations in each sequence. The process model is also user-modifiable. This is important because the testing process can vary based on the production line, the production site, or the systems and practices of the company in which the test executive is used. The process model thus provides tremendous savings in user development time.

TestStand provides a mechanism for defining a process model. A process model is in the form of a sequence file. The user can edit a process model just as he/she edits other sequences. TestStand ships with a fully functional default process model. The user can write his/her own process model, or the user can copy the default process model and then modify it.

Process models are used in the creation of a test system as follows. First the user configures a process model, or alternatively selects a previously configured process model. The process model includes functionality which is common to a plurality of different test sequences, wherein the functionality is for testing one or more UUTs. As discussed above, the process model is a separate user configurable module, e.g., is modularly separate from the test sequence file. More specifically, the process model is modularly separate from the test executive engine, the operator interfaces, the test sequences, and the test code. The process model preferably comprises a sequence file and is user editable as a sequence file. The process model comprises a plurality of steps which call code modules, wherein the code modules can be written in a variety of different programming languages. The process model includes one or more of pre-test operations, post-test operations, variables, parameters, types and code modules. The process model is also independent of language and construction of the operator interface program. Finally, the user can configure the process model to provide one or more of: customized report generation, database logging, UUT identification, and UUT status notification, among others.

Configuration of the process model includes one or more of: configuring pre-test operations and/or post-test operations in the process model; configuring one or more entry points in the process model; changing the order of calls to callback functions; adding/deleting calls to callback functions; adding new callback functions in response to new calls added to the process model; overriding callback functions called by the process model; modifying existing callback functions in the process model; and configuring variables, parameters, types, steps and/or code modules.

The user then creates a test sequence file, also called a client file, for testing the unit under test. The test sequence file includes the test sequence for the desired UUT, which is called MainSequence. The MainSequence file is called by the process model file and typically includes a number of sub-sequences which call into other sequence files. The user may include one or more client callbacks in the test sequence file which operate to override or replace callbacks in the process model.

The process model is comprised in a process model sequence file. The process model sequence file is a file that has multiple sequences that can be of types entry point sequence, callback sequence or normal sequence. A normal sequence is typically a utility subsequence. The entry point sequence is a place where the user can begin execution within the process model sequence file. The callback sequence is a sequence which typically performs test process functionality and is called by a call in the process model. The callback sequence in the process model sequence file is the default callback for the call, and is replaceable or overrideable by callback sequences inserted into the client file.

The process model and the test sequence file are then executed to test the unit under test. In the preferred embodiment, the user invokes execution of the process model, and during execution of the process model the process model operates to call the test sequence file. The user preferably begins execution of the process model through a graphical user interface, such as through a menu. The user preferably selects a desired execution entry point to begin execution within the process model sequence file. The test sequence file or client file can operate with any of the execution entry points, i.e., the test sequence file is not customized for a particular entry point in the process model. This way, for a single client file, the user can choose different ways of running the respective client file.

According to the present invention, the operator interface program automatically updates in response to user configuration of one or more entry points. When the user defines one or more entry points in the process model, the operator interface program queries the test executive engine to determine the one or more entry points defined by the user in the process model. The operator interface program then displays the one or more entry points in response to the query. Thus the one or more entry points configured by the user in the process model automatically appear in the operator interface program. Stated another way, changes to the process model which affect the appearance of the operator interface do not require manual changes to the operator interface, but rather the operator interface automatically updates itself.

Automatic Result Collection

The TestStand test executive system of the present invention performs automatic result collection to automatically collect the results of each step. The user can enable or disable result collection for a particular sequence or for the entire test station.

In the preferred embodiment, each sequence has a local array that stores the results of each step. The contents in the results for each can vary depending on the step type. When TestStand stores the results for a step into the array, it adds information such as the name of the step and its position in the sequence. For a step that calls a sequence, TestStand also adds the result array from the subsequence.

The automatic result collection of the present invention preferably operates as follows. First, the user creates a test sequence file for testing the unit under test, wherein the test sequence file includes at least one sequence comprising a plurality of steps. The user then configures the results to be collected. This, for example, includes defining one or more properties to be collected for one or more of the steps in the sequence. Where one or more step types are included in the test sequence file, the user preferably configures results for the step types.

The user then enables automatic result collection. In one embodiment, automatic result collection may be the default result collection technique. Also, the user can enable or disable automatic result collection for particular sequence(s) or for the entire test station. The test sequence file is then executed to test the unit under test. Execution of the test sequence file includes executing each of the plurality of steps in the sequence.

The TestStand Engine operates to automatically collect the results of each step in the sequence during execution. If the user enabled automatic result collection, then the automatic collection is performed in response to this enabling, and for the specified sequences. During execution, the user can also dynamically configure the results to be collected, as desired.

The automatic collection includes collecting standard result information, wherein the standard result information is automatically collected regardless of the user input configuring the results to be collected. The standard result information includes one or more of: a name of the step and a position of the step in the sequence.

Where the test sequence file includes a step type and the user has configured results for the step type, the method automatically collects results depending on the step type of the steps in the sequence. For example, when the steps in the test sequence file include a first step having a first step type and a second step having a second step type, the automatic collection operates to collect first results for the first step according to the first step type and operates to collect second results for the second step according to the second step type.

In the preferred embodiment, the sequence includes a local array that stores the results of each step in the sequence, and the automatic collection includes storing the results of each step in the local array of the sequence. The local array is preferably a local variable of the sequence, wherein the local array is accessible from within the sequence and from other code modules.

When a step in a sequence of the test sequence file calls a sub-sequence, the automatic collection includes storing a result array from the sub-sequence as the result for the first step. Further, when a first step calls a sub-sequence, and the sub-sequence includes a step which in turn calls another-subsequence, the result for the first step includes a chronology and hierarchy of step execution made in response to the first step.

Where the first step calls a sub-sequence, and the sub-sequence executes asynchronously, the automatic collection includes allocating an array element for the result when the first step calls the sub-sequence and storing the result in the array element after completion of the sub-sequence.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of the present invention can be obtained when the following detailed description of the preferred embodiment is considered in conjunction with the following drawings, in which:

FIG. 1 illustrates an instrumentation control system according to one embodiment of the present invention;

FIG. 2 illustrates the architecture of the test executive software according to the preferred embodiment of the present invention;

FIG. 3 is a flowchart illustrating the Test UUTs sequence in the default process model;

FIG. 4 is a screen shot illustrating the Test UUTs entry point sequence in the default process model;

FIG. 5 is a screen shot illustrating a list of all of the sequences in the TestStand process model file;

FIG. 6 is a flowchart illustrating a method for managing types in a test executive system;

FIG. 7 is a flowchart illustrating a method for creating types in the test executive system;

FIG. 8 is a screen shot illustrating the "Type Conflict In File" dialog box which is displayed in the preferred embodiment;

FIG. 9 is a flowchart illustrating a method for creating a test system for testing one or more units under test (UUTs), including creation of a step type and one or more instances of the step type;

FIG. 10 is a flowchart illustrating detail regarding configuration of a step type;

FIG. 11 illustrates the submenu for the Insert Step item;

FIG. 12 illustrates the Step Types tab of the Type Palette window;

FIG. 13 illustrates the custom properties for the Numeric Limits step type;

FIG. 14 illustrates the General tab of the Step Type Properties dialog box for the Action step type;

FIG. 15 illustrates the Menu tab of the Step Type Properties dialog box for the Action step type;

FIG. 16 illustrates the Substeps tab of the Step Type Properties dialog box for the Numeric Limit Test step type;

FIG. 17 illustrates the Disable Properties tab of the Step Type Properties dialog box for the Numeric Limit Test step type;

FIG. 18 illustrates the Code Templates tab of the Step Type Properties dialog box for the Numeric Limit Test step type;

FIG. 19 illustrates the Create Code Templates dialog box;

FIG. 20 illustrates the Edit Code Template dialog box;

FIG. 21 illustrates the common custom step properties;

FIG. 22 illustrates the Edit Pass/Fail Source dialog box;

FIG. 23 illustrates the step properties for the Pass/Fail Test step type;

FIG. 24 illustrates the Limits tab on the Edit Numeric Limit Test dialog box;

FIG. 25 illustrates the Data Source tab of the Edit Numeric Limit Test dialog box;

FIG. 26 illustrates the step properties for the Numeric Limit Test step type;

FIG. 27 illustrates the Limits tab on the Edit String Value Test dialog box;

FIG. 28 illustrates the Data Source tab of the Edit String Value Test dialog box;

FIG. 29 illustrates the step properties for the String Value Test step type;

FIG. 30 illustrates the Specify Module dialog box for a Sequence Call step;

FIG. 31 illustrates the Edit Statement Step dialog box;

FIG. 32 illustrates the Configure Message Box Step dialog box;

FIG. 33 illustrates the step properties for the Message Popup step type;

FIG. 34 illustrates the Configure Call Executable dialog box;

FIG. 35 illustrates the step properties for the Call Executable step type;

FIG. 36 illustrates an example sequence file that might use the limits file;

FIG. 37 illustrates the Limits File tab of the Edit Limit Loader Step dialog box;

FIG. 38 illustrates the Layout tab of the Edit Limit Loader Step dialog box;

FIG. 39 illustrates the step properties for the Limit Loader step type;

FIG. 40 illustrates the Import/Exports Sequence Limits dialog box;

FIG. 41 illustrates the Edit Goto Step dialog box;

FIG. 42 is a flowchart illustrating configuration of a process model;

FIG. 43 is a flowchart illustrating automatic updating of the operator interface in response to user-specified entry points in the process model; and

FIG. 44 is a flowchart illustrating automatic result collection of the present invention;

FIG. 45 illustrates the settings for a process model file in the Advanced tab of the Sequence File Properties dialog box;

FIG. 46 illustrates the pull-down menu for the Type ring control;

FIG. 47 illustrates the contents of the Model tab for the Test UUTs execution entry point;

FIG. 48 illustrates a list of all the sequences in the default TestStand process model;

FIG. 49 is a flowchart illustrating creation of a test system, including configuration of a process model; and

FIG. 50 illustrates the result of a Numeric Limit Test step in expanded form on the Execution window Context tab.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Cross-Reference to Related Applications

The following applications are related to the present application:

U.S. patent application Ser. No. 09/259,161 titled "Test Executive System and Method Including Process Models for Improved Configurability" and filed Feb. 26, 1999;

U.S. patent application Ser. No. 09/259,162 titled "Test Executive System and Method Including Step Types for Improved Configurability" and filed Feb. 26, 1999;

U.S. patent application Ser. No. 09/259,165 titled "Test Executive System and Method Including Automatic Result Collection" and filed Feb. 26, 1999.

Incorporation by Reference

The TestStand user documentation, available from National Instruments Corporation, is hereby incorporated by reference as though fully and completely set forth herein.

U.S. Provisional Application Ser. No. 60/097,391 titled "Test Executive System and Method with Improved Configurability" filed on Aug. 21, 1998, is hereby incorporated by reference as though fully and completely set forth herein.

FIG. 1--Instrumentation System

FIG. 1 illustrates an example instrumentation control system 100. FIG. 1 is exemplary only, and the present invention may be used in any of various systems, as desired.

The system 100 comprises a host computer 102 which connects to one or more instruments. The host computer 102 comprises a CPU, a display screen, memory, and one or more input devices such as a mouse or keyboard as shown. The computer 102 connects through the one or more instruments to analyze, measure or control a unit under test (UUT) or process 150.

The one or more instruments may include a GPIB instrument 112 and associated GPIB interface card 122, a data acquisition board 114 and associated signal conditioning circuitry 124, a VXI instrument 116, a PXI instrument 118, a video device 132 and associated image acquisition card 134, a motion control device 136 and associated motion control interface card 138, and/or one or more computer based instrument cards 142, among other types of devices.

The GPIB instrument 112 is coupled to the computer 102 via a GPIB interface card 122 provided by the computer 102. In a similar manner, the video device 132 is coupled to the computer 102 via the image acquisition card 134, and the motion control device 136 is coupled to the computer 102 through the motion control interface card 138. The data acquisition board 114 is coupled to the computer 102, and optionally interfaces through signal conditioning circuitry 124 to the UUT. The signal conditioning circuitry 124 preferably comprises an SCXI (Signal Conditioning eXtensions for Instrumentation) chassis comprising one or more SCXI modules 126.

The GPIB card 122, the image acquisition card 134, the motion control interface card 138, and the DAQ card 114 are typically plugged in to an I/O slot in the computer 102, such as a PCI bus slot, a PC Card slot, or an ISA, EISA or MicroChannel bus slot provided by the computer 102. However, these cards 122, 134, 138 and 114 are shown external to computer 102 for illustrative purposes. The cards 122, 134, 138 and 114 may also be implemented as external devices coupled to the computer 102, such as through a serial bus.

The VXI chassis or instrument 116 is coupled to the computer 102 via a serial bus, MXI bus, or other serial or parallel bus provided by the computer 102. The computer 102 preferably includes VXI interface logic, such as a VXI, MXI or GPIB interface card (not shown), which interfaces to the VXI chassis 116. The PXI chassis or instrument is preferably coupled to the computer 102 through the computer's PCI bus.

A serial instrument (not shown) may also be coupled to the computer 102 through a serial port, such as an RS-232 port, USB (Universal Serial bus) or IEEE 1394 or 1394.2 bus, provided by the computer 102. In typical instrumentation control systems an instrument will not be present of each interface type, and in fact many systems may only have one or more instruments of a single interface type, such as only GPIB instruments. The instruments are coupled to the unit under test (UUT) or process 150, or are coupled to receive field signals, typically generated by transducers. Other types of instruments or devices may be connected to the system, as desired. The system 100 may be used in a data acquisition and control application, in a test and measurement application, a process control application, an industrial automation application, or a man-machine interface application, among others.

The computer system 102 preferably includes a memory medium on which computer programs according to the present invention are stored. The term "memory medium" is intended to include an installation media, e.g., a CD-ROM, or floppy disks 104, a computer system memory such as DRAM, SRAM, EDO RAM, etc., or a non-volatile memory such as a magnetic medium, e.g., a hard drive, or optical storage. The memory medium preferably stores test executive software for creating and/or controlling an automated test system. The test executive software allows the user to create, configure, and/or control test sequence execution for various test applications, such as production and manufacturing test applications. The host computer CPU executing code and data from the memory medium comprises a means for creating and executing test programs according to the methods described below.

In the preferred embodiment, the present invention is comprised in the TestStand test executive software, available from National Instruments.

TestStand Architecture Overview

The TestStand test executive software of the present invention is a flexible, powerful test executive framework which provides a number of novel features. In particular, the test executive software of the present invention provides greatly improved configurability and modularity, thus simplifying the creation, modification and execution of test sequences. The test executive software provides numerous ways for the user to modify the out-of-the-box configuration and components or to add new components. These extensibility mechanisms enable the user to create the test executive that meets the user's particular requirements without modifying the TestStand test execution engine.

The test executive software expands on the traditional test executive concepts and introduces many new ones. The test executive software of the present invention includes new concepts and features, including process models, step types, step properties, sequence variables, sequence parameters, module adapters, type conflict resolution and automatic result collection. In the present description, the test executive software of the present invention is referred to as the TestStand test executive software or simply as TestStand.

TestStand Software Components

FIG. 2 shows the high-level relationships between elements of the TestStand system architecture. As shown, the TestStand test executive software includes operator interface programs 202 for interfacing to various software programs. The operator interface programs 202 shown in FIG. 2 are for interfacing to the LabVIEW, LabWindows CVI, and Visual Basic programs. However, additional operator interface programs 202 may be included for interfacing to other programs.

The TestStand test executive software also includes a sequence editor 212 for editing sequences. The sequence editor 212 and the operator interface programs 202 interface to the test executive engine 220, referred to as the TestStand Engine. One or more process models 222 couple to the TestStand Engine 220. The TestStand Engine 220 interfaces through an adapter interface 232 to one or more adapters 240. The adapters shown in FIG. 2 include the LabVIEW standard prototype adapter, the C/CVI prototype adapter, the DLL flexible prototype adapter, and the sequence adapter. The LabVIEW standard prototype adapter interfaces to programs having a .VI extension, i.e., LabVIEW programs. The C/CVI prototype adapter interfaces to programs having a .dll, .lib, .obj, or .c extension. The DLL flexible prototype adapter interfaces to programs having a .dll extension. The sequence adapter interfaces to sequence file programs.

As shown in FIG. 2, the TestStand Engine 220 plays a pivotal role in the TestStand architecture. The TestStand Engine 220 runs sequences. Sequences contain steps that can call external code modules. By using module adapters 240 that have the standard adapter interface 232, the TestStand Engine 220 can load and execute different types of code modules. TestStand sequences can call sub-sequences through the common adapter interface 232. TestStand uses a special type of sequence called a process model to direct the high-level sequence flow. The TestStand Engine 220 exports an ActiveX Automation API used by the TestStand sequence editor 212 and run-time operator interfaces 202.

TestStand Sequence Editor

The TestStand sequence editor 212 is an application program in which the user creates, modifies, and debugs sequences. The sequence editor 212 provides the user easy access to all of the powerful TestStand features, such as step types and process models. The sequence editor 212 includes debugging tools found in application development environments such as LabVIEW, LabWindows/CVI, and Microsoft Visual C/C++. These include breakpoints, single stepping, stepping into or over function calls, tracing, a variable display, and a watch window.

In the TestStand sequence editor 212, the user can start multiple concurrent executions. Multiple instances of the same sequence can be executed, and different sequences can be executed at the same time. Each execution instance has its own execution window. In trace mode, the execution window displays the steps in the currently executing sequence. When execution is suspended, the execution window displays the next step to execute and provides single-stepping options.

TestStand Run-Time Operator Interfaces

TestStand preferably includes three run-time operator interfaces 202 provided to the end user in both source and executable form. Each run-time operator interface 202 is preferably a separate application program. The operator interfaces 202 differ primarily based on the language and ADE in which each is developed. As shown in FIG. 2, in the currently implemented embodiment, TestStand includes run-time operator interfaces developed in LabVIEW, LabWindows/CVI and Visual Basic.

Although the user can use the TestStand sequence editor 212 at a production station, the TestStand run-time operator interfaces 202 are simpler and are fully customizable. Like the sequence editor 212, the run-time operator interfaces 202 allow the user to start multiple concurrent executions, set breakpoints, and single step. Unlike, the sequence editor 212, however, in the present embodiment the run-time operator interfaces 202 do not allow the user to modify sequences, and they do not display sequence variables, sequence parameters, step properties, and so on.

The user can customize one of the run-time operator interfaces 202 by modifying the source code for the program and the source documents for the manual. If the user desires to write his/her own run-time operator interface 202, the source code of one of the run-time operator interfaces 202 is used as a starting point.

TestStand Test Executive Engine

The TestStand Test Executive Engine 220 is used for creating, editing, executing, and debugging sequences. In the preferred embodiment, the TestStand Test Executive Engine 220 comprises a set of DLLs that export an object-based or component-based API, preferably an ActiveX Automation API. The TestStand sequence editor 212 and run-time operator interfaces 202 use the TestStand Test Executive Engine API (Engine API). The user can call the Engine API from any programming environment that supports access to ActiveX Automation servers. Thus, the user can call the Engine API from test modules, including test modules that are written in LabVIEW and LabWindows/CVI.

The TestStand Test Executive Engine 220 implements numerous features of the present invention including process models, step types, type conflict resolution and automatic result collection.

Module Adapters

Most steps in a TestStand sequence invoke code in another sequence or in a code module. When invoking code in a code module, TestStand must know the type of the code module, how to call it, and how to pass parameters to it. The different types of code modules include LabVIEW VIs, C functions in DLLs, and C functions in source, object, or library modules that are created in LabWindows/CVI or other compilers. TestStand also must know the list of parameters that the code module requires.

In the preferred embodiment, TestStand uses module adapters 240 to obtain this knowledge. In the currently implemented embodiment, TestStand provides the following module adapters:

DLL Flexible Prototype Adapter--Allows the user to call C functions in a DLL with a variety of parameter types.

LabVIEW Standard Prototype Adapter--Allows the user to call any LabVIEW VI that has the TestStand standard G parameter list.

C/CVI Standard Prototype Adapter--Allows the user to call any C function that has the TestStand standard C parameter list. The function can be in an object file, library file, or DLL. The C function can also be in a source file that is in the project that the user is currently using in the LabWindows/CVI development environment.

Sequence Adapter--Allows the user to call subsequences with parameters.

The module adapters 240 contain other important information besides the calling convention and parameter lists. If the module adapter 240 is specific to an application development environment (ADE), the adapter knows how to bring up the ADE, how to create source code for a new code module in the ADE, and how to display the source for an existing code module in the ADE. The DLL Flexible Prototype Adapter can query a DLL type library for the parameter list information and display it to the sequence developer.

TestStand Building Blocks

This section provides an overview of the TestStand features and building blocks that are used to create test sequences and entire test systems.

Variables and Properties

TestStand provides the user various places, referred to as variables and properties, in which data values can be stored. Variables are properties that the user can freely create in certain contexts. Variables can be global to a sequence file or local to a particular sequence. The values of station global variables are persistent across different executions and even across different invocations of the sequence editor 212 or run-time operator interfaces 202. The TestStand Engine maintains the value of station global variables in a file on the run-time computer.

Each step in a sequence can have properties. For example, a step might have an integer error code property. The type of a step determines the set of properties which are included in the step. Step Types are discussed in greater detail below.

TestStand variables are used to share data among tests that are written in different programming languages, even if they do not have compatible data representations. Values that are stored in variables and properties can be passed to code modules. The TestStand ActiveX API is useable to access variable and property values directly from code modules. When executing sequences, TestStand maintains a "sequence context" that contains references to all global variables and all local variables and step properties in active sequences. The contents of the sequence context change depending on the currently executing sequence and step. If the user passes a sequence context object reference to the code module, the TestStand ActiveX API can be used to access the variables and properties in the sequence context.

1. Expressions

In TestStand, the values of variables and properties can be used in numerous ways, such as passing a variable to a code module or using a property value to determine whether to execute a step. Sometimes the user desires to use an expression, which is a formula that calculates a new value from the values of multiple variable or properties. An expression can be used anywhere a simple variable or property value is used. In expressions, the user can access all variables and properties in the sequence context that is active when TestStand evaluates the expression. The following is an example of an expression:

Locals.MidBandFrequency=(Step.HighFrequency+Step.LowFrequency)/2

TestStand supports all applicable expression operators and syntax that are used in C, C++, Java, and Visual Basic. TestStand also provides an expression browser dialog box that the user can access, if the user is not familiar with expressions in these standard languages. The expression browser allows the user to interactively build an expression by selecting from lists of available variables, properties, and expression operators. The expression browser also lists a number of functions that the user can use in expressions. The expression browser has help text for each expression operator and function.

Categories of Properties

A property is a container of information. A property can contain a single value, an array of values of the same type, or no value at all. A property can also contain any number of sub-properties. Each property has a name.

A value is a number, a string, or a Boolean. TestStand stores numbers as 64-bit floating-point values in the IEEE 754 format. Values are not containers and thus cannot contain sub-properties. Arrays of values can have multiple dimensions.

The following are the major categories of properties according to the kinds of values they contain:

A "single-valued" property contains a single value. Because TestStand has three types of values, TestStand has three types of single-valued properties: Number properties, String properties, and Boolean properties.

An "array" property contains an array of values. TestStand has Number Array properties, String Array properties, and Boolean Array properties.

A "property-array" property contains a value that is an array of subproperties of a single type. In addition to the array of sub-properties, property-array properties can contain any number of subproperties of other types.

An "object" property contains no values. Typically, object properties contain multiple sub-properties. Object properties are analogous to structures in C/C++ and to clusters in LabVIEW.

Standard and Custom Named Data Types

When the user creates a variable or property, the user specifies its data type. In some cases, a simple data type such as a number or a Boolean is used. In other cases, the user can define his/her own data type, by creating a named data type, in which sub-properties are added to create an arbitrarily complex data structure. When a named data type is created, the user can reuse the named data type for multiple variables or properties. Although each variable or property that the user creates with a named data type has the same data structure, the values they contain can differ.

TestStand defines certain standard named data types. The user can add sub-properties to the standard data types, but cannot delete any of their built-in sub-properties. The standard named data types are Path, Error, and CommonResults.

The user can define his/her own custom named data types. The user must choose a unique name for each of the custom data types. Sub-properties in each custom data type can be added or deleted without restriction. For example, the user might create a "Transmitter" data type that contains sub-properties such as "NumChannels" and "Power Level".

When the user creates a variable or property, the user can select from among the simple property types and the named data types.

Built-In and Custom Properties

TestStand defines a number of properties that are always present for objects such as steps and sequences. An example is the step run mode property. TestStand normally hides these properties in the sequence editor, although it allows the user to modify some of them through dialog boxes. Such properties are called built-in properties.

The user can define new properties in addition to the built-in properties. Examples are high and low limit properties in a step or local variables in a sequence. Such properties are called "custom" properties.

Steps

A sequence comprises a series of steps. In TestStand a step can do many things, such as initializing an instrument, performing a complex test, or making a decision that affects the flow of execution in a sequence. Steps can performs these actions through several types of mechanisms, including jumping to another step, executing an expression, calling a sub-sequence or calling an external code module. The term "step module" is used to refer to the code module that a step calls.

In TestStand, steps can have custom properties. For steps that call code modules, custom step properties are useful for storing parameters to pass to the code module for the step. They also serve as a place for the code module to store its results. The TestStand ActiveX API can be used to access the values of custom step properties from code modules.

Not all steps call code modules. Some steps perform standard actions that the user configures using a dialog box. In this case, custom step properties are useful for storing the configuration settings that the user specifies.

Built-In Step Properties

TestStand steps have a number of built-in properties that the user can specify using the various tabs on the Step Properties dialog box. These built-in step properties include:

Preconditions allow the user to specify the conditions that must be true for TestStand to execute the step during the normal flow of execution in a sequence.

Load/Unload Options allow the user to control when TestStand loads and unloads the code modules or subsequences that each step invokes.

Run Mode allows the user to skip a step or to force it to pass or fail without executing the step module.

Record Results allow the user to specify whether TestStand stores the results of the step in a list. This is discussed further below with respect to Results Collection.

Step Failure Causes Sequence Failure allows the user to specify whether TestStand sets the status of the sequence to "Failed" when the status of the step is "Failed".

Ignore Run-Time Errors allows the user to specify whether TestStand continues execution normally after the step even though a run-time error occurs in the step.

Post Actions allows the user to execute callbacks or jump to other steps after executing the step, depending on the pass/fail status of the step or any custom condition.

Loop options allow the user to cause a single step to execute multiple times before executing the next step. The user can specify the conditions under which to terminate the loop. The user can also specify whether to collect results for each loop iteration, for the loop as a whole, or for both.

Pre Expressions allows the user to specify an expression to evaluate before executing the step module.

Post Expression allows the user to specify an expression to evaluate after executing the step module.

Status Expression allows the user to specify an expression to use to set the value of the "status" property of the step automatically.

Step Types

Each step has a step type in a similar manner to which each variable or property has a data type. A step type can contain any number of custom properties. Each step of that step type, also referred to as an instance of the step type, has the custom step properties in addition to the built-in step properties. All steps of the same type have the same properties, but the values of the properties can differ. The step type specifies the initial values of all the step properties. When the user creates the step in the sequence editor, TestStand sets the initial or default values of the step properties from the values that the step type specifies. The user can modify the values of the built-in step properties by using the Step Properties dialog box. Usually, the user can modify the values of custom step properties using a dialog box specific to the step type. If the step type does not have a dialog box for the custom properties, the user can view the custom properties by selecting "View Contents" from the context menu for the step. Although, step modules typically do not modify the values of the built-in step properties at run-time, they often modify and interrogate the values of the custom step properties.

A step type can also define standard behavior for each step of that type. The step type does this using a set of "substeps". Substeps are actions that the TestStand Engine performs for a step besides calling the step module. The substeps of a step type perform the same actions for every step of that type. The different types of substeps are as follows:

Edit substep

Pre-step substep

Post-step substep

The sequence developer invokes the edit substep by selecting a menu item in the context menu for the step or by clicking a button on the Step Properties dialog for the step. The step type specifies the name of the menu item and the caption of the button. The edit substep displays a dialog box in which the sequence developer edits the values of custom step properties. For example, an edit substep might display a dialog box in which the sequence developer specifies the high and low limits for a test. The edit substep might then store the high and low limit values as step properties.

The Engine 220 calls the pre-step substep before calling the step module. The user can specify an adapter and a module to invoke in the pre-step substep. For example, a pre-step substep might call a code module that retrieves measurement configuration parameters into step properties for use by the step module.

The Engine 220 calls the post-step substep after calling the step module. The user can specify an adapter and a module to invoke in the post-step substep. A post-step substep might call a code module that compares the values that the step module stored in step properties against limit values that the edit substep stored in other step properties.

TestStand contains a set of predefined step types. These include

Action

Numeric Limit Test

String Value Test

Pass/Fail Test

Label

Goto

Statement

Limit Loader

Message Popup

Call Executable

Call Sequence

For a description of each of these step types, refer to the discussion of Built-in Step Types below. Although the user can create a test application using only the predefined step types, the user can also create his/her own step types. By creating his/her own step types, the user can define standard, reusable classes of steps that apply specifically to the user's application. For example, the user might define a Switch Matrix Configuration step or a Transmitter Adjacent Channel Power Test step.

The sequence developer creates a new step by selecting the "Insert Step" item in the context menu that appears when the user right click on a sequence window. The "Insert Step" item brings up a hierarchical submenu containing the step types available on the computer. When the user creates a new step type, the user specifies its name and position within the submenu.

Source Code Templates

When the user creates a step type, the user can also define source code templates for that step type. When the sequence developer creates a new step of that type, the developer can use a source code template to generate source code for the step module. For a particular step type, the user can specify different source code templates for the different module adapters.

Sequences

In TestStand a sequence comprises:

Any number of local variables

Any number of parameters

A main group of steps

A group of setup steps

A group of cleanup steps

Built-in sequence properties

Sequence Parameters

Each sequence has its own list of parameters. The user can specify the number of parameters and the data type of each parameter. The user can also specify a default value for each parameter. When the sequence developer creates a step that calls one sequence from another, the developer can specify the values to pass for the parameters of the subsequence. If the developer does not specify the value of a parameter, the TestStand Engine 220 passes the default value. The user can use the TestStand ActiveX API to access sequence parameter values from code modules that steps in the sequence call.

Sequence Local Variables

The user can create an unlimited number of local variables in a sequence. The user can use local variables to store data relevant to the execution of the sequence. The user can use the TestStand ActiveX API to access local variables from code modules that steps in the sequence call. The user can also pass local variables by value or by reference to any step in the sequence that calls a subsequence or that calls a DLL using the Flexible DLL Prototype Adapter.

Lifetime of Locals Variables, Parameters, and Custom Step Properties

Multiple instances of a sequence can run at the same time. This can occur when the user calls a sequence recursively or when a sequence runs in multiple concurrent executions. Each instance of the sequence has it own copy of the sequence parameters, and local variables, and custom properties of each step. When a sequence completes, the TestStand Engine 220 discards the values of the parameters, local variables, and custom properties.

Step Groups

A sequence can contain the following groups of steps: Setup, Main, and Cleanup. When TestStand executes a sequence, the steps in the Setup group execute first. The steps in the Main group execute next. The steps in the Cleanup group execute last. Typically, the Setup group contains steps that initialize instruments, fixtures, or a UUT. The Main group usually contains the bulk of the steps in a sequence, including the steps that test the UUT. The Cleanup group contains steps that power down or de-initialize the instruments, fixtures, and UUT.

One of the reasons for having separate step groups is to ensure that the steps in the Cleanup group execute regardless of whether the sequence completes successfully or a run-time error occurs in the sequence. If a Setup or Main step causes a run-time error to occur, the flow of execution jumps to the Cleanup step group. The Cleanup steps always run even if some of the Setup steps do not run. If a Cleanup step causes a run-time error, execution continues at the next Cleanup step.

If a run-time error occurs in a sequence, the TestStand Engine 220 reports the run-time error to the calling sequence. Execution in the calling sequence jumps to the Cleanup group in the calling sequence. This process continues up through the top-level sequence. Thus, when a run-time error occurs, the TestStand Engine 220 terminates execution after running all the Cleanup steps of the sequences that are active at the time of the run-time error.

Built-in Sequence Properties

Sequences have a few built-in properties that the user can specify using the Sequence Properties dialog. For example, the user can specify that the flow of execution jumps to the Cleanup step group whenever a step sets the status property of the sequence to "Failed".

Sequence Files

Sequence files can contain one or more sequences. Sequence files can also contain global variables. Sequence file global variables can be accessed by all sequences in the sequence file. Sequences files have a few built-in properties that the user can specify using the Sequence File Properties dialog. For example, the user can specify Load and Unload Options that override the Load and Unload Options of all the steps in all the sequences in the file.

Sequences contain steps that conduct tests, set up instruments, or perform other actions necessary to test a UUT. Sequence files also contain the definitions for the data types and step types that are used by the sequences in the file. The sequence editor is used to create and edit sequence files. Sequences can be executed from the sequence editor or from any other TestStand operator interface program.

The various types of sequence files include normal sequence files, model sequence files, station callback sequence files, and front-end callback sequence files. Normal sequence files contain sequences that test UUTs, and these are the most common type. Model sequence files contain process model sequences. Station callback sequence files contain station callback sequences, and front-end callback sequence files contain front-end callback sequences.

Storage of Types in Files

Each sequence file contains the definitions of all the property and step types that the variables, parameters, and steps in the sequence file use. This is true for all TestStand files that use types.

In memory, the TestStand Engine 220 allows only one definition for each type. If the user loads a file that contains a type definition and a type definition of the same name already exists in memory, the TestStand Engine 220 verifies that the two type definitions are identical. If they are not identical, the TestStand Engine 220 informs the user of the conflict and/or automatically selects the appropriate type. The user can select one of the definitions to replace the other, or the user can rename one of them so that they can coexist. The operation of type conflict resolution is discussed below.

Process Models

Testing a Unit Under Test (UUT) requires more than just executing a set of tests. Typically, the test executive must perform a series of operations before and after it executes the sequence that performs the tests. Common operations include identifying the UUT, notifying the operator of pass/fail status, generating a test report, and logging results. These operations define the testing "process". The set of such operations and their flow of execution is called a "process model". Some traditional test executives implement their process model functionality internally and do not allow user modification. Other test executives do not define a process model at all. TestStand comes with a default process model that the user can easily modify or replace. The TestStand process model is a modular and user configurable entity.

The process model of the present invention provides tremendous savings in user development time. The process model enables the user to write different test sequences without repeating standard testing operations in each sequence. The process model is also user-modifiable. This is important because the testing process can vary based on the production line, the production site, or the systems and practices of the company in which the test executive is used.

TestStand provides a mechanism for defining a process model. A process model is in the form of a sequence file. The user can edit a process model just as he/she edits other sequences. TestStand ships with a fully functional default process model. The user can write his/her own process model, or the user can copy the default process model and then modify it.

Station Model

The user can select a process model file to use for all sequence files. This process model file is called the "station model" file. The TestStand installation program establishes "TestStandModel.seq" as the station model file. The user can use the Station Options dialog box to select a different station model. The user can also use the Station Options dialog box to allow individual sequence files to specify their own process model file, but typically this is not necessary.

Main Sequence and Client Sequence File

In TestStand, the sequence that initiates the tests on a UUT is called the "main sequence". The user preferably names each main sequence "MainSequence". When the user creates a new sequence file, TestStand automatically inserts a MainSequence sequence in the file. The process model invokes the main sequence as part of the overall testing process. The process model defines what is constant about the user's testing process, whereas main sequences define the steps that are unique to the different types of tests that are run.

An execution is typically begun from a main sequence in one of the user-created sequence files. TestStand determines which process model file to use with the main sequence. TestStand uses the station model file unless the sequence file specifies a different process model file and the Station Options is set to allow sequence files to override the user's station model setting.

After TestStand identifies the process model to use with a main sequence, the file that contains the main sequence becomes a "client sequence file," also referred to as a test sequence file, of the process model.

Model Callbacks

By default, each main sequence that is executed uses the process model selected by the user for the entire test station. TestStand has a mechanism called a model callback that allows the sequence developer to customize the behavior of a process model for each main sequence that uses it. By defining one or more model callbacks in a process model, the user specifies the set of process model operations that the sequence developer can customize.

The user defines a model callback by adding a sequence to the process model file, marking it as a callback, and calling it from the process model. The sequence developer can override the callback in the model sequence file by using the Sequence File Callbacks dialog to create a sequence of the same name in the client sequence file (test sequence file.

For example, the default TestStand process model defines a "TestReport" callback that generates the test report for each UUT. Normally, the TestReport callback in the default process model file is sufficient because it handles many types of test results. The sequence developer can, however, override the default TestReport callback by defining a different TestReport callback in a particular client sequence file.

Process models use callbacks to invoke the main sequence in the client sequence file. Each client sequence file must define a sequence by the name of "MainSequence". The process model contains a MainSequence callback that is merely a placeholder. The MainSequence in the client sequence file overrides the MainSequence placeholder in the model file.

To alter the behavior of the process model for all sequences, the user can modify the process model or replace it entirely. To redefine the set of customizable operations, the user can define new callbacks to, or delete existing callbacks from, the process model file.

Entry Points

A process model defines a set of "entry points". Each entry point is a sequence in the process model file. The user marks a sequence in the model file as an entry point in the Sequence Properties dialog box.

By defining multiple entry points in a process model, the test station operator has different ways to invoke a main sequence. For example, the default TestStand process model provides two entry points: "Test UUTs" and "Single Pass". The Test UUTs entry point initiates a loop that repeatedly identifies and tests UUTs. The Single Pass entry point tests a single UUT without identifying it. Such entry points are called "execution entry points". Execution entry points appear in the Execute menu of the sequence editor or operator interface when the active window contains a non-model sequence file that has a MainSequence callback.

FIG. 3 is a flowchart of the major operations of the "Test UUTs" entry point sequence in the default process model. As shown, the sequence implements many of its operations as callbacks. The box on the left shows the flow of control. The box on the right shows the action that each callback in the default model performs if the user does not override it.

Like any other sequence, the sequence for a process model entry point can contain calls to DLLs, calls to subsequences, Goto steps, and so on. FIG. 4 shows the entire set of steps for the Test UUTs entry point in the default process model.

The user can execute a sequence without a process model by selecting the Run <Sequence Name> item in the "Execute" menu, where <Sequence Name> is the name of the sequence the user is currently viewing. This option is useful for debugging. It executes the sequence directly, skipping the process model operations such as UUT identification and test report generation. Any sequence can be executed this way, not just main sequences.

A process model can define other types of entry points, such as "configuration entry points". An example is the "Report Options" entry point, which appears as "Report Options" in the View menu of the sequence editor or run-time operator interface. Process model entry points are discussed further below.

FIG. 5 shows a list of all the sequences in the default TestStand process model. The first three sequences are entry points. The last sequence is a utility subsequence that the execution entry points call. The other sequences are callbacks that the user can override in a client sequence file.

Process models are discussed in greater detail below.

Automatic Result Collection

TestStand can automatically collect the results of each step. The user can enable or disable result collection for a particular sequence or for the entire test station.

Each sequence has a local array that stores the results of each step. The contents in the results for each can vary depending on the step type. When TestStand stores the results for a step into the array, it adds information such as the name of the step and its position in the sequence. For a step that calls a sequence, TestStand also adds the result array from the subsequence. TestStand result collection is discussed further below.

Callback Sequences

Callbacks are sequences that TestStand calls under specific circumstances. The user can create new callback sequences or replace existing callbacks to customize the operation of the test station. The Sequence File Callbacks dialog box is used to add a callback sequence to a sequence file.

TestStand defines three categories of callbacks. The categories are based on the entity that invokes the callback and the location in which the callback is defined. The following table titled "Callback Types" shows the different types of callbacks.

                          Callback Types
                  Where the User Defines the
    Callback Type Callback               Who Calls the Callback
    Model         Process model file or client Sequences in the
    Callbacks     sequence file          process model file
    Engine        StationCallbacks.seq,  Engine
    Callbacks     the process model file, or
                  a regular sequence file
    Front-End     FrontEndCallbacks.seq  Operator interface program
    Callbacks

                              TABLE
                     Adapter Dialog Box Names
    Adapter                             Dialog Box Title
    DLL flexible Prototype Adapter      Edit DLL Call
    LabVIEW Standard Prototype Adapter  Edit LabVIEW VI Call
    C/CVI Standard Prototype Adapter    Edit C/CVI Module Call
    Sequence Adapter                    Edit Sequence Call
    ActiveX Automation Adapter          Edit Automation Call