Method and system for computer based testing using an amalgamated resource file

Abstract

A system for computer-based testing for producing a test and delivering the test to an examinee includes a storage device that has a first storage location, which stores a first segment of a test definition language, and a second storage location, which stores a second segment of the test definition language, a validation expansion module that validates the first segment and the second segment of the test definition language, a test packager that amalgamates the first storage location and the second storage location and transmits the amalgamated segment to the validation expansion module such that the validation expansion module can determine whether the amalgamated segment forms a complete and valid set, and a test driver that has an executable code that controls functionality that enables the test driver to deliver the test to an examinee. A method of computer-based testing includes validating a first segment of the test definition language, amalgamating the first segment and the second segment of the test definition language, validating an amalgamated segment, such that the amalgamated segment is valid if the amalgamated segment forms a complete and valid set, and amalgamating the first segment and the second segment of the test definition language during a test delivery cycle.

Claims

1. A system for computer based testing for at least one test, the at least one test having a presentation format and data content, comprising:

a storage device comprising a first storage location and a second storage location, the first storage location storing a first segment of a test definition language and the second storage location storing a second segment of the test definition language, wherein the first segment and the second segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and results reporting of the at least one test;

a validation expansion module, in operative data communication with the storage device, that validates the first segment and the second segment of the test definition language by determining whether the first segment and the second segment are correctly formatted, and stores the first segment to one of the first storage location and the second storage location and the second segment to another one of the first storage location and the second storage location in the storage device;

a test packager, in operative data communication with the storage device and the validation expansion module, that, during production of the at least one test, transmits the first segment and the second segment of the test definition language to the validation expansion module, determines to which of the first storage location and the second storage location in the storage device the first segment and the second segment are stored by the validation expansion module, amalgamates the first storage location and the second storage location and stores an amalgamated segment of the test definition language in a first virtual storage location, and transmits the amalgamated segment to the validation expansion module such that the validation expansion module is capable of determining whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language; and

a test driver, in operative data communication with the storage device and the validation expansion module, comprising an executable code that controls functionality that enables the test driver to deliver the at least one test to an examinee using a display device, manage the at least one test, control the progression of the at least one test, control the scoring of the at least one test, control the printing of the at least one test, control the timing of the at least one test, and control the results reporting of the at least one test based on the test definition language, the test driver, during delivery of the at least one test, amalgamating the first storage location and the second storage location into a second virtual storage location such that the validation expansion module is capable of retrieving the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.

2. The system of claim 1, wherein the test definition language comprises extensible markup language format and wherein the test packager comprises a compiler.

3. The system of claim 1, wherein the first segment and the second segment of the test definition language further comprise a same category of information, the same category of information comprising at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.

4. The system of claim 2, wherein the same category of information further comprises at least one of non-interactive display material, test navigation, test navigation controls, items, timing, selection, scoring, results, and reporting.

5. The system of claim 1,

wherein the validation expansion module comprises a plug-in,

wherein the first segment of the test definition language defines the plug-in, and

wherein the second segment of the test definition language defines a usage of the plug-in on at least one unit of the at least one test.

6. The system of claim 5, wherein the similar feature defined by the first segment and the second segment of the test definition language further comprises at least one of test navigation, timing, selection, scoring, results, and reporting.

7. The system of claim 1, further comprising a function interface that enables communication between the test packager and the validation expansion module, wherein the function interface enables the test packager to transmit the first segment and the second segment of the test definition language to the validation expansion module such that the validation expansion module is capable of validating the first segment and the second segment.

8. The system of claim 1, further comprising a persistence interface that enables communication between the validation expansion module and the storage device such that the validation expansion module is capable of storing the first segment to one of the first storage location and the second storage location and the second segment of the test definition language to another one of the first storage location and the second storage location in the storage device.

9. The system of claim 8, wherein the persistence interface enables storing of the first segment and the second segment as one of a set of data and a directory.

10. The system of claim 8, wherein the persistence interface further enables the test packager to transmit the amalgamated segment from the storage device to the validation expansion module such that the validation expansion module is capable of validating the amalgamated segment.

11. The system of claim 8, wherein the persistence interface further enables the validation expansion module to retrieve the amalgamated segment from the storage device such that the validation expansion module is capable of enabling the functionality of the test driver.

12. The system of claim 1, further comprising a function interface that enables communication between the test packager and the validation expansion module such that the test packager is capable of transmitting the amalgamated segment to the validation expansion module and such that the validation expansion module is capable of determining whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language.

13. The system of claim 1, further comprising a function interface that enables communication between the test driver and the validation expansion module such that such that the test packager is capable of instructing the validation expansion module to retrieve the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.

14. The system of claim 1, wherein the storage device further comprises a third storage location and the test definition language further comprises a third segment:

wherein the validation module validates the third segment and stores the first segment to one of the first storage location, the second storage location, and the third storage location, stores the second segment to another one of the first storage location, the second storage location, and stores the third storage location, and the third segment of the test definition language to another one of the first storage location, the second storage location, and the third storage location,

wherein the test packager, during production of the at least one test, transmits the third segment to the validation expansion module such that the validation expansion module is capable of validating the third segment, determines to which of the first storage location, the second storage location, and the third storage location in the storage device the first segment, the second segment, and the third segment of the test definition language are stored by the validation expansion module, amalgamates the first storage location, the second storage location, and the third storage location and stores an amalgamated segment of the test definition language in a first virtual storage location, and transmits the amalgamated segment to the validation expansion module such that the validation expansion module is capable of determining a complete and valid set of the first segment, the second segment, and the third segment of the test definition language, and

wherein the test driver, during delivery of the at least one test, amalgamates the first storage location, the second storage location, and the third storage location into the second virtual storage location such that the validation expansion module is capable of retrieving the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.

15. The system of claim 14:

wherein the validation expansion module comprises a plug-in,

wherein the first segment of the test definition language defines the plug-in,

wherein the second segment of the test definition language defines an area in a template in which the plug-in is to be used, the template determining a visual presentation of the at least one test on the display device, and

wherein the third segment of the test definition language defines a presentation in which the plug-in is to be used, the presentation determining the visual presentation of the at least one test on the display device at a particular instance during the at least one test.

16. The system of claim 14, wherein the first segment, the second segment, and the third segment of the test definition language further comprise a same category of information, the same category of information comprising at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.

17. The system of claim 16, wherein the same category of information further comprises at least one of non-interactive display material, test navigation controls, and items.

18. The system of claim 14:

wherein the validation expansion module comprises a plug-in,

the first segment of the test definition language defines the plug-in;

wherein the second segment of the test definition language defines an area in a template in which the plug-in is to be used, the template determining a visual presentation of the at least one test on the display device,

wherein the third segment of the test definition language defines an item, which includes at least one question delivered to the examinee during the at least one test, and

wherein the similar feature defined by the first segment, the second segment, and the third segment of the test definition language is items.

19. The system of claim 14, further comprising a function interface that enables communication between the test packager and the validation expansion module, wherein the function interface enables the test packager to transmit the first segment, the second segment, and the third segment of the test definition language to the validation expansion module such that the validation expansion module is capable of validating the first segment, the second segment, and the third segment.

20. The system of claim 14, further comprising a persistence interface that enables communication between the validation expansion module and the storage device such that the validation expansion module is capable of storing the first segment to one of the first storage location and the second storage location and the second segment of the test definition language to another one of the first storage location and the second storage location in the storage device.

21. The system of claim 20, wherein the persistence interface enables storing of the first segment, the second segment, and the third segment as one of a set of data and a directory.

22. The system of claim 20, wherein the persistence interface further enables the test packager to transmit the amalgamated segment from the storage device to the validation expansion module such that the validation expansion module is capable of validating the amalgamated segment.

23. The system of claim 20, wherein the persistence interface further enables the validation expansion module to retrieve the amalgamated segment from the storage device such that the validation expansion module is capable of enabling the functionality of the test driver.

24. The system of claim 14, further comprising a function interface that enables communication between the test packager and the validation expansion module such that the test packager is capable of transmitting the amalgamated segment to the validation expansion module and such that the validation expansion module is capable of determining whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language.

25. The system of claim 14, further comprising a function interface that enables communication between the test driver and the validation expansion module such that such that the test packager is capable of instructing the validation expansion module to retrieve the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.

26. A system for computer based testing for at least one test, the at least one test having a presentation format and data content, comprising:

a storage device comprising a first storage location, a second storage location, and a third storage location, the first storage location storing a first segment of a test definition language, the second storage location storing a second segment of the test definition language, and the third storage location storing a third segment of the test definition language, wherein the first segment, the second segment, and the third segment comprise at least one of the data content, the presentation format, progression, scoring, printing, timing, and results reporting of the at least one test;

a validation expansion module, in operative data communication with the storage device, that validates the first segment, the second segment, and the third segment of the test definition language, and stores the first segment to one of the first storage location, the second storage location, and the third storage location, the second segment to another one of the first storage location, the second storage location, and the third storage location, and the third segment to another one of the first storage location, the second storage location, and the third storage location;

a test packager, in operative data communication with the storage device and the validation expansion module, that, during production of the at least one test, transmits the first segment, the second segment, and the third segment of the test definition language to the validation expansion module such that the validation expansion module is capable of validating the first segment, the second segment, and the third segment, determines to which of the first storage location, the second storage location, and the third storage location in the storage device the first segment, the second segment, and the third segment are stored by the validation expansion module, amalgamates the first storage location, the second storage location, and the third storage location and stores an amalgamated segment of the test definition language in a first virtual storage location, and transmits the amalgamated segment to the validation expansion module such that the validation expansion module is capable of determining a complete and valid set of the first segment, the second segment, and the third segment of the test definition language; and

a test driver, in operative data communication with the storage device and the validation expansion module, comprising an executable code that controls functionality performed by the test driver that enables the test driver to deliver the at least one test to an examinee using a display device, manage the at least one test, control progression of the at least one test, control scoring of the at least one test, control printing of the at least one test, control timing of the at least one test, and control reporting of test results based on the test definition language, the test driver, during delivery of the at least one test, amalgamating the first storage location, the second storage location, and the third storage location into a second virtual storage location such that the validation expansion module is capable of retrieving the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.

27. The system of claim 26,

wherein the validation expansion module comprises a plug-in,

wherein the first segment of the test definition language defines the plug-in,

wherein the second segment of the test definition language defines an area in a template in which the plug-in is to be used, the template determining a visual presentation of the at least one test on the display device, and

wherein the third segment of the test definition language defines a presentation in which the plug-in is to be used, the presentation determining the visual presentation of the at least one test on the display device at a particular instance during the at least one test.

28. The system of claim 26, wherein the first segment, the second segment, and the third segment of the test definition language further comprise a same category of information, the same category of information comprising at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.

29. The system of claim 28, wherein the same category of information further comprises at least one of non-interactive display material, test navigation controls, and items.

30. The system of claim 26:

wherein the validation expansion module comprises a plug-in,

wherein the first segment of the test definition language defines the plug-in,

wherein the second segment of the test definition language defines an area in a template in which the plug-in is to be used, the template determining a visual presentation of the at least one test on the display device

wherein the third segment of the test definition language defines an item, which includes at least one question delivered to the examinee during the at least one test, and

wherein the first segment, the second segment, and the third segment further comprise a same category of information, the same category of information comprising items.

31. The system of claim 26, further comprising a function interface that enables communication between the test packager and the validation expansion module, wherein the function interface enables the test packager to transmit the first segment, the second segment, and the third segment of the test definition language to the validation expansion module such that the validation expansion module is capable of validating the first segment, the second segment, and the third segment.

32. The system of claim 26, further comprising a persistence interface that enables communication between the validation expansion module and the storage device such that the validation expansion module is capable of storing the first segment to one of the first storage location and the second storage location and the second segment of the test definition language to another one of the first storage location and the second storage location in the storage device.

33. The system of claim 32, wherein the persistence interface enables storing of the first segment, the second segment, and the third segment as one of a set of data and a directory.

34. The system of claim 32, wherein the persistence interface further enables the test packager to transmit the amalgamated segment from the storage device to the validation expansion module such that the validation expansion module is capable of validating the amalgamated segment.

35. The system of claim 32, wherein the persistence interface further enables the validation expansion module to retrieve the amalgamated segment from the storage device such that the validation expansion module is capable of enabling the functionality of the test driver.

36. The system of claim 26, further comprising a function interface that enables communication between the test packager and the validation expansion module such that the test packager is capable of transmitting the amalgamated segment to the validation expansion module and such that the validation expansion module is capable of determining whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language.

37. The system of claim 26, further comprising a function interface that enables communication between the test driver and the validation expansion module such that such that the test packager is capable of instructing the validation expansion module to retrieve the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.

38. A system for computer based testing for at least one test, the at least one test having a presentation format and data content, comprising:

first storage means for storing a first segment of a test definition language;

second storage means for storing a second segment of the test definition language, wherein the first segment and the second segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and results reporting of the at least one test;

validation means, in operative data communication with the first storage means and the second storage means, for validating the first segment and the second segment of the test definition language, and storing the first segment to one of the first storage means and the second storage means and the second segment to another one of the first storage means and the second storage means;

test packager means, in operative data communication with the first storage means, the second storage means, and the validation means, for, during production of the at least one test, transmitting the first segment and the second segment of the test definition language to the validation means such that the validation means is capable of validating the first segment and the second segment, determining to which of the first storage means and the second storage means the first segment and the second segment are stored by the validation means, amalgamating the first storage means and the second storage means and storing an amalgamated segment of the test definition language in a third storage means, and transmitting the amalgamated segment to the validation means such that the validation means is capable of determining a complete and valid set of the first segment and second segment of the test definition language; and

test driver means, in operative data communication with the first storage means, the second storage means, and the validation means, for amalgamating the first storage means and the second storage means into a third storage means such that the validation means is capable of retrieving the amalgamated segment from the third storage means, the test driver means comprising an executable code that controls functionality performed by the test driver means that enables the test driver means to deliver the at least one test to an examinee using a display device, manage the at least one test, control progression of the at least one test, control scoring of the at least one test, control printing of the at least one test, control timing of the at least one test, and control reporting of test results based on the test definition language, wherein the validation means enables the functionality of the test driver means.

39. The system of claim 38, wherein the first segment and the second segment of the test definition language further comprise a same category of information, the same category of information comprising at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.

40. The system of claim 38, further comprising function interface means that enables communication between the test packager means and the validation means.

41. The system of claim 38, further comprising persistence interface means that enables communication between the validation means and the first storage means and the second storage means.

42. The system of claim 38, further comprising function interface means that enables communication between the test packager means and the validation means such that the test packager means is capable of transmitting the amalgamated segment to the validation means and such that the validation means is capable of determining whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language.

43. The system of claim 38, further comprising function interface means that enables communication between the test driver means and the validation means such that such that the test packager means is capable of instructing the validation means to retrieve the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.

44. A system for computer based testing for at least one test, the at least one test having a presentation format and data content, comprising:

first storage means for storing a first segment of a test definition language;

second storage means for storing a second segment of the test definition language;

third storage means for storing a third segment of the test definition language, wherein the first segment, the second segment, and the third segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and results reporting of the at least one test;

validation means, in operative data communication with the first storage means, the second storage means, and the third storage means, for validating the first segment, the second segment, and the third segment of the test definition language, and storing the first segment to one of the first storage means, the second storage means, and the third storage means, the second segment to another one of the first storage means, the second storage means, and the third storage means, and the third segment to another one of the first storage means, the second storage means, and the third storage means;

test packager means, in operative data communication with the first storage means, the second storage means, the third storage means, and the validation means, for, during production of the at least one test, transmitting the first segment, the second segment, and the third segment of the test definition language to the validation means such that the validation means is capable of validating the first segment, the second segment, and the third segment, determining to which of the first storage means, the second storage means, and the third storage means the first segment, the second segment, and the third segment are stored by the validation means, amalgamating the first storage means, the second storage means, and the third storage means and storing an amalgamated segment of the test definition language in a fourth storage means, and transmitting the amalgamated segment to the validation means such that the validation means is capable of determining a complete and valid set of the first segment, the second segment, and the third segment of the test definition language; and

test driver means, in operative data communication with the first storage means, the second storage means, the third storage means, and the validation means, for, during delivery of the at least one test, amalgamating the first storage means, the second storage means, and the third storage means into a fourth storage means such that the validation means is capable of retrieving the amalgamated segment from the fourth storage means, the test driver means comprising an executable code that controls functionality performed by the test driver means that enables the test driver means to deliver the at least one test to an examinee using a display device, manage the at least one test, control progression of the at least one test, control scoring of the at least one test, control printing of the at least one test, control timing of the at least one test, and control reporting of test results based on the test definition language, wherein the validation means enables the functionality of the test driver means.

45. The system of claim 44, wherein the first segment, the second segment, and the third segment of the test definition language further comprise a same category of information, the same category of information comprising at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.

46. The system of claim 44, further comprising function interface means that enables communication between the test packager means and the validation means.

47. The system of claim 44, further comprising persistence interface means that enables communication between the validation means and the first storage means, the second storage means, and the third storage means.

48. The system of claim 44, further comprising function interface means that enables communication between the test packager means and the validation means such that the test packager means is capable of transmitting the amalgamated segment to the validation means and such that the validation means is capable of determining whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language.

49. The system of claim 44, further comprising function interface means that enables communication between the test driver means and the validation means such that such that the test packager means is capable of instructing the validation means to retrieve the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.

50. A method for computer based testing for at least one test, the at least one test having a presentation format and data content, the at least one test being controlled by a test driver, the test driver having an executable code that controls the test driver and functionality that enables the test driver to deliver the at least one test to an examinee using a display device, manage the at least one test, control progression of the at least one test, control scoring of the at least one test, control printing of the at least one test, control timing of the at least one test, and control results reporting of the at least one test based on a test definition language, the test definition language comprising a plurality of segments, the method comprising the steps of:

validating a first segment of the test definition language during a test production cycle;

validating a second segment of the test definition language during the test production cycle, wherein the first segment and the second segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and results reporting of the at least one test;

amalgamating the first segment and the second segment of the test definition language during the test production cycle, wherein an amalgamated segment is formed;

validating the amalgamated segment during the test production cycle, wherein a validated amalgamated segment is created and wherein the amalgamated segment is valid if the amalgamated segment forms a complete and valid set; and

amalgamating the first segment and the second segment of the test definition language during a test delivery cycle, wherein the validated amalgamated segment is reformed and retrieved by a validation expansion module to enable the functionality of the test driver.

51. The method of claim 50, wherein the first segment and the second segment of the test definition language comprise a same category of information, the same category of information being at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.

52. The method of claim 51, wherein the same category of information further comprises at least one of test navigation, timing, selection, scoring, printing, timing, results, and reporting.

53. The method of claim 50, further comprising the step of instantiating the validation expansion module during the test production cycle, wherein validating the first segment and the second segment of the test definition language is performed by the validation expansion module.

54. The method of claim 53, wherein instantiating the validation expansion module is facilitated by standard Microsoft object instantiation using a component object model server.

55. The method of claim 53, wherein the first segment and the second segment of the test definition language are transmitted to the validation expansion module by a test packager and wherein transmitting to the validation expansion module is facilitated by a function interface.

56. The method of claim 55, wherein the test packager comprises a compiler.

57. The method of claim 50, wherein the test definition language comprises extensible markup language format, validating the first segment and the second segment of the test definition language further comprising the step of determining whether the first segment and the second segment are correctly formatted.

58. The method of claim 57, wherein a correct format for the first segment and the second segment of the test definition language is defined in a schema.

59. The method of claim 57, further comprising the step of instantiating the validation expansion module during the test production cycle, wherein validating the first segment and the second segment of the test definition language is performed by the validation expansion module, instantiating the validation expansion module further comprising the step of calling the validation expansion module using a product identification defined in extensible markup language in the test definition language.

60. The method of claim 50, further comprising the step of storing the first segment to one of a first storage location and a second storage location and the second segment to another one of the first storage location and the second storage location in a storage device after the first segment and the second segment have been validated.

61. The method of claim 60,

wherein the first segment and the second segment of the test definition language are stored the storage device by the validation expansion module, and

wherein a test packager determines into which of the first storage location and the second storage location the first segment and the second segment are stored.

62. The method of claim 60, wherein storing the first storage location and the second storage location into the storage device is facilitated by a persistence interface.

63. The method of claim 60, wherein the test packager comprises a compiler.

64. The method of claim 50, amalgamating the first segment and the second segment of the test definition language further comprising the steps of:

combining the first segment and the second segment of the test definition language according to an amalgamation rule; and

storing the amalgamated segment in a virtual storage location in a storage device.

65. The method of claim 50, further comprising the step of instantiating the validation expansion module during the test production cycle, wherein validating the amalgamated segment is performed by the validation expansion module.

66. The method of claim 65, wherein instantiating the validation expansion module is facilitated by standard Microsoft object instantiation using a component object model server.

67. The method of claim 65, wherein an amalgamated interface is retrieved from a virtual storage object in a storage device by the validation expansion module and wherein communication between the virtual storage location and the validation expansion module is facilitated by a function interface.

68. The method of claim 65, wherein the first segment of the test definition language is stored to one of a first storage location and a second storage location and the second segment of the test definition language to another one of the first storage location and the second storage location in the storage device, the first segment and the second segment forming the amalgamated segment, wherein the first segment and the second segment are retrieved from the storage device by the validation expansion module in amalgamated form using a persistence interface.

69. The method of claim 65, wherein the validation expansion module determines what requirements are necessary for the amalgamated segment to form a complete and valid set.

70. The method of claim 50, further comprising the step of instantiating the validation expansion module during the test production delivery cycle, wherein validating the amalgamated segment is performed by the validation expansion module.

71. The method of claim 70, further comprising the step of loading the validated amalgamated segment into the validation expansion module, wherein the validated amalgamated segment is retrieved from a virtual storage location in a storage device by the validation expansion module and wherein communication between the virtual storage location and the validation expansion module is facilitated by a function interface.

72. The method of claim 70, wherein the first segment of the test definition language is stored to one of a first storage location and a second storage location and the second segment of the test definition language to another one of the first storage location and the second storage location in the storage device, the first segment and the second segment forming the amalgamated segment, wherein the first segment and the second segment are retrieved from the storage device by the validation expansion module in amalgamated form using a persistence interface.

73. A method for computer based testing for at least one test, the at least one test having a presentation format and data content, the at least one test being controlled by a test driver, the test driver having an executable code that controls the test driver and functionality that enables the test driver to deliver the at least one test to an examinee using a display device, manage the at least one test, control progression of the at least one test, control scoring of the at least one test, control printing of the at least one test, control timing of the at least one test, and control results reporting of the at least one test based on a test definition language, the test definition language comprising a plurality of segments, the method comprising the steps of:

validating a first segment of the test definition language during a test production cycle;

validating a second segment of the test definition language during the test production cycle;

validating a third segment of the test definition language, wherein the first segment, the second segment, and the third segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and results reporting of the at least one test;

amalgamating the first segment, the second segment, and the third segment of the test definition language during the test production cycle, wherein an amalgamated segment is formed;

validating the amalgamated segment during the test production cycle, wherein a validated amalgamated segment is created and wherein the amalgamated segment is valid if the amalgamated segment forms a complete and valid set; and

amalgamating the first segment, the second segment, and the third segment of the test definition language during a test delivery cycle, wherein the validated amalgamated segment is reformed and retrieved by a validation expansion module to enable the functionality of the test driver.

74. The method of claim 73, further comprising the step of instantiating the validation expansion module during the test production cycle, wherein validating the first segment, the second segment, and the third segment of the test definition language is performed by the validation expansion module.

75. The method of claim 74, wherein instantiating the validation expansion module is facilitated by standard Microsoft object instantiation using a component object model server.

76. The method of claim 74, wherein the first segment, the second segment, and the third segment of the test definition language are transmitted to the validation expansion module by a test packager and wherein transmitting to the validation expansion module is facilitated by a function interface.

77. The method of claim 76, wherein the test packager comprises a compiler.

78. The method of claim 73, wherein the test definition language comprises extensible markup language format, validating the first segment, the second segment, and the third segment of the test definition language further comprising the step of determining whether the first segment, the second segment, and the third segment are correctly formatted.

79. The method of claim 78, wherein a correct format for the first segment, the second segment, and the third segment of the test definition language is defined in a schema.

80. The method of claim 78, further comprising the step of instantiating the validation expansion module during the test production cycle, wherein validating the first segment, the second segment, and the third segment of the test definition language is performed by the validation expansion module, instantiating the validation expansion module further comprising the step of calling the validation expansion module using a product identification defined in extensible markup language in the test definition language.

81. The method of claim 73, further comprising the step of storing the first segment to one of a first storage location, a second storage location, and a third storage location, storing the second segment to another one of the first storage location, the second storage location, and the third storage location, and storing the third segment of the test definition language to another one of the first storage location, the second storage location, and the third storage location after the first segment, the second segment, and the third segment have been validated.

82. The method of claim 81,

wherein the first segment is stored to one of a first storage location, a second storage location, and a third storage location, the second segment is stored to another one of the first storage location, the second storage location, and the third storage location, and the third segment is stored to another one of the first storage location, the second storage location, and the third storage location by the validation expansion module, and

wherein a test packager determines into which of the first storage location, the second storage location, and the third storage location the first segment, the second segment, and the third segment are stored.

83. The method of claim 82, wherein storing the the first storage location, the second storage location, and the third storage location into the storage device is facilitated by a persistence interface.

84. The method of claim 82, wherein the test packager comprises a compiler.

85. The method of claim 73, amalgamating the first segment, the second segment, and the third segment of the test definition language further comprising the steps of:

combining the first segment, the second segment, and the third segment of the test definition language according to an amalgamation rule; and

storing the amalgamated segment in a virtual storage location in a storage device.

86. The method of claim 73, further comprising the step of instantiating the validation expansion module during the test production cycle, wherein validating the amalgamated segment is performed by the validation expansion module.

87. The method of claim 86, wherein instantiating the validation expansion module is facilitated by standard Microsoft object instantiation using a component object model server.

88. The method of claim 86, wherein an amalgamated interface is retrieved from a virtual storage object in a storage device by the validation expansion module and wherein communication between the virtual storage location and the validation expansion module is facilitated by a function interface.

89. The method of claim 86, wherein the first segment is stored to one of a first storage location, a second storage location, and a third storage location, the second segment is stored to another one of the first storage location, the second storage location, and the third storage location, and the third segment is stored to another one of the first storage location, the second storage location, and the third storage location, and the first segment, the second segment, and the third segment of the test definition language forming the amalgamated segment, and wherein the first segment, the second segment, and the third segment are retrieved from the storage device by the validation expansion module in amalgamated form using a persistence interface.

90. The method of claim 86, wherein the validation expansion module determines what requirements are necessary for the amalgamated segment to form a complete and valid set.

91. The method of claim 73, further comprising the step of instantiating the validation expansion module during the test delivery cycle, wherein validating the amalgamated segment is performed by the validation expansion module.

92. The method of claim 91, further comprising the step of loading the validated amalgamated segment into the validation expansion module, wherein the validated amalgamated segment is retrieved from a virtual storage location in a storage device by the validation expansion module and wherein communication between the virtual storage location and the validation expansion module is facilitated by a function interface.

93. The method of claim 91, wherein the first segment is stored to one of a first storage location, a second storage location, and a third storage location, the second segment is stored to another one of the first storage location, the second storage location, and the third storage location, and the third segment is stored to another one of the first storage location, the second storage location, and the third storage location, the first segment, the second segment, and the third segment of the test definition language forming the validated amalgamated segment, and wherein the first segment, the second segment, and the third segment are retrieved from the storage device by the validation expansion module in amalgamated form using a persistence interface.

94. The system of claim 73, wherein the first segment, the second segment, and the third segment of the test definition language further comprise a same category of information, the same category of information being at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the at least one test.

95. The method of claim 94, wherein the same category further comprises at least one of non-interactive display material, test navigation controls, and items.

Description

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention generally relates to the field of computer-based testing, and in particular, the present invention relates to amalgamating an exam resource file to combine first defined and later defined test content and specification of a computer-based test to reduce the amount of authored XXL and to reduce the size of the size of the exam resource file during delivery of the computer-based test to an examinee.

2. Background of the Related Art

For many years, standardized testing has been a common method of assessing examinees as regards educational placement, skill evaluation, etc. Due to the prevalence and mass distribution of standardized tests, computer based testing has emerged as a superior method for providing standardized tests, guaranteeing accurate scoring, and ensuring prompt return of test results to examinees.

Tests are developed based on the requirements and particulars of test developers. Typically, test developers employ psychometricians or statisticians and psychologists to determine the specific requirements specific to human assessment. These experts often have their own, unique ideas regarding how a test should be presented and regarding the necessary contents of that test, including the visual format of the test as well as the data content of the test. Therefore, a particular computer based test has to be customized to fulfill the client's requirements.

FIG. 1 illustrates a prior art process for computerized test customization, denoted generally by reference numeral 10. First, a client details the desired test requirements and specifications, step 12. The computerized test publisher then creates the tools that allow the test publisher to author the items, presentations, etc., required to fulfill the requirements, step 14. The test publisher then writes an item viewer, which allows the test publisher to preview what is being authored, step 16.

An item presenter is then written to present the new item, for example, to the test driver, step 18. Presenting the new item to the test driver requires a modification of the test driver's executable code. The test driver must be modified so that it is aware of the new item and can communicate with the new item presenter, step 20. The test packager must then also be modified, step 22. The test packager, which may also be a compiler, takes what the test publisher has created and writes the result as new object codes for the new syntax. Subsequently, the scoring engine must also be modified to be able to score the new item type, step 24. Finally, the results processor must be modified to be able to accept the new results from the new item, step 26. This process requires no less than seven software creations or modifications to existing software.

During authoring of the test, the test author might desire to define some portion of the test specification for re-use. Alternatively the test author might redefine or override certain portions of the test specification that were defined early in the test specification. Previous test definition languages attempted to solve this problem by repeating selected pieces of the test specification at a higher level of the delivery hierarchy. However, it has been determined that this solution does not cover the content of the test specification and assumes that the pattern of repetition is based on the unit of delivery. For example, test specification based on the order of delivery and the hierarchy may be: a two-section exam of math followed by English. All math questions have a minimum response value of 2 and all English questions minimum response value of 3. An example that will not support repetition of test definition is a two-section exam of math and English. Where half the math questions have a minimum response of 2 and the other half of the math questions have a minimum response of 3. In this situation, the hierarchy does not match the pattern of repetition.

Furthermore, it has been determined that this solution requires specific units of the test to have detailed knowledge of shared specification and to communicate the shared specification to other units of the test.

U.S. Pat. No. 5,827,070 (Kershaw et al.) and U.S. Pat. No. 5,565,316 (Kershaw et al.) are incorporated herein by reference. The '070 and '316 patents, which have similar specifications, disclose a computer-based testing system comprising a test development system and a test delivery system. The test development system comprises a test document creation system for specifying the test contents, an item preparation system for computerizing each of the items in the test, a test preparation system for preparing a computerized test, and a test packaging system for combining all of the items and test components into a computerized test package. The computerized test package is then delivered to authorized examinees on a workstation by the test delivery system.

FIG. 2 illustrates the relationship among session scripts 30, test scripts 32, and units. A script consists of a series of files and further specifies the option settings and configuration data, which the Test Delivery Application (TDA) needs for operation. During test preparation, scripts are prepared and combined with the items prepared during item preparation. Scripts control the sequence of events during a testing session. Two types of scripts are preferably used, for example: the session script 30 and one or more test scripts 32. The session script 30 controls the order in which units within the testing session are presented. Units provide specific services to the examinee, such as delivering a test or presenting a score report. Just as the session script controls the session, the test script controls what is presented to the examinee during the testing unit. Each testing unit may include one or more delivery units, which are separately timed and scored subdivisions of a test. The system can dynamically select, or spiral, scripts and other test components so that examinees are given what appear to be different tests. FIG. 24 shows the relationship among session scripts 30, test scripts 32, and units.

The session script is the second-level component of the testing package. It performs two primary functions: First, it specifies the Session Control Information, which defines the default options that are in effect for the entire examinee testing session. Second, it controls the order in which units within the testing session are presented and the options used to present them. The units that can be presented within a session script are: General information screen units, Tutorial units, Break units, Data collection units, Scoring and Reporting units, and Testing units.

The session control information contains the default options in effect for the entire session. Control information can be provided at multiple levels within the testing session. Thus, the control information provided at the session level can be overridden by information that occurs later in the session. The information provided at the session level would generally include the following: Name—the session script name to be used by administrators in selecting a specific session script from Administrative Application menus; Input device—the input device to be used during the session (e.g., mouse or keyboard); Color—the colors to be used during the session; Messages—program-specific messages to override default messages during the session; Demo Script—indicates whether the script presents a demonstration or operational test; Research Indicator—indicates whether the script presents a research pilot test; Special Timing—indicates whether the script is standard or specially timed version.

The testing unit presents a test, based on the contents of a test script that may have been selected at runtime. The following units can be included within a testing unit: general information screen unit; tutorial unit; break unit; delivery unit, which delivers items to the examinee. This permits testing programs to interleave general information screens, tutorials, and breaks with sections of a test. The testing unit contains the following information: script selection mode indicates whether dynamic runtime selection is to be used to select the test script; reference to a test script which controls the sequence of events and options used during the testing unit. If dynamic runtime selection is to be used, the reference is to a set of test scripts. Like the session script, the test script performs two primary functions. First, it specifies the test and delivery unit control information. Test control information defines the options that are in effect for the testing unit. Delivery unit control information defines the options that are in effect for a particular delivery unit within a testing unit. It controls the order in which units are presented within the testing unit and the options used to present them. The rules for presentation of units are the same as those for the session script, except that an additional unit, the delivery unit, can be included within a test script.

U.S. Pat. No. 5,513,994 (Kershaw et al.), which is incorporated herein by reference, discloses a centralized administrative system and method of administering standardized test to a plurality of examinees. The administrative system is implemented on a central administration workstation and at least one test workstation located in different rooms at a test center. The administrative system software, which provides substantially administrative functions, is executed from the central administration workstation. The administrative system software, which provides function carried out in connection with a test session, is executed from the testing workstations.

None of the Kershaw et al. patents appear to make any mention of a test definition language that does not required the overriding language to be repeated at higher levels of the delivery hierarchy. What is required is a computer based testing system that supports a non-deterministic test definition language that, for example, allows subsequent specifications to be added to the test definition language without requiring repetitive definitions and that enables a reduction in size of exam source files, and an exam resource file, which contains the test definition language during delivery of the test. Other features and advantages in addition to the above, or in the alternative to the above, are described in the Summary of the Invention and the Detailed Descriptions provided below.

SUMMARY OF THE INVENTION

It is one feature and advantage of the present invention to amalgamate test specification and content to reduce the size of a resource file that contains the test specification and content, where a test driver and an expansion module uses the test specification and content to deliver a test to an examinee.

It is another optional feature and advantage of the present invention to allow a test publisher to define different aspects of test specification and content for a particular feature of the test in multiple locations and to amalgamate those multiple locations at the time of delivery of the test such that the test publisher does not have to repeat the same aspects if the test specification and content for the particular feature at the multiple locations.

It is another optional feature and advantage of the present invention to allow later defined elements of a particular feature to override previously defined elements that exist at a higher level of the test specification and contents.

It is another optional feature and advantage of the present invention to allow later defined elements to define previously omitted non-common test specification and contents.

It is another optional feature and advantage of the present invention that specific units of the test need not have detailed knowledge of shared specification and need not communicate the shared specification with other units of the test.

These and other features and advantages of the present invention are achieved in a system for computer-based testing for producing a test and delivering the test to an examinee. The test has a presentation format that determines the visual presentation of the test and data content that determines the functional properties of the test. The system includes a storage device that has a first storage location and a second storage location. The first storage location stores a first segment of a test definition language and the second storage location stores a second segment of the test definition language. The first segment and the second segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and/or results reporting of the test. In an alternative embodiment, the first segment and the second segment of the test definition language further comprise the same category of information, where the category is at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and/or the results reporting of the test.

The system also includes a validation expansion module that validates the first segment and the second segment of the test definition language by determining whether the first segment and the second segment are correctly formatted. The validation expansion module also stores the first segment to one of the first storage location and the second storage location and the second segment to another one of the first storage location and the second storage location in the storage device. In an alternative embodiment, the validation expansion module is a plugin.

The system further includes a test packager that transmits the first segment and the second segment of the test definition language to the validation expansion module during delivery of the test. The test packager determines to which of the first storage location and the second storage location in the storage device the first segment and the second segment are stored by the validation expansion module. The test packager also amalgamates the first storage location and the second storage location and stores an amalgamated segment of the test definition language in a first virtual storage location and transmits the amalgamated segment to the validation expansion module such that the validation expansion module can determine whether the amalgamated segment forms a complete and valid set of the first segment and second segment of the test definition language. In an alternative embodiment, the test packager is a compiler.

The system further includes a test driver that has an executable code that controls functionality that enables the test driver to deliver the test to an examinee using a display device, manage the test, control the progression of the test, control the scoring of the test, control the printing of the test, control the timing of the test, and control the results reporting of the test based on the test definition language. During delivery of the test, the test driver amalgamates the first storage location and the second storage location into a second virtual storage location such that the validation expansion module can retrieve the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.

In another embodiment of the present invention, a system for computer-based testing includes a storage device that has a first storage location, a second storage location, and a third storage location. The first storage location stores a first segment of a test definition language, the second storage location stores a second segment of the test definition language, and the third storage location stores a third segment of the test definition language. The first segment, the second segment, and the third segment comprise at least one of the data content, the presentation format, progression, scoring, printing, timing, and/or results reporting of the test. In an alternative embodiment, the first segment, the second segment, and the third segment of the test definition language further comprise the same category of information, where the category is at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the test.

The system also includes a validation expansion module that validates the first segment, the second segment, and the third segment of the test definition language, and stores the first segment to one of the first storage location, the second storage location, and the third storage location, the second segment to another one of the first storage location, the second storage location, and the third storage location, and the third segment to another one of the first storage location, the second storage location, and the third storage location. In an alternative embodiment, the validation expansion module is a plugin.

The system further includes a test packager that transmits the first segment, the second segment, and the third segment of the test definition language to the validation expansion module during production of the test such that the validation expansion module is capable of validating the first segment, the second segment, and the third segment. The test packager determines to which of the first storage location, the second storage location, and the third storage location in the storage device the first segment, the second segment, and the third segment are stored by the validation expansion module. The test packager also amalgamates the first storage location, the second storage location, and the third storage location and stores an amalgamated segment of the test definition language in a first virtual storage location, and transmits the amalgamated segment to the validation expansion module such that the validation expansion module is capable of determining a complete and valid set of the first segment, the second segment, and the third segment of the test definition language. In an alternative embodiment, the test packager is a compiler.

The system also includes a test driver that has an executable code that controls functionality performed by the test driver that enables the test driver to deliver the test to an examinee using a display device, manage the test, control progression of the test, control scoring of the test, control printing of the test, control timing of the test, and control reporting of test results based on the test definition language. During delivery of the test, the test driver amalgamates the first storage location, the second storage location, and the third storage location into a second virtual storage location such that the validation expansion module is capable of retrieving the amalgamated segment from the second virtual storage location to enable the functionality of the test driver.

In another embodiment of the present invention, a method of computer-based testing for a test is provided, where the test has a presentation format that determines the visual presentation of the test and data content that determines the functional properties of the test. Delivery of the test is controlled by a test driver that has an executable code that enables the test driver to deliver the test to an examinee using a display device, manage the test, control progression of the test, control scoring of the test, control printing of the test, control timing of the test, and control results reporting of the test.

The method includes the sequential, non-sequential, and/or sequence independent steps of validating a first segment of the test definition language during a test production cycle and validating a second segment of the test definition language during the test production cycle. The first segment and the second segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and/or results reporting of the test. In an alternative embodiment, the first segment and the second segment further define the same category of information, which is at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the test.

The method also include amalgamating the first segment and the second segment of the test definition language during the test production cycle such that an amalgamated segment is formed, validating the amalgamated segment during the test production cycle, such that a validated amalgamated segment is created and such that the amalgamated segment is valid if the amalgamated segment forms a complete and valid set. The method further includes amalgamating the first segment and the second segment of the test definition language during a test delivery cycle. The validated amalgamated segment is reformed and retrieved by a validation expansion module to enable the functionality of the test driver.

In another embodiment of the present invention, a method for computer-based testing includes validating a first segment of the test definition language during a test production cycle, validating a second segment of the test definition language during the test production cycle, and validating a third segment of the test definition language. The first segment, the second segment, and the third segment define information comprising at least one of the data content, the presentation format, progression, scoring, printing, timing, and/or results reporting of the test. In an alternative embodiment, the first segment, the second segment, and the third segment of the test definition language further define the same category of information, the same category of information being at least one of the data content, the presentation format, the progression, the scoring, the printing, the timing, and the results reporting of the test.

The method also includes amalgamating the first segment, the second segment, and the third segment of the test definition language during the test production cycle, such that an amalgamated segment is formed and validating the amalgamated segment during the test production cycle, such that a validated amalgamated segment is created and such that the amalgamated segment is valid if the amalgamated segment forms a complete and valid set. The method further includes amalgamating the first segment, the second segment, and the third segment of the test definition language during a test delivery cycle, such that the validated amalgamated segment is reformed and retrieved by a validation expansion module to enable the functionality of the test driver.

In another embodiment of the present invention, a method for computer-based testing is provided, which includes defining the presentation format and the data content in at least two locations comprising a plugin element and actual usage of the plugin element on at least one unit of the test, the at least one unit comprising form, section and group associated with the test and amalgamating the presentation format and the data content defined in the two locations by at least one test driver to deliver the test to an examinee.

In another embodiment of the present invention, a method for computer-based testing includes validating by a plugin at least partial exam source information that is received and amalgamating exam resource data associated with the test. The method also includes validating the exam resource data that has been amalgamated to provide a substantially complete amalgamated exam specification and content and delivering the substantially complete amalgamated exam specification and content validated by the validating step.

In another embodiment of the present invention, a method for computer-based testing is provided for a test that has a first presentation format, a second presentation format, a first data content, and a second data content. The method includes defining the first presentation format and the second presentation format in at least two locations comprising a plugin element and actual usage of the plugin element on at least one unit of the test, the at least one unit comprising form, section and group associated with the test. The method further includes defining the first data content and the second data content in at least two locations comprising the plugin element and the actual usage of the plugin element on at least one unit of the test, the at least one unit comprising form, section and group associated with the test. The method also includes amalgamating at least one of the first and second presentation format and the first and second data content defined in the two locations by at least one test driver to deliver the test to an examinee.

There has thus been outlined, rather broadly, the more important features of the invention and several, but not all, embodiments in order that the detailed description thereof that follows may be better understood, and in order that the present contribution to the art may be better appreciated. There are, of course, additional features of the invention that will be described hereinafter and which will form the subject matter of the claims appended hereto.

In this respect, before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not limited in its application to the details of construction and to the arrangements of the components set forth in the following description or illustrated in the drawings. The invention is capable of other embodiments and of being practiced and carried out in various ways. Also, it is to be understood that the phraseology and terminology employed herein are for the purpose of description and should not be regarded as limiting.

As such, those skilled in the art will appreciate that the conception, upon which this disclosure is based, may readily be utilized as a basis for the designing of other structures, methods and systems for carrying out the several purposes of the present invention. It is important, therefore, that the claims be regarded as including such equivalent constructions insofar as they do not depart from the spirit and scope of the present invention.

Further, the purpose of the foregoing abstract is to enable the U.S. Patent and Trademark Office and the public generally, and especially the scientists, engineers and practitioners in the art who are not familiar with patent or legal terms or phraseology, to determine quickly from a cursory inspection the nature and essence of the technical disclosure of the application. The abstract is neither intended to define the invention of the application, which is measured by the claims, nor is it intended to be limiting as to the scope of the invention in any way.

These, together with other objects of the invention, along with the various features of novelty, which characterize the invention, are pointed out with particularity in the claims annexed to and forming a part of this disclosure. For a better understanding of the invention, its operating advantages and the specific objects attained by its uses, reference should be had to the accompanying drawings and descriptive matter in which there is illustrated preferred embodiments of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram of a prior art method for computerized test customization;

FIG. 2 is a block diagram of a prior art testing script;

FIG. 3 is a schematic diagram of a computer-based testing system according to the present invention;

FIG. 4 is a block diagram illustrating different types of plugins that are used with the computer-based testing system according to the current invention;

FIG. 5 illustrates various components that comprise an exam source file;

FIGS. 6A and 6B are schematic illustrating the components, classes, and interfaces that comprise a test definition language compiler according to the present invention;

FIG. 7 is a schematic illustrating the components that comprise a test driver and a test administration system according to the present invention;

FIGS. 8A and 8B are schematics illustrating the classes and interfaces that comprise the test driver;

FIG. 9 illustrates the interfaces that comprise a structured storage according to the present invention;

FIGS. 10A and 10B are schematics illustrating the classes and interfaces that comprise the structure storage and associated operations;

FIG. 11 is a block diagram of main storage branches of an exam resource file according to the present invention;

FIG. 12 is a block diagram illustrating an exams branch of the exam resource file;

FIGS. 13A and 13B is a block diagram illustrating a forms branch of the exam resource file;

FIG. 14 is a block diagram illustrating an items branch of the exam resource file;

FIG. 15 is a block diagram illustrating a categories branch of the exam resource file;

FIG. 16 is a block diagram illustrating a templates branch of the exam resource file;

FIG. 17 is a block diagram illustrating a sections branch of the exam resource file;

FIG. 18 is a block diagram illustrating a groups branch of the exam resource file;

FIGS. 19A, 19B, 19C, and 19D are block diagram illustrating an events sub-branch of the groups branch of the exam resource file;

FIG. 20 is a block diagram illustrating a plugins branch of the exam resource file;

FIG. 21 is a block diagram illustrating a data branch of the exam resource file;

FIG. 22 is a block diagram illustrating a formGroups branch of the exam resource file;

FIG. 23 is a block diagram illustrating an attributes branch of the exam resource file;

FIG. 24 is a block diagram illustrating a scripts branch of the exam resource file;

FIG. 25 is a block diagram illustrating a message box branch of the exam resource file;

FIGS. 26A, 26B, 26C, and 26D are block diagrams of an exam instance file according to the present invention;

FIG. 27 is a flow diagram of a method for computerized test customization according to the present invention;

FIG. 28 is a diagram of a life cycle of a plugin according to the present invention;

FIG. 29 is a flow diagram of a process for compiling plugins according to the present invention;

FIGS. 30A, 30B, 30C, and 30D are flow diagrams of a process for delivering plugins to an examinee during a computer-based test;

FIG. 31 is a flow chart illustrating a process for amalgamation of invisible plugins according to the present invention;

FIG. 32 is a flow chart illustrating a process for amalgamation of visible plugins according to the present invention;

FIG. 33 is a flow chart of a process for amalgamation according to the present invention;

FIG. 34 is a flow chart for a process for validating exam source according to the present invention;

FIG. 35 is a flow chart for a process for validating amalgamated test specification and content using a plugin according to the present invention;

FIG. 36 is a flow chart for a process of delivering amalgamated test specification and content using a test driver and a plugin according to the present invention; and

FIG. 37 is a flow diagram for an example of amalgamation of test specification and content relating to items according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Reference now will be made in detail to the presently preferred embodiments of the invention. Such embodiments are provided by way of explanation of the invention, which is not intended to be limited thereto. In fact, those of ordinary skill in the art may appreciate upon reading the present specification and viewing the present drawings that various modifications and variations can be made.

For example, features illustrated or described as part of one embodiment can be used on other embodiments to yield a still further embodiment. Additionally, certain features may be interchanged with similar devices or features not mentioned yet which perform the same or similar functions. It is therefore intended that such modifications and variations are included within the totality of the present invention.

The present invention discloses a system and method of computer-based testing using a test driver that is, for example, object-oriented and is architected to dynamically add functionality through, for example, the use of an expansion module, and preferably through the use of plugins. The test driver preferably references component object model servers using standard interfaces, and uses, for example, class names (that can be an Active Document) defined in a custom test definition language entitled eXtensible eXam Language ("XXL") based on eXtensible Markup Language ("XML") format to interact with existing applications while offering the flexibility of allowing development of new plugins. These new plugins can be customized to a client's needs without changing the core test driver. The specific format and protocol of XXL is also described in the co-pending application filed on the same date, entitled "EXTENSIBLE EXAM LANGUAGE (XXL) PROTOCOL FOR COMPUTER BASED TESTING," incorporated herein by reference.

The plugins advantageously enable the test driver to support, for example, new item types, navigation algorithms, information displays, scoring algorithms, timing algorithms, test unit selection algorithms, results persistence reporting, printed score reporting, and/or helm types without change to the test driver's executable. Plugins also allow expansion of the test driver's functionality without requiring the test driver to be recompiled or re-linked, and without requiring the test publisher to learn to program. Since plugins are written independently of the test driver, plugins can be written long after the test driver is built.

The client and the software developer can design and test the plugins and distribute the plugins to each test site. By using this method, large-scale regression testing of other examinations will not usually be necessary unless changes are made to the plugins that may be used by many examinations.

A test publisher defines common plugin test specification and common plugin test content in early XXL elements. These previous elements can be referenced by later XXL elements. If the later element defines nothing more then it will receive just the common specification and content. Optionally, the later elements can define omitted non-common plugin specification and contents. Further, the later elements can override previously defined common plugin specification and contents with exception plugin specification and contents. This process of new definition and overriding of specification and contents by later XXL elements is not limited to two stages. Multiple later stages are allowed.

When a plugin delivers it uses its specification and contents to determine how and what to deliver. This will be the amalgamation of all relevant XXL elements. The amalgamation is invisible to the plugin and requires no special effort for the plugin during delivery. The amalgamation occurs at delivery time and therefore common specification and content appears only once in the compile exam resource file. Thus the size of the resource file is reduced.

I. Overview of Computer-Based Test Delivery System

FIG. 3 shows an overview of the software architecture for the computer-based test delivery system of the present invention, denoted generally by reference numeral 100. Test driver 110 is responsible for controlling all aspects of the computer-based test. Test driver 110 identifies examinees scheduled to take the computer-based test and identifies and creates the appropriate test. Test driver 110 then presents all of the test components to examinees using a display device (not shown), such as a computer monitor, and enables examinees to enter responses to test questions through the use of an input device (not shown), such as a keyboard, a mouse, etc. Test driver 110 also monitors the security of the test. For example, test driver 110 can prevent access to the Internet and can validate examinees, although, these functions are preferably performed by the test center administration system. Test driver 110 also monitors the timing of the test, providing relevant warnings to examinee regarding the elapsed time of the test and the time remaining for a particular section of the test or for the entire test. Test driver 110 is also responsible for scoring the test, once the test is completed or while the test is in progress, and for reporting the results of the test by physical printout using printer 182 or in a file format using candidate exam results file 180. If the test is interrupted while in progress, for example, due to a power failure, test driver 110 restarts the test, preferably at the point at which the test was interrupted, as will be described subsequently in more detail. Finally, if the test is left incomplete, test driver 110 cleans up the incomplete test. An incomplete test will have an exam instance file in the examinee's directory but will not have created a results file. A results file is created even though generally the candidate will fail. The number of items delivered to the examinee is recorded in the results file. Test driver 110 picks up where the event was interrupted and invisibly deliveries the rest of the units of the test.

A test specification is authored by a test publisher according to the specifications of the client and stored in exam source files 130. Exam source files 130 include data files 132, XXL files 134, multimedia files 136, and hypertext markup language ("HTML") files 138. XXL files 134 include the test specification, which contains the client's requirements for the test, a bank of test items or questions, templates that determine the physical appearance of the test, plugins, and any additional data necessary to implement the test. Additional data is also stored in data files 132. For example an adaptive selection plugin may need a, b & c theta values. These values are stored in a binary file created by a statistical package.

HTML files 130 include, for example, any visual components of the test, such as the appearance of test items or questions, the appearance of presentations on the display device, the appearance of any client specified customizations, and/or the appearance of score reports. HTML files 130 preferably also include script, for example, VBscript and Jscript, or Java script. HTML files 130 are preferably authored using Microsoft's FrontPage 2000. FrontPage 2000 is preferably also used to manage the source files in a hierarchy that is chosen by the test publisher. Multimedia files 136 include, for example, any images (.jpg, .gif, etc.) and/or sound files (.mp3, .wav, .au, etc.) that are used during the test.

XXL compiler 140 retrieves XXL files 134 from exam source files 130 using interface 190 and compiles the XXL test content stored in XXL files 134. XXL compiler 140 stores the compiled test files in exam resource file 120. In another embodiment, exam source files 130 do not contain XXL files 134 and contains, for example, only multi-media files. In this embodiment, XXL compiler 140 is merely a test packager that writes the data directly to exam resource file 120 without modification or validation. The data appears in a stream under the "data" branch of exam resource file 120. The name of the stream is specified by the test author.

In a preferred embodiment, XXL files 134 also include XXL language that defines plugins 150, in which case, plugins 150 assist XXL compiler 140 in compiling XXL files 134. Test driver 110 preferably supports, for example, nine different types of plugins 150, including, for example: display plugin 152; helm plugin 154; item plugin 156; timer plugin 158; selection plugin 160; navigation plugin 162; scoring plugin 164; results plugin 166; and report plugin 168. Plugins 150, which are also included in XXL files 134, are the first XML files compiled into exam resource file 120.

Plugins 150 allow a test designer to customize the behavior of test driver 110 and are divided into two types, for example: visible plugins and invisible plugins, as shown in FIG. 4. The visible plugins, which include display plugin 152, helm plugin 154, and item plugin 156, enable the test driver to control what is presented visually to an examinee on the display device. The invisible plugins, which include timer plugin 158, selection plugin 160, navigation plugin 162, scoring plugin 164, results plugin 166, and report plugin 168, enable the test driver to control more functional aspects of the test. Plugins 150 are used to validate data stored in exam source files 130 that is to be used by one of plugins 150 during delivery of the test to the examinee, as is described below in greater detail. Plugins 150 are, preferably, component object model ("COM") objects, as described below. Plugins 150, may also utilize Java implementation. Plugins 150 are preferably written using Microsoft Visual C++ or Visual Basic 6.0 or any fully COM enabled language. Plugins 150 may be in or out-of-process, and, therefore, can exist as executable (".EXE") files or as dynamic link library (".DLL") files.

An application or component that uses objects provided by another component is called a client. Components are characterized by their location relative to clients. An out-of process component is an .exe file that runs in its own process, with its own thread of execution. Communication between a client and an out-of-process component is therefore called cross-process or out-of-process communication.

An in-process component, such as a .dll or .oxc file, runs in the same process as the client. it provides the fastest way of accessing objects, because property and method calls don't have to be marshaled across process boundaries. However, an in-process component must use the client's thread of execution.

Exam resource file 120 receives the compiled test content from XXL compiler 140 and plugins 150, if applicable, and stores the compiled test content in an object-linking and embedding ("OLE") structured storage format, called POLESS, which is described in greater detail below. Other storage formats may optionally be used. OLE allows different objects to write information into the same file, for example, embedding an Excel spreadsheet inside a Word document. OLE supports two types of structures, embedding and linking. In OLE embedding, the Word document of the example is a container application and the Excel spreadsheet is an embedded object. The container application contains a copy of the embedded object and changes made to the embedded object affect only the container application. In OLE linking, the Word document of the example is the container application and the Excel spreadsheet is a linked object. The container application contains a pointer to the linked object and any changes made to the linked object change the original linked object. Any other applications that link to the linked object are also updated. POLESS supports structured storage such that only one change made to an object stored in exam resource file 120 is globally effective. Test driver 110 comprises Active Document container application 112 for the visible plugins, display plugin 152, helm plugin 154, and item plugin 156, which function as embedded objects, preferably COM objects.

FIG. 3-1 shows an example of Active Document container application 112 being used with several item plugins 156. Each item plugin 156, in this case representing a multiple choice ("multi-choice") item, a hot area item, and a fill in the blank item, are linked to Active Document container application 112 through the IItem and IPlugin COM interfaces 169.

Both XXL compiler 140 and plugins 150 are involved in storing the compiled test content into exam resource file 120, if any of plugins 150 are being used. Exam resource file 120 comprises, for example, a hierarchical storage structure, as will be described in further detail below. Other storage structures may optionally be used. XXL compiler 140 determines to which storage location a specific segment of the compiled test content is to be stored. However, if any of plugins 150 are used to validate the portion of any of the data from exam source files 130, then the plugins 150 store the data directly to the exam resource file, based upon directions from XXL compiler 140. XXL compiler uses IPersistResource interface 192, co-located with I-Plugin interface 167 in FIG. 3, to control the persistence of the data to exam resource file 10. XXL compiler 140 and plugins 150 write the data to exam resource file 120 using POLESS interfaces 191.

FIG. 5 illustrates the contents of exam source file 130, which are compiled into exam resource file 120 by XXL compiler 140 and plugins 150. FrontPage 2000 Web 200 is used, for example, to author the test. Exam source files 130 contain media files 210, visual files 220, and logic files 230. Media files 210 are multimedia files used to enhance the presentation of the test, including, for example, XML data files 212, sound files 214, image files 216, and binary files 218. XML data files 212 include the XXL test definition language and the XXL extensions from the plugins 150 that use XML. The test specification, presentation, scoring and other information is specified in the XML files. Sound files 214 include any sounds that are to be used during the test, such as .mp3 files, .au files, etc. Image files 216 include any images to be used during the test, such as .jpg files, .gif files, etc. Binary files 218 include any data needed by a plugin 150 that is not in XXL format. Visual files 220 are HTML files that specify the visual presentation of the test as presented to the examine on the display device, including items files 222, presentation files 224, score report files 226, and custom look files 228. Items files 222 include HTML files that are used to specify the visual component of test questions, e.g., stems and distractors. Items files 222 are capable also of referencing external exhibits. An exhibit could be a chart, diagram or photograph. Formats of exhibits include, for example: .jpg, .png, etc. Presentation files 224 define what is seen by the examinee on the display device at a particular instant during the test. Score report files 226 include is typically an HTML file with embedded script that includes, for example candidate demographics, appointment information, and candidate performance. The performance might include pass/fail, achievement in different content areas, etc. Custom look files 228 include are typically HTML files with embedded script to layout, for example, the title bar and information contained therein. Logic files 230 are XML files that specify the functional aspects of the test, including test specification files 232, plugin files 234, item bank files 236, and template files 238. Test specification files 232 specify the content and progression of the test as provided by the client. Plugin files 234 define plugins 150 and contain any data necessary to implement plugins 150. Item bank files 236 include the data content and properties of the items, or test questions, that are to be presented to the examinee during the test. Properties of an item include the correct answer for the item, the weight given to the item, etc. Template files 238 define visual layouts that are used with the display screen during the test.

Referring again to FIG. 3, once a test has begun, test driver 110 accesses exam resource file 120 for the instructions and files needed to implement the test, using POLESS interfaces 193. Test driver 110 also accesses plugins 150 for additional data that expands the functionality of test driver 110 in the areas of items, navigation algorithms, information displays, scoring algorithms, timing algorithms, test unit selection algorithms, results persistence reporting, printed score reporting, and/or helm types. Test driver 110 communicates with plugins 150 using various COM interfaces 169. COM interfaces facilitate OLE linking. As stated previously, test driver 110 is an Active Document container application and plugins 150 are embedded objects. The COM interfaces function as communications paths between the container application and the objects.

There are, for example, ten COM interfaces utilized in computer-based test delivery system 100. IPlugin interface 167, which is also a COM interface, is supported by all of plugins 150. COM interfaces 169, therefore, includes the IPlugin interface. The IPlugin interface contains generic operations such as loading and unloading required of all plugins 150. In addition to the global IPlugin interface, each plugin 150 also uses, for example, a second, individual COM interface 169 to communicate with test driver 110. Alternative structures of the IPlugin interface may also be used. Table 1 shows the relationship between each plugin 150 and the COM interface 169 used with that particular plugin 150.