Subclass 130	Subclass 131	Subclass 132
Including analysis of program execution

Apparatus and methods for analyzing software systems

6118447

Abstract

A system and method for mode error troubleshooting including software system structure generation including prompting a developer to define a first plurality of tasks to be performed by a software system, to define a second plurality of modes in which the software system is to operate and to define for at least one task, at least one inappropriate mode in which the task cannot be performed and troubleshooting including prompting an end user to select a target task, searching for inappropriate modes in which the target task cannot be performed and providing an alert indicating when the end user is in one of the inappropriate modes.

Claims

What is claimed is:

1. A mode error troubleshooting system including:

a software system structure generator operative to prompt a developer to define a first plurality of task to be performed by the software system, to define a second plurality of modes in which the software system is to operate and to define, for at least one task, at least one inappropriate mode from among the second plurality of modes in which the at least one task cannot be performed; and

a troubleshooter operative to prompt an end user to select an individual one of the first plurality of tasks as his target task, to search among modes in which the software system is currently operating for inappropriate modes in which the target task cannot be performed and to provide an alert in respect of any inappropriate modes found.

2. A system according to claim 1 and wherein said troubleshooter is operative to provide said alert to said end user.

3. A system according to claim 1 and wherein said troubleshooter is operative to provide said alert to said developer.

4. A mode error troubleshooting method including:

generating a software system structure which is operative to prompt a developer to define a first plurality of tasks to be performed by the software system, to define a second plurality of modes in which the software system is to operate and to define, for at least one task, at least one inappropriate mode from among the second plurality of modes in which the at least one task cannot be performed; and

troubleshooting by prompting an end user to select an individual one of the first plurality of tasks as his target task, searching among modes in which the software system is currently operating for inappropriate modes in which the target task cannot be performed and providing an alert in respect of any inappropriate modes found.

5. A method according to claim 4 and wherein said troubleshooting includes providing said alert to said end user.

6. A method according to claim 4 and wherein said troubleshooting includes providing said alert to said developer.

Description

FIELD OF THE INVENTION

The present invention relates to apparatus and methods for analyzing software systems.

REFERENCE TO MICROFICHE APPENDIX

A microfiche appendix is attached hereto. The total number of microfiche is six (6) and the total number of frames is five-hundred-seventy-five (575).

BACKGROUND OF THE INVENTION

Conventional sources of information for development and maintenance of computer system regarding its usability, include its end users, e.g. in beta sites, testing groups, and professional observers operating in usability labs. Collection of this usability information is typically performed manually and non-systematically and consequently suffers from defects such as disregarding a multiple of seemingly minor usability problems.

Prior art regarding the usability of computerized systems include those described in U.S. Pat. No. 4,500,964 to Nickle, U.S. Pat. No. 5,086,393 to Kerr et al, U.S. Pat. No. 5,220,658 to Kerr et al, U.S. Pat. No. 5,590,330 to Coskun and Tate, U.S. Pat. No. 5,600,789 to Parker et al., Published European Application 0 687 988 A2 (95303298.4) to AT&T, and Published PCT Application WO 95/16949 to Software Publishing Corporation.

Other prior art systems and technologies are described in the following U.S. Patents: U.S. Pat. No. 5,490,249 to Miller, U.S. Pat. No. 5,565,316 to Kershaw et al, U.S. Pat. No. 5,566,291 to Boulton et al, U.S. Pat. No. 5,321,611 to Clark and Bramlett, U.S. Pat. No. 5,513,994 to Kershaw and Romano, U.S. Pat. No. 5,433,615 to Clark, U.S. Pat. No. 5,437,544 to Clark et al, U.S. Pat. No. 5,590,057 to Fletcher and Ruuska, U.S. Pat. No. 5,581,684 to Dudzik et al, U.S. Pat. No. 5,627,958 to Potts and Vershel, U.S. Pat. No. 5,535,422 to Chiang et al, U.S. Pat. No. 5,481,667 to Bieniek et al, U.S. Pat. No. 5,465,358 to Blades and Kiel.

The disclosures of all publications mentioned in the specification and of the publications cited therein are hereby incorporated by reference.

SUMMARY OF THE INVENTION

The present invention seeks to provide improved apparatus and methods for analyzing usability problems in computerized systems.

There is thus provided, in accordance with a preferred embodiment of the present invention, computerized apparatus for identifying human difficulties in operating a computerized system, the apparatus including a human difficulty identifier operative to identify putative instances of an end user's experience of difficulty in operating the computerized system, an operation recorder operative to store a record of the end user's operations during each putative instance, an intention recorder operative to prompt an end user to indicate his intention during each the putative instance and to store the intention in association with the record of operations for the putative instance, and an output generator operative to generate an output indication of the record of operations and of the end user's intention for each of the putative instances of experiences of difficulty.

Further in accordance with a preferred embodiment of the present invention, the intention is time-stamped and the record of operations is time-stamped, thereby to provide the association between the intention and the record of operations.

Also provided, in accordance with another preferred embodiment of the present invention, is computerized apparatus for identifying usability problems occurring in the course of operating a computerized system, the apparatus including a candidate usability problem identifier operative to generate records of occurrences of putative usability problems by monitoring an end user who is using the computerized system, a candidate usability problem database operative to store the records of occurrences of putative usability problems, and a database accessing unit operative to access the database and to derive therefrom information useful for resolving the usability problems.

Further in accordance with a preferred embodiment of the present invention, the database accessing unit includes a helpdesk access unit operative to access the database and to derive therefrom, on-line, information useful for operating a helpdesk.

Still further in accordance with a preferred embodiment of the present invention, the database accessing unit includes a developer's access unit operative to access the database and to derive therefrom information useful for redesigning the computerized system.

Further in accordance with a preferred embodiment of the present invention, the intention recorder is operative to prompt the end user to explain his intention in his own words.

Also provided, in accordance with another preferred embodiment of the present invention, is a method for computerized identification of human difficulties in operating a computerized system, the method including identifying putative instances of an end user's experience of difficulty in operating the computerized system, storing a record of the end user's operations during each the putative instance, prompting the end user to indicate his intention during each the putative instance and to store the intention in association with the record of operations for the putative instance, and generating an output indication of the record of operations and of the end user's intention for each of the putative instances of experiences of difficulty.

Also provided, in accordance with still another preferred embodiment of the present invention, is computerized apparatus for identifying human difficulties in operating a computerized system, the apparatus including a human difficulty identifier operative to identify putative instances of an end user's experience of difficulty in operating the computerized system, an operation recorder operative to store a record of the end user's operations during each the putative instance, and an output generator operative to generate an output indication of the record of operations for each of the putative instances of experiences of difficulty.

Also provided, in accordance with yet another preferred embodiment of the present invention, is a mode error troubleshooting system including a software system structure generator operative to prompt a developer to define a first plurality of tasks performed by the software system, to define a second plurality of modes in which the software system operates and to define, for at least one task, at least one inappropriate mode from among the second plurality of modes in which the task cannot be performed, and a troubleshooter operative to prompt an end user to select an individual one of the first plurality of tasks as his target task, to search among modes in which the software system is currently operating for inappropriate modes in which the target task cannot be performed and to alert the end user of any inappropriate modes found.

Additionally provided, in accordance with another preferred embodiment of the present invention, is a method for monitoring an end user's terminology, the method including accepting a definition of a computer system's terminology including a multiplicity of terms, repeating the following steps:

prompting an end user to select one of the multiplicity of terms, and, if the end user indicates that a desired intention is not associated with any of the multiplicity of terms, prompting the end user to give a name to the desired intention,

and displaying to a developer the incidence of each of the names given by the end user.

Additionally provided, in accordance with still another preferred embodiment of the present invention, is apparatus for identifying usability problems encountered when using developer-defined computerized systems each having a user interface, the apparatus including a user interface description elicitor operative to elicit from a developer of a developer-defined computerized system, and to record, a description of the user interface of the computerized system, a usability problem recorder operative to record usability problem data, and a usability problem occurrence analyzer using the description to analyze occurrences of usability problems recorded by the usability problem recorder.

Further in accordance with a preferred embodiment of the present invention, the usability problem recorder is operative to monitor an end user's interaction with the computerized system in order to identify automatically detectable usability problems.

Still further in accordance with a preferred embodiment of the present invention, the usability problem recorder is also operative to request the end user's confirmation of automatically detectable usability problems.

Additionally in accordance with a preferred embodiment of the present invention, the usability problem recorder is operative to accept a user-initiated report of a usability problem.

Still further in accordance with a preferred embodiment of the present invention, the user interface description elicitor is operative to store associations between components of a user interface of a computerized system and between functions of the computerized system.

Additionally in accordance with a preferred embodiment of the present invention, the user interface includes a plurality of input options, the apparatus also including an input option frequency recorder operative to accumulate input option frequency information including frequency of utilization for each of said plurality of input options.

Further in accordance with a preferred embodiment of the present invention, the apparatus also includes a user interface redeveloper operative to recommend a modification of at least a portion of the user interface based at least partly on the input option frequency information.

Still further in accordance with a preferred embodiment of the present invention, the apparatus also includes a user interface redeveloper operative to recommend a modification of at least a portion of the user interface based at least partly on the usability problem data.

Further in accordance with a preferred embodiment of the present invention, the usability problem data includes a record of each occurrence of a usability problem, expressed in terms of the description.

Still further in accordance with a preferred embodiment of the present invention, the usability problem recorder is operative, when recording the occurrence of a usability problem, to prompt the end user to indicate his intention when the usability problem occurred.

Additionally in accordance with a preferred embodiment of the present invention, the usability problem recorder is operative to accept from an end user an explanation of his intention phrased in his own words.

Further in accordance with a preferred embodiment of the present invention, the usability problem recorder is operative to prompt an end user to indicate his intention as a combination of at least one functions of a computerized system.

Further in accordance with a preferred embodiment of the present invention, the apparatus also includes a troubleshooter operative to troubleshoot by analyzing an end user's indicated intention.

Also provided, in accordance with another preferred embodiment of the present invention, is apparatus for developing a computerized system operative to perform a multiplicity of system functions responsive to a corresponding multiplicity of user interface components respectively, the multiplicity of user interface components forming a user interface, the apparatus including a computerized system usage statistics accumulator operative to accumulate computerized statistics regarding usage of the computerized system, and a user interface modification aid operative to propose a modification of the user interface based on the computerized statistics.

Further in accordance with a preferred embodiment of the present invention, the computerized system usage statistics accumulator includes a user difficulty accumulator operative to accumulate computerized statistics regarding difficulties encountered by end users of the computerized system.

Still further in accordance with a preferred embodiment of the present invention, the computerized system usage statistics accumulator includes a user interface component counter operative to count the number of times each user interface component is activated by at least one end user of the computerized system.

Still further in accordance with a preferred embodiment of the present invention, if the statistics indicate that an individual user interface component causes substantial difficulty, the user interface modification aid is operative to propose that the user interface be modified such that the system function is not performed responsive to the individual user interface component.

Further in accordance with a preferred embodiment of the present invention, the user difficulty accumulator is operative to identify mode errors based on the computerized statistics regarding difficulties.

Preferably, modification of at least a portion of the user interface also takes into account other factors such as mnemonics to aid end users in remembering the correspondence between input options and system functions and such as hierarchical structure to aid end users by assigning related or similar input options to related or similar system functions.

Modification of at least a portion of the user interface typically includes a recommendation of a new user interface to replace the existing user interface, the new user interface being different from the existing user interface in at least one respect. Preferably, the recommendation takes into account not only the negative cost of those portions of the existing user interface which are replaced but also the negative cost of the alternatives (in the new user interface) to the replaced portions of the existing user interface. For example, if a particular error shortcut key combination corresponding to a particular action according to the existing user interface is removed, then when the user wishes to perform that action, s/he should use an alternative input option such as menu selection or a push button. The alternative input option also may have a negative cost and preferably the system is operative to estimate this cost and take this cost into account in developing recommendations.

BRIEF DESCRIPTION OF THE DRAWINGS AND APPENDICES

The present invention will be understood and appreciated from the following detailed description, taken in conjunction with the drawings in which:

FIG. 1 is a simplified block diagram of computerized apparatus for identifying difficulties encountered by a human operator when operating a given computerized system;

FIGS. 2-12 are pictorial illustrations of screen displays generated by unit 20 of FIG. 1 according to a first embodiment of the present invention;

FIGS. 13-32 are pictorial illustrations of screen displays generated by unit 30 of FIG. 1 according to a first embodiment of the present invention;

FIGS. 33-36 are pictorial illustrations of screen displays generated by unit 32 of FIG. 1 according to a first embodiment of the present invention;

FIGS. 37-56 are pictorial illustrations of screen displays generated by unit 35 of FIG. 1 according to a first embodiment of the present invention;

FIGS. 57-68 are pictorial illustrations of screen displays generated by unit 90 of FIG. 1 according to a first embodiment of the present invention;

FIGS. 69-76 are pictorial illustrations of screen displays generated by unit 20 of FIG. 1 according to a second embodiment of the present invention;

FIGS. 77-84 are pictorial illustrations of screen displays generated by unit 30 of FIG. 1 according to a second embodiment of the present invention;

FIGS. 85-86 are pictorial illustrations of screen displays generated by unit 32 of FIG. 1 according to a second embodiment of the present invention;

FIGS. 87-106 are pictorial illustrations of screen displays generated by unit 35 of FIG. 1 according to a second embodiment of the present invention;

FIGS. 107-123 are pictorial illustrations of screen displays generated by unit 80 of FIG. 1 according to a second embodiment of the present invention;

FIG. 124 is a simplified functional block diagram of a preferred embodiment of the user action capture block 36 of FIG. 1, which is particularly suited for applications in which the computerized system is a Windows system, the embodiment of FIG. 124 also being useful in implementing units 20, 30 and 32 of FIG. 1;

FIG. 125 is a simplified flowchart illustration of a preferred mode of operation for unit 20 of FIG. 1;

FIG. 126 is a simplified flowchart illustration of a preferred mode of operation for unit 30 of FIG. 1;

FIG. 127 is a simplified flowchart illustration of a preferred implementation of the "Associate components" process of FIG. 119;

FIG. 128 is a simplified flowchart illustration of a preferred implementation of the "Associate conditions" process of FIG. 119;

FIG. 129 is a simplified flowchart illustration of a preferred mode of operation for unit 32 of FIG. 1;

FIG. 130 is a simplified flowchart illustration of a preferred mode of operation for unit 36 of FIG. 1;

FIG. 131 is a simplified flowchart illustration of a preferred mode of operation for unit 40 of FIG. 1;

FIG. 132 is a simplified flowchart illustration of a preferred implementation of the "Prompt user to report difficulty" process of FIG. 124;

FIG. 133 is a simplified flowchart illustration of a preferred implementation of the "Eliciting user intention" process of FIG. 125;

FIG. 134 is a simplified flowchart illustration of a preferred mode of operation for unit 90 of FIG. 1;

FIG. 135 is a simplified flowchart illustration of a preferred mode of operation for unit 100 of FIG. 1;

FIG. 136 is a simplified diagram of relationships between tables within databases 34 and 70 of FIG. 1, according to a first implementation of the present invention, in which tables within database 34 are indicated in dotted lines and tables within database 70 are indicated in dashed lines; and

FIG. 137 is a simplified diagram of relationships between tables within databases 34 and 70 of FIG. 1, according to a second preferred implementation of the present invention, in which tables within database 34 are indicated in dotted lines and tables within database 70 are indicated in dashed lines.

Attached herewith are the following microfiche appendices which aid in the understanding and appreciation of one preferred embodiment of the invention shown and described herein:

Appendix 1 is a software listing of utilities written in Pascal, which are useful for implementing a first embodiment of the present invention;

Appendix 2 contains software listings for the project file and for Pascal units of each of the forms of FIGS. 2-11, corresponding to unit 20 of FIG. 1;

Appendix 3 contains software listings for the project file and for Pascal units of each of the forms of FIGS. 12-32, corresponding to unit 30 of FIG. 1;

Appendix 4 contains software listings for the project file and for Pascal units of each of the forms of FIGS. 33-36, corresponding to unit 32 of FIG. 1;

Appendix 5 contains software listings for the project file and for Pascal units of each of the forms of FIGS. 37-56, corresponding to unit 35 of FIG. 1;

Appendix 6 contains software listings for the project file and for Pascal units of each of the forms of FIGS. 57-68, corresponding to unit 80 of FIG. 1;

Appendix 7 is a software listing of utilities written in Pascal, which are useful for implementing a second preferred embodiment of the present invention;

Appendix 8 contains software listings for the project file and for Pascal units of each of the forms of FIGS. 69-76, corresponding to unit 20 of FIG. 1 according to a second preferred embodiment of the present invention;

Appendix 9 contains software listings for the project file and for Pascal units of each of the forms of FIGS. 77-84, corresponding to unit 30 of FIG. 1 according to a second preferred embodiment of the present invention;

Appendix 10 contains software listings for the project file and for Pascal units of each of the forms of FIGS. 85-86, corresponding to unit 32 of FIG. 1 according to a second preferred embodiment of the present invention;

Appendix 11 contains software listings for the project file and for Pascal units of each of the forms of FIGS. 87-106, corresponding to unit 35 of FIG. 1 according to a second preferred embodiment of the present invention; and

Appendix 12 contains software listings for the project file and for Pascal units of each of the forms of FIGS. 107-123, corresponding to unit 80 of FIG. 1 according to a second preferred embodiment of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

Reference is now made to FIG. 1 which is a simplified block diagram of computerized apparatus for identifying difficulties encountered by a human operator when operating a given computerized system such as but not limited to a MS-Windows application.

The apparatus of FIG. 1 preferably comprises a computerized system characterization unit 10 operative to learn the system, preferably by holding a dialog with an expert regarding the computerized system. Preferably, the learning unit includes the following two subunits:

a. User interface learning unit 20--Performs a breakdown of the computerized system's user interface into components, in any suitable format such as a list or hierarchy of components. For example, if the computerized system is Motif, this step preferably includes characterization of the "widgets". If the computerized system is MS-Windows, this step preferably includes characterization of "controls" (such as menu items, edit fields, buttons and list boxes).

b. User task breakdown unit 30--Performs a breakdown of user tasks in terms of the user interface components characterized in step (a). Preferably, the user task breakdown unit 30 prompts the expert to specify all known user tasks and break down these user tasks into a hierarchy of sub-tasks down to procedures, until the user interface component level is reached. For example, a common user task in word processors is "move text". This task, in MS-Windows, comprises the following sub-tasks: "Select block to move", "Cut to clipboard", "Mark new location", and "Paste from clipboard". At the component level, the menu items "Edit/Cut" and "Edit/Paste" are specified together with the shortcut keys combination Ctrl+X and Ctrl+V.

A hierarchy of all known user tasks is typically stored in a user tasks database 34.

A test set-up unit 32 initializes the apparatus system and allows set-up parameters to be defined preferably for each user group, and/or for each individual user.

The apparatus of FIG. 1 also includes a usability problem identifier 35. The usability problem identifier 35 includes a user action capture unit 36 and a computerized unit 40 for identifying putative user difficulties by analyzing user actions captured by the user action capture unit 36. The user action capture unit 36 is preferably operative to capture and time-stamp various types of user input such as key-strokes and mouse-events. Putative user difficulties may be identified by any suitable criteria, as described in detail below.

Each time a putative user difficulty is identified, a dialog conducting unit 50 is actuated which conducts a dialog with the end-user in order to confirm that the putative user difficulty is in fact an instance of difficulty and in order to prompt the end-user to characterize the difficulty s/he is experiencing. The dialog unit 50 updates a usability problem database 70 to include the information collected regarding putative user difficulties.

Preferably, the characterization of difficulty which the dialog unit 50 solicits comprises an identification of the user's intention or goal which was entered by systems' experts to the user task database 34. Preferably, the characterization of difficulty comprises characterization of the information that the user examined in order to learn the operational procedure required to accomplish the task.

A preferred method for eliciting from the user a statement of his intention or goal is to display to the user, level by level beginning at the top level, the hierarchy of known user tasks and sub-tasks down to operational procedures stored in the user task database 34. Preferably, if the user cannot find an operational procedure in the task stored in database 34, the user can characterize the intended task explicitly.

A user interface usage confirmation unit 54 is preferably provided which is operative to prompt the user to indicate the user interface components s/he believes herself to have employed when attempting to perform the desired task. Preferably, this information is compared to captured information regarding the actual user interface components employed and a suitable message is provided the user if there is a discrepancy between the user interface components s/he actually used and those she thought she used. Preferably, the user interface usage confirmation unit 54 collects statistics regarding these discrepancies. These statistics are useful in indicating a users' tendency to accidentally misuse or confuse certain input options. For example, it may be the case that users tend to accidentally depress the CAPS LOCK key.

The system typically includes a usability problem solver 80 which typically includes at least one units operative to access the usability problem database 70 in order to derive various types of information from the database, such as problems in understanding the operational procedures and problems in the system response to accidental misuse of input options. In the illustrated embodiment, the usability problem solver 80 includes:

a. A developer's access unit 90 operative to derive from the database, preferably off-line, information useful for evaluating and further re-designing the computerized system. The information derived by the developer's access unit 90 preferably includes statistics regarding the costs, e.g. in terms of wasted time, associated with various categories of user difficulties. Preferably, the developer's access unit 90 includes a computerized human factors engineering expert unit 94 which is operative to generate a recommended solution for usability problems stored in database 70.

b. A helpdesk access unit 100 operative to derive from the database, preferably on-line, information useful for operating a helpdesk which may be computerized and form part of the system or alternatively may be external to the system and may comprise a human-operated help-desk.

Preferably, the apparatus of FIG. 1 also includes a computerized helpdesk 60 receiving input from the dialog unit 50 which is operative to provide conventional helpdesk services regarding known user difficulties identified by the dialog unit 50.

A particular advantage of the system shown and described herein is that, preferably, statistics are collected regarding all identified usability problems including usability problems which are not costly in terms of time. This is important because a prevalent usability problem may cause significant deterioration in performance not because each single occurrence wastes a significant amount of time but because of the problem's prevalence.

For example, it is believed that the "minor" mode error which occurs when a user unintentionally presses the Alt key, the Caps Lock key or the Insert key would have been rectified if statistics had been available regarding the high rate of recurrence of this error.

A particular advantage of the computerized collection of usability problems provided by the present invention, relative to conventional manual collection of usability problems is that, severe usability problems may be detected even when they are very rare. This is not always the case for manual collection of usability problems. For example, Version 6.0 of Word for Windows includes a severe usability problem in that if the Ctrl+Shift+H shortcut key combination is pressed erroneously, instead of Ctrl+H, in an attempt to Search And Replace a string within a selected text, the result is that the selected text is Hidden and no Search or Replace is performed. It is believed that this problem was not detected during manual collection of usability problems, otherwise it would have been fixed in Word version 7.0.

A first implementation of the apparatus of FIG. 1 is described herein with reference to FIGS. 2-68 and to appendices 1-6. A second preferred implementation of the apparatus of FIG. 1 is described herein with reference to FIGS. 69-123 and to appendices 7-12.

The first implementation of the apparatus of FIG. 1 is now described.

FIGS. 2-12 are pictorial illustrations of screen displays generated by unit 20 of FIG. 1 according to a first embodiment of the present invention. FIG. 2 is an initial screen display. The relationship between the screen displays of FIGS. 2-12 is described below including the relationships between the Delphi forms of the various screen displays, the Pascal units corresponding thereto, and an indication of the form component within a first screen display which, when selected, causes a particular second screen to be displayed.

FIGS. 13-32 are pictorial illustrations of screen displays generated by unit 30 of FIG. 1 according to a first embodiment of the present invention. FIG. 13 is the initial screen display. The relationship between the screen displays of FIGS. 13-32 is described below including the relationships between the Delphi forms of the various screen displays, the Pascal units corresponding thereto, and an indication of the form component within a first screen display which, when selected, causes a particular second screen to be displayed.

FIGS. 33-36 are pictorial illustrations of screen displays generated by unit 32 of FIG. 1 according to a first embodiment of the present invention. FIG. 33 is the initial screen display. The relationship between the screen displays of FIGS. 33-36 is described below including the relationships between the Delphi forms of the various screen displays, the Pascal units corresponding thereto, and an indication of the form component within a first screen display which, when selected, causes a particular second screen to be displayed.

FIGS. 37-56 are pictorial illustrations of screen displays generated by unit 35 of FIG. 1 according to a first embodiment of the present invention. FIG. 37 is the initial screen display. The relationship between the screen displays of FIGS. 37-56 is described below including the relationships between the Delphi forms of the various screen displays, the Pascal units corresponding thereto, and an indication of the form component within a first screen display which, when selected, causes a particular second screen to be displayed.

FIGS. 57-68 are pictorial illustrations of screen displays generated by unit 90 of FIG. 1 according to a first embodiment of the present invention. FIG. 57 is the initial screen display. The relationship between the screen displays of FIGS. 57-68 is described below including the relationships between the Delphi forms of the various screen displays, the Pascal units corresponding thereto, and an indication of the form component within a first screen display which, when selected, causes a particular second screen to be displayed.

Enclosed herewith are Appendices 1-6 including computer listings useful in generating the first embodiment of the present invention, shown and described above with reference to FIGS. 2-68.

Delphi form transitions within each of units 20, 30, 32, 35 and 90 according to a first embodiment of the present invention are now described. It is emphasized that the particular form transition structure described herein is merely an example and is provided for illustrative purposes only.

The Delphi form transitions, in the present example, within unit 20 are now described:

    ______________________________________
    From form
            To form Component Id    Component Text
    ______________________________________
    FIG. 2  FIG. 3  button: bbYourProduct
                                    label: Your Product
    FIG. 2  FIG. 4  button: bbProcess
                                    label: Process
    FIG. 2  FIG. 5  button: bbDiaBox
                                    label: Dialog Box
    FIG. 2  FIG. 6  button: bbMenuBar
                                    label: Menu Bar
    FIG. 2  FIG. 7  button: bbMenuItem
                                    label: Menu Item
    FIG. 2  FIG. 8  button: bbPopup label: Popup Menu
    FIG. 2  FIG. 9  button: bbButton
                                    label: Button
    FIG. 2  FIG. 10 button: bbSpeed label: Toolbar
    FIG. 2  FIG. 11 button: bbMoreControls
                                    label: More . . .
    FIG. 2  Quit    button: bbOK    label: OK
    ______________________________________

wherein FIGS. 2-11 describe the following Pascal unit files and Delphi forms:

    ______________________________________
    FIG.      Pascal Unit file (*.pas)
                              Delphi Form
    ______________________________________
    2         Resource        ResourceForm
    3         SelectProduct   SelectProductForm
    4         Process         ProcessForm
    5         Diabox          DiaboxForm
    6         MenuBar         MenuBarForm
    7         MenuItem        MenuItemForm
    8         Popup           PopupForm
    9         Button          ButtonForm
    10        Toolbar         ToolbarForm
    11        MoreControls    MoreControlsForm
    ______________________________________

The Delphi form transitions, in the present example, within unit 30 are now described:

    __________________________________________________________________________
    From form
          To form
               Component Id
                           Component Text
    __________________________________________________________________________
    FIG. 12
          FIG. 3
               button: BitBtn1
                           label: Product
    FIG. 12
          FIG. 13
               button: bbSpecify
                           label: Interface
    FIG. 13
          FIG. 14
               button: bbProcesses
                           label: Processes
    FIG. 13
          FIG. 15
               button: bbTriggers
                           label: Triggers
    FIG. 13
          FIG. 16
               button: bbTask
                           label: Tasks
    FIG. 16
          FIG. 17
               button: bbSetSubTasks
                           label: Specify Subtasks
    FIG. 17
          FIG. 18
               button: bbSetGoals
                           label: Specify Goals
    FIG. 18
          FIG. 19
               button: bbSetMethods
                           label: Specify Methods
    FIG. 19
          FIG. 20
               button: bbSetSteps
                           label: Specify Procedure
    FIG. 20
          FIG. 21
               button: bbSetNewTrigger
                           location: next to Standard option
    FIG. 20
          FIG. 22
               button: bbGetOldTrigger
                           location: below Operation group
    FIG. 22
          FIG. 15
               button: bbChange
                           label: Change
    FIG. 12
          FIG. 23
               button: BitBtn2
                           label: Intervention
    FIG. 23
          FIG. 32
               field: DBEdit1
                           label: Hot Key
    FIG. 23
          FIG. 32
               field: DBEdit2
                           label: Bypass
    FIG. 12
          FIG. 24
               button: bbEvaluate
                           label: Integrity
    FIG. 24
          FIG. 25
               button: bbSubTasks
                           label: Subtasks with no Goals
    FIG. 24
          FIG. 26
               button: bbGoals
                           label: Goals with no Methods
    FIG. 24
          FIG. 27
               button: bbMethods
                           label: Methods with no Steps
    FIG. 24
          FIG. 28
               button: bbSteps
                           label: Steps without Operations
    FIG. 24
          FIG. 29
               button: bbOperations
                           label: Missing Triggers
    FIG. 24
          FIG. 30
               button: bbFeatures
                           label: Redundant Triggers
    FIG. 24
          FIG. 31
               button: bbStandards
                           label: From Trigger to Task
    FIG. 12
          Quit button: bbOK
                           label: OK
    __________________________________________________________________________

wherein FIGS. 12-32 describe the following Pascal unit files and Delphi forms:

    ______________________________________
    FIG.     Pascal Unit file (*.pas)
                             Delphi Form
    ______________________________________
    12       BetaSpecMain    BetaSpecMainForm
    13       Interface       InterfaceForm
    14       Process         ProcessForm
    15       FindTrigger     FindTriggerForm
    16       MainTask        MainTaskForm
    23       InterventionControl
                             InterventionForm
    32       GetShortcutKey  GetShortcutKeyForm
    24       Integrity       IntegrityForm
    25       NulSubT         NulSubTaskForm
    26       NulGoal         NulGoalForm
    27       NulMethod       NulMethodForm
    28       NulStep         NulStepForm
    29       NulOperation    NulOperationForm
    30       MissingTask     MissingTaskForm
    31       TriggerToTask   TriggerToTaskForm
    32       GetShortcutKey  GetShortcutKeyForm
    ______________________________________

The Delphi form transitions, in the present example, within unit 32 are now described:

    __________________________________________________________________________
    From form
          To form
              Component Id
                          Component Text
    __________________________________________________________________________
    FIG. 33
          FIG. 3
              button: bbSelectProduct
                          label: Product
    FIG. 33
          FIG. 23
              button: bbControls
                          label: Intervention Controls
    FIG. 33
          FIG. 34
              button: bbIntervention
                          label: Default Intervention Policy
    FIG. 34
          FIG. 32
              field: BypassKey
                          label: Bypass
    FIG. 33
          FIG. 35
              button: bbDetails
                          label: Users
    FIG. 33
          FIG. 36
              button: bbStatus
                          label: Status
    FIG. 33
          Quit
              button: bbOK
                          label: OK
    __________________________________________________________________________

wherein FIGS. 33-36 describe the following Pascal unit files and Delphi forms:

    ______________________________________
    FIG.    Pascal Unit file (*.pas)
                            Delphi Form
    ______________________________________
    33      MainSetup       MainSetupForm
    23      InterventionControl
                            InterventionControlForm
    34      DefaultIntervention
                            DefaultInterventionForm
    32      GetShortcutKey  GetShortcutKeyForm
    35      UserProfile     UserProfileForm
    36      Status          StatusForm
    ______________________________________

The Delphi form transitions, in the present example, within unit 35 are now described:

    __________________________________________________________________________
    From form
            To form
                 Component Id
                             Component Text
    __________________________________________________________________________
    FIG. 37 FIG. 3
                 button: bbselectProduct
                             location: upper left corner
    FIG. 37 FIG. 38
                 button: bbUserId
                             image: person
    FIG. 38 FIG. 39
                 button: bbOK
                             label: OK
    FIG. 37 FIG. 40
                 button: bbActions
                             label: Actions
    FIG. 37 FIG. 41
                 button: bbProblems
                             label: Problems
    FIG. 42 FIG. 43
                 button: bbIgnore
                             label: Ignore
    FIG. 43 FIG. 49
                 button: bbIntention
                             label: I intended to do . . .
    FIG. 42 FIG. 49
                 button: bbHelp
                             label: Guide
    FIG. 44,45,46
            FIG. 47
                 button: bbIgnore
                             label: Ignore
    FIG. 47 FIG. 49
                 button: bbIntention
                             label: I intended to do . . .
    FIG. 44,45,46
            FIG. 48
                 button: bbMore
                             label: Tips
    FIG. 44,45,46
            FIG. 49
                 button: bbHelp
                             label: Guide
    FIG. 49 FIG. 50
                 button: bbNotFound
                             label: Report
    FIG. 50 FIG. 51
                 button: bbInform
                             label: Inform
    FIG. 49 FIG. 52
                 button: bbFound
                             label: but . . .
    FIG. 52 FIG. 50
                 button: bbNotFound
                             label: Report
    FIG. 52 FIG. 53
                 button: bbFound
                             label: but . . .
    FIG. 53 FIG. 50
                 button: bbNotFound
                             label: Report
    FIG. 53 FIG. 54
                 button: bbFound
                             label: but . . .
    FIG. 54 FIG. 50
                 button: bbNotFound
                             label: Report
    FIG. 54 FIG. 55
                 button: bbFound
                             label: but . . .
    FIG. 55 FIG. 50
                 button: bbNotFound
                             label: Report
    FIG. 55 FIG. 56
                 button: bbFound
                             label: but . . .
    FIG. 56 FIG. 51
                 button: bbInform
                             label: Inform
    FIG. 56 FIG. 40
                 button: bbOK
                             label: OK
    __________________________________________________________________________

wherein in FIGS. 37-56 describe the following Pascal unit files and Delphi forms:

    ______________________________________
    FIG.     Pascal Unit file (*.pas)
                             Delphi Form
    ______________________________________
    37       BetaSpy         BetaSpyForm
    38       UserDetails     UserIdForm
    39       CtrlF1          CtrlF1Form
    40       Actions         ActionForm
    41       ProblemReport   ProblemForm
    42       OnDelay         OnDelayForm
    43       OnIgnoreDelay   OnIgnoreDelayForm
    44       OnCancelForm    OnCancelForm
    45       OnHelp          OnHelpForm
    46       OnUndo          OnUndoForm
    47       OnIgnoreCancel  OnIgnoreEscapeForm
    48       OnBypass        BypassForm
    49       InquireMainTask InquireTaskForm
    50       TaskNotFound    TaskNotFoundForm
    51       Promise         PromiseForm
    52       InquireSubTask  InquireSubTaskForm
    53       InquireGoal     InquireGoalForm
    54       InquireMethod   InquireMethodForm
    55       InquireStep     InquireStepForm
    56       InquireStandard InquireStandardForm
    ______________________________________

The Delphi form transitions, in the present example, within unit 90 are now described:

    __________________________________________________________________________
    From form
          To form
               Component Id
                           Component Text
    __________________________________________________________________________
    FIG. 57
          FIG. 58
               button: bbUsage
                           label: Component Usage
    FIG. 58
          FIG. 3
               button: bbYourProduct
                           label: YourProduct
    FIG. 58
          FIG. 59
               button: bbProcess
                           label: Process
    FIG. 58
          FIG. 60
               button: bbDiabox
                           label: Dialog Box
    FIG. 58
          FIG. 61
               button: bbMenuBar
                           label: Menu Bar
    FIG. 58
          FIG. 62
               button: bbMenuItem
                           label: Menu Item
    FIG. 58
          FIG. 63
               button: bbPopup
                           label: Popup Menu
    FIG. 58
          FIG. 64
               button: bbButton
                           label: Button
    FIG. 58
          FIG. 65
               button: bbSpeed
                           label: Toolbar
    FIG. 58
          FIG. 66
               button: bbMoreControls
                           label: More . . .
    FIG. 57
          FIG. 67
               button: bbAccess
                           label: Access Difficulties
    FIG. 57
          FIG. 68
               button: bbConflict
                           label: Conflicting Components
    __________________________________________________________________________

wherein FIGS. 57-68 describe the following Pascal unit files and Delphi forms:

    ______________________________________
    FIG.     Pascal Unit file (*.pas)
                             Delphi Form
    ______________________________________
    57       ProblemSolver   SolverForm
    58       ComponentUsage  ComponentUsageForm
    59       ProcessUsage    ProcessForm
    60       DiaboxUsage     DiaboxForm
    61       MenuBarUsage    MenuBarForm
    62       MenuItemUsage   MenuItemForm
    63       PopupUsage      PopupForm
    64       ButtonUsage     ButtonForm
    65       ToolbarUsage    ToolbarForm
    66       MoreControls    MoreControlsForm
    67       AccessProblem   AccessProblemForm
    68       ConflictSummary ConflictForm
    ______________________________________

Advantages of the first embodiment based on FIGS. 2-68 and on Appendices 1-6, including problems existing in the prior art which are overcome by the first embodiment, preferably include at least some and preferably all of the following:

To the End User

The end user will able to compare his actual action with the intended one. This will facilitate the acquisition of the required psycho motoric skills

End users, such as in beta sites, are often required to report on usability problems. Typically, the reporting procedure is a burden, and end users try to avoid it. Computer guided reporting may be much easier to follow. As a result, the number of reported problems may dramatically increase.

To the Help Desk

Making the help desk more efficient, by:

Automatic problem reporting. User problems are recorded at the user site. The help desk personnel may get these reports automatically, rather than by manual recording of user complains on the phone

The database of usability errors will be much larger, because the users will report on many problems they do not report using conventional means

The help desk may apply statistics to evaluate the frequency and severity of user problems, in terms of time waste.

To System Design

Making the system more user friendly, by:

Adding wizards: for example, to facilitate the procedure of moving text, the designers can add a wizard that prompts the user to first select the text to be moved, then confirm selection in order to cut it to the clipboard, then mark the new location and finally confirm the new location, in order to paste the text from the clipboard

Disable certain keys: for example, disabling the Insert key may prevent the mode error of Overwrite, which is not very useful in Windows style text editing

Changing the control of mode transition: for example by changing the toggle control between upper case and lower case from Caps Lock to, say, Ctrl+Caps Lock, the number of unintentional mode changing may dramatically decrease

Removing excessive shortcut key combinations: for example, by removing the shortcut key combination Ctrl+Alt+F and Ctrl+Alt+H, which are error prone, the user can avoid confusing situations of getting the wrong dialog box or hiding of the selected text.

Definition of a Usability Problem

2 types of usability problems are concerned:

Procedural problems: when the user cannot find the sequence of operation required to accomplish a task: for example (Text editing), if the user wishes to move a text block. S/he may fail to know the sequence: Select, Cut to clipboard, Mark new location, Paste from clipboard

System response to erroneous operation: for example (Text editing), if the user erroneously pressed the Caps Lock key, the computerized system starts inserting upper case letters instead of lower case letters.

The Need for the Invention

Currently, only a small portion of the total amount of problems is detected.

Currently, the main source of information on usability problems is user reports.

Usually, reports based on in-house testing are not reliable, because testers cannot represent the end user properly. To get reliable information on usability problems, the computerized system should be tested by the end users, at the place where they use it.

However, reports of end users are not comprehensive, because:

Users are not aware of many of the problems: for example, many users are not aware of erroneous depression of the Alt key, since they do not notice the corresponding screen change

Even when they are aware of a problem, they are not willing to report on it: Sometimes, for example, if they do not know how to move a text block, they try to solve it themselves, by referring to the user documentation or by asking for help of their colleagues. Some other times, for example, if they hit the Caps Lock key accidentally, they consider it to be their own fault, rather than the design's fault.

Partially, Usability labs can help to solve these problems. However:

They are very expensive. As a result, those systems that are tested, are only partially tested. The majority of "minor" problems is neglected

Some usability problems are very hard to detect manually. For example, in MS Word 6.0, there are several situations of conflicting shortcut key combination. For example, Ctrl+H is used to activate the "Replace" dialog box and the Ctrl+Shift+H is (rarely) used to hide text. When pressing the Ctrl+Shift+H key combination instead of Ctrl+H, the selected text becomes hidden, when the user expects the dialog box "Replace" to appear on screen. Although Microsoft tested Word 7.0 extensively in usability labs and in many beta sites, they did not detect this problem, and did not remove the problematic key combination.

The way to overcome these shortcomings is by automation. The conventional means for automation is by the technology of Record/Playback, used in many utilities of software testing, such as WinRunner of Mercury Interactive, Vermont NighTest 2.0 of Vermont Creative Software Inc., QA Partner 2.0.1 of Segue Software Inc., PureVision of Pure Software, Inc., AutoTester 3.1 of AutoTester Inc., CA-Verify of Computer Associates International Inc.

However, these utilities do not detect usability problems as those described above: they are intended mainly for regression testing, typically, they do not look for indications of user problems, such as delay in the user response, the user request for Help or the cases when the user regrets his recent action.

Also, they do not record the user intention. Thus, if the user has a problem, they do not have the data for analyzing the reason for the user delay. For example, if the user invoked the Help facility to find out how to move text, they have not means to realize that s/he does not know how to move text.

The Invention

1. Automatic recording of the user actions, as conventionally implemented for regression testing.

2. Detection of events typical to cases of user problems

3. Inquiring the user for his intention, and recording the user response

4. In case that the user succeeded in following the operational procedure, but was surprised by the system response, inquiring the intended action, and recording it as well.

Using this procedure, the invention allows to automatically detect many of the user problems, to help him to solve certain problems and to prompt him to record the data required for the problem analysis.

A description of preferred modes of operation of the embodiment based on Appendices 1-6 is now provided:

How It Works

Detecting and Reporting on a Procedural Problem

Detection

Suppose the user fails to follow a procedure (such as moving text). In this case the user may do any of the following:

Pause the interaction, looking for help from outside, (colleagues, user documentation)

Invoke the on-line Help facility

Try the components on screen (trial and error)

Invoke the apparatus in hope to get useful instructions

The apparatus will intervene as follows:

In the first case, a screen display as in FIG. 42 will pop up

In the second case, a screen display as in FIG. 45 will pop up

In the third case, if the trial ended up in error, a screen display as in FIG. 46 will show up, if the user chose to Undo his last action, or a screen display as in FIG. 44 will show up, if the user has chosen to Cancel his last action

In the fourth case, the user initiates the interaction with the apparatus, for example, by pressing the Ctrl+F1 key combination.

Starting a Reporting Session

In the first three cases, different screen displays will show up, allowing the user to get instructions, aimed for reporting but also to help the user to resume normal operation.

The user may normally choose to click the Guide button from the screen display that pops up. In this case, he will get a screen display such as that of FIG. 49, prompting him to specify his intention.

If the user wishes to skip the reporting procedure, s/he may click the Ignore button. In this case, a screen display as in FIG. 43 will show up. There, s/he will get a second chance to report the problem, if s/he clicks the second option and then the "I intended to do . . . " button that shows up. In this case, he will get a screen display such as that of FIG. 49, prompting him to specify his intention. If s/he chooses to click any of the other options, no reporting will take place, and the user will resume normal operation of the computerized system. In the fourth case, the user will get the screen display as in FIG. 49 immediately

Reporting the User Intention

Task Reporting

Reporting the intention starts with the screen display as in FIG. 49. The user is prompted to specify his intention by selection from the task list. On selection, the second option is activated, and the "but . . . " button shows up. On pressing the "but . . . " button, a screen display as that of FIG. 52 is activated, for specifying the subtask.

If the user cannot find the intention in the task list, s/he will normally click the first option. In this case, s/he will be prompted to specify the intended task.

Subtask Reporting

Reporting the subtask starts with the screen display as in FIG. 52. The user is prompted to specify his subtask by selection from the list. On selection, the second option is activated and the "but . . . " button shows up. On pressing the "but . . . " button, a screen display as that of FIG. 53 is activated, for specifying the user goal.

If the user cannot find the subtask in the list, s/he will normally click the first option. In this case, s/he will be prompted to specify the intended subtask. On pressing the Report button, a screen display such as in FIG. 50 will pop up, to specify the information source that should be fixed. On pressing the OK button, the reporting session terminates.

Goal Reporting

Reporting the goal starts with the screen display as in FIG. 53. The user is prompted to specify his goal by selection from the list. On selection, the second option is activated and the "but . . . " button shows up. On pressing the "but . . . " button, a screen display as that of FIG. 54 is activated, for specifying the intended method.

If the user cannot find the goal in the list, s/he will normally click the first option. In this case, s/he will be prompted to specify the intended goal. On pressing the Report button, a screen display such as in FIG. 50 will pop up, to specify the information source that should be fixed. On pressing the OK button, the reporting session terminates.

Method Reporting

Reporting the method starts with the screen display as in FIG. 54. The user is prompted to specify his intended method by selection from the list. On selection, the second option is activated and the "but . . . " button shows up. On pressing the "but . . . " button, a screen display as that of FIG. 55 is activated, for specifying the desired method.

If the user cannot find the method in the list, s/he will normally click the first option. In this case, s/he will be prompted to specify the intended method. On pressing the Report button, as screen display such as in FIG. 50 will pop up, to specify the information source that should be fixed. On pressing the OK button, the reporting session terminates.

Step Reporting

Reporting the step starts with the screen display as in FIG. 55. The user is prompted specify the step in the operational procedure by selection from the step list. On selection, the second option is activated and the "but . . . " button shows up. On pressing the "but . . . " button, a screen display as that of FIG. 56 is activated, for specifying the intended action.

If the user cannot find the step in the list, s/he will normally click the first option. In this case, s/he will be prompted to specify the intended step. On pressing the Report button, a screen display such as in FIG. 50 will pop up, to specify the information source that should be fixed. On pressing the OK button, the reporting session terminates.

Detecting and Reporting a Problem of Incompatible System Response to a User Error

Detection

Suppose the user took the wrong action. If the system is properly designed, it will provide an error message. However, sometimes the erroneous action is legal in terms of the system.

Examples:

Wrong menu selection

Unintentional depression of the Caps Lock, Insert or Alt key (mode errors)

Unintentional usage of the wrong key combination, such as Ctrl+Shift+F instead of Ctrl+F or Ctrl+Shift+H instead of Ctrl+H

In these cases, the user often thinks that the system response is not compatible to his action.

In this case the user may do any of the following:

Undo or cancel the last action

Pause the interaction, looking for help from outside (colleagues, user documentation)

Invoke the on-line Help facility

Using conventional reporting means, most likely, the user will not report on these problems.

Using the apparatus, report will be initiated by the apparatus, as follows:

In the first case, a screen display as in FIG. 46 will show up, if the user chose to Undo his last action, or a screen display as in FIG. 44 will show up, if the user has chosen to Cancel his last action.

In the second case, a screen display as in FIG. 42 will pop up

In the third case, a screen display as in FIG. 45 will pop up.

Starting a Reporting Session

As above

Reporting the User Intention

As above

Facilitating the User Recovery Procedure

The apparatus optionally informs the user of his actual action.

A system, constructed and operative in accordance with a first embodiment of the present invention, for monitoring the usability of a Windows application is now described. The system is termed "ErgoLight" for simplicity.

Overview

ErgoLight is a family of solutions, that helps the developers of a Windows application to make it efficient and easy to use. Efficiency and ease of use are obtained by providing detailed and statistical information about the usability of the application:

How do the users get the features they need

When and how do they fail to use the features

ErgoLight provides this information, that is not available otherwise, using a knowledge base of human factors. Based on this information, application engineers can monitor usability issues through the whole project life cycle, starting from the demo phase, continuing with the testing and deployment phases, up to the stage of intensive operation.

Using ErgoLight, the organization can benefit from:

Increase in the user satisfaction

Saving much of the expenses due to calls to technical support.

The technologies integrated in ErgoLight include:

A knowledge base of human factors

A knowledge base of style guides and design rules

Rule based analysis of error prone procedures

Journalling and playback of the user actions

On-line analysis of usability problems

Statistical computation of the cost-effectiveness of design features

On the job learning (knowledge acquisition) and training (skill acquisition)

The Usability Issue

The usability issue is mainly that of making the proper design decisions. The shortest and easiest way for decision making is to let the programmers to decide, according to their "common sense". When budget and time are short, this might be the best way.

It is commonly agreed in the community of Usability engineers that applications designed this way might suffer from many usability problems. The main reason for this is that SW designers and system analysts do not represent the end user properly. While the end users are task oriented, system analysts are feature oriented and SW designers are control oriented. The transition from tasks to features and controls is problematic. Often, SW designers do not anticipate the situations when end users will fail to find a feature they need to accomplish their task This problem turn to be critical for applications aimed for non-technical users.

A convenient "solution" to this problem relies on style guides. Style guides define various aspects of the appearance of controls that are common to many applications. Obviously, style guides are not applicable to application specific features.

A common approach to designing application specific features is by asking the end user for their preferences. Occasionally, users can provide valuable information concerning the functionality of the application. In testing, they can complain about tasks that they could not accomplish.

The main problem with this approach is that end users are not aware of their actual needs. Eventually, their preferences are adequate for the deployment phase, but not for the phase of intensive usage. Typically, end users are not aware of the change expected in their preferences as they become more skilled in intensive operation. As a result, new applications defined with the help of the end users tend to become a burden when the operation is intensive.

Another problem with this approach is that end users are not aware of a multitude of failure modes during their interaction with the application. Users errors reduce operational performance by more than 50%, slow down the learning rate and raise the user's frustration and stress level. For example, end users often do not notice psycho motoric errors, such as pressing the wrong key, neither do they follow the sequence of events leading to the failure to accomplish a task. End users are not accustomed to analyze the sensitivity of the application to such errors.

Usability Testing

During the testing phase, designers look also for usability problems, not detected at the design phase. Traditionally, usability testing is performed by testing teams at the development site. Eventually, the operational skills of a tester in the development site are quite different from those of the end user. The observations of such a tester may often lead to wrong conclusions. In addition, typical testing teams are not aware of usability design rules, and do not have the means to evaluate usability features. Neither can they compare reliably alternative implementations.

Alpha-testing (at the development site) may contribute only when the application is for use by persons with technical background. Otherwise, the conclusion from these testing might be wrong. Beta-testing (at the customer site) contribute to finding critical usability problems, such as failure to follow a procedure after a lengthy (several hours) research, or to errors that result in data destruction (only if they can repeat the sequence of events that lead to the data destruction). Typically, end users do not bother to report about the multitude of minor problems, the cumulative effect of which is enormous. The main reason for this is that end users are not aware of most of the errors they make. In many other situations, the user is aware of the failure, but not to the action or the sequence of events that lead to the failure. In case that they are faced with an operational error, they tend to attribute the reason for the mal operation to themselves, rather than to the design.

The way large SW companies examine usability issues is through Usability Labs. In those labs, typical users are being observed while they perform their tasks. Eventually, small SW houses may use external usability lab services (e.g. OrCAD Systems; Cadence Design Systems).

The problems with this approach are:

Observations provide partial data. Data collection in Usability Labs is manual, ending up with having a small portion of the huge bulk of user problems

Observations do not provide quantitative measurements of cost-effectiveness

The validity of the conclusions depends on the ability to simulate the real operational environment

The validity of the interpretations depends on the skills of the observers (typically, usability engineers)

Observations are time consuming.

Observations are expensive.

ErgoLight is the only tool available today, that measures usability data in a real operational environment and that automatically provides valuable and meaningful information on usability issues.

Who May Benefit from ErgoLight

As a tool for finding bugs, ErgoLight might be considered as an automated testing tool. Actually, ErgoLight appeals to the same market as do the Automated Testing tools, namely, SW houses, system houses, technical consulting companies and corporate IS organizations. However, the information obtained by ErgoLight is totally different; while the "traditional" testing tools are aimed to find implementation deviation from the design, ErgoLight is aimed to find problems in the design itself.

ErgoLight can be used as:

A testbed in beta sites

A means for effective data collection in Usability labs

An add-on to Windows applications, to examine problems in setup parameters

A system for life cycle examination, from the demo phase to the phase of intensive usage.

ErgoLight Technologies

ErgoSpec Detection and identification of problems in the specification of the user interface, such as:

Insufficient guidance to the application features

Redundant complexity, caused by useless features and by multiple accessibility

Incompatibility with style guides

Conflicting access, resulting in low user performance and in steep learning curve.

ErgoSpy Capturing usability problems "on the fly":

Capture all keyboard and mouse user actions

Support of all Windows controls defined in resource files

Capture situations of user confusion

Capture situations of wrong operation

ErgoTest Journalling and on-line analysis of the users errors:

Filter real mistakes from intentional "Trial and Error" procedures

Detect failure modes in compound procedures

Find out gaps in the Help system, User's manual and Tutorial

Detect conflicting shortcut key combinations

Detect confusing terminology

Reveal the operational sequence resulting in data destruction

ErgoGuide Extending the capability of "on the job" learning of the operational procedures, by providing help in one or more of the following ways:

Guidance, by presenting material prepared by the application engineer, such as a Help window

Awareness, by showing controls available to invoke a task

Reassurance, by confirming that a problem is of wrong design, rather than of wrong usage.

ErgoTrain On the job acquisition of operational skills, by providing:

Tracking, by showing the sequence of events leading to a problematic situation

Awareness, by showing the deviation from the procedure

ErgoStat Playback and statistical measurement of Time To Recover (TTR) from a user error, classified by failure modes:

Problems of incompatibility to style guides

Problems typical to naive users

Problems in learning the operational procedures

Problems in learning the features by trial and error

Spontaneous user errors, typical to experienced users

ErgoWiz GUI optimization, by quantitative evaluation and comparison of alternative design solutions:

User options

Operation modes

Setup parameters.

ErgoLight Solutions

BetaLight--The Usability Test Bed for the Beta Site

BetaLight is effective for capturing usability problems of the real end users in their real operational environment. BetaLight reduces development time and save testing costs by automation, much the same as do other automated testing tools.

BetaLight is the most effective means to capture user confusion and erroneous operation. It is the only means available today that captures the multitude on "minor" usability problems.

Data Collection BetaLight uses the BetaSpy technology, that runs in the background to the application, "listening" to the computer-user interaction and recording it in a file. While recording, BetaSpy analyses the user reaction and detects situations when the user is either confused or responds erroneously. In such situations, BetaLight invokes the BetaGuide technology, that intervenes and offers on-line help.

Evaluation BetaLight uses the BetaStat technology, that performs statistical operations on the data, to obtain information about the effectiveness of the usability features, in terms of time waste.

ErgoLab--The Meter for Usability Engineering

ErgoLab can be integrated in Usability labs for initial testing of the application. There, much of the tedious work done by recording, playback and manual inspection, will be automated. Later, BetaLight can be integrated in the real application, for use in the real environment.

ErgoPlus--On The Job Usability Enhancement

ErgoPlus is the only means to evaluate usability problems attributed to the customization of Windows application. The evaluation, based on usability measures, directs the users in optimizing the options and setup parameters.

InfoLight--Life Cycle Examination of UI Effectiveness

Corporate IS organization do the usability testing with the actual end users, in the real environment. InfoLight can be used instead of the expensive Computer-Human-Interaction consultation, based on observations. It provides a mass of usability data, used for rapid, objective conclusions. InfoLight is used for testing through the complete life cycle, from the demo phase to the phase of intensive usage.

At the Demo Phase

InfoLight uses BetaSpec to capture problems in the product definition

At the Testing Phase

InfoLight uses BetaSpy to collect usability data, BetaGuide for providing extra help to the user and BetaStat for evaluation, much the same as BetaLight does.

At the Deployment Phase

InfoLight uses three technologies:

ErgoGuide--used mainly at the initial stage of the deployment phase, when user learn how to use the application.

ErgoTrain--used mainly after the user had acquired the basic skills, for performance improvement.

ErgoWiz--used to optimize the setup parameters to eliminate usability problems at both the personal and the team levels.

Following is a description of the data structures of tables used in a first embodiment of the present invention:

The following Paradox database tables are preferably stored in the [nonel]user task database 34 of FIG. 1 according to a first embodiment of the present invention:

Button

Diabox

Goal

Intervention

Maintask

Menubar

MenuItem

Method

Popup

Process

Product

Step

Subtask

Toolbar

Trigger

UserClass

UserProfile

The following Paradox database tables are preferably stored in the [none2]usability problem database 70 of FIG. 1 according to a first embodiment of the present invention:

Action

Problem

Following is a description of the data structures of the tables listed above:

Actions.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    TheTime           @
    ControlType       A      12
    ControlIndex      S
    ControlName       A      20
    Event             A      12
    ActiveForm        A      16
    ActiveControl     A      16
    MouseX            A       4
    MouseY            A       4
    KeyChar           A       1
    ______________________________________

Button.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    Diabox           A      20
    Name             A      20
    Usage            S
    ______________________________________

Diabox.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    Name             A      20
    Usage            S
    ______________________________________

Goal.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    Id               A      2
    SubTask          A      2
    MainTask         A      2
    Text             A      60
    Children         S
    Usage            S
    ______________________________________

Intervention.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    Diabox            A      20
    Name              A      20
    Type              A      10
    NuInMenuBar       S
    NuInMenuItem      S
    NuInPopup         S
    Button            A      20
    ShortKey          A      20
    BypassKey         A      20
    ______________________________________

MainTask.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    Id               A       2
    Text             A      40
    Children         S
    Usage            S
    ______________________________________

MenuBar.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    DiaBox           A      20
    Order            S
    Text             A      20
    Name             A      20
    Accel            A       1
    Usage            S
    ______________________________________

MenuItem.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    Diabox           A      20
    NuInBar          S
    Order            S
    Name             A      20
    Text             A      20
    Usage            S
    ______________________________________

Method.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    Id               A      2
    Goal             A      2
    SubTask          A      2
    MainTask         A      2
    Text             A      60
    Children         S
    Usage            S
    ______________________________________

Popup.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    Diabox           A      20
    Order            S
    Name             A      20
    Usage            S
    ______________________________________

Problem.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    Time              @
    UserName          A      10
    SessionNumber     S
    What              A      60
    MainTask          A      2
    SubTask           A      2
    Goal              A      2
    Method            A      2
    Step              A      2
    Operation         A      20
    HelpContents      L
    HelpSearch        L
    Tutorial          L
    UserGuide         L
    ______________________________________

Process.DB

    ______________________________________
    Field name         Type   Size
    ______________________________________
    Name               A      20
    Usage              S
    ______________________________________

Product.DB

    ______________________________________
    Field name         Type   Size
    ______________________________________
    FileName           A      30
    Path               A      60
    OnCancel           L
    OnDelay            L
    OnHelp             L
    OnUndo             L
    TimeToIntervention I
    DelayBypass        A      20
    CurrentUser        A      10
    CurrentUserClass   A      12
    WelcomeCount       S
    ______________________________________

Step.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    Number           A      2
    Method           A      2
    Goal             A      2
    SubTask          A      2
    MainTask         A      2
    Text             A      60
    Operation        A      20
    Trigger          A      20
    Usage            S
    ______________________________________

SubTask.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    Id               A      2
    MainTask         A      2
    Text             A      60
    Children         S
    Usage            S
    ______________________________________

ToolBar.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    Diabox           A      20
    Name             A      20
    Usage            S
    ______________________________________

Trigger.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    SrcBox           A      20
    Name             A      20
    TgtBox           A      20
    Process          A      20
    NuInBar          S
    NuInMenu         S
    NuInPopup        S
    Button           A      20
    Toolbar          A      20
    Shortkey         A      20
    Usage            S
    ______________________________________

UserClass.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    Name             A      12
    Percent          S
    Criterion        S
    NuOfUsers        S
    OnDelay          L
    OnCancel         L
    OnHelp           L
    OnUndo           L
    DelayTime        S
    ______________________________________

UserProfile.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    UserName         A      10
    UserGroup        A      12
    NuSessions       S
    OnDelay          L
    OnCancel         L
    OnHelp           L
    OnUndo           L
    DelayTime        S
    SessionNu        S
    NuProblems       S
    NuManual         S
    ______________________________________

There follows a description of a preferred method for employing the Appendices 1-6 to generate a software implementation of a system for identifying usability problems, the system being constructed and operative in accordance with a first embodiment of the present invention.

Enclosed herewith are Appendices 1-6 including computer listings useful in generating a first embodiment of the present invention.

A preferred method for employing Appendices 1-6 to generate an example of a software implemented system for identifying usability problems include the following steps:

1. Install Microsoft Windows/95 on a PC

2. Install Borland Developer version of Delphi 2.0 for Windows/95

3. Create the following directories:

    ______________________________________
    Usage            Example in Appendices 1-6
    ______________________________________
    Pascal Utilities .backslash.ABBA.backslash.UTILS
    Common Pascal units
                     .backslash.ABBA.backslash.ERGOLITE
    Paradox Database tables
                     .backslash.ABBA.backslash.ERGOLITE.backslash.DB
    Files of unit 20 .backslash.ABBA.backslash.ERGOLITE.backslash.RESOURCE
    Files of unit 30 .backslash.ABBA.backslash.ERGOLITE.backslash.SPEC
    Files of unit 35 .backslash.ABBA.backslash.ERGOLITE.backslash.BETASPY
    Files of unit 32 .backslash.ABBA.backslash.ERGOLITE.backslash.TESTPLAN
    Files of unit 80 .backslash.ABBA.backslash.ERGOLITE.backslash.USAGE
    ______________________________________

4. Define an alias name "BetaLite" for the Paradox Database tables directory (e.g. .backslash.ABBA.backslash.ERGOLITE.backslash.DB) and set it as the Working directory.

5. In the Working directory, use Delphi's Database Desktop to define the Paradox tables as depicted above:

6. Create utility Pascal units as listed in Appendix 1

7. Construct unit 20 (User interface learning) as follows:

Create a Delphi project and save it as ResourceProject.dpr in directory of unit 20 (e.g. .backslash.ABBA.backslash.ERGOLITE.backslash.RESOURCE)

Create the data modules ResourceModule.pas and ProductNodule.pas as in the pages listed in Appendix 2

Create the forms for the ResourceProject.dpr Delphi project according to FIGS. 2-11

Arrange the forms in the Delphi project in the order they appear in the listing of the ResourceProject.dpr Delphi project listed in Appendix 2

For each form, add events and code as in the Pascal units listed in Appendix 2. Use the form transition charts described above to name the components that do the transitions

Build the EXE file (compile and link) and then run

8. Construct unit 30: User task breakdown, as follows:

Create a Delphi project and save it as Bspec.dpr in directory of unit 30 (e.g. .backslash.ABBA.backslash.ERGOLITE.backslash.SPEC)

Create the data modules InterventionModule.pas and TriggerFilterModule.pas as in the pages listed in Appendix 3

Create the forms for the Bspec.dpr Delphi project according to FIGS. 12-32

Arrange the forms in the Delphi project in the order they appear in the listing of the Bspec.dpr Project pages listed in Appendix 3

For each form, add events and code as in the Pascal units listed in Appendix 3. Use the form transition charts described above to name the components that do the transitions

Build the EXE file (compile and link) and then run

Construct unit 32: Test setup, as follows:

9. Create a Delphi project and save it as TestPlan.dpr in directory of unit 32 (e.g. .backslash.ABBA.backslash.ERGOLITE.backslash.TESTPLAN)

Create the data module UserModule.pas as in the pages listed in Appendix 4

Create the forms for the TestPlan.dpr Delphi project according to FIGS. 33-37

Arrange the forms in the Delphi project in the order they appear in the listing of the TestPlan.dpr Project pages listed in Appendix 4

For each form, add events and code as in the Pascal units listed in Appendix 4. Use the form transition charts described above to name the components that do the transitions

Build the EXE file (compile and link) and then run

10. Construct unit 35: Usability problem identifier, as follows:

Create a Delphi project and save it as Bspy.dpr in directory of unit 35 (e.g. .backslash.ABBA.backslash.ERGOLITE.backslash.BETASPY)

Create the data module TestingModule.pas as in the pages listed in Appendix 5

Create the forms for the Bspy.dpr Delphi project according to FIGS. 38-56

Arrange the forms in the Delphi project in the order they appear in the listing of the Bspy.dpr Project pages listed in Appendix 5

For each form, add events and code as in the Pascal units listed in Appendix 5. Use the form transition charts described above to name the components that do the transitions

Build the EXE file (compile and link) and then run

11. Construct unit 80: Usability problem solver, as follows:

Create a Delphi project and save it as Solver.dpr in directory of unit 80 (e.g. .backslash.ABBA.backslash.ERGOLITE.backslash.USAGE)

Create the forms for the Solver.dpr Delphi project according to FIGS. 57-68

Arrange the forms in order according to the Project pages listed in Appendix 6

Arrange the forms in the Delphi project in the order they appear in the listing of the Solver.dpr Project pages listed in Appendix 6

For each form, add events and code as in the Pascal units listed in Appendix 6. Use the form transition charts described above to name the components that do the transitions

Build the EXE file (compile and link) and then run

12. To test a computerized system, use the hooks as in file JournalHooks.pas listed in Appendix 1.

The second implementation of the apparatus of FIG. 1 is now described with reference to FIGS. 69-123 and to Appendices 7-12.

FIGS. 69-76 are pictorial illustrations of screen displays generated by unit 20 of FIG. 1 according to a second embodiment of the present invention. FIG. 69 is an initial screen display. The relationship between the screen displays of FIGS. 69-76 is described below including the relationships between the Delphi forms of the various screen displays, the Pascal units corresponding thereto, and an indication of the form component within a first screen display which, when selected, causes a particular second screen to be displayed.

FIGS. 77-84 are pictorial illustrations of screen displays generated by unit 30 of FIG. 1 according to a second embodiment of the present invention. FIG. 77 is the initial screen display. The relationship between the screen displays of FIGS. 77-84 is described below including the relationships between the Delphi forms of the various screen displays, the Pascal units corresponding thereto, and an indication of the form component within a first screen display which, when selected, causes a particular second screen to be displayed.

FIGS. 85-86 are pictorial illustrations of screen displays generated by unit 32 of FIG. 1 according to a second embodiment of the present invention. FIG. 85 is the initial screen display. The relationship between the screen displays of FIGS. 85-86 is described below including the relationships between the Delphi forms of the various screen displays, the Pascal units corresponding thereto, and an indication of the form component within a first screen display which, when selected, causes a particular second screen to be displayed.

FIGS. 87-106 are pictorial illustrations of screen displays generated by unit 35 of FIG. 1 according to a second embodiment of the present invention. FIG. 87 is the initial screen display. The relationship between the screen displays of FIGS. 87-106 is described below including the relationships between the Delphi forms of the various screen displays, the Pascal units corresponding thereto, and an indication of the form component within a first screen display which, when selected, causes a particular second screen to be displayed.

FIGS. 107-123 are pictorial illustrations of screen displays generated by unit 80 of FIG. 1 according to a second embodiment of the present invention. FIG. 107 is the initial screen display. The relationship between the screen displays of FIGS. 107-123 is described below including the relationships between the Delphi forms of the various screen displays, the Pascal units corresponding thereto, and an indication of the form component within a first screen display which, when selected, causes a particular second screen to be displayed.

Delphi form transitions within each of units 20, 30, 32, 35 and 80 according to a second embodiment of the present invention are now described. It is emphasized that the particular form transition structure described herein is merely an example and is provided for illustrative purposes only.

The Delphi form transitions, in the present example, within unit 20 are now described:

    ______________________________________
    From form
            To form  Component Id   Component Text
    ______________________________________
    FIG. 69 FIG. 70  button: bbSetup
                                    label: Setup
    FIG. 69 FIG. 71  button: bbStart
                                    label: Start
    FIG. 71 FIG. 72  button: bbTasks
                                    label: Tasks
    FIG. 71 FIG. 73  button: bbViewSpy
                                    label: Debug
    FIG. 73 FIG. 74  button: bbSpy  label: Spy
    FIG. 74 FIG. 75  button: bbViewMenu
                                    label: View
    FIG. 73 FIG. 76  button: bbViewClas
                                    label: Class
    ______________________________________

wherein FIGS. 69-76 describe the following Pascal unit files and Delphi forms:

    ______________________________________
    FIG.     Pascal Unit file (*.pas)
                             Delphi Form
    ______________________________________
    69       GetComponentMain
                             MainForm
    70       MainSetup       MainSetupForm
    71       GetComponent    GetComponentForm
    72       ComponentTasks  ComponentTasksForm
    73       HyperSpy        HyperSpyForm
    74       Spy             SpyForm
    75       ViewMenu        MenuForm
    76       ViewClass       ClassForm
    ______________________________________

The Delphi form transitions, in the present example, within unit 30 are now described:

    __________________________________________________________________________
    From form To form
                   Component Id
                              Component Text
    __________________________________________________________________________
    FIG. 77   FIG. 70
                   button: bbSetup
                              label: Setup
    FIG. 77   FIG. 78
                   button: bbStart
                              label: Start
    FIG. 78   FIG. 79
                   button: bbSetSubTasks
                              label: Specify Subtasks
    FIG. 79   FIG. 80
                   button: bbSetGoals
                              label: Specify Goals
    FIG. 80   FIG. 81
                   button: bbSetMethod
                              label: Specify Methods
    FIG. 81   FIG. 82
                   button: bbSetSteps
                              label: Specify Procedure
    FIG. 82   FIG. 83
                   button: bbSetOperation
                              label: Specify Operation
    FIGS. 78,79,80,81,82
              FIG. 84
                   button: bbSetModes
                              label: SetModes
    FIGS. 83,84
              FIG. 73
                   button: bbViewSpy
                              label: Debug
    __________________________________________________________________________

wherein FIGS. 77-84 describe the following Pascal unit files and Delphi forms:

    ______________________________________
    FIG.      Pascal Unit file (*.pas)
                              Delphi Form
    ______________________________________
    77        DefTaskMain     MainForm
    78        MainTask        MainTaskForm
    79        SubTask         SubTaskForm
    80        Goal            GoalForm
    81        Method          MethodForm
    82        Steps           StepsForm
    83        Activation      ActivationForm
    84        SetCondition    SetConditionForm
    ______________________________________

The Delphi form transitions, in the present example, within unit 32 are now described:

    ______________________________________
    From form To form  Component Id  Component Text
    ______________________________________
    FIG. 85   FIG. 86  button: bbStart
                                     label: Start
    FIG. 86   FIG. 73  button: bbViewSpy
                                     label: Debug
    ______________________________________

wherein FIGS. 85-86 describe the following Pascal unit files and Delphi forms:

    ______________________________________
    FIG.      Pascal Unit flle (*.pas)
                              Delphi Form
    ______________________________________
    FIG. 85   GetIndicatorMain
                              MainForm
    FIG. 86   GetIndicators   GetIndicatorsForm
    ______________________________________

The Delphi form transitions, in the present example, within unit 35 are now described:

    __________________________________________________________________________
    From form
            To form
                 Event       Component Text
    __________________________________________________________________________
    FIG. 87 FIG. 70
                 button: bbSetup
                             label: Setup
    FIG. 87 FIG. 88
                 button: bbLearnUI
                             label: Learn UI
    FIG. 88 FIG. 73
                 button: bbViewSpy
                             label: Debug
    FIG. 88 FIG. 89
                 event: user delay
    FIG. 88 FIG. 91
                 event: Cancel, Esc request
    FIG. 88 FIG. 92
                 event: Help request
    FIG. 88 FIG. 93
                 event: Undo request
    FIG. 91,92,93
            FIG. 94
                 button: bbIgnore
                             label: Ignore
    FIG. 90,94
            FIG. 96
                 button: bbIntention
                             label: I intended to do . . .
    FIG. 89,91,92,93
            FIG. 96
                 button: bbGuide
                             label: Guide
    FIG. 89,91,92,93
            FIG. 95
                 button: bbTips
                             label: Tips
    FIG. 96 FIG. 97
                 button: bbNotFound
                             label: Task not found
    FIG. 97 FIG. 98
                 button: bbFound
                             label: but . . .
    FIG. 98 FIG. 99
                 button: bbFound
                             label: but . . .
    FIG. 99 FIG. 100
                 button: bbFound
                             label: but . . .
    FIG. 100
            FIG. 101
                 button: bbFound
                             label: but . . .
    FIG. 101
            FIG. 105
                 button: bbFound
                             label: but . . .
    FIG.    FIG. 102
                 button: bbNotFound
                             label: Report
    97,98,99,100,101
    FIG. 87 FIG. 104
                 button: bbViewActions
                             label: View
    FIG.    FIG. 106
                 event: Mode error
    97,98,99,100,101
                 detected
    FIG. 105,102
            FIG. 103
                 button: bbInform
                             label: Inform product designers
    __________________________________________________________________________

wherein FIGS. 87-106 describe the following Pascal unit files and Delphi forms:

    ______________________________________
    FIG.    Pascal Unit file (*.pas)
                            Delphi Form
    ______________________________________
    87      RTmonitor       MainForm
    88      LearnUI         LearnUIForm
    89      OnDelay         OnDelayForm
    90      OnIgnoreDelay   OnIgnoreDelayForm
    91      OnCancelForm    OnCancelForm
    92      OnHelp          OnHelpForm
    93      OnUndo          OnUndoForm
    94      OnIgnoreCancel  OnIgnoreEscapeForm
    95      OnBypass        BypassForm
    96      InquireComponent
                            InquireComponentForm
    97      InquireMainTask InquireTaskForm
    98      InquireSubTask  InquireSubTaskForm
    99      InquireGoal     InquireGoalForm
    100     InquireMethod   InquireMethodForm
    101     InquireStep     InquireStepForm
    102     TaskNotFound    TaskNotFoundForm
    103     Promise         PromiseForm
    104     GetUserAction   GetUserActionForm
    105     ShowReasons     ShowReasonsForm
    106     ModeErrors      ModeErrorForm
    ______________________________________

The Delphi form transitions, in the present example, within unit 80 are now described:

    __________________________________________________________________________
    From form
          To form
               Component Id Component Text
    __________________________________________________________________________
    FIG. 107
          FIG. 108
               bbComponentUsage
                            Component Usage
    FIG. 108
          FIG. 109
               bbConflicts  Conflicts
    FIG. 109
          FIG. 110
               bbViewDetails
                            Details of Selected
    FIG. 110
          FIG. 111
               bbViewDetails
                            Analysis of Selected
    FIG. 107
          FIG. 112
               bbTasksAndTerms
                            Tasks and Terms
    FIG. 112
          FIG. 113
               bbSolve      Solve
    FIG. 113
          FIG. 114
               bbModify     Modify
    FIG. 107
          FIG. 115
               bbProcedureKnowledge
                            Procedure Knowledge
    FIG. 115
          FIG. 116
               bbViewDetails
                            Details of Selected
    FIG. 107
          FIG. 117
               bbModeErrors Mode Errors
    FIG. 117
          FIG. 118
               bbViewDetails
                            Details of Selected
    FIG. 107
          FIG. 119
               bbByUserIntention
                            By User Intention
    FIG. 119
          FIG. 113
               bbPreviousProblems
                            Previous Problem Reports
    FIG. 119
          FIG. 120
               bbGoalIdentified
                            Goal Identified
    FIG. 120
          FIG. 116
               bbPreviousProblems
                            Previous Problem Reports
    FIG. 120
          FIG. 121
               bbProcedureIdentified
                            Procedure Identified
    FIG. 121
          FIG. 122
               bbAssociatedComponents
                            Associated Components
    FIG. 107
          FIG. 123
               bbByUserAction
                            By User Action
    FIG. 123
          FIG. 110
               bbAssociatedReports
                            Associated Reported Problems
    __________________________________________________________________________

wherein FIGS. 107-123 describe the following Pascal unit files and Delphi forms:

    ______________________________________
    FIG.  Pascal Unit file (*.pas)
                          Delphi Form
    ______________________________________
    107   ProblemSolver   SolverForm
    108   ComponentUsage  ComponentUsageForm
    109   ConflictSummary ConflictSummaryForm
    110   ConflictReport  ConflictReportForm
    111   BackTrack       BackTrackForm
    112   GoalSummary     GoalSummaryForm
    113   UserTerm        UserTermForm
    114   SolveTerm       SolveTermForm
    115   ProcedureSummary
                          ProcedureSummaryForm
    116   ProcedureKnowledgeReport
                          ProcedureKnowledgeReportForm
    117   ModeErrorSummary
                          ModeErrorSummaryForm
    118   ModeReport      ModeReportForm
    119   HelpByUserIntention
                          HelpByUserIntentionForm
    120   HelpGoal        HelpGoalForm
    121   HelpProcedure   HelpProcedureForm
    122   AssociatedComponents
                          AssociatedComponentsForm
    123   HelpByUserAction
                          HelpByUserActionForm
    ______________________________________

Following is a description of the data structures of tables used in a second preferred embodiment of the present invention:

The following Paradox database tables are preferably stored in the user task database 34 of FIG. 1 according to a second preferred embodiment of the present invention:

Component.DB

Condition.DB

DocSource.DB

File.DB

Goal.DB

Maintask.DB

Method.DB

ObjectClass.DB

ProblemIndicator.DB

Product.DB

Step.DB

Subtask.DB

The following Paradox database tables are preferably stored in the usability problem database 70 of FIG. 1 according to a second preferred embodiment of the present invention:

Action.DB

ConflictSummary.DB

DocSummary.DB

ModeError.DB

Problem.DB

TaskSummary.DB

UsageSummary.DB

Following is a description of the data structures of tables listed above:

Actions.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    The Time         @
    Indicator        L
    Type             A      10
    Identifier       A      60
    Class            A      20
    Static           L
    Caption          A      30
    Location         A      30
    ShortcutKey      A      20
    State            A      10
    FormTitle        A      60
    ______________________________________

Component.DB

    ______________________________________
    Field name       Type   Size
    ______________________________________
    Identifier       A      60
    Description      A      60
    Class            A      20
    IsStatic         L
    IsMode           L
    Caption          A      30
    Location         A      30
    ShortcutKey      A      20
    FormTitle        A      60
    ______________________________________

Condition.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    MainTask          A       2
    SubTask           A       2
    Goal              A       2
    Method            A       2
    Step              A       2
    Identifier        A      60
    Class             A      20
    Caption           A      30
    FormTitle         A      60
    Value             A      10
    Likelihood        A      10
    CurrentValue      A      10
    ______________________________________

ConflictSummary.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    TimeWaste         S
    ProblemCount      S
    MainTask          A       2
    SubTask           A       2
    Goal              A       2
    Method            A       2
    Step              A       2
    CompType          A      10
    CompId            A      10
    CompClass         A      20
    CompCaption       A      30
    ShortcutKey       A      20
    FormTitle         A      60
    Advice            A      60
    ToDesigner        A      100
    ToHelpDesk        A      100
    ______________________________________

DocSource.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    Name              A      20
    ______________________________________

DocSummary.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    TimeWaste         S
    ProblemCount      S
    MainTask          A      2
    SubTask           A      2
    Goal              A      2
    DocSource         A      20
    Advice            A      60
    ToDesigner        A      100
    ToHelpDesk        A      100
    ______________________________________

File.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    ExePath           A      60
    ExeFile           A      30
    DBPath            A      60
    ______________________________________

Goal.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    Id                A      2
    SubTask           A      2
    MainTask          A      2
    Text              A      60
    Children          S
    Usage             S
    ______________________________________

MainTask.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    Id                A      2
    Text              A      40
    Children          S
    Usage             S
    ______________________________________

Method.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    Id                A      2
    Goal              A      2
    SubTask           A      2
    MainTask          A      2
    Text              A      60
    Children          S
    Usage             S
    ______________________________________

ModeError.DB

    ______________________________________
    Field name         Type   Size
    ______________________________________
    TimeWaste          S
    Count              S
    MainTask           A      2
    SubTask            A      2
    Goal               A      2
    Method             A      2
    Step               A      2
    CompId             A      60
    ModeBtnId          A      60
    ModeForm           A      60
    ModeBtnCaption     A      30
    ModeBtnValue       A      10
    Advice             A      60
    UsageCount         S
    UserSet            A      10
    SWset              A      10
    ToDesigner         A      100
    ToHelpDesk         A      100
    ______________________________________

ObjectClass.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    Name              A      20
    IsStatic          L
    IsMode            L
    ______________________________________

Problem.DB

    ______________________________________
    Field name         Type   Size
    ______________________________________
    Time               @
    TimeWaste          S
    What               A      60
    MainTask           A      2
    SubTask            A      2
    Goal               A      2
    Method             A      2
    Step               A      2
    CompType           A      10
    CompId             A      10
    CompClass          A      20
    CompCaption        A      30
    ShortcutKey        A      20
    FormTitle          A      60
    ModeBtnCaption     A      30
    ModeBtnId          A      60
    ModeBtnValue       A      10
    DocSource          A      20
    ToDesigner         A      100
    ToHelpDesk         A      100
    ______________________________________

ProblemIndicator.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    Identifier        A      60
    Type              A      20
    ______________________________________

Product.DB

    ______________________________________
    Field name         Type   Size
    ______________________________________
    FileName           A      30
    Path               A      60
    TimeToIntervention I
    ______________________________________

Step.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    Number            A      2
    Method            A      2
    Goal              A      2
    SubTask           A      2
    MainTask          A      2
    Text              A      60
    Operation         A      20
    FormId            A      60
    ControlId         A      60
    MenuBarId         A      60
    PopupId           A      60
    ShortcutKey       A      60
    State             A      60
    Usage             S
    ______________________________________

SubTask.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    Id                A      2
    MainTask          A      2
    Text              A      60
    Children          S
    Usage             S
    ______________________________________

TaskSummary.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    TimeWaste         S
    ProblemCount      S
    MainTask          A      2
    SubTask           A      2
    ToDesigner        A      100
    ToHelpDesk        A      100
    ______________________________________

UsageSummary.DB

    ______________________________________
    Field name        Type   Size
    ______________________________________
    ComponentId       A      60
    Class             A      20
    IsStatic          L
    IsMode            L
    FormTitle         A      60
    Time              S
    UserSet           A      10
    Swset             A      10
    Count             S
    Supported         L
    AssignedTasks     S
    CurrentValue      A      10
    ______________________________________

Enclosed herewith are Appendices 7-12 including computer listings useful in generating a second preferred embodiment of the present invention.

A preferred method for employing Appendices 7-12 to generate an example of a software implemented system for identifying usability problems include the following steps:

1. Install Microsoft Windows/95 on a PC

2. Install Borland Developer version of Delphi 2.0 for Windows/95

3. Create directories as follows:

    ______________________________________
    Usage        Example: directory used in Appendices 7-12
    ______________________________________
    Common Pascal units
                 .backslash.ergolite.backslash.common
    Paradox Database
                 .backslash.ergolite.backslash.DB
    tables
    Files of unit 20
                 .backslash.ergolite.backslash.CaptureComponents
    Files of unit 30
                 .backslash.ergolite.backslash.DefineTaskBreakdown
    Files of unit 32
                 .backslash.ergolite.backslash.DefineProblemIndicators
    Files of unit 35
                 .backslash.ergolite.backslash.CaptureUserInteraction
    Files of unit 80
                 .backslash.ergolite.backslash.ViewUserDifficulties
    ______________________________________

4. Define an alias name "BetaLite" for the Paradox Database tables directory (e.g. .backslash.ergolite.backslash.db) and set it as the Working directory.

5. In the Working directory, use Delphi's Database Desktop to define the Paradox tables listed above.

6. Create utility Pascal units as listed in Appendix 7

7. Construct SharedData.dll as follows:

Create a Delphi project and save it as SharedData.dpr in the common directory (e.g. .backslash.ergolite.backslash.common)

Add the units to the project file, according to the listings of the project file SharedData.dpr in Appendix 7

Build the EXE file (compile and link)

8. Construct HookDLL.dll as follows:

Create a Delphi project and save it as HookDLL.dpr in the common directory (e.g. .backslash.ergolite.backslash.common)

Add the units to the project file, according to the listings of the project file HookDLL.dpr in Appendix 7

Build the EXE file (compile and link)

9. Construct unit 20 (User interface learning) as follows:

Create a Delphi project and save it as CaptureComponents.dpr in directory of unit 20 (e.g. .backslash.ergolite.backslash.CaptureComponents)

Create the forms for the CaptureComponents.dpr Delphi project according to FIGS. 69-75

Arrange the forms in the Delphi project in the order they appear in the listing of the CaptureComponents.dpr Delphi project listed in Appendix 8

For each form, add events and code as in the Pascal units listed in Appendix 8. Use the form transition charts described above to name the components that do the transitions

Build the EXE file (compile and link) and then run

10. Construct unit 30: User task breakdown, as follows:

Create a Delphi project and save it as DefineTasklreakdown.dpr in directory of unit 30 (e.g. .backslash.ergolite.backslash.DefineTaskBreakdown)

Create the forms for the DefineTaskBreakdown.dpr Delphi project according to FIGS. 76-83

Arrange the forms in the Delphi project in the order they appear in the listing of the DefineTaskBreakdown.dpr Project pages listed in Appendix 9

For each form, add events and code as in the Pascal units listed in Appendix 9. Use the form transition charts described above to name the components that do the transitions

Build the EXE file (compile and link) and then run

11. Construct unit 32: Test setup, as follows:

Create a Delphi project and save it as TestPlan.dpr in directory of unit 32 (e.g. .backslash.ergolite.backslash.DefineProblemIndicators)

Create the forms for the DefineProblemIndicators.dpr Delphi project according to FIGS. 84-85

Arrange the forms in the Delphi project in the order they appear in the listing of the DefineProblemIndicators.dpr Project pages listed in Appendix 10

For each form, add events and code as in the Pascal units listed in Appendix 10. Use the form transition charts described above to name the components that do the transitions

Build the EXE file (compile and link) and then run

12. Construct unit 35: Usability problem identifier, as follows:

Create a Delphi project and save it as Bspy.dpr in directory of unit 35 (e.g. .backslash.ergolite.backslash.CaptureUserInteraction)

Create the forms for the CaptureUserlnteraction.dpr Delphi project according to FIGS. 86-106

Arrange the forms in the Delphi project in the order they appear in the listing of the CaptureUserlnteraction.dpr Project pages listed in Appendix 11

For each form, add events and code as in the Pascal units listed in Appendix 11. Use the form transition charts described above to name the components that do the transitions

Build the EXE file (compile and link) and then run

13. Construct unit 80: Usability problem solver, as follows:

Create a Delphi project and save it as Solver.dpr in directory of unit 80 (e.g. .backslash.ergolite.backslash.ViewUserDifficulties)

Create the forms for the ViewUserDifficulties.dpr Delphi project according to FIGS. 107-123

Arrange the forms in order according to the Project pages listed in Appendix 12

Arrange the forms in the Delphi project in the order they appear in the listing of the ViewUserDifficulties.dpr Project pages listed in Appendix 12

For each form, add events and code as in the Pascal units listed in Appendix 12. Use the form transition charts described above to name the components that do the transitions

Build the EXE file (compile and link) and then run

Advantages of the second preferred embodiment described in FIGS. 69-123 and in Appendices 7-12, as well as problems existing in the prior art which are overcome by the second preferred embodiment, are now described:

The system shown and described herein provides information on possible reasons for user difficulties. This information is obtained by interrupting the operation of the computerized system, and by prompting the user to specify her intention. In conventional capture/playback tools, the user's normal operation is not interrupted for reporting on the difficulty of the end user.

An advantage of the system shown and described herein is that the system preferably provides hints for the system designers on possible ways to resolve identified usability problems, as in the following examples which apply to Word for MS-Windows:

Adding wizards. For example, to facilitate the procedure of moving text, the designers can add a wizard that prompts the end user to first select the text to be moved, then confirm selection in order to cut it to the clipboard, then mark the new location and finally confirm the new location, in order to paste the text from the clipboard

Disable certain keys. For example, disabling the Insert key may prevent the mode error of Overwrite, which is not very useful in Windows style text editing

Changing the control of mode transition. For example by changing the toggle control between upper case and lower case from Caps Lock to, say, Ctrl+Caps Lock, the frequency of unintentional mode changing may dramatically decrease

Removing excessive shortcut key combinations. For example, by removing the shortcut key combination Ctrl+Alt+H, which experienced typists unintentionally often use instead of the Ctrl+H key combination, the end user avoids confusing situations such as hiding of the selected text.

The system described herein is preferably "general purpose", whereas prior art systems are specific to and built onto particular software applications and require changes in the software code of the combined system/application, so that they cannot be extended to other software applications. The present invention preferably does not require any information on the software code of the computerized system. Therefore, it is applicable to various applications. This feature is useful for software solution providers and SI's (System Integrators) who wish to test applications purchased from a third party, for integration with other software applications and procedures.

A particular advantage of the invention for system integrators is that the usability problem identifier unit 35 can be extended so that certain user input options, that a system integrator consider as error prone, can be either disabled or re-mapped to alternative input options that are less error prone. For example:

The Insert key can be disabled, to avoid unintentional change to Overwrite mode

Excessive, error prone, shortcut key combinations, such as Ctrl+Shift+H, can be disabled, to avoid unintentional hiding of selected text

A function actuated by a menu item can be re-directed to a different component, such as a key combination

The toggling Caps Lock feature can be re-mapped to two distinctive key combinations, such as Ctrl+Caps Lock for upper case typing and Esc to resume lower case typing.

The system of the present invention may somewhat hamper the use of the computerized system by an end user. However, the system of the present invention, apart from its main function of aiding system developers, also preferably aids the end user in the following ways:

The end user of the computerized system can learn how to avoid psychomotor errors, by comparing his actual action with his intended one.

Another advantage for the end user of the computerized system is that mode errors are automatically detected and that the end user obtains information regarding the system mode that caused her difficulty in operating the computerized system.

Another advantage for the end user of the computerized system is that problem reporting is integrated with regular operation of the computerized system. End users, such as in beta sites, are often required to report, typically manually, on usability problems. Typically, the reporting procedure is cumbersome and distracting, and end users try to avoid it. Computer guided reporting is easier to carry out. As a result, the proportion of problems reported may dramatically increase.

Advantages for the help desk personnel, in providing technical support to the end users of the computerized system, are now described:

A particular advantage of the computerized collection of usability problems provided by the present invention, relative to conventional manual collection of usability problems, for the help desk personnel, is that difficulties of the end user are recorded at the user site, thus eliminating the extra work required to enter the problem report manually.

Another advantage of the computerized collection of usability problems provided by the present invention, relative to conventional manual collection of usability problems, for the help desk personnel, is that the database of usability errors grows, because, using the preferred embodiment of the apparatus of the present invention, the users may report on many problems they would not have reported using conventional means.

A particular advantage of automated statistics collection for the help desk personnel is that the statistical data allow the technical supporter to evaluate t