Sound operated menu method and apparatus5999895
Abstract
The apparatus and method of the invention relate to data entry and menu selection. Applications include: (a) data entry for ideographic languages, including Chinese, Japanese and Korean; (b) fast food ordering; (c) correction of documents generated by optical character recognition; and (d) computer access and speech synthesis by persons temporarily or permanently lacking normal motor capabilities. In a preferred embodiment, each option of a menu is associated respectively with a selectable region displayed adjacent an edge of a display, forming a perimeter menu and leaving a region in the center of the perimeter menu for the output of an application program. Selectable regions may be on the display, outside the display, or both. A menu option may be selected by clicking on the associated selectable region, by dwelling on it for a selection threshold period or by a cursor path toward the selectable region, or by a combination thereof Remaining dwell time required to select a selectable region is preferably indicated by the brightness of the selectable region. Submenus of a perimeter menu may also be perimeter menus and the location of a submenu option may be foretold by the appearance of its parent menu option. Menu options may be ideographs sharing a sound, a structure or another characteristic. Ideographs, which may be homophones of one another, may be associated with colored indicating regions and selection of an ideograph may be made by speaking the name of the associated color.
Claims
I claim:
1. A method of selecting a sequence of one or more graphic symbols from a first plurality of sequences of one or more graphic symbols, one or more sequences of the first plurality of sequences including one or more ideographs in an ideographic language, said method comprising the steps of:
displaying on a display the first plurality of sequences, each of the first plurality of sequences having a first common characteristic in the ideographic language and each associated respectively with one of a first plurality of displayed sound indicators, each of the first sound indicators respectively indicating one of a first plurality of indicated sounds, each of the first indicated sounds differing from each other first indicated sound;
matching a first received sound to any one of the first indicated sounds; and
selecting the sequence associated with the matched first indicated sound.
2. The method of claim 1 wherein the first received sound is not a phonetic representation of the sequence associated with the matched first indicated sound.
3. The method of claim 1 wherein one of the first sound indicators is non-alphanumeric.
4. The method of claim 3 wherein the non-alphanumeric sound indicator is a hue.
5. The method of claim 3 wherein the non-alphanumeric indicator includes any one of a pitch indicator, a volume indicator, a sound duration indicator, a change in pitch indicator, a change in volume indicator, a shape indicator, a color saturation indicator, a brightness indicator, a location indicator, and an object indicator.
6. The method of claim 1 further comprising the step of indicating the association of each of the first plurality of sequences with the associated sound indicator.
7. The method of claim 1 wherein each of the first plurality of sequences belongs to any one of the Chinese language, Japanese language, Korean language, Picture Communication Symbols symbol set, Rebus symbol set, Picsyms symbol set, Pictogram Ideogram Communication Symbols symbol set, Yerkish symbol set, Blissymbolics symbol set, and a set of depictions of the signs of a manual sign language.
8. The method of claim 1 wherein the first common characteristic includes a phonetic unit, including either one of a Chinese pronunciation and a Japanese pronunciation.
9. The method of claim 8 wherein the phonetic unit is specified by voice.
10. The method of claim 8 wherein each of the first sound indicators includes a hue of a region on the display each region having a hue different from the hue of all the regions of all other first sound indicators; wherein each region is either adjacent or nearly adjacent another of the regions; wherein the displaying step displays each sequence intersecting the region of the associated sound indicator; and wherein the first received sound is the name of the hue of the sound indicator to be matched.
11. The method of claim 10 further comprising the step of inputting the selected sequence to an application program.
12. The method of claim 8 wherein the phonetic unit is specified by one or more keystrokes.
13. The method of claim 8 wherein the phonetic unit is a phonetic unit in a first dialect of the ideographic language, and wherein the matching step includes matching the phonetic unit to a phonetic unit in a second dialect of the ideographic language.
14. The method of claim 1 wherein the first common characteristic includes any one of:
(a) an intonation;
(b) a distinct sound;
(c) a part of speech;
(d) a meaning; and
(e) a meaning class.
15. The method of claim 1 wherein the first common characteristic includes any one of:
(a) a stroke used to draw an ideograph;
(b) a number of horizontal strokes;
(c) a number of vertical strokes;
(d) a number of total strokes;
(e) a stroke order;
(f) a radical;
(g) an ideograph;
(h) a kana;
(i) a diacritic;
(j) a classification of a part of an ideograph;
(k) a root;
(l) a graphic symbol;
(m) an appearance of a graphic symbol; and
(n) an attribute for optical recognition purposes.
16. The method of claim 1 wherein the displaying step precedes the matching step.
17. The method of claim 1 wherein the first plurality of sound indicators together at least partially circumscribe a region on the display.
18. The method of claim 17 wherein each of the first sound indicators is outside the display.
19. The method of claim 1 wherein each of the first sound indicators is on the display and either adjacent or nearly adjacent another of the first sound indicators.
20. The method of claim 1 further comprising the steps of:
receiving a signal; and
responsive to the received signal, displaying on the display a second plurality of sequences of one or more graphic symbols, each of the second plurality of sequences having the first common characteristic and each associated respectively with one of a second plurality of displayed sound indicators, each of the second sound indicators respectively indicating one of a second plurality of indicated sounds, each of the second indicated sounds differing from each other second indicated sound.
21. The method of claim 20 wherein each sequence of the first plurality of sequences has a frequency of use greater than each sequence of the second plurality of sequences.
22. The method of claim 1 further comprising the steps of:
first storing a representation of the first common characteristics; and
second storing a representation of a second common characteristic.
23. The method of claim 22 wherein the first storing step and the second storing step precede the displaying step.
24. The method of claim 23 followed by the steps of:
displaying oil the display a second plurality of sequences of one or more graphic symbols, each of the second plurality of sequences having the second common characteristic and each associated respectively with one of a second plurality of displayed sound indicators, each of the second sound indicators indicating one of a second plurality of indicated sounds, each of the second indicated sounds differing from each other second indicated sound;
matching a second received sound to any one of the second indicated sounds; and
selecting the sequence of the second plurality of sequences associated with the matched second indicated sound.
25. The method of claim 1 wherein one of the first sound indicators is numeric.
26. The method of claim 1 wherein the displaying step displays each sequence of the first plurality of sequences in an order responsive to a second characteristic of the sequence other than the first common characteristic, the second characteristic including any one of:
(a) a phonetic unit, including either one of a Chinese pronunciation and a Japanese pronunciation;
(b) an intonation;
(c) a distinct sound;
(d) a stroke used to draw an ideograph;
(e) a number of horizontal strokes;
(f) a number of vertical strokes;
(g) a number of total strokes;
(h) a stroke order;
(i) a radical;
(j) a part of speech;
(k) an ideograph;
(l) a kana;
(m) a diacritic;
(n) a classification of a part of an ideograph;
(o) a meaning;
(p) a meaning class;
(q) a root;
(r) a graphic symbol;
(s) an appearance of a graphic symbol;
(t) an attribute for optical recognition purposes;
(u) a frequency of use of each of the first plurality of sequences;
(v) a number of graphic symbols; and
(w) a frequency of use of each of a second plurality of sequences, of graphic symbols, each of the second plurality of sequences including one of the first plurality of sequences.
27. The method of claim 1 further comprising the step of inputting the selected sequence to an application program.
28. The method of claim 1 further comprising the steps of:
inputting a first sequence of one or more graphic symbols including one or more ideographs to an application program; and
replacing the first sequence with the selected sequence.
29. The method of claim 1 wherein the first common characteristic includes a first graphic symbol and the displaying step displays only a single instance of the first graphic symbol indicating that the first graphic symbol is included in each of the first plurality of sequences.
30. The method of claim 29 wherein the first graphic symbol is an ideograph.
31. The method of claim 29 wherein the displaying step displays the single instance of the first graphic symbol in a region of the display separate from the display of at least part of one of the first plurality of sequences or in a manner visibly distinct from the display of at least part of one of the first plurality of sequences.
32. The method of claim 1 wherein one of the first sound indicators includes at least part of a particular graphic symbol of the associated sequence.
33. The method of claim 32 wherein the first received sound includes any one of:
(a) a pronunciation of at least part of the particular graphic symbol;
(b) a name of at least part of the particular graphic symbol;
(c) a meaning of at least part of the particular graphic symbol;
(d) a meaning class of at least part of the particular graphic symbol;
(e) a part of speech of at least part of the particular graphic symbol;
(f) a classification of at least part of the particular graphic symbol;
(g) a stroke used to draw at least part of the particular graphic symbol;
(h) a number of horizontal strokes used to draw the particular graphic symbol;
(i) a number of vertical strokes used to draw the particular graphic symbol; and
(j) a number of total strokes used to draw the particular graphic symbol.
34. The method of claim 1 wherein the number of sequences in the first plurality of sequences is less than or equal to a predetermined number thereby increasing the likelihood of matching of the first received sound to the first indicated sound intended by a user.
35. For use in a language rich in homophones, a method of selecting a sequence of one or more graphic symbols in the language from a plurality of sequences of one or more graphic symbols in the language, said method comprising the steps of:
(a) displaying the plurality of sequences, each of the plurality of sequences associated respectively with a sound which is not a phonetic representation of the associated sequence, each of the plurality of sounds differing from each other sound in the plurality of sounds;
(b) receiving a sound signal;
(c) matching the received sound signal to any one of the plurality of sounds; and
(d) selecting the sequence associated with the matched sound.
36. The method of claim 35 wherein at least two of the sequences are homophones.
37. The method of claim 35 wherein each sequence represents a word in the language.
38. The method of claim 35 further comprising the steps of:
(e) displaying a plurality of sound indicators, each sound indicator associated respectively with one of the plurality of sounds; and
(f) indicating the association of each sequence to the sound indicator associated with the sound associated with the sequence.
39. A method of editing a first sequence of two or more words in an ideographic language, said method comprising the steps of:
displaying on a display:
(a) the first sequence, the first sequence including a plurality of second sequences of one or more graphic symbols in the language, each of the second sequences associated respectively with a sound indicator, each of the sound indicators respectively indicating a sound, each of the indicated sounds differing from each of the other indicated sounds; and
(b) the sound indicators;
receiving a sound;
matching the received sound to any one of the indicated sounds; and
replacing the second sequence associated with the matched indicated sound with a third sequence of one or more graphic symbols in the language, the third sequence including an ideograph in the language.
40. For use with a general purpose computer system including a display, an apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including an ideograph in an ideographic language, said apparatus comprising:
(a) a computer readable medium; and
(b) a program, stored on the medium and executable by the general purpose computer system, for:
(1) displaying the plurality of sequences on the display, each of the sequences associated respectively with a displayed sound indicator, each of the sound indicators respectively indicating a sound, each of the plurality of indicated sounds differing from each of the other indicated sounds, and wherein a particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator;
(2) matching a received sound to the sound indicated by the particular sound indicator; and
(3) selecting the sequence associated with the particular sound indicator.
41. The apparatus of claim 40 wherein each of the plurality of sequences has a common characteristic in the ideographic language.
42. The apparatus of claim 40 wherein the medium may be coupled to and uncoupled from the general purpose computer system.
43. The apparatus of claim 40 wherein the medium includes any one of:
(a) a random access memory;
(b) a magnetic store; and
(c) an optical store.
44. An apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language, said apparatus comprising:
(a) a display on which may be displayed the plurality of sequences, each of the plurality of sequences having a common characteristic in the ideographic language and each associated respectively with a displayed sound indicator, each of the sound indicators respectively indicating a sound, each of the plurality of indicated sounds differing from each of the other indicated sounds, and wherein a particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator; and
(b) a selection device for matching a received sound to the sound indicated by the particular sound indicator and for selecting the sequence associated with the particular sound indicator.
45. The apparatus of claim 44 wherein the common characteristic includes any one of the meaning classes:
(a) actions;
(b) amounts;
(c) animals;
(d) articles of clothing;
(e) bodily functions;
(f) buildings;
(g) business activities;
(h) cleaning activities;
(i) colors;
(j) communication activities;
(k) computer peripherals;
(l) days;
(m) devices used to maintain personal hygiene;
(n) directions;
(o) drinks;
(p) emergency conditions;
(q) emotions;
(r) financial activities;
(s) foods;
(t) government services;
(u) greetings;
(v) holidays;
(w) household appliances;
(x) illnesses;
(y) items of office equipment;
(z) jokes;
(aa) lengths;
(ab) locations, including locations frequented by a user of the apparatus;
(ac) meals;
(ad) means of transportation;
(ae) months;
(af) names;
(ag) numbers;
(ah) parts of the human body;
(ai) persons known to a user of the apparatus;
(aj) plants;
(ak) prosthetic device;
(al) recreational activities;
(am) rehabilitation activities;
(an) relative locations;
(ao) school activities;
(ap) shapes;
(aq) shopping activities;
(ar) sizes;
(as) smells;
(at) sports;
(au) tactile attributes;
(av) tastes;
(aw) telephone numbers;
(ax) temperatures;
(ay) times;
(az) topics of study;
(ba) utterances used as acknowledgements in conversation without conveying new substantive information;
(bb) utterances used to bid for a turn to speak in conversation;
(bc) weights; and
(bd) work activities.
46. The apparatus of claim 44 further comprising:
(a) a computer system including the display and a sound receiver for receiving the received sound; and
(b) a program, executable on the computer system;
and wherein the selection device is formed by the combination of the progam and the computer system.
47. The apparatus of claim 44 wherein the selection device includes a processor, a program executable oil the processor, and a medium readable by the processor; wherein the processor, during execution of the program, compares a representation of the received sound to a representation of each of the indicated sounds, the representation of each of the indicated sounds stored on the medium and read by the processor therefrom; and wherein the sound indicated by tile particular sound indicator is tile indicated sound whose representation is closest to the representation of the received sound.
48. The apparatus of claim 44 wherein the selection device includes a speech recognizer for comparing the received sound to the sound indicated by the particular sound indicator.
49. The apparatus of claim 40 wherein the program is transferred from the medium to the general purpose computer system via a telecommunications or data communications network.
50. The apparatus of claim 44 wherein the sound indicated by the particular sound indicator is either one of:
(a) a sound found in normal speech; and
(b) a sound not found in normal speech.
51. The apparatus of claim 44 wherein each of the plurality of sequences belongs to any one of the Chinese language, Japanese language, Korean language, Picture Communication Symbols symbol set, Rebus symbol set, Picsyms symbol set, Pictogram Ideogram Communication Symbols symbol set, Yerkish symbol set, Blissymbolics symbol set, and a set of depictions of the signs of a manual sign language.
52. The apparatus of claim 44 wherein the plurality of sound indicators together at least partially circumscribe a region on the display.
53. The apparatus of claim 52 wherein each of the sound indicators is outside the display.
54. An apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of tile plurality of sequences including one or more ideographs in an ideographic language, said apparatus comprising:
(a) means for displaying tile plurality of sequences, each of tile plurality of sequences having a common characteristic in the ideographic language and each associated respectively with a displayed sound indicator, each of the sound indicators respectively indicating a sound, each of the plurality of indicated sounds differing from each of the other indicated sounds, and wherein a particular one of the sound indicators is not a phonetic representation of the sequence associated with tile particular sound indicator: and
(b) means for:
(1) matching a sound to the sound indicated by the particular sound indicator; and
(2) selecting the sequence associated with the particular sound indicator.
55. A device controller comprising:
(a) a display on which may be displayed a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language, each of the plurality of sequences having a common characteristic in the ideographic language and each associated respectively with a displayed sound indicator, each of the sound indicators respectively indicating a sound, each of the plurality of indicated sounds differing from each of the other indicated sounds, and wherein a particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator; and
(b) a signal generating device, coupled to a controlled device, for:
(1) matching a sound to the sound indicated by the particular sound indicator, the sequence associated with the particular sound indicator representing a function of the controlled device; and
(2) generating a device control signal corresponding to the function of the controlled device.
56. The apparatus of claim 55 wherein the controlled device includes any one of:
(a) a computer peripheral;
(b) a device capable of playing previously recorded sound;
(c) a device capable of playing previously recorded video;
(d) a household appliance;
(e) a lamp;
(f) a microprocessor;
(g) a motorized transport device including either one of a scooter and a wheelchair;
(h) a radio;
(i) a robot;
(j) a security system;
(k) a television;
(l) a thermostat;
(m) a voice output device;
(n) a workstation;
(o) an alarm; and
(p) an office appliance.
Description
FIELD OF THE INVENTION
The present invention relates generally to interactive display terminals and interactive display methods, and more particularly, to interactive display terminals and interactive display methods for use by persons temporarily or permanently lacking normal motor capabilities. It also relates to systems and methods for the assessment of the motor capabilities of persons lacking normal motor capabilities. It further relates to interactive display terminals and interactive display methods for use in speech synthesis for persons having impaired speech. It also relates to systems and methods for the control of devices, including appliances, by persons lacking normal motor capabilities. It further relates to interactive display terminals and interactive display methods for selecting one menu option from a menu. It further relates to systems and methods utilizing sound recognition for selecting a menu option from a menu. It further relates to data and order entry systems including, and data and order entry methods utilizing, an interactive display terminal. It also relates to interactive display terminals and interactive display methods for displaying and selecting ideographic characters, such as are used in the Chinese, Japanese and Korean languages. It also relates to interactive display terminals and interactive display methods for producing an indication of progress toward and/or away from selection of a menu option.
A portion of the disclosure of this patent document contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure, as it appears in the World Intellectual Property Organization patent file or records, but otherwise reserves all copyright rights whatsoever.
BACKGROUND ART
Many persons suffer from various neurogenic muscular disorders, such as Cerebral Palsy ("CP"), Traumatic Brain Injury, Spinal Cord Injury, Muscular Dystrophy, Amyotrophic Lateral Sclerosis and Multiple Sclerosis. These conditions can result in a reduced ability to voluntarily control or prevent the movement of parts of the body, including the head, limbs and digits, muscle stiffness, weakness, limited range of motion, abnormal posture, involuntary muscle tremors, involuntary muscle activity causing involuntary motion, impaired ability to voluntarily stop motion, impaired ability to coordinate muscle activity, and/or impaired ability to sense the position of a part of the body. Any one of these symptoms may impair an affected individual's fine motor control. Moreover, while some individuals affected by a neuromuscular disorder may be able to exercise fine motor control with enormous effort, the struggle to do so often fatigues the individual, limiting the period of time the individual is capable or comfortable performing the fine motor control task.
Neuromuscular disorders are often systemic in effect, impairing an individual's ability to operate prosthetic devices, such as a wheelchair, and to perform the activities of daily life, such as speaking, walking and operating household appliances. Speech is frequently affected since the mechanics of producing speech require coordination of many muscle groups--the muscles of the diaphragm which push air over the vocal cords, the muscles of the larynx, jaws, tongue and lips. The inability to use or coordinate these muscle groups may result in impaired speech. Depending upon the degree of impairment, speech may be totally absent, present but impaired to the point of unintelligibility, or intelligible on the whole but with occasional unintelligible words. The ability to walk is often affected since walking requires coordination and voluntary control of many muscle groups. Furthermore, impaired fine motor control may prevent or impede an individuals from effectively operating household appliances or computer input devices.
Devices are available which produce speech, control appliances and facilitate computer access for some persons having neuromuscular disorders ("NMD operators"). Devices which produce speech for individuals whose own speech is impaired, called Augmentative and Alternative Communication ("AAC") devices, allow the operator to select words or phrases by spelling the words, by specifying an abbreviation for the phrase or by selecting a sequence of symbols, and then speak the selected words or phrases using an electronic speech synthesizer. However, due to the systemic nature of neuromuscular disorders, NMD operators are often unable to efficiently use a standard keyboard and mouse. For example, an NMD operator who is unable to stop the movement of a limb with precision, when attempting to use a keyboard or mouse, may move his arm toward the target key or move the cursor toward the target object on the display but overshoot the target. If he has involuntary tremors and cannot hold a limb still, then, when attempting to use a keyboard, he may hit keys adjacent to his target key. If he has involuntary motion moving left to right, then, when attempting to use a keyboard, he may have difficulty accessing an intended key on the right side of the keyboard.
The benefits of interfacing an NMD operator to a general purpose computer so that he may control the computer and devices attached to it ("computer access") are both numerous, because many of the problems faced by the disabled are susceptible to a computer-driven solution, and profound, because of the psychological deprivation occasioned by a severe physical disability. The benefits potentially obtained through computer access for individuals affected by neuromuscular disorders include:
a. Speech synthesis. A computer connected to a speech synthesizer enables an NMD operator with impaired speech to direct the computer to speak for him.
b. Device control. A user who is physically unable to operate a household appliance, for example, a television, video cassette recorder, compact disc player, radio, alarm clock, telephone, light, thermostat, dimmer or power switch, may be able to control the appliance via a computer equipped with an interface he can control.
c. Access to general purpose computer applications. NMD operators may make use of the same general purpose computer application programs ("applications") as able-bodied users, including applications for word processing, database, computer-aided instruction, access to literature accessible via computer, spreadsheet, time management and computer utilities.
d. Enhanced self-esteem and peer approval. Adolescents with CP are obviously different from their peers. They are often surrounded by non-normative assistive technology, e.g. wheelchair or walker, special school bus equipped with a chair lift, stair lift, standing aid, AAC device, feeding apparatus, bath seat, toiletting apparatus, etc. In addition, they may drool, lacking the ability to coordinate lip closure with swallowing. Nonetheless they are adolescents and need peer approval to support them in their maturation from dependent children to independent adults. Today, demonstrated facility with a computer is an emblem of intelligence among adolescents, so computer use provides adolescents the opportunity to prove their intelligence and thus potentially rewards NMD operators with both self-esteem and peer approval.
e. Privacy. Some severely disabled school-aged children require nearly constant physical assistance to transfer them to and from bed, to feed them, help them with toiletting and personal hygiene, etc. Because they are constantly attended, all their mistakes in class or when doing homework are known to their attendant, often a family member. They do not have the opportunity to make mistakes in private. Computer use, if it can be done without assistance, affords the NMD operator the opportunity to avoid the embarrassment of showing their failings to their attendant.
f. Expanded personal interaction. Some severely disabled individuals, e.g. quadriplegics, are essentially incarcerated by their disability. They interact with their family or their caretakers, depending upon whether they live at home or in an institution. Their circle of friends is often very small. Using a computer and a modem, they can expand their circle of friends to include the tens of thousands of people who periodically connect to worldwide electronic networks to trade information on topics of mutual interest. Moreover, the interaction via present computer networks is mostly textual; there is no voice or visual interaction between users. Since messages are customarily composed and read off-line to minimize connect time charges, no one even knows how long it took the sender to enter the text. Electronic networks thus afford the disabled user an opportunity to relate to others on an equal footing, not as a disabled person to his able-bodied peers, something many NMD operators dearly want to do but were never able to.
NMD operators vary widely in their motor capabilities. Even individuals having the same medical diagnosis may require completely different technologies for computer access. Many NMD operators are able to use an oversize keyboard, a device having a pressure-sensitive surface divided into squares, each square associated with a letter of the alphabet. The squares may be sized to match the operator's abilities, but typically each square is two inches on either side. NMD operators who are unable to efficiently use an oversize keyboard may use another conventional computer access solution, called an "on-screen keyboard", which, as illustrated in FIG. 1, is a picture of keyboard drawn on a computer display (1101). The operator selects a letter by pointing to that letter's key image on the display with a pointing device ("pointer"), then indicating that he has reached his target either by operating a switch, a process called selection by click, or by maintaining the location indicated by the pointer ("dwelling") on the key image for a predetermined period of time (the "selection threshold"), a process called selection by dwell. Switch operation includes, but is not limited to, each of the following: opening the switch, closing the switch, opening the switch multiple times within a predetermined period, and closing the switch multiple times within a predetermined period. A program executing on the computer determines which letter the operator has selected and processes the letter or passes it to some other application program which processes the letter as if it came from a true keyboard.
Conventional pointing devices include a mouse, trackball, joystick (which may be integrated into a keyboard, e.g. TrackPoint II.RTM.), stylus and graphics tablet, lightpen, thumb wheel, touch screen, touch panel, head pointer, occulometer, intraoral pointer and eye tracker. They may be active, e.g. a lightpen that emits an infrared beam, or passive, e.g. an eye tracker that uses images of an individual's eyes to determine where his eyes are focusing. Conventional switches include a button on the mouse, a switch in the tip of the stylus actuated by pressure or the release of pressure, a switch mounted on the user's wheelchair operated by a turn of the head to or the switch below a keyboard key.
Dwell time may be continuous or discontinuous depending upon the operator's motor capabilities. In continuous dwelling, if the operator moves the cursor from one key image to another region of the display, the time accumulated on the key image is discarded so that if the operator returns to that key image he must dwell on it for the full selection threshold to select it. Discontinuous dwelling, by contrast, compensates for involuntary tremors which pull the operator off the desired key image. Accumulated dwell time on a key image is remembered, so that on return to a key image, the operator need only dwell for a period equal to the difference between the selection threshold and the previously accumulated dwell time for that key image. Accumulated dwell time is reset to zero for all key images following the selection of any one key image. Conventional on-screen keyboards do not indicate to the operator the dwell time associated with any key image.
There may be a single selection threshold period for all key images or each key image may be associated with its own selection threshold period. In the latter case, keys associated with shorter selection threshold periods are easier to select than keys associated with longer selection threshold periods.
As was mentioned earlier, computer access permits an NMD operator to run a variety of applications. One such application is speech synthesis. In a computer-based speech synthesis system, a computer system displaying an on-screen keyboard is connected to a speech synthesizer. The operator spells the desired word or words using the on-screen keyboard. These are then spoken by the speech synthesizer. Another application of the on-screen keyboard is word processing. FIG. 2 illustrates an example of a combined display of an on-screen keyboard and a word processing application program. The on-screen keyboard (0201) is shown on the lower portion of a display connected to a computer system (not shown) which also executes the word processing application program whose output (0203) appears on the upper portion of the display. Letters selected by the operator are input to the word processing application program.
Due to impaired fine motor control, many NMD operators have difficulty selecting a key image by click or by dwell and this difficulty increases as the size of the key image decreases. FIG. 1 shows an on-screen keyboard containing 81 total keys including 26 alphabetic keys, 10 numeric keys, 12 function keys, 4 arrow keys and 29 special purpose keys. Drawing this many key images on a display restricts the size of each key image making each very difficult for many NMD operators to select.
When a display is shared between application program output (0203) and an on-screen keyboard (0201), as is the display shown in FIG. 2, the size of each key image must be reduced from its size in FIG. 1 to allow space for the application program output. Thus, as more display space is allotted to application program output, the key images become more difficult for an NMD operator to select.
Many NMD operators have difficulty using the conventional dwell selectable on-screen keyboard because they cannot maintain a steady pointer position. The body member with which they control the pointer may move slightly ("drift") when they want it to remain still. One approach to this problem is a variation of the on-screen keyboard, depicted in FIGS. 3, 4 and 5 and called a quaternary on-screen keyboard ("quaternary keyboard"). The quaternary keyboard provides for larger key images. The alphabet is divided into four groups of letters, each displayed in one of the four quadrants (1302), (1304), (1306) and (1308) of the display, as shown in FIG. 3. The operator selects one of the four groups by, for example, pointing to and dwelling on one quadrant of the display. The selected group is then exploded into four subgroups, each displayed in one quadrant of the display, as shown in FIG. 4. Once more the operator selects one of the four. The selected group is exploded into four letters and each letter displayed in one quadrant of the display, as shown in FIG. 5. The operator again selects one of the four. This letter is then input to an application program (not shown).
The quaternary keyboard illustrates the use of a menu hierarchy in computer access. Each of the four groups of letters (1302), (1304), (1306) and (1308) is a menu option. Each of these menu options is itself a menu which includes other menu options. A menu hierarchy exists if at least one of a menu's menu options is itself a menu. Hereinafter, a menu accessed from another menu may be called a submenu, and the options of the submenu may be called submenu options. If a menu hierarchy is narrow and deep, many selections are required to make the desired choice. If a menu hierarchy is broad and shallow, each layer is composed of many menu options.
The quaternary keyboard greatly expands the size of a single key image and thus accommodates certain NMD operators with drift or involuntary tremors. The cost of this adjustment is high Instead of selecting a letter with one pointing motion and dwelling for one selection threshold, the quaternary keyboard requires three pointing motions and dwelling for three selection thresholds. Thus the operator's productivity is dramatically reduced from the standard on-screen keyboard depicted in FIG. 1.
The computer access advantage gained from the quaternary keyboard is greatest when the quaternary keyboard occupies the entire display. In this configuration the size of each of the four active display regions is maximized, making them easier to hit and dwell on for the operator. However, this configuration allows no room on the display for the output of the application program being run by the operator, the reason he is sitting at the computer in the first place. This does not prevent the on-screen keyboard from passing letters to the application program since an application program need not be visible to be active, but it does prevent the operator from seeing what the application program has to show him. The more of the application program output that is displayed, the smaller the on-screen keyboard, the smaller each active region of the on-screen keyboard and the more difficult access becomes. FIG. 6 illustrates a display combining a quaternary keyboard and output from two application programs.
Another conventional structure for selecting of a menu option from a menu is a pie menu A pie menu is an opaque region on a display divided into selectable slices, each slice associated with a menu option. The pie menu suffers some of the drawbacks discussed above, particularly that, while displayed the pie menu occupies more space than a linear menu and obscures much of the output of the operator's application program. For illustrations and a discussion of pie menus, see Callahan, Jack et. al., "An Empirical Comparison of Pie vs. Linear Menus", Computer Science Technical Report Series, CS-TR-1919, University of Maryland, College Park, Md., September 1987.
NMD operators who cannot effectively use a conventional keyboard or a pointing device may use a computer access method called "joystick patterns". FIG. 7 depicts a conventional joystick pattern device. The device (1602) is connected to a joystick and to a computer. The operator pushes the joystick to the top, bottom, left, right, top left corner, top right corner, lower left corner or lower right corner, closing one of eight switch contacts within the joystick housing. That switch position is then indicated on the display (1604). A sequence of consecutive of switch closures encodes a letter or other programmed output that the device (1602) displays on an LCD display (1606) and sends to the connected computer, simulating keyboard input.
The conventional joystick pattern device is ill-suited for many NMD operators. The involuntary tremors common some neuromuscular disorders may result in unintended switch closures. In addition, the device does not provide an indication that the operator is moving a body member in an unintended direction until switch closure occurs. For example, an operator with CP who intends to move the joystick the right but actually moves it to the upper right receives no indication from the device, prior to switch closure, that he's not on target. Moreover, the device requires that the operator memorize the encoding of each letter or other output since there's no indication on the display (1606) which sequence encodes which letter. Further, the device provides no support for head pointing, although the head is often the best controlled part of an NMD operator's body.
NMD operators who cannot effectively use either a conventional keyboard or a pointing device but can reliably actuate a switch may use a computer access method called "scanning", which is subdivided by cursor control technique into three types of scanning: automatic, directed and step. In automatic scanning all the operators' options, for example, the letters of the alphabet, appear on either a static or dynamic display (depending upon the implementation), organized in rows and columns. At the scanning interval, usually about one second, a cursor moves from one row to the next. When the cursor indicates the row containing the letter the operator wants, he closes a switch. The machine now moves the cursor from one letter to the next within the selected row until the operator closes the switch again. The operator has now selected one letter. In directed scanning, like automatic scanning, the cursor moves at the frequency determined by the scanning interval, however, it moves only when the switch is closed. To select an option, such as a row or a letter in a row, the operator opens the switch while the cursor indicates the desired option. In step scanning, the cursor moves with each switch activation.
As one can well imagine, writing a sentence via any of these scanning techniques is an extremely slow process, since selecting a single letter may take many seconds.
Problems of computer access cascade and affect the quality of verbal interactions between AAC device operators ("AAC operators") and others. People speak much faster than they type. Not surprisingly, operators who speak with AAC devices, particularly NMD operators whose motor deficits impair their ability to use a keyboard, lag substantially in their conversations. The slow pace of an AAC operator's word production disrupts normal verbal interaction. Speaking persons, accustomed or not to the AAC operator's slow rate, often lose patience in conversations with AAC operators. They may prematurely terminate the conversation, read the AAC device display in an attempt to guess at the AAC operator's intended utterance and so accelerate the interaction, lead the AAC operator, ask predominantly yes/no questions, change the topic of conversation with little input from the AAC operator and otherwise dominate the interaction. The AAC operator often has difficulty participating as an equal partner in the conversation. He may be unable to change the topic, interject a humorous comment in a timely fashion or respond to a question before the speaking person changes the topic. Slow AAC operators may be perceived as mentally slow. Thus the quality of verbal interactions where one party uses an AAC device to speak depends significantly upon the AAC operator's rate of word production.
Increasing an operator's letter or menu option selection rate proportionately increases his word production rate and increases the operator's productivity in data entry generally. Letter or menu option selection time includes the time the operator requires (a) to comprehend the menu options displayed, (b) to move the pointer to the desired menu option on the display, and, in selection by dwell, (c) the selection threshold period, or, in selection by click, (c) the time required to operate the switch. Decreasing any one of these increases the operator's productivity, assuming all other steps in the selection process are unaffected.
Personal interactions are composed of more than speech alone. People in conversation gesture to one another, use facial expressions, change the object of their gaze and make non-speech utterances (e.g. "hmmm-mmmm") to bid for a turn to speak, to grant such a bid made by the other party, to request to continue speaking and to acknowledge, accept or dispute what has been said. Ideally, the production of speech from an AAC devices does not distract the AAC operator from the personal interaction and subject matter of the conversation. This is possible if the operator habituates the AAC device access technique and menu structure, producing speech without focusing on each step of the process, much as automobile drivers habituate mechanical tasks, such as changing gears and switching between foot pedals, and focus their attention on pedestrians or traffic lights while operating their vehicle.
Another consequence of personal interaction during conversation for an AAC device operator is that the operator needs a way to easily enable and disable the AAC device operator interface so that movement the operator makes during personal interaction, for example, nodding his head, is not interpreted by the AAC device.
As noted previously, neurogenic muscular disorders may impair the ability of an individual to sense the position of a body member. An NMD operator thus relies more than his able-bodied peer on the location of a cursor or similar automated indication of body member position. Conventional access methods which use a pointer do not provide additional feedback to the operator of the position of a body member.
Access methods which require the NMD operator to make the same movement for most selections, such as single switch access, mouth sticking (the use of a small rod held in the mouth and used to depress keys on a keyboard) and head sticking (the use of a rod mounted on the head and used to depress keys on a keyboard), may result in repetitive motion injury, particularly after years of use.
The need for quick selection from a menu also arises from the use of optical character recognition ("OCR") systems which attempt to recognize graphic symbols and words based on attributes for optical recognition purposes, for example, the appearance of graphic symbols, the ratio of dark space to light space within part or all of a graphic symbol, the ratio of dark space in one part of a graphic symbol to the dark space in another part of the graphic symbol, and the derivative of darkness over the scan of the graphic symbol. OCR systems convert the contents of a typewritten document into a computer encoding of the same. OCR systems at times are unable to recognize a graphic symbol or word, or may err in selecting a graphic symbol or word from a plurality of candidates. Therefore, following optical character recognition, a human may proofread and correct the computer encoded document. The proofreader may indicate where an error or omission in the computer encoded document occurred and may select from a plurality of menu options, each representing a candidate graphic symbol or word.
There are several aspects of the invention, each addressing one or more of the problems described above and/or one or more problems specifically addressed by that aspect of the invention. The objects, disclosure and description of each aspect is separately described below under one of the headings: (A) Perimeter Menu, (B) Confinement, (C) Dwell, (D) Path Directness, (E) Intersection, (F) Alignment, (G) Length Order, (H) Location Indication, (I) Sound Match, and (J) Ideographic Languages,. Where there is background art applicable to an aspect in addition to that described above, the additional background art is described below.
A & B. Perimeter Menu and Confinement
One object of the invention is to facilitate computer access by a disabled operator.
A further object of the Perimeter Menu and Confinement aspects of the invention is to facilitate menu selection by an operator having impaired ability to maintain a body member in a steady position.
Another object of the Perimeter Menu and Confinement aspects of the invention is to facilitate menu selection by an operator having impaired ability to stop motion.
Yet another object of the Perimeter Menu and Confinement aspects of the invention is to simultaneously display an application program window and a computer access menu which does not obstruct the application program window.
Another object of the Perimeter Menu and Confinement aspects of the invention is to allow an operator to enable and disable a menu.
A still further object of the Perimeter Menu and Confinement aspects of the invention is to synthesize speech for an operator having impaired speech and impaired fine motor control.
Still another object of the Perimeter Menu and Confinement aspects of the invention is to facilitate device control for a disabled operator.
Another object of the Perimeter Menu and Confinement aspects of the invention is to reduce the cognitive demand of speech synthesis for the disabled.
A still further object of the Perimeter Menu and Confinement aspects of the invention is to enlarge the effective area of a selectable region without concomitantly reducing the area available for information display.
Another object of the Perimeter Menu and Confinement aspects of the invention is to speed data entry.
Yet another object of the Perimeter Menu and Confinement aspects of the invention is to facilitate computer access for an operator having impaired ability to sense the position of a body member used for computer access.
Another object of the invention is to facilitate the correction of errors or completion of omissions in a computer encoded document produced, at least in part, by optical character recognition.
C. Dwell
Conventional systems allowing selection by dwell do not provide an indication to the operator of either how much dwell time has been accumulated for any selectable region or how much more dwell time is required to select a selectable region. Consequently, an operator of a conventional system who is dwelling on a intended selectable region has no indication, other than his estimation from prior use of the system, that he has nearly made his selection and can plan his next movement to the next selectable region or that he has very nearly made his selection and can begin moving to the next selectable region. Furthermore, an operator of a conventional system who is dwelling on an unintended selectable region, has no indication, other than his estimation from prior use of the system, of how close he is to making an unintended selection and thus how important it is to act quickly. Conventional systems using discontinuous dwell give no indication of the accumulated dwell time associated with a selectable region either when the operator dwells on that region or when the operator ceases dwelling on that region. Some disabled users can dwell relatively easily on their intended targets for short periods of time, but have difficulty dwelling for long periods. If such an operator knows that only a little more dwell time is needed he may be able to satisfy the dwell time required for selection, without preparing himself to dwell for an extended period.
Conventional menu-driven data entry and order entry systems incorporating pointing at intended selections employ a two step selection procedure. In the first step the operator indicates, with a pointer, his intended selection. The system then provides feedback, for example, by highlighting the indicated selection, showing which selection the operator has indicated. In the second step, the operator selects the indicated selection, for example, by operating a switch. Thus, conventional data entry and order entry systems are ill-suited to circumstances where the operator cannot easily operate a switch while maintaining the pointer on the intended selection.
While the two step procedure is not complicated, many operators require some training to learn it, and, if they are infrequent users of the system, these operators may require refresher training. Simplifying the procedure further would lessen the need for initial and refresher training.
One object of the invention is to facilitate the use of systems allowing selection by dwell.
Still another object of the invention is to facilitate device control by the disabled.
A further object of the invention is to increase the independence of the disabled.
Yet another object of the invention is to facilitate the use of a data entry or order entry system by an intermittent operator.
Another object of the invention is to facilitate ordering by someone seated in a vehicle.
D. Path Directness
The on-screen keyboard with dwell selectable key images is ill-suited for use by many NMD operators. Selection by dwell may fatigue NMD operators or may require greater fine motor control than they bring to this task. Operators with impaired ability to stop motion and those having involuntary tremors have difficulty maintaining the location indicated by a pointer on a key image for a period sufficient to distinguish intentional dwelling from unintentional dwelling. Consequently, some NMD operators who try to use on-screen keyboards often miss their target key images and/or accidentally select unintended key images. Following such an error, the operator must erase his erroneous selection by selecting the backspace or undo key. As the number of erroneous selections increases, the operator's productivity decreases markedly, since each error requires a correction in which there might be another error.
Conventional on-screen keyboards require the ability to select by dwell or by click and thus are limited to operators with these capabilities. Conventional on-screen keyboards do not utilize the relatively intact motor capabilities of some NMD operators to compensate for impaired ability to select by dwell or by click or to speed up the slow process of selecting by dwell. For example, while an NMD operator may overshoot a key image, his directional control may be relatively intact. Conventional on-screen keyboards do not exploit this capability.
The dominant computer operating system for graphic applications on general purpose computer systems today is the Windows.RTM. Operating System. Windows.RTM. assigns meaning to the cursor location. When the operator moves the cursor on top of a menu item and clicks, Windows.RTM. interprets the action as manifesting an intent to choose that menu item. The operator's path to that menu item, whether direct or circuitous, is irrelevant. Operators who can move toward a target accurately but cannot maintain the location indicated by a pointer on the target cannot effectively use standard Windows.RTM. applications through the conventional interface to these applications.
Often NMD operators cannot steady a pointer while operating a switch; the act of operating the switch triggers involuntary muscle activity pulling the cursor off target. For these operators, conventional selection by click is not practicable. Conventional selection by dwell also requires greater fine motor control than many NMD operators bring to this task. Operators with impaired ability to stop motion may overshoot their intended target. Operators whose voluntary muscle activity is accompanied by some involuntary muscle activity affecting their directional control often cannot point accurately. Operators with involuntary tremors often cannot maintain the location indicated by a pointer on a key image. Consequently, NMD operators who try to use on-screen keyboards often miss their target key images and accidentally select unintended key images. Following such an error, the operator must erase his erroneous selection by selecting the backspace or undo key. As the number of erroneous selections increases, the operator's productivity decreases markedly, since each error requires a correction in which there might be another error.
Measures of an individual's fine motor control assist a physician or therapist in evaluating the effectiveness of a treatment program, including assistive technology, and in gauging the severity of a disability. Such measures help the physician or therapist in determining what treatment course to pursue and whether the severity of a certain disability justifies the risk of a particular treatment option, such as neurosurgery.
One object of the invention is to facilitate selection of an option from a menu.
Another object of the invention is to indicate to an operator moving a cursor toward an option in a menu displayed by a computer system, which option the computer system believes the operator is moving toward.
Another object of the invention is to display a menu on a display so that a large contiguous area on the display is not obstructed by the menu.
Still another object of the invention is to make use, in computer access, of relatively unimpaired directional control in persons having impaired fine motor control.
A further object of the invention is to indicate to an operator moving a cursor toward a dwell-selectable option in a menu, a changed selection threshold of the dwell-selectable option.
Yet another object of the invention is to more efficiently select an option from a menu on a display.
Another object of the invention is to speed up selection of an option from a menu by an operator having impaired ability to operate a switch while the operator simultaneously keeps a cursor location within a region on a display.
Another object of the invention is to help an operator with a disability control a pointer.
Yet another object of the invention is to speed data entry by an individual with a disability.
A still further object of the invention is to facilitate artificial speech generation by a person having impaired speech due to a neurogenic muscular disorder.
Yet another object of the invention is to facilitate device control by a person having a neurogenic muscular disorder.
Another object of the invention is to measure an individual's ability to move one of the individual's body members in a direct path from a starting position to an ending position.
Yet another object of the invention is to facilitate computer access for an individual who cannot stop movement cleanly.
E. Intersection
One object of the invention is to facilitate computer access by an operator having impaired ability to maintain a body member in a steady position.
Another object of the invention is to facilitate the selection of a desired menu option by an operator having impaired fine motor control.
A further object of the invention is to synthesize speech for an operator having impaired speech and impaired motor control.
Yet another object of the invention is to use an operator's directional control in computer access.
A still further object of the invention is, in selecting a menu option from a menu of dwell-selectable menu options, to compensate for an operator's impaired ability to maintain a body member in a steady position by using the operator's relatively intact motor capability .
F. Alignment
Conventional on-screen keyboards do not compensate for NMD operators' inability to stop motion. Suppose the operator has been fitted with a head pointing device so that his head motion moves the cursor and that he's using the quaternary keyboard shown in FIGS. 3, 4 and 5. Assume further that, as he attempts to point to the quadrant containing the "j" key image (1308) in FIG. 3, he is unable to stop on that quadrant and continues turning 20 more degrees to the left. There are two known ways of responding to this situation: (1) the cursor may continue to track the operator's motion and disappear from the display, leaving no indication to the operator of the location of the cursor and consequently causing some operator disorientation, or (2) the cursor may "stick", i.e. remain confined to the display, at, for example, point (1312). Conventional on-screen keyboards respond in this way. The operator's line of sight is now 20 degrees to the left of the cursor location. After the dwell period, the quadrant (1308) is selected and FIG. 4 is displayed. The cursor hasn't moved. It is now at point (1320). Assume that again the operator attempts to point to the quadrant containing the "j" key image (1324) in FIG. 4. As the operator turns his head to the right, the cursor immediately moves with him. Thus, the operator's line of sight remains 20 degrees to the left of the cursor location as the cursor moves to the right across the display. The operator must watch the cursor out of his right eye. The problem is aggravated if either the operator cannot cleanly stop or if he drifts as he dwells. Assume that while attempting to dwell on quadrant (1324) the operator drifts 25 degrees past the bottom of the screen. His line of sight is now 25 degrees below and 20 degrees to the left of the cursor. To correct this misalignment in the conventional quaternary keyboard, the operator must turn his head to the right, "stick" the cursor against the right edge of the display, and continue turning 20 degrees until he has the cursor in his line of sight. Then he must lift his head until he sticks the cursor against the top edge and continue lifting 25 degrees more. Alternatively, in this scenario, the operator could stick the cursor in the upper right corner of the display and simultaneously rotate his head up and to the right until he brought the cursor into his line of sight.
Alignment is also a problematic for NMD operators who use a pointer, such as a mouse, with which the operator indicates by a location on a surface, e.g. a desk top, which corresponds to a desired location on the display, and achieve alignment by removing the pointer from the surface, e.g. lifting the mouse, moving the mouse, then replacing it on the surface. Due to impaired fine motor control, many NMD operators cannot remove a pointer from the surface and replace it on the surface at a desired location without unintentional movement or extraordinary effort. For these operators, alignment cannot be effectively achieved through conventional means.
In summary, misalignment interferes with accurate pointing and the process of correcting for misalignment may result in the selection of unintended key images.
One object of the invention is to allow an operator to align a pointer with a location on a surface.
Another object of the invention is to indicate to an operator a location on a surface with which he may align a pointer.
Still another object of the invention is to indicate to an operator when he may align a pointer with a location on a surface.
A further object of the invention is to allow an operator having impaired motor control to align a pointer with a cursor.
G. Length Order
As noted previously, one of the elements determining the menu option selection time is the time the operator requires to comprehend the menu options displayed. This time may be reduced if the operator can limit the number of menu options he searches in looking for his desired menu option.
Conventional word prediction systems attempt to reduce this operator search time. The operator of a conventional word prediction system may, for example, select the letter "p". The system displays some number, say six, of the most frequently used words beginning with the letter "p". Conventionally these six words are displayed either in alphabetic order or in order of frequency of use. Assuming the operator does not see his desired word on the display, he selects another letter, say "r". The system then displays the six most frequently used words beginning with the letters "pr".
Searching a displayed list of words in alphabetic order requires that the operator focus his attention on the selection task, as opposed to the information content of the conversation or other task the operator is engaged in. Further, determining whether a given word is alphabetically greater or lesser than a desired word takes substantial time, slowing the selection process. An alphabetically ordered list is of limited use to an individual who has below normal spelling ability, a frequent problem among individuals with impaired speech. Ordering words by frequency of use often does not limit the number of words the operator must search. The word at the bottom of the displayed list, for example, the sixth most frequently used word beginning with the letters "pr" may be a very common word, even though it is less frequently used than the other five displayed words.
One object of the invention is to reduce the time an operator requires to comprehend displayed menu options.
Another object of the invention is to reduce operator search time.
Still another object of the invention is to limit the number of menu options an operator searches for in looking for his desired menu option.
A further object of the invention is to speed data entry.
Yet another object of the invention is to increase productivity in speech synthesis for an operator having impaired speech.
H. Location Indication
The difficulties experienced by NMD operators in pointing to relatively small selectable regions have already been described. One approach to these difficulties is to enlarge the on-screen selectable region, illustrated by the quaternary expansion on-screen keyboard already described. Another approach is the conventional eye gaze system for a speech impaired individual, depicted in FIG. 8. The system consists of a plexiglass frame (6352) having a centrally located aperture (6354). The eye gaze system is positioned between the speech impaired individual and person with whom the speech impaired individual is communicating. There are eight groups of five squares each on the plexiglass frame. Each square within each group of five squares is color coded, e.g. red, blue, green, yellow and clear, matching the color on each of the four corners of the plexiglass frame. The clear square matches the aperture (6354). All squares are labeled with symbols representing items to be communicated. These labels are not shown in FIG. 8. The person with whom the speech impaired individual is communicating observes the eyes of the speech impaired individual to determine the target of the speech impaired individual's eye gaze. To communicate an item, the speech impaired individual gazes first toward the one of the eight groups of five squares, indicating that he wants to communicate one of the symbols in that group, and gazes second toward one of the four corners and aperture (6354) matching the color of the square labeled with the item to be communicated in the previously indicated group.
Two types of selectable regions are conventionally used in a point and click menu interface in a graphical user interface. The first, shown in FIG. 9, depicts a menu having three menu options, labeled "High", "Medium" and "Low", each displayed on a display (4807), each associated respectively with selectable regions (4801), (4803) and (4805), and each located adjacent the associated selectable region. FIG. 10 depicts a menu having the same three menu options, each displayed on a display (4807), each associated respectively with selectable regions (4901), (4903) and (4905), and each intersecting the associated selectable region. In both these conventional menus, a menu option is selected by pointing to and clicking on the associated selectable region.
Conventional menu hierarchies in automated systems, built from menus of the type shown in FIG. 9 or FIG. 10, require that the operator proceed sequentially through the steps of searching menu options, selecting one of them, and, assuming a menu option including a submenu was selected, searching the submenu options, and selecting one of them. Where selection from menu hierarchies constitutes a substantial component of the operator's activities, the slowness of the selection process diminishes productivity.
Locating selectable regions or parts thereof outside the display, in accordance with the Perimeter Menu aspect of the invention, allows the large areas outside the display to be used, a major advantage for operators having impaired fine motor control who are unable to maintain a pointer on a small selectable region while selecting by click or by dwell. However, if menu options are displayed on the display near the perimeter of display and near their associated selectable region, the operator has an indication of the location of each selectable region but may not be able to see all the displayed menu options in a glance. Because an operator usually searches a displayed menu for his intended menu option, placing the menu only near the perimeter of the display may increase menu search time, thus increasing menu option selection time.
One object of the invention is to indicate to an operator of a menu system having selectable regions outside the display, the menu option associated with each selectable region and the location of each selectable region.
Another object of the invention is to facilitate selection from a menu by an operator having impaired motor control.
Still another object of the invention is to speed selection of a menu option from a menu and of a submenu option from a menu hierarchy.
A further object of the invention is to speed speech synthesis for a person having impaired speech and impaired motor control.
I. Sound Match
Conventional speech recognition systems facilitate computer access for individuals unable to use a standard keyboard whose speech is relatively unimpaired, for example, an individual with quadriplegia, and hands-free computer access for able-bodied individuals. The operator of such a speech recognition system reads a menu option out loud, for example, "open file", and the system, which includes sound receiving means, for example, a microphone coupled to a sound board having a Digital Signal Processor ("DSP"), receives the sound of the read menu option, digitizes the sound of the read menu option, and then provides the digitized sound to another component of the speech recognition system, sound matching means which includes an application program for matching the digitized sound to one of a plurality of sounds, each representing respectively the sound of a spoken menu option. The system determines which sound best matches the sound of the read menu option and selects the menu option associated with this best matched sound.
Individuals whose speech is impaired are often unable to effectively use conventional speech recognition systems because they often cannot produce a large distinct variety of sounds characteristic of phonetic languages. For example, such an individual may produce similar sounds for the two consonants "t" and "d" so that these sound are indistinguishable to a conventional speech recognition system, or such an individual may not be able to consistently produce sounds distinguishable by a speech recognition system, resulting in false matches. Other symptoms of impaired speech, for example, similarities among certain phonemes and impaired ability to start or stop sound production appropriately, may substantially limit the variety of sounds distinguishable to a conventional speech recognition system an individual may consistently produce.
Conventional speech recognition systems provide limited capabilities in languages rich in homophones, for example, Chinese, because in such languages, a distinct sound is often insufficient to specify a word, as is described in the Background Art section of the Ideographic Languages aspect of the invention. The problem may be briefly illustrated by an example. Suppose a Chinese data entry operator using a conventional speech recognition system speaks the phonetic unit "fu" with a particular intonation. This distinct sound may well have over 15 homophones. Although the operator could use the keyboard to select one of these 15 homophones, this defeats the purpose of speech recognition, which is to facilitate hands-free computer access.
One object of the invention is to facilitate selection from a menu, and, in particular, from a menu of homophones.
Another object of the invention is to facilitate speech synthesis and voice activated computer access by individuals with speech impairments.
Still another object of the invention is to speed data entry in Ideographic Languages.
J. Ideographic Language
The use of ideographs as the graphic symbols in written languages is found in many parts of the world. An ideograph, as used herein, is a graphic symbol used to represent an object, an idea or a word, without expressing, as in a phonetic system, the specific sounds forming the verbal expression of the object, idea or word. Ideographic languages include Chinese, Japanese and Korean. A graphic symbol, as used herein, includes, but is not limited to, each of the following: a letter of an alphabet, a Japanese kana, and an ideograph. For purposes of illustrating the concepts of the present invention specific reference will be made herein to a preferred embodiment of the system and method as it applies to the Chinese language.
In modern Chinese, a repertoire of between 2500 and 3000 ideographs is necessary to achieve normal business adequacy in reading and writing, while the language itself has approximately 50,000 ideographs that have been identified historically, with about 10,000 ideographs in current use. The conventional keyboard, with approximately 100 keys, is designed for languages with phonetic scripts, such languages having a small set of graphic symbols, i.e. letters. If such a keyboard were to be used in a corresponding manner for the direct input of Chinese ideographs, it would require many thousands of keys since, unlike western phonetic languages, Chinese has many thousands of ideographs. Selection of an ideograph from such a keyboard would require the operator to search a great many keys for the desired key, and thus be impracticably slow.
Prior art methods for selecting Chinese ideographs make use of various ideograph classification systems known to Chinese speakers. The operator first specifies a class of ideograph, based on a first characteristic common to many ideographs. Ideographs having that common characteristic are displayed and the operator selects from among them, either directly, by selecting an individual ideograph, or indirectly, by specifying a second common characteristic usually dependent upon the first characteristic, thus further limiting the displayed ideographs to those having both the first and second common characteristics. In some prior art methods, the operator may continue to specify characteristics until he has specified a unique ideograph.
One ideograph classification system is called the Pin Yin System. This classification system uses the phonetic structure of the Chinese language. In spoken Chinese there are approximately 412 basic phonetic units, each having a monosyllabic sound, for example, "nee", "how" and "ma". Four intonations can potentially be applied to each phonetic unit, resulting in approximately 1280 distinct sounds. With 10,000 ideographs in current use, each represented by one of approximately 1280 distinct sounds, it is evident that many Chinese ideographs are homophones, i.e. have the same sound. Over 80% of Chinese ideographs have homophones. The Pin Yin System uses this limited number of phonetic units as the basis for its classification. Ideographs which are homophones are classified together; the common characteristic of the Pin Yin System is the distinct sound.
According to the Pin Yin and Zhu Yin coding methods, known in the prior art, the operator specifies a distinct sound using a keyboard labeled with symbols representing the Latin alphabet (Pin Yin method) or Chinese phonetic units (Zhu Yin method). The first key operation or sequence of key operations specifies the phonetic unit. The second key operation specifies the intonation. In general, less than 15 ideographs have this sound, though in some cases there are many more homophones. These are displayed and the user selects from among them. In such cases, the operator, depending upon the system, may page through matching ideographs or specify another common characteristic to further limit the number of ideographs displayed. A common characteristic which may be used at this stage exploits another feature of the Chinese language. The majority of Chinese words are expressed by a combination of two ideographs, the meaning of the paired ideographs has its own meaning which may or may not be related to that of the constituent ideographs. Assuming the operator has specified a first distinct sound matching 40 ideographs, he may specify a second distinct sound which alone may match, for example, 20 ideographs, but there may be only two ideograph pairs having the specified first and second distinct sounds in that order. Thus, a second common characteristic may limit matching ideograph pairs to a number sufficiently small for the operator to efficiently search and select from, or may uniquely specify an ideograph pair. Another common characteristic the operator may specify to limit the number of matching ideographs is a meaning or meaning class to which one or more sequences of one or more ideographs belong.
Yet another feature of the Chinese language which may be exploited to limit the number of matching sequences of ideographs is the ideograph block. An ideograph block is a sequence of four ideographs which together has its own meaning which may or may not be related to that of the constituent ideographs. As above, where the operator specified a distinct sound for the second of two ideographs of an ideograph pair, so may the operator specify a distinct sound for the second, third and/or fourth ideograph of an ideograph block, to limit the number of matching ideograph blocks.
Another conventional ideograph classification system makes use of a classification of parts of ideographs. Ideographs are built from a set of 214 components, called radicals. Different radicals, perhaps placed within different locations within an ideograph, are combined to create an ideograph. According to the Chan Jie coding method, known in the prior art, the operator specifies one or more radicals appearing in the ideograph he wishes to enter. He may, for example, use a keyboard having at least 214 keys, each corresponding to a radical, or may actuate a sequence of keys, the sequence corresponding to a radical. Other common characteristics the operator may specify to limit the number of matching ideographs include a phonetic unit, the first brush stroke, and the last brush stroke used to draw the ideograph.
Another conventional ideograph classification system makes use of a classification of parts of ideographs. According to the Four Corner coding method, known in the prior art, the operator specifies the classification of the four corners of the ideograph he wishes to enter. Other common characteristics the operator may specify to further limit the number of matching ideographs include the number of horizontal strokes used to draw the ideograph, and the classification of a certain part of the ideograph above the lower right corner.
Yet another conventional ideograph classification system makes use of a classification ideographs based on the basic strokes from which each ideograph is built. In Chinese, there are a limited number of basic strokes, each ideograph being composed of between 1 and 33 such strokes. Ideographs may be classified by a small number of basic strokes, preferably according to strict rules regarding the order of stroke entry. In one conventional application of this coding method, the operator specifies only the first and last basic strokes of the desired ideograph, then selects from a display of all ideographs sharing this first-last basic stroke combination.
Japanese is somewhat more complicated than Chinese. In addition to ideographs, the Japanese language uses graphic symbols called kana, which includes hiragana and katakana. In written Japanese, ideographs are frequently combined with kana. Kana may be may specified phonetically, for example, to designate the hiragana pronounced "ko" an operator of a Japanese word processing system may type "k" and then "o" on a Latin alphabetic keyboard or may type a single key associated with this hiragana. Kana has multiple uses in a Japanese word processing system. Kana may represent itself, since kana may stand alone in Japanese text. Alternatively, kana may be used to specify Japanese ideographs, either by specifying the radicals which compose Japanese ideographs or by specifying the pronunciation of Japanese ideographs. A sequence of phonetic units specified by kana may represent that sequence of kana, a single Japanese ideograph, multiple Japanese ideographs, or a combination of one or more Japanese ideographs and one or more kana. In addition, a single Japanese ideograph may have multiple pronunciations, including a Japanese pronunciation and a Chinese pronunciation, and may have multiple kana spellings.
Conventional word processing systems for ideographic languages suffer from certain deficiencies. First, in systems where the operator specifies common characteristics until he has uniquely specified an ideograph, the operator must be extensively trained in the particular classification system. Depending upon the system, the operator may need to know, for example, how may horizontal brush strokes are required to draw a desired ideograph, or each of the 214 radicals and the encoding of each of them on a keyboard having less than 214 keys. Second, in systems where the operator uses both hands on the keyboard to specify a common characteristic, then selects from among ideographs, ideograph pairs or ideograph blocks by operating a function key or by pointing to one of the options with a mouse or other hand operated pointer and then operating a switch, the operator lifts one of his hands from the keyboard, makes the selection and then moves his hand back to the keyboard to specify another common characteristic. This sequence occurs often and contributes to the slow average rate of word entry (approximately 20 words per minute) for Chinese relative to alphabetic languages. Another problem in these systems is that the display of ideographs for selection may obscure part of the image of the previously entered ideographs or other information on the display.
Another drawback of many word processing systems for ideographic languages relates to the ease of copying a document. Ideally, the operator concentrates on the document to be copied, only occasionally scanning text he has input. For those word processing systems that display ideographs on a display for the operator's selection, the operator must frequently shift his gaze from the document to the display and back again. The operator cannot concentrate on both the document and the display simultaneously.
Ideographs, as used herein, also include the symbols of symbol sets used for communication by individuals who have hearing, speech or language impairments, for teaching literacy skills to those lacking them, including pre-literate children and individuals with intellectual disabilities, and for international written communication. These symbol sets include, but are not limited to, each of the following: Picture Communication Symbols, Rebus, Picsym, Pictogram Ideogram Communication Symbols, Yerkish, Blissymbolics and depictions of the signs of a manual sign language. Examples of symbols of the Picture Communication Symbols, Rebus, Picsyms, and Blissymbolics symbol sets are shown in FIG. 11, Pictogram Ideogram Communication Symbols in FIGS. 12(a)-12(d) and Yerkish in FIGS. 13(a)-13(j). Picture Communication Symbols, Rebus, Picsyms, Pictogram Ideogram Communication Symbols, Yerkish, and Blissymbolics are each described in Beukelman, David R. & Mirenda, Pat, Augmentative and Alternative Communication. Management of Severe Communication Disorders in Children and Adults, Paul H. Brookes Publishing Co., 1992, pp. 22-29.
Individuals who have not acquired or who have lost their literacy skills may use symbolic symbol sets in learning to read. If the individual lacks fine motor control, for example, due to cerebral palsy, the individual's disability may inhibit the acquisition of literacy skills by, for example, inhibiting repetition of an exercise by the individual, by limiting the individual's ability to participate in the classroom, or by making skill assessment by a teacher difficult so that the teacher may incorrectly believe that remediation is necessary or that a particular skill has been mastered. If the individual also has impaired speech, literacy acquisition is more difficult still.
Conventional literacy training systems for individuals who are unable to use a standard keyboard or mouse may use switch access, often in combination with scanning. As already described, scanning is an extremely slow process. Moreover, as the number of symbols in the symbol set increases, the time required to select a symbol also increases. Of the symbol sets mentioned above, Picture Communication Symbols contains approximately 1800 symbols, Rebus contains approximately 800 symbols, Picsyms contains approximately 1800 symbols, Pictogram Ideogram Communication Symbols contains approximately 400 symbols and Blissymbolics contains approximately 1400 symbols. When using a system with a static display, the operator may expend considerable time and effort finding the desired symbol; when using a system with a dynamic display, the operator may expend considerable time effort memorizing and recalling where a particular symbol is located within a hierarchy of symbols. This time and effort generally does not contribute to the acquisition of literacy skills.
One object of the invention is to display a menu of sequences of one or more ideographs on a display so that a large contiguous area on the display is not obstructed by the menu.
Another object of the invention is to facilitate ideograph entry in word processing systems for the Chinese, Japanese and Korean languages.
Still another object of the invention is to speed selection of sequences of graphics including one or more ideographs.
Yet another object of the invention is to allow an operator of a word processing system for an ideographic language to select a sequence of one or more ideographs without lifting either hand from the keyboard.
A further object of the invention is to indicate to an operator the progress toward selection of a dwell-selectable sequence of one or more ideographic characters.
A still further object of the invention is to synthesize speech for an operator having impaired speech.
Additional objects, advantages and novel features of the invention will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following or may be learned by practice of the invention. The objects and advantages of the invention may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the appended claims.
SUMMARY OF THE INVENTION
According to the present invention, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by a method of selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language. The method includes several steps. The first step is displaying on a display the plurality of sequences. Each of the plurality of sequences has a common characteristic in the ideographic language and each is associated respectively with one of a plurality of displayed sound indicators. Each of the sound indicators respectively indicates one of a plurality of indicated sounds. Each of the indicated sounds differs from each other indicated sound. Another step is matching a received sound to any one of the indicated sounds. Yet another step is selecting the sequence associated with the matched indicated sound.
Additionally, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by a method of selecting a sequence of one or more graphic symbols in a homophone-rich language from a plurality of sequences of one or more graphic symbols in the language. The method includes several steps. One step is displaying the plurality of sequences. Each of the plurality of sequences is associated respectively with a sound which is not a phonetic representation of the associated sequence. Each of the plurality of sounds differs from each other sound in the plurality of sounds. Another step is receiving a sound signal. Still another step is matching the received sound signal to any one of the plurality of sounds. Yet another step is selecting the sequence associated with the matched sound.
Additionally, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by a method of editing a first sequence of two or more words in an ideographic language. The method includes several steps. One step is displaying on a display the first sequence. The first sequence includes a plurality of second sequences of one or more graphic symbols in the language. Each of the second sequences is associated respectively with a sound indicator. Each of the sound indicators respectively indicates a sound. Each of the indicated sounds differs from each of the other indicated sounds; and (b) the sound indicators. Another step is receiving a sound. Still another step is matching the received sound to any one of the indicated sounds. A further step is replacing the second sequence associated with the matched indicated sound with a third sequence of one or more graphic symbols in the language, the third sequence including an ideograph in the language.
Additionally, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by an apparatus, for use with a general purpose computer system including a display, for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols. One or more sequences of the plurality of sequences includes an ideograph in an ideographic language. The apparatus includes a computer readable medium, and a program, stored on the medium and executable by the general purpose computer system. The program is operative to display the plurality of sequences on the display. Each of the sequences is associated respectively with a displayed sound indicator. Each of the sound indicators respectively indicates a sound. Each of the plurality of indicated sounds differs from each of the other indicated sounds. A particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator. The program is further operative to match a received sound to the sound indicated by the particular sound indicator, and to select the sequence associated with the particular sound indicator.
Additionally, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by an apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language. The apparatus includes a display on which may be displayed the plurality of sequences. Each of the plurality of sequences has a common characteristic in the ideographic language and each is associated respectively with a displayed sound indicator. Each of the sound indicators respectively indicates a sound. Each of the plurality of indicated sounds differs from each of the other indicated sounds. A particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator. The apparatus further includes a selection device for matching a received sound to the sound indicated by the particular sound indicator and for selecting the sequence associated with the particular sound indicator.
Additionally, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by an apparatus for selecting a sequence of one or more graphic symbols from a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language. The apparatus includes means for displaying the plurality of sequences. Each of the plurality of sequences has a common characteristic in the ideographic language and each associated is respectively with a displayed sound indicator. Each of the sound indicators respectively indicates a sound. Each of the plurality of indicated sounds differs from each of the other indicated sounds. A particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator. The apparatus further includes means for: (1) matching a sound to the sound indicated by the particular sound indicator; and (2) selecting the sequence associated with the particular sound indicator.
Additionally, one or more of the objects mentioned in the Background Art section describing the Sound Match aspect of the invention and other objects and advantages are attained by a device controller. The device controller includes a display on which may be displayed a plurality of sequences of one or more graphic symbols, one or more sequences of the plurality of sequences including one or more ideographs in an ideographic language. Each of the plurality of sequences has a common characteristic in the ideographic language and each is associated respectively with a displayed sound indicator. Each of the sound indicators respectively indicates a sound. Each of the plurality of indicated sounds differs from each of the other indicated sounds. A particular one of the sound indicators is not a phonetic representation of the sequence associated with the particular sound indicator. The device controller further includes a signal generating device, coupled to a controlled device, for: (1) matching a sound to the sound indicated by the particular sound indicator, the sequence associated with the particular sound indicator representing a function of the controlled device; and (2) generating a device control signal corresponding to the function of the controlled device.
Still other objects and advantages of the present invention will become readily apparent to those skilled in this art from the following detailed descriptions, wherein I have shown and described the preferred embodiment of each aspect of the invention, simply by way of illustration of the best mode contemplated by me of carrying out each aspect of my invention. As will be realized, each aspect of the invention is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the invention. Accordingly, the drawings and descriptions are to be regarded as illustrative in nature, and not as restrictive.
BRIEF DESCRIPTION OF DRAWINGS
FIG. 1 is an illustration of a display showing a conventional on-screen keyboard.
FIG. 2 is an illustration of a display showing a conventional on-screen keyboard and output from a word processing application program.
FIGS. 3, 4 and 5 are each illustrations of the display of each step of letter selection using a conventional quaternary on-screen keyboard.
FIG. 6 is an illustration of a display showing a conventional quaternary on-screen keyboard and output from two application programs.
FIG. 7 is an illustration of a conventional device implementing joystick patterns.
FIG. 8 is an illustration of a display of a conventional eye gaze system.
FIGS. 9 and 10 are each illustrations of a display showing a conventional menu.
FIG. 11 depicts examples of symbols from the Picture Communication Symbols, Rebus, PicSym and Blissymbols symbol sets.
FIGS. 12(a)-FIG. 12(d) depict examples of symbols from the Pictogram Ideogram Communication Symbols symbol set.
FIGS. 13(a)-FIG. 13(j) depict examples of symbols from the Yerkish symbol set.
FIG. 14 is a block diagram of a computer which may be utilized in accordance with the present invention.
FIG. 15 is a block diagram of a speech synthesis system which may be utilized in accordance with the present invention.
FIG. 16 is an illustration of software components of an apparatus in accordance with an embodiment of the Perimeter Menu aspect of the invention.
FIGS. 17 and 18 are each illustrations of a display and structures in accordance with an embodiment of the Perimeter Menu aspect of the invention.
FIG. 19 is an illustration of a display and structures in accordance with another embodiment of the Perimeter Menu aspect of the invention.
FIG. 20 is an illustration of a display and structures in accordance with still another embodiment of the Perimeter Menu aspect of the invention.
FIG. 21 is an illustration of a display and structures in accordance with another embodiment of the Perimeter Menu aspect of the invention.
FIG. 22 is an illustration of a display and structures in accordance with yet another embodiment of the Perimeter Menu aspect of the invention.
FIG. 23 is an illustration of an apparatus in accordance with a further embodiment of the Perimeter Menu aspect of the invention.
FIGS. 24 and 25 are each illustrations of a display and structures in accordance with another embodiment of the Perimeter Menu aspect of the invention.
FIGS. 26 and 27 illustrate an apparatus in accordance with still another embodiment of the Perimeter Menu aspect of the invention. FIG. 26 depicts a front view of the apparatus. FIG. 27 depicts a cut away view from the top of the apparatus.
FIG. 28 is a top view of a headrest in accordance with an embodiment of the Perimeter Menu aspect invention.
FIGS. 29 and 30 illustrate the state table aPocketFsm in accordance with the preferred embodiment of the Perimeter Menu aspect of the invention.
FIG. 31 is an illustration of a display and structures in accordance with the preferred embodiment of the Confinement aspect of the invention.
FIG. 32 is an illustration of a display and structures in accordance with another embodiment of the Confinement aspect of the invention.
FIG. 33 is an illustration of a display and structures in accordance with another embodiment of the Confinement aspect of the invention.
FIG. 34 is an illustration of an apparatus in accordance with the Dwell aspect of the invention and with the Path Directness aspect of the invention.
FIG. 35 is an illustration of another apparatus in accordance with the Dwell aspect of the invention.
FIG. 36 is an illustration of still another apparatus in accordance with the Dwell aspect of the invention.
FIG. 37 is an illustration of yet another apparatus in accordance with the Dwell aspect of the invention.
FIG. 38 is an illustration of another apparatus in accordance with the Dwell aspect of the invention.
FIG. 39 is an illustration of a display and structures in accordance with the preferred embodiment of the Path Directness aspect of the invention.
FIG. 40 is an illustration of a display and structures in accordance with an embodiment of the Path Directness aspect of the invention.
FIG. 41 is an illustration of a display and structures in accordance with another embodiment of the Path Directness aspect of the invention.
FIG. 42 is an illustration of a display and structures in accordance with another embodiment of the Path Directness aspect of the invention.
FIG. 43 is an illustration of a display and structures in accordance with another embodiment of the Path Directness aspect of the invention.
FIGS. 44, 45 and 46 are each illustrations of a display and structures in accordance with the preferred embodiment of the Intersection aspect of the invention.
FIGS. 47 and 48 are each illustrations of a display and structures in accordance with another embodiment of the Intersection aspect of the invention.
FIGS. 49, 50 and 51 are each illustrations of a display and structures in accordance with the preferred embodiment of the Alignment aspect of the invention.
FIG. 52 is an illustration of a display and structures in accordance with the preferred embodiment of the Location Indication and the Length Order aspects of the invention.
FIG. 53 is an illustration of a display and structures in accordance with an embodiment of the Location Indication aspect of the invention.
FIGS. 54 and 55 are each illustrations of a display and structures in accordance with another embodiment of the Location Indication aspect of the invention.
FIG. 56 is an illustration of a display and structures in accordance with a still further embodiment of the Location Indication aspect of the invention.
FIG. 57 is an illustration of a display and structures in accordance with a further embodiment of the Location Indication aspect of the invention.
FIGS. 58 and 59 are illustrations of a display and structures in accordance with the preferred embodiment of the Sound Match aspect of the invention.
FIG. 60 is an illustration of a display and structures in accordance with another embodiment of the Sound Match aspect of the invention.
FIG. 61 is an illustration of a display and structures in accordance with another embodiment of the Sound Match aspect of the invention.
FIG. 62 is a block diagram of a speech recognition system which may be utilized in accordance with the Sound Match aspect of the invention.
FIG. 63 is an illustration of software components of an apparatus in accordance with an embodiment of the Sound Match aspect of the invention.
FIG. 64 is an illustration of a display and structures in accordance with the preferred embodiment of the Ideographic Language aspect of the invention.
FIG. 65 is an illustration of a display and structures in accordance with another embodiment of the Ideographic Language aspect of the invention.
FIG. 66 is an illustration of a display and structures in accordance with still another embodiment of the Ideographic Language aspect of the invention.
FIG. 67 is an illustration of a display and structures in accordance with yet another embodiment of the Ideographic Language aspect of the invention .
BEST MODE FOR CARRYING OUT THE INVENTION
The hardware and software operating environment of the preferred embodiment of all aspects of the invention will now be described with reference to a particular embodiment of the invention, hereinafter "prototype". The prototype of the invention illustrates the best mode of practicing each aspect of the invention known by me except where a preferred mode is described.
FIGS. 15 depicts a block diagram of the hardware components of the prototype (2214), including a conventional general purpose computer system (2218), an optional pointer (2202), an optional printer (2220) and a speech synthesizer (2206). The general purpose computer system (2218) includes a conventional computer system (2116), a storage unit (2208), a keyboard (2210), and a diskette drive (2216). FIG. 14 depicts a block diagram of the conventional computer system (2116), including a processing unit (2102) and a display (2112). The processing unit (2102) includes a processor (2104), a memory (2106) and control circuitry (2108). The prototype employs the Toshiba T6400DXC general purpose computer system manufactured by Toshiba Corporation, Kawasaki, Japan. However, the T6400DXC is preferably substituted with the IBM ThinkPad 755C computer system, part number 9545F0C, manufactured by IBM Corporation, Armonk, N.Y., USA., because the former requires a 110VAC power source while the latter is powered by an integral battery. An integral battery allows an NMD operator to use the system when a 100VAC source is not available or when attaching to a 110VAC power source is inconvenient. The prototype further includes a head mounted pointer communicating via an infrared link with the computer system so that there are no cables tethering the operator to the computer system. Any cable between the operator and the computer system would have to be connected, probably by an attendant since the operator may lack the fine motor skills required to make such a connection. Preferably computer access can be accomplished independently by the operator. The choice of a pointing device is primarily dictated by the particular capabilities of the operator. Usually the best pointing device for a particular operator is the one drawing on that operator's best motor control. For example, if an operator's foot control is superior to his head control, a pointing device using his foot is preferably to a head pointer.
The prototype employs the Remote Headmaster.RTM. manufactured by the Prentke Romich Company, Wooster, Ohio, USA. However, the combination of the HeadMaster.RTM. Plus, part number HM-1P, HeadMaster.RTM. Plus Remote Adapter, part number HM-RA, and HeadMaster.RTM. Plus Laptop Adapter, part number HM-LA, all available from the Prentke Romich Company, is preferable because the headset is more comfortable and the HeadMaster.RTM. Plus ultrasonic transmitter mounts more easily on a laptop computer system than the Remote Headmaster ultrasonic transmitter.
The preferred embodiment further includes a battery powered printer, the MobileWriter.RTM., part number 730879, manufactured by Mannessmann Tally Corporation, Kent, Wash., USA and a speech synthesizer, the Multivoice Speech Synthesizer, part number MV2-SS, manufactured by The Institute on Applied Technology, Children's Hospital, Boston, Mass., USA.
The pointer (2202) is a device which provides data concerning the relative or absolute position of the operator or any body member of the operator. The display (2112) and pointer (2202) together provide for the interactive nature of the general purpose computer system (2218) in that, in accord with the various aspects of the invention, the interpretation that the processor (2104) gives to a certain pointer action made by the operator depends, in the majority of situations, upon what is being displayed to the operator at that time.
The prototype (2214) shown in FIG. 15 further includes a keyboard (2210), which functions to provide input from an able-bodied operator to the general purpose computer system (2218). The keyboard (2210) is useful for configuration, diagnostic and backup purposes, functions which are performed relatively infrequently and usually require an able-bodied person for ancillary activities, for example, loading backup media into the general purpose computer system. The prototype (2214) also optionally includes a printer (2220) which functions to provide hard copy output of data developed or stored in the general purpose computer system, and a speech synthesizer (2206), which functions to provide speech output for utterances and words composed using or retrieved from the general purpose computer system (2218).
The couplings between the devices depicted in FIG. 15 may be made by any means which permits the orderly and timely exchange of data across the interface. In the preferred embodiment, the interfaces between the pointer (2202) and the general purpose computer system (2218) and between the general purpose computer system (2218) and the speech synthesizer (2206) conform to the Electronic Industries Association RS-232 interface specification. The interface between the general purpose computer system (2218) and the printer (2220) conform to the Centronix 50 pin parallel interface specification.
The software component of the prototype are stored in memory (2106) and executed on the processing unit (2102). The software component of the prototype, depicted in FIG. 16, include a software driver (1202), an operating system (1204), an optional database program (1210), and the prototype access program code and data, hereinafter collectively referred to as the "access program" (1206). In the preferred embodiment, one or more application programs (1208) may also execute on the processing unit (2102) and accept control and data from the access program (1206) via the operating system (1204). The software driver (1202) of the prototype is the Logitech Mouse Driver included with Windows.RTM. version 3.1. The operating system (1204) of the prototype is Windows.RTM. version 3.1 in combination with MS-DOS.RTM. version 6.2. Hereinafter, the operating system is referred to simply as "Windows.RTM.", available from Microsoft Corporation, Redmond, Wash., USA.
The optional database program (1210) is described in the detailed description of the Length Order aspect of the invention. The prototype access program (1206) is described in detail below.
As stated earlier, the software components of the prototype are stored in memory (2106). Depending on the capacity of memory (2106) and the size of the application programs, portions of these programs may be transferred as needed between memory (2106) and the storage unit (2206) or between memory (2106) and a diskette in the diskette drive (2216) depicted in FIG. 15. The basic function of the storage unit (2206) and the diskette drive (2216) is to store programs and data that are employed by the general purpose computer system (2218) and which may readily be transferred to the memory (2106) when needed.
It is to be understood that components others than those used in the prototype may be utilized in accordance with the invention. It is only necessary that the substitute component or components have the capacity to carry out the functions described. For example, the processing unit of the general purpose computer system may be substituted with a microprocessor coupled to custom electronics for performing the functions of the various aspects of the invention, or the color display of the prototype may be substituted with a monochrome display.
A. Perimeter Menu
The preferred embodiment of the Perimeter Menu aspect of the invention will now be described in detail from a functional perspective using an example. This description refers to selectable regions which include one or more subregions. A selectable region is a region, delimited with respect to a display or a surface, and associated with a menu option which may be selected, usually by a selection event. A subregion is a selectable region that is included within another selectable region. Thus a subregion is, by itself, a selectable region. Assuming that a certain selectable region includes subregions A and B, dwell time on subregions A and B may be combined, for example, by summing, so that dwelling on either subregion A or B or a combination of both for the selection threshold period selects the menu option associated with the selectable region. Reference will now be made to FIGS. 17 and 18 which depict an example of the preferred embodiment of the Perimeter Menu aspect of the invention. FIG. 17 shows the display (2112) of a general purpose computer system (2218 in FIG. 15) and eight selectable regions. Each of the eight selectable regions consists of the union of a visible subregion on the display (2112) and an invisible subregion located outside the display (2112) and adjacent the visible subregion. For example, the selectable region at 11 o'clock in FIG. 17 labeled with menu option "vort<space>x" consists of invisible subregion (0104) and visible subregion (0106), and within this description of the Perimeter Menu aspect of the invention is referred to as selectable region (0104/0106). The other selectable regions shown in FIG. 17, proceeding counter clockwise from selectable region (0104/0106) are (0108/0110), (0112/0114), (0116/0118), (0120/0122), (0124/0126), (0128/0130) and (0132/0134). Each subregion may be sized to suit the operator's preferences and abilities. Each selectable region is associated respectively with a menu option. In FIG. 17, selectable region (0104/0106) is associated with menu option vort<space>x, selectable region (0108/0110) with menu option "sumac", selectable region (0112/0114) with menu option "wizen'", selectable region (0116/0118) with the menu option undo indicated by an icon on visible subregion (0118) representing an undo function, selectable region (0120/0122) with menu option "words", selectable region (0124/0126) with menu option "talk", selectable region (0128/0130) with menu option "ldhbfk" and selectable region (0132/0134) with menu option "ypgqj,". Together, the eight visible subregions circumscribe region (0150) on the display.
Selectable regions may be delimited by data indicative of one or more boundaries of the selectable region. Equivalently, the delimit means may be detectors operative to determine when the location indicated by the movement related signal has crossed one of those boundaries or intersects a selectable region. A partially delimited region or subregion is one which is unbounded on at least one side.
Resuming, now, with the example, to select a menu option associated with a selectable region the operator moves a pointer (2202 in FIG. 15) coupled to the general purpose computer system (2218 in FIG. 15) to indicate a location on the selectable region, including either subregion, associated with the desired menu option and maintains the indicated location on the selectable region for the selection threshold period. The period of time required for selection may vary responsive to the proximity of the indicated location to the location of a cursor on the display or to the proximity of the indicated location to a point within the intersected selectable region. Dwell time may be continuous, discontinuous or dynamic (described below) for either or both subregions of the selectable region.
Selection in the above example is in response to a dwell event. A dwell event includes, but is not limited to, each of the following: (a) the durations of one or more periods of intersection of locations indicated by a movement related signal, a body member or a cursor (including any part of the cursor) and a selectable region equalling or exceeding a predetermined period; (b) a first quantity responsive to the durations of the periods referred to in (a) equalling or exceeding a predetermined quantity; (c) dwell event (a) or (b) followed by a location indicated by the movement related signal, the body member or the cursor no longer intersecting the intersected selectable region; and (d) dwell event (a) or (b) wherein the period of intersection required for selection of a selectable region increases in response to a non-intersection or a period of non-intersection of locations indicated by the movement related signal, the body member or the cursor and the selectable region ("dynamic dwell event"). The use of non-intersection or a period of non-intersection in determining the duration of a period of intersection required for selection is called dynamic dwell. Associated with each type of dwell event is an intersected selectable region. This is the selectable region intersected by the location indicated by the movement related signal, body member or cursor which triggers the dwell event by causing the period or the first quantity to equal or exceed the predetermined period or the predetermined quantity, respectively.
Selection may also be in response to a selection event. A selection event includes, but is not limited to: (a) a dwell event; (b) a switch operation at or near the time of an intersection of a location indicated by a movement related signal, a body member or a cursor and a selectable region; (c) an intersection of a location indicated by a movement related signal, a body member or a cursor and a selectable region; and (d) selection event (c) followed by a location indicated by the movement related signal, the body member or the cursor no longer intersecting the selectable region it previously intersected. Associated with each type of selection event is an intersected selectable region. This is the selectable region intersected by the location indicated by the movement related signal, body member or cursor. The fact that a selection event has occurred may be indicated to the operator, for example, visually by changing the cursor appearance or location, by changing location, size, shape, hue, brightness, contrast, tone, dithering, pattern, hatching, font or fill of an object on the surface, or by displaying a graphic or a point distinguishable from its immediate surroundings on a surface or removing a graphic or point distinguishable from its immediate surroundings from a surface; auditively by generating a sound or changing the pitch or volume of an extant sound; tactilely by changing the surface or temperature of a contact area or the pressure exerted by a contact area; or by other means. In the prototype, following selection, the hue of the visible subregion of the selected selectable region is changed from green to magenta.
As used herein, a cursor includes a temporary marking on a display which emphasizes to an operator, in an optical manner, a momentarily important location or object. As used herein, body member means any part of the body including, but not limited to, each of the following: the shoulder, arm, elbow, wrist, hand, finger, thumb, leg, knee, ankle, foot, toe, hip, trunk, neck, tongue, lip, eye and head. The received movement related signal includes, but is not limited to, a signal indicative of movement or from which movement can be derived, such as a plurality of relative or absolute positions or a difference between two relative or absolute positions. Movement related signal receiving means includes, but is not limited to, each of the following: (a) pointer interface circuitry found in a general purpose computer system; (b) one or more detectors operative to detect movement of a pointer; and (c) one or more detectors operative to detect movement of a body member of an operator. In the prototype, the movement related signal receiving includes electronic circuitry in the general purpose computer system (2218) operative to receive the movement related signal generated in part by the movement of the pointer (2202).
In the prototype subregions are displayed on the display (2112). However, other means for displaying may be substituted for the means used in the prototype, for example, a projector for projecting an image, a surface having a static display thereon, or other suitable means.
Resuming, now, with the example of FIG. 17, and assuming that the operator has selected menu option "vort<space>x", the display is changed to that shown in FIG. 18. In FIG. 18, each of six selectable regions is now associated with a submenu option of the selected menu option "vort<space>x". Selectable region (0104/0106) is now associated with submenu option "<space>", selectable region (0108/0110) with submenu option "o", selectable region (0112/0114) with submenu option "t", selectable region (0124/0126) with submenu option "x", selectable region (0128/0130) with submenu option "v", and selectable region (0132/0134) with submenu option "r". Selectable regions (0116/0118) and (0120/0122) remains associated with the same menu options with which each was associated in FIG. 17. The operator may now select one of these submenu options.
Assuming that the selected submenu option is one of v, o, r, t, x and space, the selected character, or a corresponding computer encoding of that character, may be input to an apparatus coupled to the general purpose computer system (2218 in FIG. 15), or input to an application program (1208) executing on the general purpose computer system (2218) coupled to the display (2112). Inputting, as used herein, includes, but is not limited to, generating or passing signals representative of the selected menu option along a path toward the destination apparatus or program. Preferably, the computer program displays at least some of its output in the circumscribed region (0150).
Given a display having eight selectable regions, an operator may, with a single selection indicate one of eight menu options, with two selections indicate one of up to 64 different menu options, with three selections indicate one of up to 256 menu options, etc. Each of these menu options may represent a sequence of one or more characters, a sequence of one or more data or control inputs to an application program (1208), or a control function for one or more devices or speech synthesizers coupled to the general purpose computer system (2218). As used herein, a character includes a space, a control character as defined by the American National Standards Institute (ANSI) or the American Standard Code for Information Exchange (ASCII), and a letter from one of the Afrikaans, Albanian, Amharic, Arabic, Armenian, Assamese, Assyrian, Avar, Azerbaijani, Balinese, Bamara, Bantu, Bashkir, Basque, Bengali, Birhari, Bulgarian, Buluba-Lulua, Burmese, Buryat, Byelorussian, Caddoan, Catalan, Chechen, Chikaranga, Chippewa, Choctaw, Church Slavik, Chuvash, Coptic, Cree, Croatian, Cyrillic, Czech, Dakota, Danish, Dari, Devanagari, Dutch, Dzongkha, English, Eskimo, Esperanto, Estonian, Ewe, Farsi, Fijian, Filipino, Finnish, Flemish, French, Fulani, Gaelic, Galician, Georgian, German, Greek, Gujarati, Gurmakhi, Harari, Hausa, Hawaiian, Hebrew, Hindi, Hiragana, Ibo, Icelandic, Indonesian, Irish, Irogquoian, Italian, Kabardian, Kalmyk, Kannada, Kanuri, Kashmiri, Katakana, Kazakh, Khasi, Khmer, Kirghiz, Kishmiri, Komi, Kongo, Kurdish, Lao, Latin, Latvian, Lithuanian, Lu-Guanda, Macedonian, Magahi, Maithili, Makua, Malagasy, Malay, Malayalam, Maltese, Mandingo, Manipuri, Marathi, Masai, Mizo, Moldavian, Mongolian, Munda, Naga, Navaho, Nyanja, Nepalese, Norwegian, Oriya, Oromo, Ossetian, Pashto, Polish, Portugese, Punjabi, Rajasthani, Rhaeto-Romanic, Rumanian, Russian, Samoan, Sangs, Serbian, Serbo-Croatian, Sinhalese, Sinhi, Sioux, Slovak, Slovenia, Spanish, Sundanese, Swahili, Swedish, Syriac, Tadzhik, Tagalog, Tajik, Tamil, Tatar, Telugu, Thai, Tibetan, Turkish, Turkmen, Udmurt, Uighur, Ukranian, Umbundu, Urdu, Uzbek, Vietnamese, Visayan, Welsh, Yakut, Yoruba and phonetic alphabets. As used herein, each of a character, ideograph, control input and control function includes a computer encoding of the same. As used herein, a device includes, but is not limited to, each of a wheelchair, a household appliance, an appliance for use in an office, a workstation, a robot, and a computer peripheral. Thus, by selecting from a menu, the operator may, for example, increase the volume of an external speech synthesizer, or turn a wheelchair to the left.
The selectable regions organized as described above help an NMD operator make the menu selection he intends. Referring to FIG. 17, suppose, for example, an NMD operator intends to move a pointer (2202) that is indicating point (0154) to indicate point (0156), a location in subregion (0130), but who is unable to quickly stop motion, so that the location indicated by the pointer (2202) move |
