Hierarchical encapsulation of instantiated objects in a multimedia authoring system5680619
Abstract
An application development system, optimized for authoring multimedia titles, enables its users to create selectively reusable object containers merely by defining links among instantiated objects. Employing a technique known as Hierarchical Encapsulation, the system automatically isolates the external dependencies of the object containers created by its users, thereby facilitating reusability of object containers and the objects they contain in other container environments. Authors create two basic types of objects: Elements, which are the key actors within an application, and Modifiers, which modify an Element's characteristics. The object containers (Elements and Behaviors--i.e., Modifier containers) created by authors spawn hierarchies of objects, including the Structural Hierarchy of Elements within Elements, and the Behavioral Hierarchy, within an Element, of Behaviors (and other Modifiers) within Behaviors. The system utilizes an Element's dual hierarchies to make that Element an environmental frame of reference to the objects it contains. Through techniques known as Hierarchical Message Broadcasting, Hierarchical Variable Scoping and Hierarchical Relative Positioning, objects automatically receive messages sent to their object container and access data known to their object container. An Element's position is even determined relative to the position of its parent Element container. The system is highly extensible through a Component API in which Modifiers and Services that support them can be added and integrated seamlessly into the system. The system's architecture is substantially platform-independent, automatically allowing most author's titles to run on multiple platforms. In addition, the entire authoring environment can be ported relatively easily to a variety of platforms due to the isolation a platform-dependent layer within the system.
Claims
We claim:
1. An application development system comprising:
(a) a first class of Element objects from which one or more Elements can be instantiated, the first class defining a first set of characteristics inherent to each Element object in the first class;
(b) a second class of Modifier objects from which one or more Modifiers can be instantiated, the second class defining a second set of characteristics inherent to each Modifier object in the second class;
(c) an instantiation mechanism that enables a first Element and a second Element to be instantiated from the first class of Element objects, and further enables a first Modifier to be instantiated from the second class of Modifier objects;
(d) a first hierarchical linking mechanism that enables an author to link the first Element as a parent to the first Modifier, the first Element attaining the second set of characteristics while the first Element and the first Modifier remain linked; and
(e) a second hierarchical linking mechanism that enables an author to link the first Element as a parent to the second Element, the first Element providing an environmental frame of reference for the second Element and the first Modifier.
2. An application development system comprising:
(a) a first class of Element objects from which one or more Elements can be instantiated, the first class defining a first set of characteristics inherent to each Element in the first class;
(b) a second class of Modifier objects from which one or more Modifiers can be instantiated, the second class defining a second set of characteristics inherent to each Modifier in the second class;
(c) a third class of Element objects from which one or more Elements can be instantiated, the third class defining a third set of characteristics inherent to each Element in the third class;
(d) an instantiation mechanism that enables a first Element to be instantiated from the first class, and further enables a second Element to be instantiated from the third class, and further enables a first Modifier to be instantiated from the second class;
(e) a first hierarchical linking mechanism that enables the first Element to be linked to the first Modifier, the first Element providing an environmental frame of reference for the first Modifier, and attaining the second set of characteristics, while the first Element and first Modifier remain linked; and
(f) a second hierarchical linking mechanism that enables the first Element to be linked to the second Element, the first Element providing an environmental frame of reference for the second Element while the first and second Elements remain linked,
whereby the system enables the creation of an object hierarchy.
3. The application development system of claim 2 wherein the first and third classes are equivalent.
4. The application development system of claim 2 or 3 wherein:
(a) the first Element's set of characteristics includes a current position of the first Element;
(b) the second Element's set of characteristics includes a current position of the second Element; and
(c) the environmental frame of reference provided by the first Element is manifested at least in part by a Hierarchical Relative Positioning mechanism that determines the current position of the second Element relative to the current position of the first Element.
5. An application development system comprising:
(a) a first class of Element objects from which one or more Elements can be instantiated, the first class defining a first set of characteristics inherent to each Element in the first class;
(b) a second class of Modifier objects from which one or more Modifiers can be instantiated, the second class defining a second set of characteristics inherent to each Modifier in the second class;
(c) an instantiation mechanism that enables a first Element to be instantiated from the first class, and further enables a first Modifier to be instantiated from the second class; and
(d) a hierarchical linking mechanism that enables the first Element to be linked to the first Modifier, the first Element attaining the second set of characteristics while the first Element and first Modifier remain linked,
whereby the system enables a form of object-based authoring through the modification of the behavior of an object without modifying the system's underlying class hierarchy.
6. The application development system of claim 5 wherein:
(a) the first Element's set of characteristics includes a current position of the first Element that is visually perceptible to a user of the system; and
(b) the first Modifier's set of characteristics includes the ability to detect when the current position of the first Element crosses a predetermined boundary.
7. The application development system of claim 5 wherein the first Modifier's set of characteristics includes the ability to detect a collision between the first Element and another Element.
8. The application development system of claim 5 wherein:
(a) the first Element constitutes a first object container containing the first Modifier while the first Element and first Modifier remain linked; and
(b) the system can discern automatically dependencies of that first object container.
9. The application development system of claim 8 wherein:
(a) the first Element's set of characteristics includes a current position of the first Element;
(b) the system further comprises a relative positioning mechanism that determines the current position of the first Element relative to a current position of an ancestor Element of the first Element; and
(c) the dependencies of the first object container include the current position of the first Element.
10. The application development system of claim 8 wherein:
(a) the system further comprises an object configuration and messaging mechanism that allows the first Modifier to be configured to perform an action in response to the receipt of a specified message; and
(b) the dependencies of the first object container include the specified message to which the first Modifier is configured to respond.
11. The application development system of claim 8 wherein:
(a) the system further comprises Variable Modifiers having a third set of characteristics inherent to each Variable Modifier, the third set of characteristics including the ability to store data on behalf of an Element;
(b) the system further comprises an object configuration and variable scoping mechanism that allows the first Modifier to be configured to access a second Variable Modifier, provided that the second Variable Modifier stores data on behalf of the first Element or an ancestor Element of the first Element; and
(c) the dependencies of the first object container include the data stored by the second Variable Modifier.
12. An application development system comprising:
(a) a first class of Element objects from which one or more Elements can be instantiated, the first class defining a first set of characteristics inherent to each Element in the first class;
(b) a second class of Modifier objects from which one or more Modifiers can be instantiated, the second class defining a second set of characteristics inherent to each Modifier in the second class;
(c) an instantiation mechanism that enables a first Element to be instantiated from the first class, and further enables a first Modifier to be instantiated from the second class; and
(d) a hierarchical linking mechanism that enables the first Element to be linked to the first Modifier, the first Element providing an environmental frame of reference for the first Modifier while the first Element and first Modifier remain linked,
whereby the system enables the behavior of an Element to be modified.
13. The application development system of claim 2 or 12 wherein:
(a) the system further comprises a messaging mechanism that sends messages to Elements and Modifiers; and
(b) the environmental frame of reference provided by the first Element is manifested at least in part by a Hierarchical Message Broadcasting mechanism that uses the messaging mechanism to broadcast a message sent to the first Element to descendants of the first Element.
14. The application development system of claims 2 or 12 wherein:
(a) the system further comprises Variable Modifiers having a fourth set of characteristics inherent to each Variable Modifier, the fourth set of characteristics including the ability to store data on behalf of an Element; and
(b) the environmental frame of reference provided by the first Element is manifested at least in part by a Hierarchical Variable Scoping mechanism that renders data stored in a Variable Modifier on behalf of the first Element accessible to descendants of the first Element.
15. The application development system of claim 5 or 12 wherein:
(a) the first Element's set of characteristics includes a current position and a graphic representation of the first Element that are visually perceptible to a user of the system; and
(b) the first Modifier's set of characteristics includes the ability to change the current position of the first Element over time.
16. The application development system of claim 5 or 12 wherein:
(a) the first Element's set of characteristics includes a graphic representation of the first Element that is visually perceptible to a user of the system; and
(b) the first Modifier's set of characteristics includes graphic attributes that alter the first Element's graphic representation.
17. An application development system comprising:
(a) a first class of Behavior objects from which one or more Behaviors can be instantiated, the first class defining a first set of characteristics inherent to each Behavior in the first class;
(b) a second class of Modifier objects from which one or more Modifiers can be instantiated, the second class defining a second set of characteristics inherent to each Modifier in the second class;
(c) an instantiation mechanism that enables a first Behavior to be instantiated from the first class, and further enables a first Modifier to be instantiated from the second class; and
(d) a hierarchical linking mechanism that enables the first Behavior to be linked to the first Modifier, the first Behavior providing an environmental frame of reference for the first Modifier while the first Behavior and first Modifier remain linked,
whereby the system enables a behavioral hierarchy of Modifiers to be created.
18. The application development system of claim 17 wherein:
(a) the system further comprises a messaging mechanism that sends messages to Behaviors and Modifiers; and
(b) the environmental frame of reference provided by the first Behavior is manifested at least in part by a Hierarchical Message Broadcasting mechanism that uses the messaging mechanism to broadcast a message sent to the first Behavior to descendants of the first Behavior.
19. The application development system of claim 17 wherein:
(a) the system further comprises Variable Modifiers having a fourth set of characteristics inherent to each Variable Modifier, the fourth set of characteristics including the ability to store data on behalf of a Behavior; and
(b) the environmental frame of reference provided by the first Behavior is manifested at least in part by a Hierarchical Variable Scoping mechanism that renders data stored in a Variable Modifier on behalf of the first Behavior accessible to descendants of the first Behavior.
20. The application development system of claim 7, 12 or 17 wherein the second set of characteristics includes the ability to send a message to another Modifier.
21. An application development system comprising:
(a) a first class of Element objects from which one or more Elements can be instantiated, the first class defining a first set of characteristics inherent to each Element in the first class;
(b) a second class of Modifier objects from which one or more Modifiers can be instantiated, the second class defining a second set of characteristics inherent to each Modifier in the second class;
(c) a third class of Behavior objects from which one or more Behaviors can be instantiated, the third class defining a third set of characteristics inherent to each Behavior in the third class;
(d) a fourth class of Modifier objects from which one or more Modifiers can be instantiated, the fourth class defining a fourth set of characteristics inherent to each Modifier in the fourth class;
(e) an instantiation mechanism that enables a first Element to be instantiated from the first class, and further enables a first Modifier to be instantiated from the second class, and further enables a first Behavior to be instantiated from the third class, and further enables a second Modifier to be instantiated from the fourth class; and
(f) a hierarchical linking mechanism that enables:
(1) linking of the first Element to the first Modifier and the first Behavior; and
(2) linking of the first Behavior to the second Modifier, wherein the first Behavior attains the fourth set of characteristics while the first Behavior and second Modifier remain linked, and the first Element attains the second, third and fourth sets of characteristics, while the first Element, first Behavior and first and second Modifiers remain linked,
whereby the first, second, third and fourth sets of characteristics define the first Element's personality.
22. An application development system comprising:
(a) a plurality of object classes from which one or more objects can be instantiated, each class defining a set of characteristics inherent to each object in that class;
(b) an instantiation mechanism that enables a first object having a first set of characteristics to be instantiated from the plurality of object classes, and further enables a second object having a second set of characteristics to be instantiated from the plurality of object classes, and further enables a third object having a third set of characteristics to be instantiated from the plurality of object classes, and further enables a fourth object having a fourth set of characteristics to be instantiated from the plurality of object classes; and
(c) a hierarchical linking mechanism that enables:
(i) creation of a first object container by linking the first object to the second object, wherein the first object provides an environmental frame of reference for the second object while the first and second objects remain linked;
(ii) creation of a second object container by linking the second object to the third object, wherein the second object provides an environmental frame of reference for the third object while the second and third objects remain linked; and
(iii) replacement of the second object with the fourth object by breaking the links between the second object and the first and third objects, and establishing corresponding links between the fourth object and the first and third objects, such that the first object provides an environmental frame of reference for the fourth object, which in turn provides an environmental frame of reference for the third object, while the first, third and fourth objects remain linked,
whereby the system enables selective reusability of an object container and the objects it contains.
23. The application development system of claim 22 wherein the first, second, third and fourth objects are Elements.
24. The application development system of claim 22 wherein the first and second objects are Elements, and the third object is a Modifier, and wherein the second object attains the third set of characteristics while the second and third objects remain linked.
25. The application development system of claim 22 wherein the first object is an Element, the second object is a Behavior and the third object is a Modifier, and wherein the first object attains the second and third sets of characteristics while the first, second and third objects remain linked.
26. The application development system of claim 22 wherein the first and second objects are Behaviors, and the third object is a Modifier, and wherein the first object attains the second and third sets of characteristics while the first, second and third objects remain linked.
27. An application development system comprising:
(a) a plurality of object classes from which one or more objects can be instantiated, each class defining a set of characteristics inherent to each object in that class; and
(b) an instantiation mechanism that enables a first object having a first set of characteristics to be instantiated from the plurality of object classes, and further enables a second object having a second set of characteristics to be instantiated from the plurality of object classes, and further enables a third object having a third set of characteristics to be instantiated from the plurality of object classes; and
(c) a hierarchical linking mechanism that enables creation of:
(i) a first object container by linking the first object to the second object, wherein the first object provides an environmental frame of reference for the second object while the first and second objects remain linked; and
(ii) a second object container, replacing the first object with the third object, by breaking the link between the second object and the first object, and establishing a corresponding link between the second object and the third object, such that the third object provides an environmental frame of reference for the second object while the objects remain linked,
whereby the system enables selection of a new object container for an object.
28. The application development system of claim 27 wherein the first, second and third objects are Elements.
29. The application development system of claim 27 wherein each of the first and third objects is either an Element or Behavior, and the second object is either a Behavior or Modifier.
30. The application development system of claim 27 wherein the dynamic object configuration and hierarchical linking mechanism allows configuration of the second object such that the replacement of the first object with the third object occurs dynamically, at runtime, upon the occurrence of a specified condition.
31. The application development system of claim 27 wherein the hierarchical linking mechanism further includes a touch-up mechanism that, upon the replacement of the first object with the third object, automatically identifies and resolves dependencies of the first, second and third objects.
32. The application development system of claim 31 wherein the dependencies include a current position of the first and third objects.
33. The application development system of claim 31 wherein the dependencies include a Variable accessed by the second object.
34. The application development system of claim 31 wherein the dependencies include a message to which the second object responds.
35. The application development system of claim 31 wherein:
(a) the dependencies include a first asset constituting a dependency of the second object and a second asset constituting a dependency of the third object, the system referencing each of the first and second assets both by a name and by a unique identifier; and
(b) the touch-up mechanism, upon the replacement of the first object with the third object, resolves name and unique identifier conflicts by:
(i) replacing the unique identifier of the first asset with the unique identifier of the second asset, if the name of the first and second assets are equivalent; and
(ii) assigning a new unique identifier to the first asset if the unique identifiers, but not the names, of the first and second assets are equivalent.
36. An application development system comprising:
(a) a plurality of object classes from which one or more objects can be instantiated, each class defining a set of characteristics inherent to each object in that class;
(b) an instantiation mechanism that enables a first object, a second object and a third object to be instantiated from the plurality of object classes; and
(c) a hierarchical linking mechanism that enables creation of:
(i) a first object container, by linking the first object to the second object, wherein the first object provides an environmental frame of reference for the second object while the first and second objects remain linked; and
(ii) a first object hierarchy within the first object container, by linking the second object to the third object, thereby creating a second object container wherein the second object provides an environmental frame of reference for the third object while the second and third objects remain linked; and
(d) a Hierarchical Message Broadcasting mechanism that broadcasts a message sent to the first object container to the second and third objects included within the first object hierarchy,
whereby the system enables the broadcasting of messages within an object hierarchy.
37. The application development system of claim 36 wherein:
(a) the instantiation mechanism further enables a fourth object to be instantiated from the plurality of object classes;
(b) the hierarchical linking mechanism further enables linking of the first object to the fourth object, wherein the first object provides an environmental frame of reference for the fourth object while the first and fourth objects remain linked;
(c) the system further comprises an object configuration and messaging mechanism that enables the third object to be configured to send a message to a runtime object container that contains the third object when the message is sent at runtime; and
(d) if the first object is a runtime object container of both the third and fourth objects, the message sent by the third object to the first object will be broadcast to the fourth object by the Hierarchical Message Broadcasting mechanism,
whereby an object can be configured to send a message to another object by relatively targeting the lowest common ancestor of the two objects.
38. The application development system of claim 36 further comprising an object configuration and messaging mechanism that enables an object to be configured to send a message to a specified destination Element and limit its broadcasting, via the Hierarchical Message Broadcasting mechanism, such that the message will not cascade to any Elements contained within the destination Element.
39. The application development system of claim 36 further comprising an object configuration and messaging mechanism that enables an object to be configured to send a message to a specified destination object and limit its broadcasting, via the Hierarchical Message Broadcasting mechanism, such that the message will not be relayed beyond the first object configured to respond to the message.
40. An application development system comprising:
(a) a plurality of object classes from which one or more objects can be instantiated, each class defining a set of characteristics inherent to each object in that class;
(b) an instantiatiOn mechanism that enables a first object and a second object to be instantiated from the plurality of object classes;
(c) a hierarchical linking mechanism that enables creation of an object container by linking the first object to the second object, wherein the first object provides an environmental frame of reference for the second object while the first and second objects remain linked; and
(d) an object configuration and messaging mechanism that enables configuration of the second object to send a message to a runtime object container that contains the second object when the message is sent at runtime,
whereby the system enables the relative targeting of message destinations.
41. An application development system comprising:
(a) a first class of Element objects from which one or more Elements can be instantiated, the first class defining a first set of characteristics inherent to each Element in the first class;
(b) a second class of Modifier objects from which one or more Modifiers can be instantiated, the second class defining a second set of characteristics inherent to each Modifier in the second class;
(c) an instantiation mechanism that enables a first Element to be instantiated from the first class, and further enables a first Modifier to be instantiated from the second class;
(d) a hierarchical linking mechanism that enables the first Element to be linked to the first Modifier, the first Element attaining the second set of characteristics while the first Element and the first Modifier remain linked; and
(e) a component API that enables:
(i) creation of a third class of Modifier objects, the third class defining a third set of characteristics inherent to each Modifier in the third class; and
(ii) integration of the third class of Modifiers into the system such that the instantiation mechanism can be used to instantiate a second Modifier from the third class, and the hierarchical linking mechanism can be used to link the first Element to the second Modifier, wherein the first Element attains the third set of characteristics while the first Element and second Modifier remain linked,
whereby the system's component API enables the functionality of the system to be extended.
42. The application development system of claim 41 wherein the third class is a subclass of the second class.
43. The application development system of claim 41 wherein:
(a) the system further comprises a fourth class of Modifier objects from which one or more Modifiers can be instantiated, the fourth class constituting a subclass of the second class and inheriting at least some of the second set of characteristics, and defining a fourth set of characteristics inherent to each Modifier in the fourth class; and
(b) the component API further enables:
(i) addition of one or more methods or data structures to the second class, via the component API, without access to or recompilation of source code defining the second class, thereby defining a supplemental set of characteristics, in addition to the second set of characteristics, inherent to each Modifier in the modified second class; and
(ii) integration of the modified second class of Modifiers into the system such that the instantiation mechanism can be used to instantiate a third Modifier from the fourth class, and the hierarchical linking mechanism can be used to link the first Element to the third Modifier, wherein the first Element attains the supplemental set of characteristics, in addition to the fourth set of characteristics, while the first Element and third Modifier remain linked,
whereby the system's component API enables the functionality of an existing superclass of the system to be extended.
44. An application development system comprising:
(a) a first class of Element objects from which one or more Elements can be instantiated, the first class defining a first set of characteristics inherent to each Element in the first class;
(b) an instantiation mechanism that enables a first Element to be instantiated from the first class; and
(c) a media linking mechanism that enables:
(i) linking of the first Element to a first media item having a first media data type, such that the first Element will take on the appearance of the first media item during runtime while the first Element and first media item are linked;
(ii) breaking of the link between the first Element and the first media item; and
(iii) linking of the first Element to a second media item having a second media data type, such that the first Element will take on the appearance of the second media item during runtime while the first Element and second media item are linked,
whereby the system enables an Element to be morphed among different media data types.
45. The application development system of claim 44 wherein the media linking mechanism further comprises an object configuration and dynamic linking mechanism that enables configuration of the first Element such that the system dynamically, upon the occurrence of a specified runtime condition, breaks the link between the first Element and the first media item and establishes a link between the first Element and the second media item.
46. An application development system comprising:
(a) a first class of Element objects from which one or more Elements can be instantiated, the first class defining a first set of characteristics inherent to each Element in the first class;
(b) a second class of Element objects from which one or more Elements can be instantiated, the second class defining a second set of characteristics inherent to each Element in the second class;
(c) a third class of Variable Modifier objects from which one or more Variables can be instantiated, the third class defining a third set of characteristics inherent to each Variable in the third class, the third set of characteristics including the ability to store data on behalf of an Element;
(d) an instantiation mechanism that enables a first Element to be instantiated from the first class, and further enables a second Element to be instantiated from the second class, and further enables a first Variable to be instantiated from the third class;
(e) an alias mechanism that enables creation of an alias of the first Variable having a first value, from which aliased copies of the first Variable can be made; and
(f) a hierarchical linking mechanism that enables the first Element to be linked to a first aliased copy of the first Variable, and to link the second Element to a second aliased copy of the first Variable, such that the first aliased copy of the first Variable stores the first value on behalf of the first Element, and the second aliased copy of the first Variable stores the first value on behalf of the second Element;
(g) wherein the alias mechanism causes any change at runtime to the first value stored by the first aliased copy of the first Variable to be reflected dynamically in the second aliased copy of the first Variable, such that the first and second aliased copies of the first Variable store the same value at any give time during runtime,
whereby the system provides dynamically aliased Variables.
47. An application development system comprising:
(a) a first class of Element objects from which one or more Elements can be instantiated, the first class defining a first set of characteristics inherent to each Element in the first class;
(b) an instantiation mechanism that enables a first Element to be instantiated from the first class;
(c) a media linking mechanism that enables the first Element be linked to a first media item having a cel-based animation data format, such that the Element will take on the appearance of the cels of the first media item at a specified animation rate during runtime; and
(d) an object configuration and animation range selection mechanism that enables the first Element to be configured to play a specified range of cels of the first media item during runtime upon the occurrence of a specified condition,
whereby the system enables dynamic changes to a portion of an animation that is played during runtime upon the occurrence of a specified condition.
48. An application development system comprising:
(a) a plurality of object classes from which one or more objects can be instantiated, each class defining a set of characteristics inherent to each object in that class;
(b) an instantiation mechanism that enables a first object, a second object, a third object, a fourth object and a fifth object to be instantiated from the plurality of object classes; and
(c) a hierarchical linking mechanism that enables creation of:
(i) a first object container in a first application, by linking the first object to the second object, wherein the first object provides an environmental frame of reference for the second object while the first and second objects remain linked;
(ii) a first object hierarchy within the first object container, by linking the second object to the third object, thereby creating a second object container wherein the second object provides an environmental frame of reference for the third object while the second and third objects remain linked; and
(iii) a third object container in a second application, by linking the fourth object to the fifth object, wherein the fourth object provides an environmental frame of reference for the fifth object while the fourth and fifth objects remain linked; and
(c) an object library mechanism that enables copying of:
(i) the second object container from the first application into a first object library upon breaking the link between the first and second objects; and
(ii) the second object container from the first object library into the second application, and linking of the fourth object using the hierarchical linking mechanism, wherein the fourth object provides an environmental frame of reference for the second object container while the second, third and fourth objects remain linked,
whereby the system enables selective reusability of object containers across applications.
Description
I. BACKGROUND
A. Field of the Invention
B. Description of the Related Art
1. Reusability and the Modular Interface
2. Encapsulation and the Object Interface
3. The Need for Selective Reusability of "User Objects"
a. An OOP Windowing Example
4. The Lack of Selective Reusability in Multimedia Authoring Systems
II. SUMMARY OF THE INVENTION
III. BRIEF DESCRIPTION OF THE DRAWINGS
IV. DETAILED DESCRIPTION
A. External Architecture--"Author's-Eye View"
1. Objects: Elements and Modifiers
2. Object Containers: Elements and Behaviors
3. Object Hierarchies: Structural Hierarchy and Behavioral Hierarchy
a. Types of Elements in the Structural Hierarchy
(1) Projects
(2) Sections
(3) Subsections
(4) Scenes
(5) Media Elements
b. Isolation of External Dependencies of Object Containers
(1) Hierarchical Message Broadcasting
(2) Hierarchical Variable Scoping
(3) Hierarchical Relative Positioning
c. Selective Reusability through Adoption and Transplantation
4. Object Authoring Interface
a. Layout View
b. Layers View
c. Structure View
d. Asset Palette
e. Library
f. Alias Palette
g. Messaging Log
h. mToon Editor
i. Modifier Palettes
j. Drag and Drop
5. Object Messaging
a. Categories of Messages
(1) Author Messages
(2) Environment Messages
(3) Commands
b. Parts of a Message
(1) Message Name
(2) Message Destination
(3) Message Data
(4) Message Path
6. Object Configuration
a. Elements
b. Modifiers
(1) Behavior
(2) Variables
(3) Capabilities
B. Core System Architecture
1. Edit Mode Functionality
a. Connecting Objects
(1) Element To Element Connection
(2) Modifier Connections
(3) Player and Asset Connections
(4) Touch-Up Process
b. Aliasing
2. Saving Projects and Titles
3. Runtime Functionality
a. Message Targeting and Dispatching
b. Event Loop Processing
C. Component API--"Programmer's-Eye View"
1. Object Model
2. Component Methods
3. Registration Process
4. Core and Service Utilities
D. Examples
1. Snake
2. Fish
3. Windowing System
TABLES
Table I. System Messages And Commands
Table II. Message Destinations
Table III. Message Data Structure
Table IV. Base Component Class Functions
Table V. Sample Instance Data Structure
FIGURES
FIG. 1 Dual Hierarchy
FIG. 2 Layout View With Tool, Object Information And Modifier Palettes
FIG. 3 Layers View
FIG. 4 Structure View
FIG. 5 Asset Palette
FIG. 6 Library
FIG. 7 Alias Palette
FIG. 8 Messaging Log
FIGS. 9(a)-(d) mToon Editor
FIG. 10 Element Configuration Dialog
FIG. 11 Behavior Interface
FIGS. 12(a)-(h) Variable Interfaces
FIGS. 13(a)-(g) Messenger Interfaces
FIGS. 14(a)-(c) Scene-Based Modifier Interfaces
FIG. 15 Scene Change And Return Modifiers
FIGS. 16(a)-(d) Motion Modifier Dialogs
FIGS. 17(a)-(d) Visual Modifier Dialogs
FIGS. 18(a)-(d) Sound Modifier Dialogs
FIG. 19 Style Modifier Dialog
FIG. 20 Cursor Modifier Dialog
FIG. 21 Miniscript Modifier Dialog
FIG. 22 Classification Modifier Dialog
FIG. 23 Set Value Modifier Dialog
FIG. 24 System Architecture
FIG. 25 Object Interconnections
FIG. 26 Service Connections
FIG. 27 Title Storage
FIG. 28 Event Loop Processing
FIG. 29 Interfaces Architecture
FIG. 30 Operation Of Object Model
FIG. 31 Loading Of Components
FIG. 32 Implementation Of A Snake
FIG. 33 Implementation Of Fish
FIGS. 34(a)-(g) Selectively Reusable Windows Example
FIGS. 35(a)-(g) Implementation Of Windows Example
I. BACKGROUND
A. Field of the Invention
This invention relates to application development systems generally, and in particular to systems for authoring interactive multimedia applications.
B. Description of the Related Art
Since the advent of computers many decades ago, computer scientists have labored to build increasingly powerful computer hardware that is faster and cheaper to build than its predecessors. Today's computers can perform, in one second, many millions of "add," "shift," "load," "store" and other relatively simple functions.
To perform tasks of any significant complexity, however, one must somehow cause computers to perform vast numbers of these simple functions in some sequence. Computer software, in its most basic form, comprises programs or sequences of instructions, each of which directs a computer to perform the function corresponding to that instruction.
Yet, even as computers have become more powerful, developers of computer software continue to struggle to create complex programs without having to "reinvent the wheel" for each task they direct computers to perform. This need for "reusability" permeates virtually every aspect of software development, and is driven ultimately by the end user's desire for "ease of use."
1. Reusability and the Modular Interface.
Although there are many varied approaches to this basic problem of reusability, one technique remains constant--"modular programming," i.e., the "bootstrapping" of multiple simple functions into "modules" of greater and greater complexity that can be reused through higher-level "modular interfaces." Virtually all software relies on modular programming to some extent.
For example, assemblers and compilers enable programmers to bootstrap from the machine language defined by a computer's instruction set to a higher-level, more human-readable language. Similarly, operating systems perform many of the low-level support tasks commonly needed by applications software developers (e.g., file system, memory management, basic GUI routines, etc.), and then provide developers with a library of "reusable" modules.
In essence, all application development systems provide to their users a language, an application programming interface ("API"), or some other form of modular interface designed to facilitate development of general purpose or highly specialized applications. Such modular interfaces provide reusability of the "hidden" functionality which implements that interface.
2. Encapsulation and the Object Interface.
One of the most popular current trends designed to promote the creation of reusable software is in the field of object-oriented programming ("OOP"). There are a variety of OOP languages and application development tools on the market today (e.g., C++ and Kaleida Labs' "ScriptX.TM."), as well as a number of "pseudo-OOP" tools (e.g., "HyperCard.TM." from Apple Computer and "Visual Basic.TM." from Microsoft which borrow some, but not all, of the basic principles of OOP.
OOP systems generally are intended to extend reusability to all portions of a computer program, not just to particular modules. This is accomplished by distributing control of the various tasks performed by a program to individual "objects" with well-defined modular interfaces. These objects typically communicate by directly calling one another's capabilities or "methods."
OOP application development systems generally require their users to define abstract "classes" of objects that define the characteristics of the objects in that class. The actual objects in the user's application are instantiated as needed at "runtime" from this class "template." Such systems also typically include certain built-in reusable class libraries to provide the user with an initial base of functionality.
The instantiated objects that perform the various tasks within the user's application are reusable in large part due to a process known as "encapsulation." Encapsulation involves defining a class of objects with a well-defined external interface, and "hiding" within the objects the code and data necessary to implement that interface. In other words, the implementation of an object's interface is "encapsulated" entirely within the object.
Thus, programmers can reuse objects in various different contexts within their application. All other objects that are aware of this interface can "reuse" this object because its external dependencies are isolated within its modular "object interface."
Programmers also can use a process known as "inheritance" to "specialize" or modularly extend an object's functionality by reusing some or all of that object's characteristics via its interface or class definition--i.e. , the public methods and data structures that define the template for instances of that class. For example, programmers may wish to extend the base functionality of an existing class (whether created by the programmer or provided in a reusable class library) by adding a new method to objects in that class, or replacing (i.e., overriding) or supplementing (i.e., overloading) an existing method.
By defining a sub-class of objects that "inherit" the characteristics of the "parent" class, the programmer can reuse (as well as add to, replace or supplement) those inherited characteristics. For example, a "car" or "bus" sub-class might inherit characteristics from a more general "vehicle" class. A car is a "kind of" vehicle, and may inherit, for example, a vehicle's color and speed properties, as well as steering and braking methods.
3. The Need for Selective Reusability of "User Objects".
Despite their facilities for reusability of objects, OOP application development systems remain difficult to use. It is far simpler for application developers to create and configure instantiated objects directly, as opposed to defining abstract classes each time they need to create a slightly different object or to model the relationships or interaction among existing objects.
This problem is exacerbated when an application developer creates and desires to reuse compound or aggregate "user objects"--i.e., a group of objects that are related in the context of a particular application, but do not necessarily share common characteristics. These "user objects," in addition to requiring extensive inter-object communication, often involve "part of" or "container" relationships that occur frequently in any modularly designed application.
Inheritance and encapsulation are well-suited to model "kind of" relationships to further specialize the atomic objects from which an application will be built. All modular systems require certain atomic "building blocks" which are reusable and provide sufficient performance, even if difficult to modify.
To create applications of any significant complexity, however, one must combine these atomic objects into more complex "user objects" that interact with one another. It is this common modular process that illustrates the significant limitations on reusability imposed by traditional OOP systems.
OOP systems allow developers to "reuse" an existing class of objects by: (i) creating a class of objects that inherits from and "specializes" that existing class; (ii) creating a class of objects that encapsulates that existing class within its private (hidden) data structures and methods; or (iii) creating a distinct class of objects that communicates with (i.e., invokes methods of) that existing class of objects.
Yet, in all three cases, the new class of objects is tightly coupled to, and thus highly dependent upon, the existing class of objects. Mere communication among otherwise unrelated objects provides virtually no reusability, and creates many explicit dependencies. Although inheritance and encapsulation provide reusability of objects at various levels of complexity, a class of objects at any given level of complexity remains highly dependent upon the particular characteristics it encapsulates or inherits from its less complex superclasses.
It therefore remains quite difficult, if not impossible, to "selectively reuse" characteristics from (and interaction among) complex "user objects" in new environments. One cannot, for example, easily "mix and match" object characteristics across class libraries and choose "some from column A and some from column B." At any given level of complexity, a developer cannot simply replace undesired characteristics of a complex object from one class library with the more desirable characteristics of another complex object from a different class library. Until such selective reusability is achieved, the ultimate promise of OOP cannot be considered fulfilled.
a. An OOP Windowing Example.
Consider the following example involving simple windowing functionality, a very common application for traditional OOP systems. Imagine two independent developers ("A" and "B") creating windowing systems in a typical OOP development environment.
Developer A might create a "Window" class of window objects with a name ("Window X"), a particular size (specified by a bounding rectangle), a title bar (specified by another bounding rectangle), a color and a beveled border. Its only functionality might be to create, destroy and draw these window objects, via three respective methods.
To enhance the functionality of these window objects in a modular manner, Developer A might create a "Minimize" subclass that inherits from the Window class and adds a "minimize" capability. When a user clicks on the small minimize box in the upper right hand corner of the window, the minimize object transforms the window object into a small icon at the bottom of the screen. When the user clicks on the icon, the window is restored.
This Minimize subclass would require "minimize" and "maximize" methods to perform these functions, as well as a minimize box (specified by a bounding rectange), a minimize flag (indicating the state of the window--i.e. , whether it is currently minimized), and data for a minimize icon (pattern, size, position, etc.). In addition, the Minimize subclass would overload the create, destroy and draw methods of the Window class--i.e., to cream, destroy and draw the minimize box and/or icon as well as the window.
Finally, Developer A might add additional "drag" functionality to these window objects, which would enable the user to drag the window by its title bar. This subclass could inherit all data and methods from the Minimize subclass (which itseft inherits from the Window class), and simply add a "drag" method.
Developer B independently might take a similar approach. But, the class of Window objects created by Developer B (named Window X') might not include the beveled border. Moreover, Developer B might add a "window shade" capability that differs slightly from the minimize capability created by Developer A. When the user "double clicks" on the title bar of Window X', the Window might "roll up," showing only the title bar. Developer B might implement this capability by creating a "Window Shade" subclass that inherits from (and overloads the methods of) the Window class, and adds a drag method to implement this "window shade" capability. That subclass might also include a flag to track whether the window currently is "pulled up."
Assume that Developers A and B distribute their respective object classes in class libraries containing the public interface (i.e., the methods and data identified above), but not the private code and data that implements that interface. Selective reusability would allow Developer "C" to reuse all or any modular portion of the functionality created by Developers A and B, without recompiling these classes or obtaining access to any such private code.
It is true that Developer C could reuse the Window class, the Minimize subclass or the Drag subclass, thereby reusing functionality at various levels of abstraction. But, what if Developer C wanted to reuse the window and the drag capability created by Developer A, but integrate the window shade capability of Window X' in lieu of the minimize capability of Window X?
In short, Developer C cannot easily accomplish this task. The problem is that the private data and methods of the Minimize and Drag subclasses are dependent upon the public data and methods of the classes from which they inherit. The code might not even link.
The Drag subclass inherits from, and thus assumes the existence of, the Minimize class. The drag method might, for example, modify the bounding rectangle of a Minimize object to "move" that object (as well as the rest of the window) across the screen as the user drags the window. This dependency effectively prevents Developer C from replacing the Minimize class with the Window Shade class. The drag method would attempt to modify data that no longer exists.
Developer C would be forced to rewrite the Window Shade and Drag classes (potentially a significant amount of code) merely to reuse the Window class. Reusability therefore is substantially impaired by the external (inter-class or inter-object) dependencies of a subclass on its superclass. As noted above, encapsulating these dependencies or invoking other objects' methods within a class' private methods (as opposed to relying on inheritance) creates similar dependencies and perhaps even less reusability.
Any system of reasonable complexity will contain many levels of these external dependencies, making selective reusability of complex "user objects" across class libraries extremely difficult, if not practically impossible. As discussed below, today's application development systems create similar external dependencies that impair selective reusability.
4. The Lack of Selective Reusability in Multimedia Authoring Systems.
Consider, for example, the field of multimedia authoring systems to which an embodiment of the present invention is directed. Although there exist many such systems with almost as many different techniques, these systems all impose significant limitations on the author's ability to selectively reuse complex "user objects" across different environments.
Certain systems, such as Macromedia's Director,.TM. employ traditional metaphors with which authors are familiar and quite comfortable. Director employs a frame-based metaphor that requires authors to determine precisely which objects will be present in each frame of a sequence. This metaphor is familiar to animators, and frequently is used for constructing sequential animation sequences containing relatively little interactivity.
Constructing highly interactive applications, however, is quite difficult within the confines of a frame-based environment. Even with a scripting language included to provide greater control over the sequence in which characters appear, interactivity frequently is limited to that provided by one or more scripts within a frame.
It is quite difficult to reuse any of the characteristics of a particular object, because the object is highly dependent upon its environment (i.e., the scripts of other objects that "call" it, as well as the confines of the frames in which it appears, relative to other objects). Extensive reliance on scripting to model interactivity makes it difficult to isolate an object's external dependencies within a single script, and virtually impossible to isolate such dependencies across multiple scripts.
Rigorous OOP tools (such as the ScriptX.TM. language from Kaleida Labs and the more visually oriented Quest.TM. from Allen Communication) still require significant programming expertise on the part of their users. Despite offering extensive libraries of reusable "high-level" objects, such products remain relatively low-level development tools--i.e., OOP programming languages (or perhaps visual programming languages), as opposed to authoring tools.
As noted above, such tools impair selective reusability by creating external (inter-object) dependencies as a result of inheritance and encapsulation. Visual programming techniques facilitate the author's task of creating complex applications, but still frequently require scripting and/or programming to modify an object's characteristics or to model inter-object communication. The visual interface, at best, masks the programming that controls the user's application beneath the surface. In any event, the external dependencies remain, and reusability is impaired.
Other systems employ "pseudo-OOP" techniques in an attempt to combine the power of reusable objects with the flexibility of customizing the characteristics of particular objects and modeling inter-object communication. Forms-based authoring systems, such as Apple Computer's HyperCard.TM. and Microsoft's Visual Basic.TM., provide authors with significant freedom to create and configure individual objects using highly visual interfaces. Yet, such systems still rely heavily on scripting to implement an object's functionality, as well as inter-object communication and other forms of interaction.
Though these systems often contain many built-in types of objects, it is extremely difficult to create new object types. An object's unique characteristics therefore are determined by its script. Because objects communicate with one another directly via scripts, these scripts are highly dependent upon one another, and cannot easily be reused in other environments. Moving an object from one environment to another generally requires reading the scripts not only for that object, but for other objects in its former environment.
One category of products (e.g., Apple Computer's Apple Media Tool.TM.) provides a highly visual approach to authoring with virtually no scripting or programming involved. In other words, as opposed to visual programming languages (which, in essence, are merely "easy to use" scripting languages), such products provide true "object-based authoring"--i.e., the author creates and configures actual instantiated objects (or pseudo-objects) with little or no scripting or programming.
One problem with such tools is that they are extremely limited and inflexible. Authors typically cannot create custom events or messages, much less add functionality to an existing object or group objects together in any meaningful way. Moreover, such systems provide no mechanism for modeling modular "container" relationships among objects, and reusing these more complex "user objects" in different container environments.
All of the approaches noted above suffer from a lack of support for the creation of complex "user objects" that can be selectively reused in other environments. An object is only reusable to the extent that its dependencies on its external environment are isolated within (i.e., known to) that object. An object is only selectively reusable to the extent that it is loosely coupled to the objects it contains, thereby permitting authors to modify this relationship.
What is needed is a system that models this "container" relationship among objects in a manner that permits authors to selectively reuse object containers and the objects they contain across different container environments.
II. SUMMARY OF THE INVENTION
The present invention encompasses an application development system that enables its users to create reusable "object containers" merely by defining links among instantiated objects. Employing a technique referred to herein as Hierarchical Encapsulation, the system automatically isolates the external dependencies of the object containers created by its users. This isolation of external dependencies resolves the problems addressed above regarding selective reusability of "user objects," thereby facilitating the development of applications of increasing complexity.
Objects contained within other objects are not "hidden" within or tightly coupled to their object container environments. Rather, they are loosely coupled with those environments, and therefore can more easily be reused in other environments. By virtue of being contained within another object, the contained object automatically is afforded access to its environment. Its object container is, in essence, an "environmental frame of reference" for the objects it contains. For example, unless overridden by the author, objects automatically receive messages sent to their object container. They automatically can access data known to their object container. Their position is even determined relative to their object container.
Moreover, objects are decoupled from their characteristics. By defining two distinct types of objects (one of which modifies the characteristics of the other), and loosely coupling (i.e., temporarily linking) these two types of objects, the system provides a mechanism for authors to modify an object's characteristics merely by deeming one object to be contained within another. Removing that object from its container removes that characteristic. In this manner, authors easily can modify an object's characteristics and reuse it in other environments.
In one embodiment described herein, the system is optimized for the development of interactive multimedia applications or "titles." This multimedia authoring system provides its users ("authors") with a visual authoring interface that requires little, if any, scripting or programming. The system employs a form of object-based authoring in which authors create and configure instantiated objects directly, typically by "dragging and dropping" icons and configuring dialog boxes.
Authors can create two basic types of objects: Elements and Modifiers. Elements represent the actual characters or actors that interact with one another in the author's title. Elements generally can be linked to external media (such as text, sounds, pictures, animations and movies), and possess certain inherent characteristics relating to that media.
Authors can supplement an Element's inherent characteristics by incorporating Modifiers within that Element. These Modifiers provide the Element with properties (known as Variables) that further define what the Element is and capabilities that further determine what the Element does. A special type of Modifier, known as a Behavior, can contain additional Behaviors and other Modifiers, providing the author with a mechanism to create a complex Element "personality."
Both Elements and Behaviors are "object containers"--in this embodiment, object instances that can "contain" (i.e., be linked to) other object instances. Elements can contain Modifiers as well as other Elements; and Behaviors can contain Modifiers, including other Behaviors.
By incorporating Elements within Elements, authors create a Structural Hierarchy of Elements, each Element providing an environmental "frame of reference" for the Elements it contains. These "parent" Elements enable authors to provide structure for their titles and to model relationships among their Elements.
Elements can communicate with one another at a high "Element level," without regard to their child Elements. In one respect, Elements "encapsulate" their child Elements by creating a modular interface through which an Element's child Elements can communicate with objects external to that Element container.
Similarly, by incorporating Behaviors (and other Modifiers) within Behaviors, all inside an Element, authors create a Behavioral Hierarchy within the Element--i.e., the Element's internal "personality." Within the context of an Element "personality," each Behavior provides an environmental "frame of reference" for the Modifiers it contains. These "parent" Behaviors enable authors to model the relationships among the various Behaviors within an Element's overall personality.
Elements, in effect, "inherit" the characteristics provided by their internal Behavioral Hierarchy. Because Elements and Modifiers are distinct, loosely coupled objects, authors can modify an Element's characteristics merely by adding Modifiers to (or removing Modifiers from) an Element.
The system provides for significant reusability of object containers by utilizing the Structural and Behavioral Hierarchies to isolate the external dependencies of Elements and Behaviors. In essence, the system automatically "encapsulates" an author's object containers. Once encapsulated, they can be reused in other "environments." Moreover, by loosely coupling an Element to the Modifiers it contains, the system enables authors to modify their Elements so as to "inherit" and "disinherit" characteristics while maintaining an evolving hierarchical encapsulation vis-a-vis the Element's external environment.
Using a technique known as Adoption, an author can cause an Element to be "adopted" by a new parent Element. Using a similar technique known as Transplantation, an author can "transplant" an Element's Behavior (or its entire "personality") into another Element.
Because Hierarchical Encapsulation is integrated into the Structural and Behavioral Hierarchies determined by the author's object containers, authors obtain the benefits of this technique automatically. Their Elements and Behaviors are thus selectively reusable.
For example, a mechanism known as Hierarchical Message Broadcasting provides a structured messaging system that broadcasts messages from their initial destination down the Structural and Behavioral Hierarchies to all descendant Elements and Modifiers. This mechanism isolates an object container as a centralized abstract destination for all messages intended for "any object within that object container." This mechanism facilitates reusability of object containers in other environments in that an object container's new "parent" Element will provide it with messages automatically.
Another mechanism, known as Hierarchical Variable Scoping, makes a Variable accessible automatically to all descendant objects of the Variable's parent Element or Behavior. This mechanism isolates an object container's dependencies on Variables that are external to that object container, but still within its ancestral "environment." By making such Variables "known" to those objects in the object container that rely on that Variable, the object container can be moved to another environment with a well-defined external interface that "knows" which external Variables are assumed to be present in that environment.
Yet another mechanism, known as Hierarchical Relative Positioning, determines the position of a child Element relative to the position of its parent Element. As a result, the child Element moves with its parent Element automatically. This mechanism isolates an Element's external positional dependencies--i.e., the effects of an Element's environment on the Element's position.
In addition to the "built-in" Elements and Modifiers, the system is quite extensible via a "Component API." This Component API enables programmers to seamlessly integrate new Modifiers (and "Services" that support them) into the system.
Finally, the architecture of the system is substantially platform-independent. Titles can be "played" on multiple platforms. Moreover, the entire authoring environment can be ported to a variety of platforms with relatively little modification due to the isolation of a platform-dependent layer within the system.
III. BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is an illustration of the dual hierarchy (Structural and Behavioral) underlying the principles of the present invention.
FIG. 2 is a screen display showing the layout view window, tool palette, object information palette and menus of the graphical user interface of the implementation of one embodiment of the present invention.
FIG. 3 is screen display showing the layers view window of the graphical user interface under one embodiment of the invention.
FIG. 4 is a screen display showing the structure view window of the graphical user interface under one embodiment of the invention.
FIG. 5 is a screen display showing the asset palette of the graphical user interface under one embodiment of the invention.
FIG. 6 is a screen display of showing a library window in the graphical user interface under one embodiment of the invention.
FIG. 7 is a screen display showing the alias palette of the graphical user interface under one embodiment of the invention.
FIG. 8 is a screen display showing the messaging log window of the graphical user interface under one embodiment of the invention.
FIGS. 9(a)-(d) are screen displays showing the mToon editor windows of the graphical user interface under one embodiment of the invention.
FIG. 10 is a screen display. showing the Element configuration dialog of the graphical user interface under one embodiment of the invention.
FIG. 11 is a screen display of a Behavior configuration dialog in the graphical user interface under one embodiment of the invention.
FIGS. 12(a)-(h) are screen displays showing the Modifier configuration dialogs for Variables in the graphical user interface under one embodiment of the invention.
FIGS. 13(a)-(g) are screen displays showing the Modifier configuration dialogs for Messengers in the graphical user interface under one embodiment of the invention.
FIGS. 14(a)-(c) are screen displays showing the Modifier configuration dialogs for Scene-based Modifiers in the graphical user interface under one embodiment of the invention.
FIG. 15 is a diagram showing the operation of the Scene change Modifier of FIG. 14(a) in conjunction with the return Scene Modifier of FIG. 14(b).
FIGS. 16(a)-(d) are screen displays showing the Modifier configuration dialogs for Motion Modifiers in the graphical user interface under one embodiment of the invention.
FIGS. 17(a)-(d) are screen displays showing the Modifier configuration dialogs for Graphics Modifiers in the graphical user interface under one embodiment of the invention.
FIGS. 18(a)-(b) are screen displays showing the Modifier configuration dialogs for Sound Modifiers in the graphical user interface under one embodiment of the invention.
FIG. 19 is a screen display showing the Modifier configuration dialog for the style Modifier in the graphical user interface under one embodiment of the invention.
FIG. 20 is a screen display showing the Modifier configuration dialog for the cursor Modifier in the graphical user interface under one embodiment of the invention.
FIG. 21 is a screen display showing the Modifier configuration dialog for the set Miniscript Modifier in the graphical user interface under one embodiment of the invention.
FIG. 22 is a screen display showing the Modifier configuration dialog for the classification Modifier in the graphical user interface under one embodiment of the invention.
FIG. 23 is a screen display showing the Modifier configuration dialog for the set value Modifier in the graphical user interface under one embodiment of the invention.
FIG. 24 is a high level block diagram of the implementation of one embodiment of the present invention.
FIG. 25 is a diagram showing the interconnection of the various classes in the object model of the present invention.
FIG. 26 is a diagram showing the interconnection of the Service classes in the object model of the present invention.
FIG. 27 is a diagram showing the storage layout for a multimedia title in one embodiment of the present invention.
FIG. 28 is a flowchart showing the event processing loop under one embodiment of the invention.
FIG. 29 is a diagram showing the architecture of both the Component API and authoring GUI according to the present invention.
FIG. 30 is a diagram showing the operation of the object model of the present invention.
FIG. 31 is a flowchart showing the loading of Components (Modifiers, Services) at boot-up time of the program under one embodiment of the invention.
FIG. 32 is a screen display showing an implementation of a snake.
FIG. 33 is a screen display showing an implementation of a school of fish.
FIGS. 34(a)-(g) are screen displays of a selectively reusable windowing system.
FIGS. 35(a)-(g) are diagrams showing the implementation of the selectively reusable windowing system.
IV. DETAILED DESCRIPTION
A. External Architecture--"Author's-Eye View".
Before discussing the underlying implementation of the authoring system of the present invention, it is helpful to examine its external architecture--i.e., that portion of the system which is visible to the author.
It is through this external architecture that authors create end-user applications or "titles." As is explained in greater detail below, programmers can extend the functionality of this external architecture, seamlessly, via an interface to the "core" of the system. Moreover, the platform-independent nature of the system's implementation facilitates portability of the efforts of authors and programmers to a wide variety of computing platforms.
1. Objects: Elements and Modifiers.
In one embodiment of this invention, the author can create two basic types of objects: Elements and Modifiers. To use the metaphor of a stage play, the Elements comprise the play's structure (e.g., acts and scenes) and its actors, and the Modifiers comprise the stage directions, the motivations the director imbues in the actors to behave the way they do, and other modifications to the characteristics of the play's structure and its actors.
The system enables authors to create various types of Elements and to assign particular characteristics to each Element. These characteristics include properties (e.g., weight, color, etc.) that define what the Element is and capabilities (e.g., send messages, move along a path, etc.) that determine what the Element does.
Authors assign characteristics to an Element by incorporating Modifiers within that Element. As is illustrated below, the process of creating and configuring Elements and Modifiers is highly visual and intuitive, typically involving "dragging and dropping" icons representing Elements and Modifiers, selecting menu items and configuring dialog boxes. Little, if any, scripting or programming is required.
The system is, however, quite extensible via the "Component API," discussed in greater detail below. Programmers can use the Component API to create "Components" that become fully integrated Modifiers and "Services" (which "service" Modifiers but are not accessible directly to authors). Once created, Components are indistinguishable from the system's "built-in" Modifiers and Services.
In this embodiment, there are three general categories of Modifiers: (i) Variables that simply store data of various types; (ii) Capabilities that perform actions on behalf of the Element; and (iii) Behaviors that "contain" additional Behaviors and other Modifiers. In another embodiment, a hybrid Variable/Capability Modifier could both store data and perform actions.
An author can create two general categories of Elements: (i) Structural Elements (including Projects, Sections and Subsections, discussed in greater detail below) that serve only to "contain" other Elements and Modifiers; and (ii) Media Elements (including Scenes, discussed below) that not only can contain other Media Elements and Modifiers, but also can be "linked" to raw media, such as text, sounds, pictures, animations and movies.
There are three basic types of Media Elements that an author can create: (i) Graphic Elements (including still pictures, animations and videos), Text Elements and Sound Elements. An author can then link raw media of a particular format to the Element, causing that Element to attain certain inherent capabilities (e.g., to make a sound, display a picture, or play a movie).
By linking raw media of a particular format to a Graphic, Text or Sound Element, the author causes the system to "morph" the Element into a more specific type of Element (and among specific types) related to that format. For example, the author could create an "AIFF" sound, an "ASCII" sentence, a "PICT" picture, a "QuickTime" movie or an "mToon" (a built-in animation format). The morphing process is described in greater detail below with respect to the system's implementation.
Elements of a given type also have certain built-in properties known as Attributes. An Element's Attributes include properties applicable to the type of media to which the Element is linked (e.g., position, cel number, rate, etc.).
Thus, an Element's properties are defined by its Attributes and its author-created Variables. An Element's capabilities are determined by the type of media, if any, to which it is linked, and by its author-created Behaviors and Capabilities (i.e., its non-Variable Modifiers). Together, these properties and capabilities define an Element's individual characteristics--i.e., what the Element is and what the Element does.
In addition to creating Elements and assigning them individual characteristics, an author can specify the manner in which these Elements interact with one another, via an integrated object messaging mechanism. This messaging mechanism is accessible to authors via Modifiers, as is described in greater detail below.
2. Object Containers: Elements and Behaviors.
Unlike simple "leaf" objects, Elements and Behaviors are "object containers" that can comprise a hierarchy of objects. These object hierarchies provide "environments" (each enclosed by an object container) in which the system can isolate external dependencies. This isolation of an object container's external dependencies, referred to herein as "Hierarchical Encapsulation," results in a significant degree of reusability of the Elements and Behaviors created by an author.
As is illustrated below, authors can construct complex environments comprising a hierarchy of "Elements within Elements" interacting with one another. Within an Element, authors can create equally complex internal environments (together representing the Element's "personality") comprising a hierarchy of interrelated "Behaviors and other Modifiers within Behaviors."
Authors can selectively reuse these modular Element and Behavior object containers at virtually any level of complexity. Moreover, authors can "mix and match" Elements and Modifiers from differing environments at practically any level of the Structural and Behavioral Hierarchies. Furthermore, a team of authors can collaborate to create and test an entire application while individual authors refine their own modular, serf-contained environments.
For example, multiple authors might collaborate to build a complex model of a "car" Element that contains one "engine," four "wheel" and various other Elements, as well as a "driving" Behavior that contains "steering," "braking" and various other Behaviors. Individual authors each might create one or more of these self-contained Elements and Behaviors, and then collaborate with one another to test all or a portion of the "car" and refine the desired interaction among the various Elements and Behaviors.
Moreover, a subsequent author working on a different application could reuse the entire "car" Element or merely one or more of the Elements or Behaviors contained therein. For example, the author might apply the "braking" Behavior to a "horse" or "airplane" Element in another application.
To create complex objects, authors need not create abstract "classes" or object templates that serve to relate objects to one another with respect to their characteristics. Elements, Behaviors and other Modifiers created by the author are object instances. To permit Elements and Behaviors to "contain" other objects, the system links the Element and Behavior object instances to the other object instances contained within them (as is explained in greater detail below with respect to the system's implementation).
Thus, Elements and Behaviors are object containers--in this embodiment, object instances that can "contain" (i.e., be linked to) other object instances. To some extent, however, Elements do attain (at least temporarily) the characteristics they contain. Yet Elements merely provide an environmental frame of reference to their descendant Elements. Elements and Behaviors do not actually "inherit" the characteristics of the objects they contain.
Many embodiments of this "object container" relationship among objects are possible, including intersecting families of objects, multiple-parent relationships and various combinations of "one-to-many" and "many-to-many" child-parent relationships. In any event, by spawning object hierarchies, object containers provide the environments that facilitate their reusability via Hierarchical Encapsulation, discussed below.
Authors are thus free to create and work directly with objects that represent, at any level of complexity, the characters in their application and the individual characteristics "contained within" those characters, as well as the all-encompassing "environmental factors" that "contain" those characters and determine the nature of their interaction.
3. Object Hierarchies: Structural Hierarchy and Behavioral Hierarchy.
When an Element contains another object within it, the Element object is called a parent and the other object is called its child. If the child object in turn contains another object, the child object is considered the parent of the object it contains, and so on.
This chain of parents and children is called an Element or (Structural) Hierarchy. In one embodiment, each parent can have multiple children, but each child has exactly one parent. Elements above and below a particular Element in the Structural Hierarchy can be referred to, respectively, as ancestors and descendants of that Element. Children of the same parent are called siblings.
Just as an Element can contain other Elements in the Structural Hierarchy, each Element also contains its own Modifier or Behavioral Hierarchy of Behaviors and "leaf" Modifiers (i.e. the Element's personality). These Modifiers modify the characteristics of the Element (e.g., by storing data in the Element or performing actions on behalf of the Element), often in response to messages from other Modifiers inside the same or another Element.
In the context of the Structural Hierarchy, an Element can be viewed as an environmental "frame of reference" for its descendant Elements. It is within that Element's environment that its descendant Elements exhibit their personalities. The Structural Hierarchy determines the manner in which those descendant Elements interact with one another within that Element's environment.
Similarly, the Behavioral Hierarchy determines the manner of interaction among an Element's internal Behaviors. In the context of the Element's overall personality, a Behavior also can be viewed as an environmental "frame of reference" for its descendant Behaviors and other Modifiers.
Continuing our "car" example, the "car" Element provides the environment in which the "engine" Element functions. If the car moves, the engine will move with the car automatically (as it would in a real car). Yet, the engine Element need not have any internal "movement" Behavior. By acting as the local coordinate system for its child Elements, the car becomes the engine's environmental frame of reference. This particular manner of isolating positional dependencies of Elements within the Structural Hierarchy, referred to as Hierarchical Relative Positioning, is discussed in greater detail below.
Similarly, the "driving" Behavior provides the environment in which the "braking" Behavior functions. If the driver does not step on the brakes, the "braking" Behavior will not be invoked--i.e., it will not receive a "brake pedal depressed" message from its parent "driving" Behavior. By acting as the "gatekeeper" for messages intended for its child Behaviors, the parent "driving" Behavior becomes the "braking" Behavior's environmental frame of reference. This particular manner of isolating messaging dependencies of Elements and Behaviors (and other Modifiers) within the Structural and Behavioral Hierarchies, referred to as Hierarchical Message Broadcasting, is discussed in greater detail below.
Continuing the example, the "driving" Behavior must monitor the cat's speed for a variety of reasons, as must the driver of a real car (e.g., to know when to release the brake). The speed of the car could be stored in a "car speed" Variable inside the Element, but not necessarily inside the "driving" Behavior because other Behaviors (e.g., an "Air Bag" Behavior detecting massive deceleration) may need to access this Variable. By making its child Variables accessible to its descendants, the car becomes the frame of reference for the "driving" Behavior. This particular manner of isolating data dependencies of Elements and Behaviors (and other Modifiers) within the Structural and Behavioral Hierarchies, referred to as Hierarchical Variable Scoping, also is discussed in greater detail below.
a. Types of Elements in the Structural Hierarchy.
As noted above, there exist two basic categories of Elements in this embodiment--Structural Elements and Media Elements.
Authors can utilize Structural Elements to group the contents of a title into organized sections, like the chapters of a book or the acts in a play. In this embodiment, the purely Structural Elements comprise Projects, Sections and Subsections.
Media Elements (including the Modifiers contained within them) comprise the primary content of an author's application. They typically are the characters or actors within a "Scene" (a special Media Element the primary purpose of which is structural--to contain other Media Elements). Any Media Element can be used structurally, as containers for other Media Elements, to form a more complex Element. Media Elements as their name suggests, typically are linked to raw media, such as text, sounds, pictures, movies or animations, in order to communicate with the user of the title.
(1) Projects.
A Project is a Structural Element that contains the entire title. Its children are Sections, which are described below. A Project also can contain Modifiers (including Behaviors) that store data or perform actions on behalf of the entire Project. The score of a game, for example, could be stored in the Project so as to be available to all other Elements and Modifiers.
(2) Sections.
A Section is a Structural Element that can be used by the author to organize logical portions of his title. It is most analogous to an act in a play or a chapter in a book. For example, an author of a U.S. travel game might create a Section for all Elements and Modifiers relating specifically to the state of California. Sections are parents to Subsections, as described below. As with Projects, Sections also can contain Modifiers. A "time-in-state" Behavior might, for example, track the amount of time the user spent in a particular state.
(3) Subsections.
A Subsection is a Structural Element that can be used by the author to further organize logical portions of a Section. Continuing the above example, the author of the travel game might create a Subsection for all Elements and Modifiers relating specifically to the San Francisco Bay Area. Subsections are parents to Scenes, and also can contain Modifiers.
(4) Scenes.
A Scene is a special Media Element that not only further organizes logical portions of a Subsection, but also contains other (non-Scene) Media Elements and Modifiers. Much like the scenes in a play, the Scene Element contains those Media Elements and Modifiers the author deems necessary to impart to the user a portion of the interactive experience provided by the parent Subsection. Continuing our travel game example, one Scene might take place on the Pacific Coast, while another occurs further inland in the city of Palo Alto.
In one embodiment, there exists a special type of Scene Element known as a Shared Scene. A Shared Scene is used to organize Elements that will be visible across multiple Scenes, such as a background picture or an animated character that travels from Scene to Scene.
In a more refined embodiment, the author can share visible Elements across Subsections and Sections by having multiple Shared Scenes, or being able to select one or more Shared Scenes dynamically. In the latter case, rather than designating a Shared Scene statically at "authoring time," the author can, dynamically at "runtime," designate any Scene (or perhaps multiple Scenes) as a current Shared Scene.
(5) Media Elements.
Media Elements comprise the primary characters or actors in a Scene. They can contain other Elements (to form a more complex Element) and can be linked to raw media. A Media Element also can contain Modifiers, and thus Behaviors, which together constitute the Element's personality. As illustrated below, it is an Element's "dual hierarchy" that provides the environmental framework in which descendant Elements interact with one another and exhibit their own individual "personality."
b. Isolation of External Dependencies of Object Containers.
Elements and Behaviors are object containers that spawn hierarchies of descendant objects. Consider the sample illustration of the Structural and Behavioral Hierarchies in FIG. 1. When Media Elements are created, they are positioned automatically below the Scene in the Project's Structural Hierarchy. Thus, Element E1 108 is a child of Scene Sc1 106 in Subsection SS1 104 in Section S1 102 in Project P 101. Element E4 111 is a child of Element E1 108.
By isolating the external dependencies of the environment created by an Element or Behavior object container, the system provides for a significant degree of selective reusability of that Element or Behavior (or any object contained therein) in other environments, as is further demonstrated below. It is this isolation of an object container's external dependencies that allows the Elements and Behaviors created by an author to become an "environmental frame of reference" for their descendant Elements, Behaviors and other Modifiers, as was demonstrated in the "car" example discussed above.
In one embodiment of this system, three mechanisms are employed to isolate external dependencies within object containers: (i) Hierarchical Message Broadcasting--messages that are received by an Element or Behavior typically are "broadcast" down the Structural and Behavioral Hierarchies to its descendant Elements and Modifiers (though this effect can be overridden); (ii) Hierarchical Variable Scoping--Variables are accessible to all descendant Elements and Modifiers of that Variable's parent Element or Behavior; and (iii) Hierarchical Relative Positioning--an Element's position in the Scene is determined relative to the position of (i.e., using the local coordinate system of) its parent Element, and therefore changes as the position of its parent Element changes.
(1) Hierarchical Message Broadcasting.
A message sent to an Element or Behavior typically will be broadcast down the Structural and Behavioral Hierarchies. Elements pass down messages to their children, be they Modifiers or other Elements. Behavior pass down messages to their child Modifiers. The precise manner in which the various types of messages propagate down the Structural and Behavioral Hierarchies is discussed in greater detail below with respect to object messaging generally.
In one embodiment, the order in which messages are broadcast among Elements, Behaviors and other Modifiers at any given level of the Structural and Behavioral Hierarchies is determined by the order in which those objects appear in the "structure view window" (see, e.g., structure view window 330 in FIG. 4). Initially, this is the order in which they are created. The author can modify this order simply by "dragging and dropping" or "cutting and pasting" the object's icon in the structure view window.
For example, referring again to FIG. 1, a message sent to Scene Sc1 106, under one embodiment, initially would be sent down to Element E1 108 (but not to any sibling Elements of Scene Sc1 106, such as Shared Scene Sc0 107). It would then be sent down to Element E4 111, and then to Element E2 109. From there it would be sent down to Modifiers M1 114, M2 115 and M3 116, and then to Element E3 110. It would then be sent down to Modifier M4 117, Behavior B1 118 and then down to Modifier M5 119. Finally, it would be sent to Behavior B2 120 and then down to Modifiers M6 122 and M7 123.
Note that, in this embodiment, Variables such as Variable V3 121 solely store data and thus do not respond to messages. Their data, however, can be read and written by other Modifiers, as for example, the Miniscript Modifier.
In this manner, a message sent to an Element or Behavior typically will be broadcast to all objects contained within that Element or Behavior. Upon receiving the message, those objects (Elements, Behaviors and other Modifiers) can then respond appropriately, performing whatever actions are specified by their current configuration.
By providing for the sending of messages to an Element or Behavior object container, and the broadcasting of those messages to the objects within that container, the system facilitates the isolation of messaging traffic to and from that container. In essence, the container becomes a higher-level abstract destination for messages intended not only for the container itself, but for "objects within the container."
Continuing the above example of the travel game, assume that Section S1 102 represents California, Section S2 103 represents Kansas and Subsection SS1 104 represents the San Francisco Bay Area. The author could create Behaviors to simulate the different weather patterns in these two states.
A "Midwest Weather" Behavior in Section S2 103 could simulate the weather in Kansas and periodically update a "temperature" Variable and send a "check temperature" message to that Section S2 103, which would be broadcast down the Structural and Behavioral Hierarchies to all Elements and Modifiers in each Scene in Kansas. This Behavior and the temperature Variable could both be contained at the Section S2 103 level of the Structural Hierarchy to simulate relatively constant weather throughout the state of Kansas.
Conversely, within California, Subsection SS1 104 (the San Francisco Bay Area) could contain two Scenes, one representing the Pacific Coast and the other representing Palo Alto. These Scenes could contain "Coast Weather" and "Inland Weather" Behaviors, respectively, each updating their own "temperature" Variable and sending a "check temperature" message to their respective Scenes. These separate Behaviors could simulate the differing weather patterns that occur even within a relatively small region of California.
"People" Elements in the various Scenes within the two states might be very complex, perhaps containing five or ten different Behaviors, including a Jogging, Swimming and a "Check Temperature" Behavior that check the temperature in response to a "check temperature" message. If the temperature deviates too far from the average temperature, the Check Temperature Behavior might send a "Very High Temperature" or "Very Low Temperature" message to its person Element, which will be broadcast down to the Swimming and Jogging Behaviors. The Swimming Behavior might be enabled by "Very High Temperatures" and disabled by "Very Low Temperature" and vice-versa for the Jogging Behavior.
By permitting the various "weather" Behaviors to broadcast their messages down the Structural and Behavioral Hierarchies (from the Section level in Kansas, and from the Scene level in California), the people Elements in either state will receive this message from their respective parent Scenes. The "weather" Behaviors need not target their message directly to a particular person Element, or even know which type of Element might respond to their "check temperature" message (e.g., person, animal, etc.).
The people Elements, therefore, could be moved from any Scene in Kansas to any other Scene in Kansas or California, and still respond appropriately to the temperature in that region. Even though the "check temperature" message is sent to a different environment or level of the Structural Hierarchy in Kansas (Section) than in California (Scene), it still reaches all relevant people Elements.
This degree of reusability of Elements and Behaviors is facilitated by the system's Hierarchical Message Broadcasting mechanism, which broadcasts messages sent to an object container to the objects it contains. This mechanism isolates the dependencies of a message at the abstract level of the environment represented by the object container to which that message is sent, as opposed to requiring the author to target the message directly to a particular "leaf" object.
As is discussed below, direct targeting of messages remains an option in this embodiment in the event that the author elects to forego a degree of reusability in favor of explicit target naming.
(2) Hierarchical Variable Scoping.
As demonstrated above, Elements and Behaviors are dependent upon messages to enable, disable and trigger certain actions, including those of their descendant Modifiers. Those Modifiers frequently are dependent upon Variables to perform actions on behalf of their parent Elements. Isolating Variable dependencies within Elements and Behaviors also facilitates reusability of those object containers.
The scope of Variables (data) accessible to a Modifier is another manifestation of that Modifier's environment. Variables are accessible to all descendant objects of that Variable's parent Element or Behavior. In other words, Variables are accessible within their parent's environment.
Hierarchical Variable Scoping serves to isolate an object container's dependencies on external Variables. By making Variables accessible to descendant objects of the Variable's parent, those descendant objects automatically "know about" those external Variables. A descendant Element or Behavior therefore becomes reusable in that the external Variables upon which it depends are isolated within ("known to") that Element or Behavior.
Returning to our travel game example, the "temperature" Variable in Kansas is accessible to the "Check Temperature" Behavior within any person Element of any Scene in Kansas. Because the "temperature" Variable is located at the Section S2 103 level, its descendant Elements include all Subsections therein, each Scene contained within those Subsections, each Element in each such Scene (including any people Elements), and finally the "Check Temperature" Behavior contained within such people Elements.
Moreover, if a person Element was moved from a Scene in Kansas to the Pacific Coast Scene in California, the "temperature" Variable in that Scene would be accessible, automatically, to that person Element and to its "Check Temperature" Behavior. This degree of reusability is a direct result of the Hierarchical Variable Scoping mechanism.
In the above example, this mechanism isolates "temperature" Variable dependencies at the Section level in Kansas and at the Scene level within the San Francisco Bay Area in California. As noted above, the Hierarchical Message Broadcasting mechanism also isolates messaging dependencies (in particular the "check temperature" message) at those same levels of the Structural Hierarchy. Thus, in this example, these two mechanisms together make object containers below that level (such as the person Element) reusable across Sections of a Project.
(3) Hierarchical Relative Positioning.
Another environmental dependency of object containers, in this case limited to Elements, is the position of an Element relative to that of its ancestors. Rather than require the author to model the common circumstance in which the movement of one Element must be relative to the movement of another Element, the system, by default, determines an Element's position in the Scene relative to the position of (i.e., using the local coordinate system of) its parent Element.
In another embodiment, this effect could be made optional. In any event, a child Element can, as part of its own personality, move on its own initiative. Its position nevertheless remains relative to its parent Element. It therefore also will continue to move as its parent Element moves.
Thus, a child Element moves (changes position) within the Scene as its parent Element moves. This effect filters down the Structural Hierarchy below the Scene level. A common use of Hierarchical Relative Positioning is to model Elements physically contained within or attached to other Elements (e.g., a toy in a box or the wings of a bird). It also can be used to model Elements that, for one reason or another, tend to follow other Elements (e.g., planets orbiting the sun, or packs of animals that tend to move together).
Hierarchical Relative Positioning also serves to isolate an Element's dependencies on its environment, in that an author need not model this "follow my parent" movement. Upon reusing or reattaching an Element to another parent Element, the system automatically recalculates that Element's position relative to its new parent Element.
c. Selective Reusability through Adoption and Transplantation.
The selective reusability of object containers discussed above frequently takes one of two forms. If Elements are to be reused, they often will be placed in a new environment--i.e., given a new parent Element in the Structural Hierarchy. This process is referred to as "Adoption." In other words, the child Element has been "adopted" by a new parent Element.
The underlying implementation of Adoption is discussed in greater detail below. From the author's perspective, Adoption is a simple process at authoring time. For example, referring to FIG. 2, the author can break its current link and create a new link with the link tool 365 on the tool palette 36 with another Element in the layout view window 320. Alternatively, the author can drag and drop the Element's icon to the icon representing its new parent Element in the Structure Window 33.
Authors also may desire to move a Behavior or other Modifier from within its parent Element or Behavior to another parent Element or Behavior. This process is referred to as "Transplantation." In other words, a Behavior or other Modifier, and its associated capabilities, are "transplanted" to a new Element or Behavior, which then acquires those capabilities. Transplantation also can be accomplished quite easily at authoring time, by dragging and dropping the icon representing the Behavior or Modifier from its old parent to its new parent, either in the Layout Window 320 or in the Structure Window 33.
Adoption and Transplantation are quite similar, both from the author's perspective and in view of the manner in which they are implemented, as is discussed in greater detail below. Both involve the transfer of an Element or Modifier to a new parent Element or Behavior. In other words, both involve the selective reusability of an Element or Modifier in new environments.
In a more refined embodiment, Adoption and/or Transplantation can be performed dynamically, at runtime. One author might, for example, desire to model certain gravitational forces that cause an Element "planet" orbiting another Element "star" to begin to orbit a new parent Element "rogue star" that passes nearby. Another author might desire to model a change in a person's behavior, in effect by transplanting a Behavior Modifier within the Element representing that person.
As is discussed in greater detail below with respect to the system's implementation, this process is quite similar whether performed at authoring time (referred to as "Edit Mode") or at runtime (referred to as "Runtime Mode"). Depending upon the nature of the Adoption or Transplantation the author desires (or perhaps desires to provide to the user), different functions can be accessed via the Component API.
Similarly, a programmer can, via the Component API, provide dynamic (runtime) instantiation of objects, such as Elements and Modifiers. This process, referred to as "Dynamic Creation," also is discussed below with respect to the system's implementation.
4. Object Authoring Interface.
FIGS. 2-9 show some of the features of the object authoring interface. They include the layers view window 310 (FIG. 3), layout view window 320 (FIG. 2), structure view window 330 (FIG. 4), modifier palettes 35a and 35b (FIG. 9), tool palette 36 (FIG. 2), object information palette 37 (FIG. 2), menus 38 (FIG. 2), asset palette 70 (FIG. 5), alias palette 80 (FIG. 7), mToon editor windows 900, 910, 920 and 930 (FIG. 9), and messaging log window 420 (FIG. 8).
With reference to FIG. 2, the tool palette 36 is provided with a variety of functions, including: (i) a pointer 361 for selecting, clicking, dragging, scrolling, etc.; (ii) a graphic tool 362 for creating Graphic Elements; (iii) a text placeholder tool 363 for creating Text Elements; (iv) a cropping tool 364 to crop and link non-animated Graphic Elements; (v) a linking tool 365 for linking Elements into a parent/child relationship; (vi) a hand tool 366 to offset media within Elements, as well as (vii) a background/foreground color tool 367 for selecting the background and foreground colors of an Element.
The object information palette 37 displays an Element's position in x 328 and y 329 coordinates, size in x 336 and y 331 coordinates, scale in x 332 and y 333 coordinates, and other data related to a selected Element, such as the Element's name 326, the name of its parent 327, its layer order number 335 and, in the case of an mToon, its cel number 334. In the "Hierarchical Relative Positioning" embodiment discussed herein, the position of the Element is given relative to its parent.
The menus 38 allow access to all other functions not accessible otherwise from the present on-screen features. Menu 338a is the "Apple" menu under the Macintosh.TM. operating system with its well known features. The functions of the remaining menus are described below.
"File" menu 338b allows authors to create new Projects, libraries and mToons, to open existing Projects, to close windows, to save projects under the current or another name, to link Elements to media in a particular file or resources folder. Menu 338b further allows authors to break media links, to run a Project from either the start or from a given selection, to build a title from the current Project, or to quit the application.
"Edit" menu 338d allows authors to perform the usual undo, cut, copy, paste, clear, "select all" and duplicate functions, as well as create author messages and set application and Project preferences.
"Format" menu 338d allows authors to set font, font size, font style and text alignment characteristics for selected text.
"Arrange" menu 338e allows authors to align objects, to adjust their sizes, to move them completely forward or backward in the layer order, or incrementally so, as well as to adjust for any gaps in layer order created by such operations.
"Object" menu 338f allows authors to create new Sections, Subsections, Scenes, Graphic Elements, Sound Elements and Text Elements, to access the Element information dialog 950 (FIG. 10), to revert to the default size of an object, to lock an object's settings, to find items by name, as well as to make and break aliases.
"View" menu 338g allows authors to open the layout view window 320, the structure view window 330, the layer view window 310, the tool palette 36, the modifier palettes 35a and 35b, the alias palette 80, the asset palette 70, the object information palette 37 and the messaging log window 420. In addition, authors can use menu 338g to preview the effects of selecting different color tables, to show or hide Element frames, Modifier icons, object names, low resolution (draft) object images, the Shared Scene, as well as to synchronize all windows to reflect information for the same object.
"Window" menu 338h allows authors to access all currently open windows.
a. Layout View.
The layout view 32 is used to edit a Scene of a Project. Again with reference to FIG. 2, the basic operation of the layout view 32, through layout view window 320, is as follows. The layout view window 320 represents Projects at a Scene level, thus acting as a stage in which authors can arrange their Media Elements. To bring the appropriate Scene into the layout view window 320, pop-up menus 321, 322 and 323 are provided to navigate, respectively, among a Project's Sections, Subsections and Scenes. These pop-up menus 321, 322 and 323 also have "New Section", "New Subsection" and "New Scene" items, as an alternative to using Object menu 338f to create such Structural Elements. Back arrow 324a and forward arrow 324b are an alternate means of accessing the previous and next Scenes within the current Subsection. Forward arrow 324b is dimmed to indicate that the current Scene is the last Scene in the current Subsection. Shown within the content region of layout view window 320 is a Graphic Element 337 containing Graphic Modifier 338.
b. Layers View.
The layers view 31 is used to edit Projects a Section at a time through the layers view window 310. With reference to FIG. 3, the basic operation of the layers view window 310 is now described. The layers view window 310 presents a matrix of Media Elements, arranged by Scene order on the horizontal axis and layer order on the vertical. Layer order is the order in which Elements are drawn on the screen, meaning that a Scene is layer order 0, and all the Scene's contained Elements progress in increments of 1 therefrom. Layer order is particularly useful to create the illusion that the two dimensional (X,Y) screen has a depth (Z) dimension, an effect referred to as "2.5D." The layer order can easily be changed by dragging and dropping an Element over another Element, although this may also be accomplished through menus 38. Either way, the user performs such operations as bring to front (make layer order the highest), send to back (make layer order 1), bring forward (exchange layer order with next highest) or bring backward (exchange layer order with next lowest). Scenes are assigned layer order 0, which cannot be changed by the author, and are thus always in the background.
The layers view window 310 contains pop-up menus 311 and 312 that navigate among a Project's Section and Subsections, respectively, in a manner similar to the pop-up menus 321 and 322 of the layout view window 320. Furthermore, back and forward arrows 313a and 313b act to select Subsections much as arrows 324a and 324b navigate through Scenes in the layout view window 320.
Thus, in this example, one can see columns arranged horizontally for Shared Scene 314a, Scene 1 314b, Scene 2 314c and Scene 3 314d. Vertically, one can see rows for layer order 0 (Scene) 318a, layer order 1 318b and layer order 2 318c. Along row 318a, one can see the background pictures for the respective Scenes 315a, 315b, 315c and 315d, as well as four non-Scene Media Elements (Element 316a and 316d under Scene 1 315b, Element 316b under Scene 2 315c, and Element 316c under Scene 3 315d). Modifiers are also shown within Elements (with Modifiers 317a, 317b and 317c in Shared Scene 314a, Modifiers 317d, 317e and 317f in Scene 3 315d, Modifier 317g in Scene 1 315b, Modifier 317h in Scene 2 315c, Modifiers 317i and 317j in Element 316a, and Modifier 317k in Element 316c).
Elements and Modifiers can be dragged and dropped within layers view window 310. In particular, an Element's layer order can be modified easily by dragging and dropping it onto an Element in another layer, causing all Elements to be "pushed up" one layer higher.
c. Structure View.
The structure view 33 allows authors to move Elements and Modifiers throughout the Structural and Behavioral Hierarchies--i.e., across Projects, Subsections and Scenes, as well as within Elements and Behaviors. Structure view 33 generates and controls the operation of structure view window 330, shown in FIG. 4. The Structural Hierarchy is plainly evident in that Project "Untitled-1" 303 is at the top, with Section "untitled section" 304 immediately below and indented with respect to the Project 303 icon, Subsection "untitled subsection" 305 immediately below and indented with respect to the section 304 icon, Scene "untitled Scene" 306 immediately below and indented with respect to the Subsection 305 icon. In this instance the Scene 306 icon represents a Shared Scene, to which a Graphic Modifier 308 is attached. The Subsection 305 has another child Scene 307, being the actual first Scene, which is represented as equally indented as its sibling Scene 306. Under this Scene 307 is Media Element 300. Knobs 302a, 302b, 302c, 302d, 302e and 302f are used to reveal or conceal the objects beneath Elements 303, 304, 305, 306, 307 and 309, respectively, in the Structural Hierarchy. This is particularly useful for an author who wishes to concentrate on only part of the Project at a given time.
Changing the hierarchical relationship through the structure view window 330 is particularly simple. Elements may be dragged up and down and dropped into the Structural Hierarchy at will, subject to a few limitations. For example, a Media Element cannot be dropped directly into a Project, Section or Subsection. Similarly, Scenes cannot be dropped into to a Section or Project. In addition, only Modifiers can be dropped into Behaviors.
d. Asset Palette.
The Asset Manager 7, operating through the Asset Palette 70, illustrated in FIG. 5, is a visual database of the media that have been linked to a Project. This palette makes linking, storing, and copying Graphic and Sound Elements easy and convenient. Each item can then be dragged and dropped onto Scenes from this central location. Each time a new asset is added to the Project, it gets added to the Asset Manager 7, which adds it to the Asset Palette 70 in a manner that is transparent to the author.
Once an asset is in the Asset Palette 70, it can be used by multiple objects in the Project. For example, a single animation file can be linked simultaneously to multiple Elements. Each Element can use this animation without regard to the other Elements. Each entry in the Asset Palette 70 includes an associated count of the number of objects which use that asset. When an asset is no longer in use (i.e. its count becomes zero), it still remains in the Asset Palette 70.
Each time the author performs a link to media, an asset appears in the Asset Palette 70. "Show" pop-up menu 751 displays media files by selected types (e.g., all types, all graphics types, PICTs, mToons, sounds, QuickTime.TM. movies, or color tables). "View" pop-up menu 752 displays assets either as icons or as small or large thumbnails. Trash can icon 753 allows authors to remove assets from the palette by dragging them to the trash can. There are multiple assets 754a, 754b, 754c, 754d shown here. For example, asset 754c is a sound, represented by a sound icon and having a preview button to allow the author to hear the sound.
e. Library.
Libraries offer a convenient place to store Elements and Modifiers for use across Projects. Authors can create multiple libraries, and can have more than one open at once. Authors can move Elements and Modifiers freely between any Project and any library. Any object, from a simple Modifier to an entire Scene, Subsection, Section, or Project (at any stage of development) can be dragged between libraries and Projects.
With reference to FIG. 6, the operation of a view 520 of one of the libraries 52 is shown. It includes the library's name 726, a close box 727, a trash can icon 728 which allows authors to remove items from the library by dragging them to the trash can, a "save" pop-up menu which allows authors to save the library to disk, and, as shown here, multiple objects 729a, 729b, 729c, 729d, 729e and 729f in the library.
f. Alias Palette.
An alias is a programming tool that enables authors to maintain identical settings across multiple Modifiers. When a Modifier is "aliased" initially, a master copy is placed in the Alias Palette 80. Authors can create additional aliases that refer to that master copy merely by duplicating any aliased Modifier. The advantage of using aliases lies in the fact that they are globally updatable. Changing the settings of one "instance" of an alias alters the settings of its master copy in the Alias Palette 80 and automatically updates the settings of each of the other instances of that alias throughout the Project.
Variables are good candidates for aliasing. For example, a game score can be stored and aliased. This Variable can then be strategically placed in Elements far apart in the Structural Hierarchy. Messengers and Miniscript Modifiers in these Elements can access and update their "local copy" of this Variable, which in turn will access the master copy, thereby ensuring synchronization across all "local copies" of this Variable. In this manner, various Elements, regardless of their location in the Structural Hierarchy, will have access to the updated score as the value of this Variable changes during the game. Breaking an alias causes the formerly aliased Modifier to become an independent Modifier, no longer updated when its former aliases are modified.
With reference to FIG. 7, Alias Palette 80, controlled by Alias Manager 8, includes a close box 854, a trash can icon 851 which allows authors to remove aliases from the palette by dragging them to the trash can, and, as shown here, multiple aliases 852a, 852b, 852c, 852d, 852e and 852f, with respective user counts 853a, 853b, 853c, 853d, 853e and 853f. The user count reveals the number of instances of such alias within the current Project. Thus, if a new alias is created, a new entry in the Alias Palette 80 is added. If a modifier is de-aliased or deleted, the corresponding user count on the Alias Palette 80 is decremented.
g. Messaging Log.
Messaging Log 42 allows authors to "debug" their Projects by maintaining for authors a log of particular messages sent to selected objects during Runtime execution. It displays these messages in messaging log window 420, as illustrated in FIG. 8.
The "Enable Logging" checkbox 424 displays the path of messages that have been passed to selected objects during a Runtime execution, such objects having been selected by the author through the structure view window 330. First, one sees a message 425 sent, specifying the sender 423, the message type 422 and the recipient 421. What follows is the chain of messages that arise from message 425. For example, message 426 caused, in particular, a chain to occur with received messages 427. Turn knobs 428a, 428b, 428c, 428d, 428e and 428f allow the corresponding message lines to be revealed or hidden, thereby allowing the author to concentrate on the area of interest. More refined embodiments could easily include other standard debugging features, such as breakpoints, stepped execution and examination of selected data, as are well known in the art.
h. mToon Editor.
An mToon is a continuous series of images compiled by the system into a single file. Any animation files created in a 3D program, or in a 2D animation program which has been saved as or converted to PICT or PICS files, can be imported to the mToon Editor 53 and processed as an mToon. mToons are cel-based and very flexible. In addition to being able to specify the playback rate and duration of an mToon, a selected cel or range of cels in the mToon can be specified for playback. These ranges can be dynamically set during Runtime, opening new creative possibilities to the multimedia author.
For example, the sitting, walking, and running motions of a rabbit can be compiled into a single mToon linked to an Element. During Runtime, Messengers or the Miniscript Modifier can be used to specify which cels of the animation to play back according to predefined conditions. Thus, mToons abstract physical action with frame-by-frame control.
The system maintains a link to the actual picture images that make up the individual frames of the mToon. Thus, if any of these images is updated, as verified by the system through the modification date, the system asks the user whether the mToon should be recreated with the new images. This "hot link" feature is ideal when additions or adjustments are made frequently to individual frames.
Authors can import single or multiple PICS or PICT files into the mToon Editor, and then edit, compress and save them as an mToon. Authors also can open and edit existing mToons in this window. Moreover, authors can define ranges of eels within each animation and name them as a subset of the mToon. In the current embodiment, these cel ranges are a sequential range of integers; however, there is nothing to preclude cel ranges from being completely discontiguous and in any order. Once defined, authors can access these ranges by name via Messengers or Miniscript Modifiers during Runtime.
With reference to FIGS. 9(a)-(d), the operation of mToon editor 53 is shown. In FIG. 9(a), mToon editor window 900 is shown, having a registration point 901 which establishes the upper left corner of an inserted animation cel, a cel number field 902 which shows the number of the current cel, and a selection field 903 which displays the range of cels selected for editing. It has an mToon controller 904, which is used to preview the animation (which may assist the author in selecting a range of cels for editing), and has step buttons to step forward or backwards through the animation, cel by cel. Finally "show" pop-up menu 905 allows the author to select a specific range of cels, referenced by name to be played.
In FIG. 9(b), the mToon source file information dialog 910 is shown. Thus, the file path 911 is displayed, along with the cel numbers 912, the original file names for the cel contents 913, the index number 914, if any, when the cel is a PICS file, and the file format 915 (e.g. PICT, PICS).
In FIG. 9(c), the compression settings dialog 920 is shown. The compression type pop-up menu 921 specifies the compression method to be used (e.g., none, Animation, Cinepak, Graphics, Photo-JPEG, or Video). The color depth pop-up menu 922 sets the number of colors used by an mToon, which defaults to 256. A "random access" checkbox 923 provides authors with random access to individual cels despite compression. This is necessary for an mToon to be played backwards or forwards, cel by cel, or at a constant speed. The compression quality slider 926 controls the quality of the chosen compression method, with lower quality imparting artifacts into the mToon.
In FIG. 9(d), the ranges dialog 930 is shown. It permits the naming of sequences of cels, so that they may be accessed by name, as for example, by the Miniscript Modifier. The name field 931 shows the name assigned to the range; the start field 932 shows the cel number at the start of the range; the end field 933 shows the end; the delete button 934 allows a selected (highlighted) range to be deleted; and the "new" button allows a new range to be created.
i. Modifier Palettes.
With further reference to FIG. 2, modifier palettes 35a and 35b are shown. As discussed below, the system's extensible architecture enables programmers to create and insert additional Modifiers into the system, possibly necessitating additional modifier palettes. Each entry in a modifier palette allows an instance of a Modifier to be created and incorporated within (i.e., linked to the instance of a particular Element (Element, Scene, etc.) through a simple click, drag and drop procedure.
Modifier palettes 35a and 35b include (as discussed in greater detail below) icons for the behavior 340, the miniscript Modifier 1980, the MESSENGERS ›(basic) 1200, if messenger 1220, timer 1240, border detection 1260, collision 1280, hypertext 1300 (at bottom right) and keyboard 1320!; the VARIABLES ›integer 1000, integer range 1020, vector 1040, boolean 1060, string 1080, floating point 1100, point 1120 and time 1140!; the SCENE MODIFIERS ›change Scene 1400, return 1420 and Scene transition 1440!; the MOTION MODIFIERS ›simple 1600, drag 1620, path 1640 and vector 1660!; the VISUAL MODIFIERS ›graphic 1700, color table 1720, gradient 1740 and image effect 1760!; the SOUND MODIFIERS ›sound effect 1800 and audio fade 1820!; the style modifier 1900, and the cursor modifier 1920. The classification Modifier 1940 (FIG. 22) and set value Modifier 1960 (FIG. 23) are not shown.
j. Drag and Drop.
As will be discussed below, files containing code for Components (Modifiers and Services) may be added freely to the system by dragging and dropping them inside a special resource folder just as easily as instantiated Modifiers are dragged and dropped into Projects. As discussed herein, Elements and Modifiers may be freely dragged from window to window within a Project, or into libraries or other Projects.
5. Object Messaging.
The object messaging mechanism is integrated into the system so as to be accessible to authors via the configuration of Modifiers. Authors can use this mechanism to specify the manner in which their Elements and Modifiers will communicate with one another.
As is discussed in greater detail in the next section, authors can configure their Modifiers in Edit Mode to perform actions in response to messages received during Runtime Mode. These messages are created and sent either by the system itseft or by a special type of Modifier known as a "Messenger" (i.e., any Modifier capable of sending a message).
For example, during Edit Mode, an author could configure the simple Messenger Modifier 1200 depicted in FIG. 13(a) to respond to a "Mouse Up" message 1203. This message typically would be sent by the system during Runtime Mode when the user clicks on the Element containing this Messenger Modifier 1200. By configuring pop-up menu 1204 with a "Hide" message (a type of message known as a |
