Progressive adaptive time stamp resolution in multimedia authoring6976208Abstract Environments with unreliable delivery may result in faltering presentation of multimedia objects, due to missing time stamp deadlines. This may be alleviated by introducing more flexible time stamping. To avoid this, additional MPEG-4 object time information is sent to the client. This requires a new dedicated descriptor, carried in the Elementary Stream Descriptor. The new more flexible timing information will have two features. First, instead of fixed start and end times, the duration of an object can be given a range. And second, the start and end times are made relative to other multimedia object start and end times. This information can then be used by the client to adapt the timing of the ongoing presentation to the environment, while having more room to stay within the presentation author's intent and expectations. Claims 1. A computer-implemented method of progressive time stamp resolution in a multimedia presentation, comprising the steps of: Description BACKGROUND OF THE INVENTION
In the third step, the bounds on the durations of each object n are recalculated by using:
In the fourth step, the preferred duration of each object n is recalculated: end if In the sixth step, the general error criterion for resolving the duration of each multimedia object is defined as:
Finally, in the sixth step, the corresponding duration of multimedia object n is calculated with: BRIEF DESCRIPTION OF THE DRAWINGS The foregoing and other objects, aspects and advantages will be better understood from the following detailed description of a preferred embodiment of the invention with reference to the drawings, in which: FIG. 1 is a block diagram of one preferred computer system with multimedia inputs and outputs that uses the method of the present invention; FIG. 2 is a temporal diagram illustrating the problem solved by the present invention; FIG. 3 is a flow diagram showing the logic of the overall process according to the invention; FIG. 4 is a flow diagram showing the logic of the process for calculating the minimum and maximum times in block 302 of FIG. 3; FIG. 5 is a flow diagram showing the logic of the process for calculating tx in block 303 in FIG. 3; and FIG. 6 is a flow diagram showing the logic of the process for calculating the durations of the objects in block 304 of FIG. 3. DETAILED DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTION Referring now to the drawings, and more particularly to FIG. 1, there is shown in block diagram form a computer system 100 on which the subject invention may be practiced. The computer system 100 includes a personal computer (PC) 105 running a windowing operating system and including a multimedia audio/video capture adaptor 110. A video camera 122 connects to the adaptor 110 as does an optional playback monitor 124 for multimedia presentations composed on the computer system 100. Other multimedia hardware 130 may be included as well as various input devices, such a keyboard (not shown), a cursor pointing device (e.g., a mouse) (not shown) and a microphone 132 or other audio input device, and a monitor 134 on which a graphic user interface (GUI) of the operating system and application software is displayed. The computer 105 includes secondary memory storage (e.g., a hard drive) 140 of adequate capacity to store the multimedia presentation being authored. The solution to the problem outlined above is best illustrated by a simple example. Let us consider a presentation that is authored having three multimedia objects, a video clip (V), an audio clip (A), and a background image (B). As explained above, the Isis authoring system requires the author to specify for each multimedia object the duration range, as well as a relative start and end time. For the three objects in our exemplary presentation, the parameters are authored as:
The labels P1, P2, and P3 are to indicate how the various multimedia objects are temporarily related. This means, for example, that objects V and B start at the same time. The temporal aspect of this authored presentation can be depicted more clearly in FIG. 2. As shown in FIG. 2, the background image B starts a point P1 and ends at a point P3. The duration times are shown in brackets as 7,7,8 corresponding to 7 seconds minimum duration, 7 seconds preferred duration, and 8 seconds maximum duration. Similarly, the video clip V begins at the point P1 and ends at a point P2, and the audio clip A begins at the point P2 and ends at the point P3, again with duration times shown in the brackets. The player (the client) of the multimedia presentation first receives the multimedia object parameters for video clip V and background B. The player then initializes the time of point P1 (arbitrarily) to t1=0, and starts playing the two objects V and B with their preferred duration. For the video clip V, this means it will be played at the corresponding preferred speed. If no network or playback delays occurred, the video will finish after four seconds. However, if a delay of 12 second occurred during playback, the time of point P2 is not t2=4, but t2=4.5. The player next attempts to resolve the durations of B and A. It does this using the relations: Knowing that t1=0 and t2=4.5, we obtain: Which is combined into: With this we can recalculate the duration range for both the background B and audio clip A. Using: we get We next use these new duration ranges to redefine the preferred durations of both audio clip A and background B. For background B, we see that the preferred duration cannot be met, and we have to settle for the closest value to the preferred value, which is now 7.5 seconds. Similarly, the preferred duration for the object audio clip A changes to 3.5 seconds: Finally, we can use these now feasible preferred durations to determine a good value for the time t3 at point P3, and thus for the durations of the objects B and A. We do this by defining an error criterion on the durations as the sum of the squared deviations from the (updated) preferred durations: Using the definitions of the durations from above, and the recalculated preferred durations, this is rewritten into: Minimizing this error with respect to t3 simply yields: and the durations are From this example, it will be understood that the solution to the problem consists of two parts. First, it is defined what information must be available to the client in order to be able to determine the multimedia object durations. And second, the resolution of the durations themselves must be solved. A client (i.e., player of the multimedia presentation) must receive for each multimedia object five items of information. These items are the two labels, one for the object's start time and one for the end time, and the three durations, the minimum, maximum, and the preferred duration. In the case of video, audio, and other multimedia objects that have a playback speed, the preferred duration must correspond to the "regular" playback speed of the object. The information on a particular multimedia object must be delivered to the client prior to starting playback of the object. When playback has finished for a particular multimedia object, the absolute time of a certain label will become known. This means, that one or more label times can be resolved using this new information. The time stamp resolution is therefore progressive over time, as more information becomes available in the form of factual multimedia object durations, and arrival of information of objects that are to be played in the (near) future. To resolve the actual label time, and the corresponding duration of the multimedia objects that have that label for their respective end times, the following steps are taken: Here tn is the start time of object n, and N is the number of objects.
The entire process of steps 1 through 6 is summarized as illustrated in FIG. 3. The inputs to the process as in step 1, supra, are shown at block 301. Step 2 calculates the minimum and maximum end times over all multimedia objects in function block 302. This is described in more detail in the description of FIG. 4, infra. Next, the steps 3, 4 and 5 are combined in function block 303. This is described in more detail in the description of FIG. 5, infra. Finally, the durations of the objects are calculated in function block 304, which is described in more detail in the description of FIG. 6, infra. Step 2 (i.e., block 302 of FIG. 3) is illustrated more detail in FIG. 4. The process is initialized in function block 401 before entering the processing loop. The value of n is incremented by one in function block 402 at the beginning of the processing loop. A test is made in decision block 403 to determine if the minimum end time is less than the start time of object n plus the minimum duration of object n. If so, the minimum time is set to that value in function block 404. If not, a test is made in decision block 405 to determine if the maximum end time is greater than the start time of object n plus its maximum duration. If so, the maximum time is set to that value in function block 406. Finally, a test is made in decision block 407 to determine if all objects have been processed and, if not, the process loops back to function block 402 where the value of n is again incremented, and the maximum and minimum times for the next multimedia object are calculated. This processing continues until the minimum and maximum end times over all N multimedia objects have been calculated. Steps 3, 4 and 5 (i.e., block 303 in FIG. 3) are illustrated in more detail in FIG. 5. The process is initialized in function block 501 before entering the processing loop. The value of n is incremented by one in function block 502 at the beginning of the processing loop. A test is made in decision block 503 to determine if the preferred duration is greater than the minimum end time less the start time of a current object n. If not, the preferred duration is set to this value in function block 504; otherwise, a further test is made in decision block 505 to determine if the preferred duration is less than the maximum end time less the start time of the current object n. If not, the preferred duration is set to this value in function block 506; otherwise, the preferred duration is set to the preferred duration of the object n in function block 507. Then, in function block 508, the sum of the times is calculated. A test is made in decision block 509 to determine if all objects have been processed and, if not, the process loops back to function block 502 where the value of n is again incremented. When all objects have been processed, the time tx is computed as the sum divided by N, the number of the multimedia objects, in function block 510. Step 6 (i.e., block 304 in FIG. 3) is shown in more detail in FIG. 6. The process begins by initializing n to zero in function block 601. The value of n is incremented by one in function block 602 at the beginning of the processing loop. The duration of each object n is calculated in function block 603 as the calculated time tx minus the start time t(n) of the object n. After each calculation, a test is made in decision block 604 to determine if all objects have been processed. If not, the process loops back to function block 602 where n is again incremented and the duration of the next object is calculated. The process ends when all N objects have been processed. While the invention has been described in terms of a single preferred embodiment, those skilled in the art will recognize that the invention can be practiced with modification within the spirit and scope of the appended claims.
|
Same subclass Same class Consider this |
||||||||||||||||||||||||||||||||||||||||||||||||||