Video cataloger system with hyperlinked output6567980Abstract One aspect of the invention is directed to a system and method for video cataloging. The video is cataloged according to predefined or user definable metadata. The metadata is used to index and then retrieve encoded video. In another aspect of the invention, video metadata track processors convert metadata tracks of the video information to produce displayable frames containing hyperlinking between displayable data. Another aspect of the invention is directed to a method of browsing stored video information, including displaying hyperlinked frames of metadata track representations, and selecting and displaying links between displayable frames. Claims What is claimed is: Description BACKGROUND OF THE INVENTION
TABLE 1
Track Data Types
Metadata Data
Track Type Notes
Virtual Base untyped Defines In-time and Out-time for all
Class (void*) tracks
Keyframe image (bitmap) In-time equals Out-time, i.e., keyframe
Track is a point in time
CC-text Track Text fragment Each text fragment is typically a
sentence (but not required to be so)
and spans a time interval
Audio Class Enumerated Speech, Silence, Music, Applause,
Track Classes Siren, etc..., each spanning a time
interval when that classification was
valid
Speech Track Text fragment Each text fragment spans a time
interval
Keyword Track Word (text) keyword utterance spans a short (1/2
sec) time interval
Speaker ID Enumerated Identifiers of individuals whose
Track Classes speech is recognized... each Speaker
ID spans a time interval when that
speaker was speaking
Clip Track Label Set (user Different Label Set schemas can be
defined set of used in different applications. Each
labels): Text, Label Set is applied to all clips within
Enums, Dates, a Cataloging session. The Clip
Numbers, etc. definition spans a time interval marked
by the user. Each Label field value is
entered manually by the user.
Custom Data type Typically, a custom metadata
defined by generator uses a custom track data
plug-in type for storing its metadata. It could
also re-use existing track data types
such as Text Fragment.
Table 1 is a summary of the various standard metadata tracks, detailing the data types of each, and providing descriptive notes. 8. Video Cataloger--Architecture FIG. 8 is a global architecture illustration of the entire Video Cataloger software process 420. The main components of this software are the Media Capture Services 430, the Video Encoding and Synchronization facility 450, the Start-up Extensibility Initialization manager 470, and the core Extensible Video Engine component 440. The details of the core Extensible Video Engine 440 are provided in FIG. 9. The Video Encoding and Synchronization module 450 is responsible for communicating with the "Vidsync" daemon processes running on the video encoders, e.g., 123, 125 and 127 (FIG. 4). The Media Capture Services 430 are further described in conjunction with FIG. 9. The registration interfaces for the extensible aspects of the Extensible Video Engine 440 are explicitly shown in FIG. 8. Upon start-up of the Video Cataloger 110, registration processes are invoked for the four primary extensibility aspects of the Video Cataloger: Metadata track registration 476, Feature Extractor registration 472, Output Filter registration 478, and Event registration 472. A set of output filters 484 are installed during system start-up. These registration processes, as well as user input and output functions 550, 554, are further described in conjunction with FIG. 11 below. 9. Extensible Video Engine--Architecture FIG. 9 depicts the main architectural elements of the extensible Video Engine 440. Incoming media is processed by the Media Capture Services 430 consisting of Timecode Capture 502, Video Capture 504, Audio Capture 506, and Text Capture 508. Digital media 509 is then made available to the Feature Extractor Framework 510 for processing. Metadata from the Feature Extractors 512, 514, 516, 518, 520, 522 is then committed to the Metadata Track Index Manager 530 in a time based track representation as shown in FIGS. 6 and 7. During metadata capture, the user may mark video clips and annotate them. This input 552 is captured by the GUI Input Capture element 550. Event monitoring 540 and dispatch 544 also occurs during capture, driven by an Event Dictionary 542. Finally, when capture is complete, the metadata may be output in a variety of formats such as Virage Data Format (VDF) 562, HTML 564, XML 566, SMIL 568 and other 570, which are managed by the Output Filter Manager 560. A VDF API and Toolkit may be licensed from Virage of San Mateo, Calif. Furthermore, the use of the format is described in "Virage VDF Toolkit Programmer's Reference". One reference for the eXtensible Mark-up Language (XML) is the following URL: http://www.w3.org/TR/REC-xml which is a subpage for the W3C. Also, information on Synchronized Multimedia Integration Language (SMIL) may be accessed at the W3C site. The Metadata track Index Manager 530 represents the object that manages the multiplicity of metadata tracks. When data is committed to the track index by either a feature extractor 512-522 or GUI input 550 and 552 (i.e., user marks clips and annotates them), this can trigger display updates as follows: the particular metadata track that receives the data decides if this requires a display update. If so, it sends a message to the GUI Display Update Manager 554 which marks the relevant GUI object as "dirty" and in need of a redraw. In Windows Microsoft Foundation Classes (MFC), the event model allows Windows to detect these dirty GUI objects and issue redraw messages to them directly (see FIG. 12--Get Event) The core aspects of extensibility are: Extensible Track data types are registered with the Metadata Track Index Manager 530. Any desired data representation can be defined and installed, such as region markers, OCR text and confidence values, face identifiers, camera parameters (pan, tilt, zoom), etc. Any property that a feature extractor chooses to extract can be placed in a custom metadata track. Extensible Feature Extractors can be registered with the Feature Extractor Framework 510 to operate on digital media, or on any collateral data they may choose to collect when called. Extensible Event triggers: event criteria (e.g., cc-text == "clinton", or audio_class =="tone") can be registered in the Event Dictionary 542, and arbitrary actions can be registered and triggered (e.g., grab a keyframe right then, or stop capture). The Event Monitor 540 monitors the incoming metadata to decide if an event is triggered. If so, it sends a message to the Event Dispatcher 544 which invokes the corresponding action 546 for the event. Extensible Output Filters may be registered with the Output Filter Manager 560. Further discussion of Output Filters is provided below with respect to FIGS. 15 and 16. Time code capture 502 is typically via VLAN (as in FIG. 3), but may come from a variety of sources. Time code capture is another aspect of extensibility (though not core) since we have a plug-in for time-code extraction 10. Audio Feature Extractors FIG. 10 depicts the architectural components of the audio analysis feature extractors 516 in one embodiment of the Video Engine 440. As can be seen in the diagram, there are various cross-couplings between these feature extractors, which may not be precluded in the extensibility mechanisms managed by the feature extractor framework 510 (FIG. 9). The analog audio signal 592 is captured and digitized by audio digitization device 506, which may be any standard audio digitization device, such as a Sound Blaster audio card for a PC. The digital signal is then normalized by a software component 596 to account for variability in signal amplitude (volume). The normalized digital audio signal 598 is then fed into an Audio Class Profiler 600 which classifies the signal into one of several possible categories, such as "speech", "music", "silence", "applause", etc., where each of the categories may be trainable using well understood techniques, and is stored in a Class Dictionary 602. An Audio Classification (AC) Engine 604 is a modular component that is available from multiple vendors, or may be proprietary. One skilled in the relevant technology may evaluate and utilize a specific engine depending on the application requirements. When the Audio Class Profiler 600 detects that the class is "speech", it triggers switch 610 which then allows the normalized digital audio signal 598 to pass into additional feature extractors which are capable of processing speech. A speech transcription module 620 is designed to interface with any available Speech Recognition Engine 624 using an industry standard interface 626, such as the "Speech API", or SAPI defined by Microsoft. Typically, the Speech Recognition Engine 624 utilizes a Vocabulary Dictionary 622 to aid in the speech recognition process and improve accuracy by limiting the speech domain, although this is not required. It is a typical feature of existing speech recognition engines available on the market today. Examples include offerings from IBM, BBN, Dragon Systems, SRI, and so on. The output of the Speech Transcription Feature Extractor 620 may then be further processed as follows: the full text 628 of the transcription process may be used directly as metadata; additionally, a Keyword Spotting Feature Extractor 640 may be employed to selectively identify keywords of interest, and produce a text output 648 limited to the keywords specified by a Domain Dictionary 642. A Domain Dictionary Engine 644 is responsible for making these selections. Again, the Domain Dictionary 644 Engine is typically a modular component that may be one of several available, interfacing with the Keyword Feature Extractor normally via a standard interface 646 such as the Domain Dictionary API, or DDAPI. The normalized digital audio signal containing speech can also be fed into a Speaker ID Feature Extractor 630 to identify individual speakers by name. A Speaker ID Engine 634 may also be a modular component that is offered by several speech recognition vendors, and interfaces with the Speaker ID Feature Extractor 630 typically via an industry standard interface 636 such as the SVAPI. Typically, the Speaker ID Engine utilizes a Speaker Dictionary 632 to constrain the space of possible speakers, and store signatures or sample speech of individual speakers which are used during speaker identification. 11. Extensible Video Engine Start-up Initialization--Flowchart FIG. 11 is the process flowchart for the start-up initialization of the Video Cataloger 110 (FIG. 1). This flowchart depicts the process for registering data types, algorithms, and events which are important to the extensibility features of the Video Cataloger 110. Upon start-up of the Video Cataloger, the extensible video engine initialization process 470 is executed by the workstation 111. Starting at a begin step 702, the process 470 moves to step 704 to install metadata tracks. This occurs first since later extensions (mainly Feature Extractors) may then utilize the, track data types previously installed. Built-in Track Types are installed first at step 704, followed by installation of custom track types defined by plug-in modules at steps 706 to 710. For each track plug-in, the data representation defined by that plug-in is installed at step 708. Next, feature extractors are installed. The built-in feature extractors are first installed at step 714, followed by feature extractors defined by plug-ins at steps 716 to 722. For each plug-in feature extractor, it is first registered at step 718 with the Feature Extraction Framework 510 (FIG. 9). At step 720, each of these plug-in feature extractors may request a metadata track type to receive its metadata. Following the feature extractor initialization, the Output Filters are initialized. As with the other elements, the built-in Output Filters are, installed first at step 724, followed by the installation of plug-in Output Features at steps 726 to 730. Finally, Events are registered. All events are application specific (i.e., there are no built-in events), and are registered by plug-ins starting at steps 734 to 740. Each plug-in may define one or more events in the dictionary at step 736, and each event will have an associated event handler registered with it at step 738. The extensibility initialization process 470 completes at an end step 742. 12. Video Encoding/Synchro--Flowchart FIG. 12 details an important aspect of the present invention, which is the control and synchronization of the video encoding process with the metadata capture process. This synchronization is necessary because time-code indices within the metadata elements should correspond to correct and known points within the digital video that results from the encoding process. When video capture is initiated by the user, the video encoding process 450 starts at a begin step 762 and moves to step 764 wherein the Video Cataloger 110 (FIG. 1) first issues a Start Encoding command to each of N video encoders in parallel by spawning process threads 766 for each encoder present. A process thread or a lightweight process is well understood by computer technologists. This command/control is effected by the "Vidsync" daemon process 260 (FIG. 4) running on each encoder station. These Start commands are issued in parallel so that all the encoders begin encoding as close together in time as possible. However, their exact start times will not in general, be coincident. For this reason, the Vidsync process 260 returns the actual start times to the encoder flow control, and these times are stored by the Video Cataloger 110 with the video metadata in step 774 for later use. Next, the general process of capturing metadata occurs in step 776 until the process is stopped by the user. The details of the metadata capture process 776 are provided in FIG. 13. When capture is done, Stop Encoding commands are sent in parallel to each encoder (via Vidsync) by spawning process threads 780. It is of no consequence that the N encoders may stop encoding at slightly different times, as no metadata is associated with these time intervals. 13. Capture Metadata--Flowchart FIG. 13 details the metadata capture process 776 which is an important activity of the Video Engine 440 of FIG. 9. The metadata capture process 776 was first introduced in FIG. 12. The capture process 776 begins with the scheduling of a system timer event in step 804 set to go off 1/30 of a second in the future. The control flow of the process 776 immediately proceeds to the Get Event step 806 where other system events (besides the timer event) may be processed. When an event occurs, control passes to the Event Dispatcher 808 which decides if the event is one of the two types of events: a normal GUI event, or the scheduled timer event. For a GUI event, the event is first inspected in step 812 to determine if it is an End Capture event, in which case the capture process loop terminates. If not, processing proceeds to step 816 to handle the GUI event (such as keystroke, window resized, etc.). Some GUI events may generate metadata (if the user marked a video clip), which is determined in step 818. If metadata (a video clip) was in fact generated, that metadata is committed to the Metadata Track Index Manager 530 (FIG. 9) during step 820. This also necessitates a GUI redraw, so the affected parts of the GUI are marked for Redraw in step 822. If the event dispatched in 808 is the timer event., this signifies that feature extraction of metadata from the video signals is to take place at a feature extraction process 810. The details of the feature extraction process 810 are provided in conjunction with FIG. 14. Once feature extraction is complete, control moves to step 804 where the next timer event is scheduled. This flow of activity is tied to the event model of the operating system under which the software application is running. The flow that is shown is an event model that is typical of a Windows MFC-based application. Other operating system platforms, such as Unix, have event models that differ somewhat. The event model illustrates how the feature extraction process fits into an application event framework. Note that, in the depicted embodiment, the Get Event task 806 is a call out to Windows MFC, which processes Redraw Events by calling the Redraw method of the appropriate GUI elements directly (this process diagram does not "call" the Redraw methods directly). Note that it is acceptable if feature extraction takes more than 1/30 second. 14. Feature Extraction--Flowchart FIG. 14 details the feature extraction process 810, which is an important aspect of the present invention, relying on the innovative architecture of FIG. 9. The feature extraction process 810 begins at a start step 842 and proceeds to step 844 where the current time code is obtained by module 502 of FIG. 9. This time code is used by all feature extractors to time-stamp the metadata they extract. Next, all digital media is captured in step 846 by modules 504, 506, and 508 of FIG. 9. This digital media is then passed on to the Feature Extractor Framework 510 (FIG. 9) for processing. The Feature Extractor Framework 510 spawns a process thread 850 for each feature extractor. Each feature extractor processes the digital media in step 852 in whatever way it desires, for example, extract a keyframe, classify the audio signal, etc. In certain cases, but not all, some metadata will be generated from this process. Step 854 determines if this is the case, and if so, the metadata is passed to the Metadata Track Index Manager 530 (FIG. 9) during step 856. Since metadata is usually displayed in real-time in the GUI, the GUI is marked for redraw in step 858. One particular exemplary feature: extractor for video keyframes is described in the pending U.S. patent application entitled "Key Frame Selection" filed on Jun. 6, 1997. When all feature extractor threads complete, as determined at wait (synchronization) step 862, control is returned to the capture metadata process at end step 864. 15. HTML Output Filter--Architecture The Output Filter Manager 560 (FIG. 8) may utilize a HTML output filter 564 in one embodiment. Referring to FIG. 15, elements of FIGS. 1, 2 and 9 are shown together as utilized in generating HTML output. The user may invoke a GUI command such as the "Save-As" command on the "File" menu 553, which in turn provides a list of output filter choices (HTML, Real Networks SMIL, XML, custom, etc.). When the HTML filter 564 is invoked, it accesses the metadata in the Metadata Track Index Manager 530 and processes it into HTML form in a browser window 916 (FIG. 17), which also involves keyframe images in a keyframe frame 176 (FIG. 2) or 904 (FIG. 17), and the digital video 142 (FIG. 1) or as seen in a video frame 896 (FIG. 17). For instance, hyperlinks may be formed from displayed keyframes to video sequences. The digital video 142 may or may not be served by a content server 140. For instance, it could be a simple file on the file system of the client computer or, say, a networked mass storage device visible to the computer. Some key features of the Video Cataloger HTML output are: a. The HTML files used to generate the display in the browser window 916 (FIG. 17) are completely stand-alone, internally linked HTML, such that no Web server is required. Exemplary HTML files are provided in the Appendix and are described in conjunction with FIG. 17 below. b. It incorporates play-back of digital video 142 from a file or from a video server 140. That is, the digital video may be streamed directly to the browser, or it may simply be played from a local file on disk. The stand-alone aspect is strengthened when the digital video is a local file. This way, all of the content (HTML, keyframes, digital video) could be packaged up, compressed, and e-mailed to someone. c. All metadata is cross-referenced/cross-linked based on time-codes. d. Digital video is independent of the HTML representation--any digital video source can be linked into the playback frame. 16. HTML Output Filter--Flowchart FIG. 16 details a HTML export process 890 from the Video Cataloger. This process 890 is performed by module 564 identified in FIGS. 9 and 15. The output process 890 starts at a begin step 892 and proceeds to step 894 to process the session level metadata. This metadata is not time-based, but rather is descriptive of the entire logging session. The session level metadata corresponds to the information 404 generated by the Metadata Track Index Manager 402 shown in FIG. 7. The nature of the session level metadata is a schema which may be defined by the user, in addition to standard items such as the location where the video is taken. This information is encapsulated in an HTML frame 896 used to view this data on request, and is linked to the main HTML frame 916. The next step is to process the keyframe track in step 898. Keyframe images, which are captured raster images, may be converted to JPEG images suitable for display in a web browser. JPEG is but one possible viewable format. For convenience, the JPEG image files 900 may be stored in a separate subdirectory of the Cataloger file system. At step 902, the keyframe track is then further processed by constructing an HTML keyframe frame containing the keyframe time code information used to invoke video playback in 896, and establishes hyperlinks directly to the corresponding JPEG images 900. Next, the close caption text track is processed in step 906. The cc-text is output into an HTML frame, with hyperlinks created from time-codes into the keyframes of the HTML keyframe frame 904. This allows the user to click on cc-text elements, and invoke the corresponding set of related keyframes. Video Clips are processed in step 910. The clips (defined by in- and out-times, and a user defined set of text labels) are output into an HTML Clip frame 912. The time codes are used to establish hyperlinks into the corresponding close caption text 908, and the corresponding keyframes in keyframe frame 904. Finally, a main HTML page that incorporates the above frames is constructed in step 914. This HTML page embeds all the other frames for display and navigation. A video play-out helper application to decode and display video can be embedded in the web page frame. Examples of helper applications include RealPlayer (for RealVideo), Compcore SoftPEG (for MPEG) and Apple Quicktime. Exemplary reference guides which could be uselful to write the code to automatically generate HTML are HTML: The Definitive Guide, The second Edition (1997) Chuck Musciano and Bill Kennedy , O'Reilly & Associates, Inc. and "Treat Yourself Web Publishing with HTML", Laura LeMay, Sams Publishing, 1995, which are hereby incorporated by reference. Note that this process flow is one example which incorporates a subset of all available metadata tracks. The output process 890 described above generated the exemplary screen shot in FIG. 17. 17. Example HTML Output--Screen Shot Referring to FIGS. 16 and 17, a screen shot of the HTML output as seen at a client browser and as generated by the HTML output process 890 (FIG. 16) will be described. Element 896 corresponds to the video frame in the upper left portion of the screen display. Element 904 corresponds to the keyframe frame in the lower left portion of the screen display. Element 908 corresponds to the cc-text frame in the lower right portion of the screen display. Element 912 corresponds to the clip frame in the upper right portion of the screen display. Element 916 corresponds to the whole browser window. As with most browsers, including Microsoft Explorer and Netscape Navigator, if the displayable page is larger than the physical display, the browser will cause the page to be scrolled. Video data is retrieved by sending a time code to the embedded player application. The player application then retrieves the video, seeks to the requested time code (in-time), and begins playback. The user can interrupt the playback using standard VCR type controls on the player. The HTML code for an exemplary screen display is provided in the Appendix. Sheets A, B, C, D, E, G, H1, N and P are hereby incorporated by reference in their entirety and can be found in the USPTO file for application Ser. No. 09/134,499. Sheet A of the Appendix lists the directory names (clip and icons) and file names at a top level. Sheet B lists the files in the clip directory, while sheets C, D and E list the files in the icons directory. Sheet F lists the HTML code for the top level index.html file which provides the framework for the display shown in the browser window 916 (FIG. 17). Sheet G lists the contents of the topr.html file (as would be seen in the clip frame 912 (FIG. 17)). Sheet H lists the contents of the video_label.html file. Sheet I lists the contents of the video_mbase.html file. Sheet J lists the contents of the video_netshow.html file. Sheet K lists the contents of the video_noproxy.html file. Sheet L lists the contents of the video_ovs.html file. Sheet M lists the contents of the video_real.html file. Sheets J, K, L, and M may be used to provide the proxy video to allow different video formats to be displayed in the video frame 896 (FIG. 17). Sheet N lists the contents, including a set of keyframes and corresponding timecodes (as would be seen in the keyframe frame 904 (FIG. 17)), of the 000l.html file in the clips directory. Sheet P lists the contents, including a set of icons in a closed-caption text frame (as would be seen in the cc-text frame 908 (FIG. 17)), of the 000r.html file in the clips directory. The remaining sheets in the Appendix are alternate instances of the contents shown in exemplary sheets N and P. Of course, other programming languages besides HTML code could be used to implement hyperlinked output conversion. 18. Alternative System An alternate embodiment 940 of the video encoding process, which involves a video server 942, is shown in FIG. 18. In this scenario, digital video is encoded in a MPEG stream on the Cataloger workstation 111. The data stream is broadcast as a set of UDPs (Universal Datagram Packets) 946 on a specific port number (configurable). UDPs is a standard which is a member of the IP family of protocols. When cataloging begins, the Video Cataloger 110 sends a START command 944 to a Vidsync process 260 which is running on the content server 140 where the video server software process 942 is running. Vidsync 260 in turn tells the video server 942 to "start listening" for UDP packets 946 on the specific port number. The video server 942 then begins "catching" the UDP packets 946, and converting the MPEG data into a digital video asset on that server 942. As always, metadata 112 is sent from the Video Cataloger 110 to the metadata server 130 in parallel to this encoding process. When a STOP command 944' is issued, Vidsync 260 signals the video server 942 to stop listening for the UDP packets 946. In point of fact, the allocations of support hardware, computer workstations and software processes are only described here as but one example. Many other functional partitions can be defined to implement the present invention. While the above detailed description has shown, described, and pointed out the fundamental novel features of the invention as applied to various embodiments, it will be understood that various omissions and substitutions and changes in the form and details of the system illustrated may be made by those skilled in the art, without departing from the concepts of the invention.
|
Same subclass Same class Consider this |
||||||||||