Methods and systems for providing random access to structured media content6807550Abstract Methods and systems for providing random access of structured media content are described. One aspect provides a novel file structure or "Vfile", and method of using the same, for organizing and managing portions of a structured media content file that are or can be downloaded from a content provider such as a multimedia content server. The Vfile is configured to emulate a server file structure that a server uses to store structured media content files. The Vfile is established at a client location when a client accesses a structured media content file and is used to buffer and cache structured media content file portions. In addition to having an infrastructure that emulates the server's file structure, the Vfile includes one or more media management tags. The tags are used by the client to manage a cache of structured media content file portions that are downloaded from the server. Exemplary tags include, but are not limited to (1) a validity tag that indicates whether a particular portion of a structured media content file is available in the Vfile cache, (2) a hit count tag that gives an indication of how many times a particular portion has been accessed, (3) an importance tag that gives a weight to the hit count and determines how long the portion can stay in cache, and (4) a lock tag that indicates whether or not to lock selected structured media content file portions in the Vfile. Cache management strategies make use of the management tags to ensure that relevant portions of the media file are maintained by the client for as long as they are needed. Those file portions that are needed, but which are not present in the Vfile, can be streamed to the client by the server while those file portions that are present are used to render a view for the user. Claims What is claimed is: Description TECHNICAL FIELD
Unit_update ()
{
while (memory is not enough to hold the new media unit
received from the server)
{
find the media unit which is unlocked and has the smallest
value of (importance + hit count);
remove media unit from Vfile;
store the new media unit in Vfile
In one exemplary implementation, a client application, i.e. browser, implements software code that renders the current view over and over again using the file portions that are present in the Vfile. A user may interact at any time with the browser to change the region of interest and viewing resolution (for a JPEG 2000 browser), or to change the viewing position, angle, and FOV (for an IBR browser). In any instance, the browser renders the view based upon the content available in the Vfile. An important feature of the invention is that information that is not present in the Vfile, but which is necessary for the currently rendered view, is supplemented while the view is being rendered. Supplementation takes place by calling the server so that the additional file portions can be downloaded to the client and stored in the Vfile. Supplementing the Vfile can take place in any suitable manner. As an example, techniques used in error concealment can be used to supplement information that is not present in the Vfile. Exemplary techniques are described in a document authored by Wenjun Zeng and Bede Liu entitled "Rate Shaping by Block Dropping for Transmission of MPEG-Precoded Video over Channels of Dynamic Bandwidth", ACM Multimedia 1996: pps. 385-393. One exemplary approach to implementing this functionality is described in the pseudo code set forth immediately below. Of course, this represents but one exemplary approach and is not intended to limit the invention in any way.
Create_Vfile();
Obtain_global_header();
for (;;)
{
update_view_point() //For JPEG 2000, it is the resolution
and position of the viewing window. For IBR, it is the updated
viewing position, angle, and camera FOV.//
if (new_view)
stop Vfile update;
clear all temporary locks; clear decode list;
for (i=1; i<L;i++)
if (structure s.sub.i is useful in rendering current view)
{
temporarily lock structure s.sub.i ;
if (query_Vfile: is structure s.sub.i valid?)
for (j=1; j<n; j++)
if (media unit b.sub.j is useful in rendering
current view)
if (query_Vfile: is media unit b.sub.j valid?)
temporarily lock unit b.sub.j ; add b.sub.j to the
decode list;
update Vfile;
decode all media units within the decode list, release
temporary lock during decoding;
render current view;
}
In the example, the center of a Vfile-enabled media browser is a loop which renders the current view over and over again. Cache misses that are generated by the loop indicate that certain portions of a structured media content file are needed for rendering. Accordingly, when this happens, software code executing on the client knows that it must send a request to the server to obtain the missing file portions. The Vfile enhances the browsing experience because information that is streamed to the client over the network is reused. As an exemplary implementation using the media units and sub media units of FIG. 4, consider the following: When an application call is made to access a particular media segment, the call is broken down into access calls of multiple SMUs which cover the entire media segment. The validity of the SMU is first checked, and a continuous header portion of available SMUs are returned to the calling application. For unavailable SMUs, a pending network request is generated to stream the SMU from the server. SMUs are stored within the cache space of its corresponding MU, which is allocated from the system memory. Any time an SMU is accessed, the hit count of the associated MU is increased by one. Whenever the hit count of a MU reaches the maximum value, it is halved together with the hit counts of all the MUs. The priority and importance of the calling media segments is applied to all of the MUs and SMUs affected. If a MU or a SMU is accessed multiple times, its priority and importance are determined by the function call with the highest priority and importance value. The priority and importance of the MUs and SMUs may be temporarily raised through a lock operation, and released through an unlock operation. Such lock and unlock operation may be used when a user shifts the region of interest (ROI) and to prevent the MUs needed in the current region to be swapped out of cache. Exemplary Architecture for Managing Structured Media Content FIG. 7 shows a portion of an exemplary architecture that can be used to manage and oversee random access to structured media content files. The illustrated architecture can be implemented in any suitable hardware, software, firmware, or combination thereof. To get the best browsing experience, all media content should not be treated equally. For example, in JPEG 2000 browsing, low resolution coefficients should be transmitted from the server before high resolution image content. The low resolution file portions should also be maintained in the Vfile as long as possible because they are useful in generating a broader viewing region. In the described embodiment, a cache manager 700 is provided and manages the content of the Vfile. The cache manager ensures that the most pertinent portions of a structured media content file are maintained in the Vfile. Those portions that are not important or likely to be used by a client are removed from the Vfile to make room for other portions that are important or likely to be used. A plurality of modules are provided, in this example, that assist the cache manager in performing its job. In the illustrated and described embodiment, the following modules are provided: an importance evaluator module 704, a priority evaluator module 706, and a preemptive loader module 708. Importance Evaluator Module One role of importance evaluator module 704 is to assign importance values to a plurality of portions of the structured media content file. These importance values are used by the cache manager 700 to determine whether or not to remove individual portions of the structured media content file when, for example, additional portions of a file are streamed from the server to the client. The importance evaluator module can be implemented at the server or the client end. Although, for reductions in the server processing overhead, it is more advantageous to implement the importance evaluator module at the client end. The implementation of the importance evaluator is media specific. That is, as each of the media with which the described embodiment can be utilized varies, so too can the factors that are considered when assigning importance values. An exemplary rule of thumb in determining the importance of file portions was given above, e.g. lower resolution file portions are more important than higher resolution file portions for JPEG 2000 image data. The importance values that are assigned by the importance evaluator module may be implemented at the time the media is encoded and stored with the media bit stream. Alternately, the server or client may calculate the importance values based upon factors such as resolution and quality level of the file portion or media unit. FIG. 8 shows a flow diagram that describes steps in a method in accordance with the described embodiment. In the present example, these steps are implemented by both the importance evaluator module 704 and the cache manager 700. Step 800 assigns individual importance values to a plurality of structured media content file portions. Step 802 stores the structured media content file portions in a client cache structure, i.e. the Vfile. Step 804 uses the individual importance values to determine which stored structured media content file portions in the client cache should be removed to accommodate additional portions of the structured media content file that are obtained from the server. In the described embodiment, these steps can be performed while randomly accessing structured media content file portions located on a plurality of different computers such as the client and server computers discussed above. Priority Evaluator Module One role of the priority evaluator module 706 is to assign priority values to a plurality of structured media content file portions. The priority values are used to determine an order in which the portions are to be provided or streamed from one or more servers to a client. The priority evaluator module 706 can be implemented at the server or the client end. Although, for reductions in the server processing overhead, it is more advantageous to implement the importance evaluator module at the client end. The priority values are calculated and assigned to the individual file portions in much the same way as the importance values. That is, more important file portions (such as low resolution portions) should be streamed earlier than higher resolution file portions. One goal of the priority evaluator module is to have the most important and necessary file portions sent to the client first so that the browsing experience can be enhanced. This way, image data is rendered more quickly and efficiently for the user. For example, by having the lower resolution file portions sent first and used in the rendering process, a user can, for example, view a less detailed image while the higher resolution file portions are streamed from the server. FIG. 9 shows a flow diagram that describes steps in a method in accordance with the described embodiment. These steps can be implemented by the priority evaluator module 706 and/or one or more server computers. Step 900 assigns individual priority values to a plurality of structured media content file portions. The assignment of priority values, like the importance values, is media specific. In some instances, priority values are assigned based upon the resolution and/or quality of the structured media content file portions. In other instances other considerations are taken into account. For example, an exemplary alternative approach can be based on visual progressive coding, i.e. assignment of priority values based on a combination of SNR (signal-to-noise ratio) layer and visual weight as described in an article authored by J. Li entitled "Visual progressive coding", SPIE Visual Communication and Image Processing (VCIP'99), Vol. 3653, pp. 1143-1154, San Jose, Calif., January 1999. Step 902 evaluates the individual priority values. In the described embodiment, one or more servers that are responsible for streaming media content to one or more clients evaluate the priority values to ascertain an order in which to stream requested media content. Responsive to the evaluation, step 904 sends at least one structured media content file portion having a higher priority value to a client before sending at least one structured media content file portion having a lower priority value. Thus, the priority evaluator module facilitates provision of random access to structured media content file portions. Preemptive Loader Module One role of the preemptive loader module is to predict what is the most likely media content file portion that a user is going to require or want, based upon one or more factors including the currently provided media content that the user has accessed. Accordingly, by predicting the likely needed file portions, the server can preemptively provide (or be preemptively requested to provide) the file portions to the client, without the client specifically requesting so when the portions are actually needed. Many factors can be considered in determining which file portions to preemptively provide to the client. For example, file portions that correspond to the "neighborhood" of the currently rendered view, i.e. close in proximity, as well as file portions that correspond to higher resolution content of the currently rendered view can be preemptively loaded. One assumption concerning these file portions is that users most probably will pan around their current view, or zoom into the current view. Thus, by anticipating where a user might navigate, file portions can be preemptively loaded to reduce the access time in the event they are needed. This further enhances the user's browsing experience. FIG. 10 shows a flow diagram that describes steps in a method in accordance with the described embodiment. Step 1000 determines one or more structured media content file portions that may likely be requested by a client, but which have not specifically been requested by a client. As indicated above, this step can be implemented by ascertaining a current client view and either identifying file portions that are close in proximity with the portions that correspond to the current view and/or identifying file portions that corresponds to a higher resolution of the current view. Of course, other factors can be considered as well. Step 1002 sends one or more structured media content file portions to the client. The file portions can be sent responsive to a client-side request for the preemptive subject matter, or responsive to a server determination that the particular preemptive subject matter should be sent. By determining which file portions may likely be needed by a user for rendering on a client machine, those portions can be preemptively streamed from the server before they are actually specifically queried for use by the client. It is possible that the client may preemptively request the file portions from the server though. It should be understood, however, that this particular type of client request is not a request associated with a per se use of the file portion, but rather is a request that is associated with a likely (but not guaranteed) use of the file portion. This results in a quicker, more efficient browsing experience. The functionality of the above-discussed modules can be implemented at the server side, the client side, or shared at both ends. However, since the server may store a large number and category of media, and since the server handles requests from tens to hundreds of client simultaneously, it is more advantageous to implement the modules on the client side. Exemplary Network Architecture for Managing Structured Media Content One aspect of the invention provides a layered software architecture that implements a system for randomly accessing structured media content files over a network, such as the Internet. In the described embodiment, the architecture includes an application program interface (i.e. Vfile API) between the media program (i.e. client API) and the network (i.e. network API). By using the Vfile API, the media programmer can access a remote media file as a "virtual local file", and is relieved from the chores of network programming. The Vfile API operates along principles that are similar to the standard file operation API (fopen, fread, etc.). The Vfile API maps the media file at the server side virtually to the client side. The virtual local copy "looks" exactly the same as the remote media file. It varies in that segments or portions of the local file may be not available. FIG. 11 is a diagram that shows an exemplary multi-layered architecture in accordance with the described embodiment. The architecture on the client end is designated at 1100, and the architecture on the server end is designated at 1102. Client architecture 1100 includes a Vfile 1104 and three layered sets of application program interfaces (APIs) that assist in managing and organizing both the client browsing experience and the Vfile 1104. An API is a set of routines (e.g. function or method calls) that an application program uses to request and carry out lower-level services performed by a computer's operating system. In the illustrated example, a client API 1106 is provided and interfaces with a user through, for example, the user's browser. The client API 1106 enables a user, through software executing on their computer, to request structured media content and have it rendered on their computer. In the described embodiment, the client API 1106 continuously monitors the Vfile 1104 and renders content based upon the contents of the Vfile. A Vfile API 1108 is provided and constitutes the middle layer in the client-side architecture. The client API 1106 depends on the Vfile API 1108 to get the content of the Vfile 1104. For example, the client API 1106 uses the Vfile API to perform file operations on the Vfile. For example, the client API 1106 opens and closes files in the Vfile through the Vfile API 1108. The client API 1106 also sends requests for needed portions of the Vfile to the Vfile API 1108. If the needed portions of the Vfile are not stored on the client side, the Vfile API generates a request that is used to gain access to the file portions from the server. The Vfile API 1108 also includes functions that can be used to lock and unlock file portions in the Vfile. A network API 1110 is provided and supports the Vfile API 1108. The network API 1110 accepts requests for file portions from the Vfile API and sends them to a remote server. The network API 1110 also receives responses from the server, e.g. the requested file portions, and can then make function calls on the Vfile API to ensure that the requested file portions are copied into the Vfile 1104. In the described embodiment, the pending sub media unit (SMU) requests are sent to the network API 1110, which packages multiple requests into one or more large requests. The network API 1110 then sends the request to the media server and retrieves streamed media content from the server. The network API 1110 also handles packet loss and error resilience. In the described embodiment, a media list is built and maintained by the network API 1110. The media list records the SMU requests that are sent to the server. An identifier is attached to each SMU request so that the returning SMU can be easily identified. A timer is maintained for each SMU request that is sent. If a defined time frame is exceeded, the network API assumes that either the SMU request or the returned SMU packet has been lost in the transmission. The network API will again attempt to access the SMU packet. When the requested SMU packet arrives, it is stored in the Vfile cache 1104 and the corresponding request is removed from the media list. In the described embodiment, the media request and the returned packet are transmitted with the same channel using the same TCP protocol, regardless of their priority level. Performance can be improved, however, by transmitting information through different channels. For example, the media request and high priority packets can be transmitted using TCP protocol; low priority packets can be transmitted using UDP protocol. The Vfile API may access the media file through a network other than the current Internet. In such a case, the only part of the API that needs modification is the network API 1110. In the described embodiment, the server comprises a network API 1112 that communicates with the client-side network API 1110 for retrieving particular structured media content file portions from a content storage source 1116. As an example of the work flow that occurs between a media application (client API) and the various APIs mentioned above, consider the following: The media application typically calls the Vfile API 1108 to access the remote media. Upon the receipt of the application's call, Vfile API 1108 checks to determine whether the remote media exists. If it exists, a virtual file identifier (vfid) is established, and the future Vfile API access uses the vfid, just like file operation calls uses a file identifier. Whenever a segment or portion of the remote media is accessed, Vfile API 1108 first checks to determine whether the segment or portion is in the Vfile cache 1104. If the entire segment is in Vfile cache 1104, the Vfile API 1108 assembles and returns the entire segment to the calling application. If none of the segment is in the Vfile cache 1104, the Vfile API 1108 returns immediately. If only a partial portion of the requested segment is in Vfile cache 1104, a continuous header portion of the segment is returned. In the latter two cases, a pending network request is also generated by the Vfile API 1108, which is later sent to the server so that the server can stream the missing segments or portions to the client. The pending request is processed by the network API 1110, which packages and prioritizes multiple requests, and handles network packet loss and errors. When a request packet arrives, it is stored in the Vfile local cache and waits to be accessed by the application program. The Vfile API 1108 may or may not report the arrival of the packet to the application. If the media application chooses to not be notified of the packet arrival, which is generally the common case, it may just access the Vfile 1104 over and over, and render the scene based on the available data. Compared with handling the network connection all by itself, the above workflow of the media application is very simple. Through a group of unified Vfile APIs, the media programmer can disregard chores such as the network accessing protocol, cache management, error resilience and error correction, etc. The task of programming a browser supporting random media access becomes much simpler. The media browsing experience is also improved by the Vfile in three aspects: first, the streaming media content is cached locally and can be reused; second, the media content is prioritized and the most needed data is streamed first; third, the potentially needed media may be preemptively loaded. All these features and functionalities ensure that the user has a better rendered view as fast as possible within the available network bandwidth constraint. Synchronous and Asynchronous Work Mode In one aspect, the Vfile architecture can be configured to work in either a synchronous transmission mode or an asynchronous transmission mode. The media application may call the Vfile API in a synchronous transmission mode. This means that the media segment must be available before the call continues. Most often though, the media application should work in the asynchronous transmission mode. In the asynchronous mode, when the media application calls the Vfile API to access a media segment, the Vfile API will return control immediately to the calling application, regardless of the outcome. The Vfile API attempts to fill the request on a "best effort" basis, i.e., if the entire accessed media segment is in the Vfile cache, it is returned to the calling application. If only part of the accessed media segment is in the Vfile cache, a continuous header portion is returned to the media application. If none of the accessed media segment is in the Vfile cache, nothing is returned to the calling media application. In this aspect, it is the responsibility of the calling media application to be aware that not all media segments are accessible immediately, and it should therefore have ways to handle the missing media segment. One possible strategy is for the media application to repeatedly render a scene using the data that is currently available. With this strategy, the user will, at first, get a coarsely rendered scene when some data is available. The scene will gradually improve as more and more data is received. In a media file, there is usually a very important portion, i.e., the file head that contains scene size, coding method and parameters, etc, that must be streamed before other media access functional calls can be made. To speed up the media streaming experience, a companion file is built for each media file accessed. The companion file is maintained on the server. The companion file indicates the general structure of the media file. Initially, when a media application accesses a media file, the media content or file portions that are above a certain priority level are streamed to the media application. In addition, a preemptive download strategy may be adopted by the media application. Specifically, by accessing the media content surrounding the current view with a low priority, content that is most probably needed by the user may be streamed to the browser when the bandwidth is available. Exemplary Application Program Interfaces for Implementing a Management System for Managing Structured Media Content The following APIs and data structure can be used to implement the described embodiment. It is to be understood that the following constitutes but one example of how APIs might be configured, and is not intended to limit the invention in any way. Accordingly, other arrangements of APIs can be used. The description given below is best understood in the context of FIG. 11. The client API generates requests for needed portions of the Vfile, and forwards the requests on to the Vfile API 1108 for execution. Vfile API In the illustrated example, the Vfile API includes twelve file operation functions. The Vfile also includes a synchronous mode and an asynchronous mode. In the synchronous mode, the function waits until data has arrived from the server. Additionally, the Read function will return an indication of the Network status so that the client API can remain informed. In the asynchronous mode, the function will test to see whether there is data at the client end. If there is data, the function will fill the appropriate buffer provided by the calling application. If there is no data, the function will return immediately. The following table gives a list of the twelve Vfile APIs, their parameters list, and return values:
ID APIs Parameters List Return Value
1 VFOpen "URL", proc1*, [bRefresh=FALSE], vfid (-1 means
[iPriority=0] error)
2 VFRead vfid,RequestStruct*, iLength
NetworkStatus*=NULL
3 VFSeek vfid, iOffset, iPosition iOffset
4 VFSync vfiid, bAsyncFlag = TRUE iRequestNum
5 VFStat vfid struct* (return
comp file)
6 VFClear vfid, [importance] void
7 VFLockCache vfid, offset, size bFlag
8 VFUnLockCache vfid, offset, size bFlag
9 VFTell vfid iCurrentPos
10 VFClose vfid, bWriteCacheFlag = TRUE bFlag
11 VFSetReportImportance iReportId, importance iImportance
12 VFCopyToCache RequestStruct* iLength
The following is a more detailed explanation of the Vfile APIs listed in the above table:
(1) int VFOpen(char* pszUrl, void* pProc1, BOOL bRefresh, int
ipriority) ;
RETURN VALUE :
vfile id for the vfile that is specified by parameter
pszUrl;
PARAMETERS
pszUrl, the vfile that needs to be opened;
pProc1, a function pointer that is provided by the Media
Layer (client API); the function will be called by the VFile API when
a particular condition has been met.
bRefresh, Flag to indicate whether to use current local
disk file cache
TRUE: doesn't use local disk file cache;
FALSE: use the vfile's local disk file cache that has
been defined.
iPriority, PreLoad level. The media content may have
different priority; iPriority indicates that the content
whose priority level is higher than iPriority needs to be
preloaded.
REMARK:
This function opens the vfile that is specified by pszUrl.
If the bFrefresh parameter is FALSE, then load the disk cache of this
file into memory (if it exists), and preload the data that has a
priority that is higher than iPriority from the server side.
(2)int VFRead(int vfid,
RequestStruct* pRequestStr, NetworkStatus* pNetStatus) ;
RETURN VALUE
The byte size that VFRead obtains.
PARAMETERS
vfid : The vfile id;
pRequestStr: The VFRead request.
pNetStatus: The Network status that obtained from Network
API.
REMARK:
This function accepts a request from the client API that
indicates a particular block of data that the caller needs, checks the
memory cache and the disk cache file to determine if the data is
already at the client side. If a part of the data is valid and the
request provides a buffer, then this function copies the valid data
into the buffer, otherwise it does nothing. But at both times, it will
return the data byte size that is valid. If a part of data is not
valid, then the VFile API will send a request to the Network API to
ask for the data from the server.
(3) int VFSeek(int vfid, int iOffset, int iPosition);
RETURN VALUE
The offset, in bytes, of the new position from the
beginning of the file.
PARAMETERS
vfid : vfile id
iOffset: The offset to be sought.
iPosition: The start position in this seek operation
REMARK
The function moves the file pointer associated with the
vfile to a new location that is offset bytes from start position.
This function locates a portion of the file that is desired to be
read.
(4) int VFSync(int vfid, BOOL bAsyncFlag) ;
RETURN VALUE
-1 if Fail , 0 if success.
PARAMETERS
vfid : vfile id
bAsyncFlag: mode flag.
If flag is TRUE then asynchronous requests need be
flash out
Otherwise, synchronous requests should be flash out
REMARK
Whether in the VFile layer or in the Network layer, the requests
that are to be sent may be buffered to improve efficiency.
Accordingly, there may be some requests that are not sent out
immediately. This function flashes out all the synchronous or
asynchronous requests, which depend on the value of the parameter
bAsyncFlag. If the bAsyncFlag parameter is FALSE, it indicates a
synchronous read, so that the expected contents should be returned.
Additionally, the function will wait until all of the content arrives
before it returns. Otherwise, if the bAsyncFlag parameter is TRUE,
only asynchronous requests are sent out, and the function will return
immediately.
(5)FileStat* VFStat(int vfid) ;
RETURN VALUE
File status that stores in FileStat record
PARAMETERS
vfid : vfile id
REMARK
This function gets the companion file that comes with the
media file.
(6) int VFClear(int vfid, int iImportance) ;
RETURN VALUE
reserved
PARAMETERS
vfid : vfile id
iImportance: reserve importance.
REMARK
This function uses the Network API to inform the server to
dismiss the content that has not been send to client side, unless the
content has an importance that is higher than the parameter
iImportance.
(7)int VFLockInCache(int vfid, int iOffset, int iSize) ;
RETURN VALUE
reserved.
PARAMETERS
vfid : vfile id
iOffset: start offset of data
iSize: size of data segment
REMARK
This function locks a data segment in the vfile starting from
iOffset having length iSize.
(8)int VFUnLockInCache(int vfid, int iOffset, int iSize) ;
RETURN VALUE
reserved
PARAMETERS
vfid : vfile id
iOffset: start offset of data
iSize: size of data segment
REMARK
This function Unlocks the data segment that was locked with
VFLockInCache.
(9)int VFTell(int vfid) ;
RETURN VALUE
Current vfile pointer position.
PARAMETERS
vfid: vfile id
REMARK
This function returns the current vfile pointer position.
(10)BOOL VFClose(int vfid, BOOL bWriteCacheFlag) ;
RETURN VALUE
FALSE for error, TRUE for success.
PARAMETERS
vfid : vfile id
bWriteCacheFlag: Flag that enables the memory cache to be
written into the disk cache file before closing the vfile.
REMARK
This function closes the vfile. It also enables the memory
cache to be written to the disk cache so that browsing activities
on a certain file portion can be resumed after a browsing session
has been closed.
(11)int VFSetReportImportance(int vfid, int iImportance) ;
RETURN VALUE
reserved
PARAMETERS
vfid : vfile id
iImportance: The importance of a particular file portion
that is used to determine if it is to be sent (streamed) before
other file portions.
REMARK
This function is used by the Media Layer to specify certain
Importance/Priority levels. Based on the specified levels, if the
incoming data's Importance/Priority is above the level appointed
by this function, a message will be sent to the Media Layer to
inform it that new data has arrived. This function is used in
the Asynchronous model that is used by the Media Layer.
(12)int VFCopyToCache(RequestStruct* pRequestStr,
BYTE* pBuffer) ;
RETURN VALUE
-1 for cache busy. Otherwise return copy data byte size
PARAMETER
pRequestStr : Request pointer, to indicate what the data is
pBuffer: Buffer that holds the data.
REMARK
A function pointer pointing to this function will be transferred
to the Network Layer when the NFOpen (an API of the Network Layer) is
called. This pointer will be used when data comes from the network.
This function tries to copy the corresponding data from the Network
Layer's buffer to the VFile's Cache. If coping data fails, it will
return -1, else return the copied size.
Network API The network API supports the Vfile API, accepts requests from the Vfile API and sends the requests to the remote server. The network API also receives data from the remote server. The network API uses a function pointer to interact with the Vfile API. When data arrives at the network API from the server, the network API calls the function pointer which was previously provided by the Vfile API when the Vfile API called the NFOpen function on the network API. The function pointer is translated in as the "pro c2" parameter, as described in more detail below. The network API will then copy the data received from the remote server into the Vfile cache. The following table gives a list of nine network APIs, their parameters list, and return values:
ID Network APIs Parameters List Return Value
1 NConnect "host name" nid
2 NFOpen nid, "filename", proc2*, int *len nfid
3 NSyncRequest nfid, RequestStruct* void
4 NSync nfid bFlag
5 NASyncRequest nfid, RequestStruct*, void
NetworkStatus*=NULL
6 NASync nfid bFlag
7 NFClear nfid, iImportance/iPriority, bFlag
8 NFClose nfid bFlag
9 NConnectClose nid bFlag
The following is a more detailed explanation of the network APIs listed in the above table:
(1) int NConnect(CString hostname);
RETURN VALUE
id for connecting to the server
PARAMETER
hostname : hostname of server
REMARK
This function sets up a connection to a server, and returns
the connection id.
(2) int NFOpen(int nid, char* filename, int
(*vfproc) (RequestStruct*, BYTE*), long& len);
RETURN VALUE
id for the file
PARAMETER
nid : connect id
filename : file name to be open
vfproc: the Network Layer will call this function pointer
when asynchronous data is reached.
len : the length of file
REMARK
This function tries to open a file on a server that has a
connection with the client which has an id number that equals the nid
parameter. The function stores the vfproc function pointer for future
use, and returns the remote vfile length in len parameter.
(3) void NSyncRequest(int nfid, RequestStruct* request);
RETURN VALUE
TRUE for success, FALSE for fail.
PARAMETER
nfid : file id
request : data request to be sent
REMARK
This function sends out the request to the network layer,
and waits until the data is received from server side.
(4) BOOL NSync (int nfid);
RETURN VALUE
TRUE for success, FALSE for fail.
PARAMETER
nfid : file id
REMARK
Flashes out the buffered synchronous requests to the
server
(5) BOOL
NASyncRequest(int nfid, RequestStruct* request, NetworkStatus*
netstatus = NULL)
RETURN VALUE
TRUE for success, FALSE for fail.
PARAMETER
nfid : file id
request: the data request that needs to be sent
netstatus: the network layer status
REMARK
This function sends out the request to the network layer,
and returns the network status in netstatus parameter. The function
returns immediately, i.e. it does not wait for the completion of the
network function call.
(6)BOOL NASync(int nfid);
RETURN VALUE
TRUE for success, FALSE for fail.
PARAMETER
nfid : file id
REMARK
Flashes out the buffered asynchronous requests to server.
(7)BOOL NFClear(int nfid, BYTE priority);
RETURN VALUE
TRUE for success, FALSE for fail.
PARAMETERS
nfid : file id
priority: the priority level
REMARK
This function is the lower API of the VFClear from the
Vfile APIS above. This function informs the server and clears the
unsent data for the file which is specified by nfid, unless the
priority of that data is higher than the priority level that is given
as a parameter.
(8)BOOL NFClose(int nfid);
RETURN VALUE
TRUE for success, FALSE for fail
PARAMETERS
nfid: file id on a connect
REMARK:
This function closes the file that is specified by nfid.
(9)BOOL NConnectClose(int nid);
RETURN VALUE
TRUE for success, FALSE for fail
PARAMETERS
nid : connect id
REMARK
This function disconnects the connection that is specified
by nid.
The following data structures can be used in connection with the above described APIs:
typedef struct _RequestStruct
{
BYTE* pBuff ; // if pBuff is NULL, VFRead will only send
request out and return,
//buffer is used to hold data, if data is cache
int iOffset ; // start offset of data
int iSize ; //data segment length
BYTE iPriority ; //priority of data
BYTE iImportance ; //importance of data
int vfid ; //vfid of this request
BOOL bAsyncFlag ; // default is TRUE, which means async
model.
} RequestStruct;
typedef struct _NetworkStatusStruct
{
int requestLen; //total number of uncompleted
requests in the network layer
int requestSize; //total request data size of
uncompleted requests in network layer
} NetworkStatus;
typedef struct _tagFilestat {
int vfid ;
long iFileSize ;
BYTE* pBuffer ;
} FileStat ;
This data structure should be defined by the VFile Layer, which
is used to store the companion file's information. It should be filled
in when the function VFopen(..) is called and can be accessed by the
function VFStat(..) :
The above APIs and data structures constitute but one example implementation of the described embodiment. Accordingly, the described APIs and data structures are not intended to limit the invention in any way. Example Using JPEG 2000 File Structure JPEG 2000 constitutes but one example of a structured media content file. An exemplary JPEG 2000 bit stream is shown in FIG. 12 at 1200. The bit stream is led by a global head 1202 that contains crucial information for the decoding operation, e.g. the image and tile size, the transform and coding method, the color component and the bit depth of the image. In JPEG 2000 coding, a large image is segmented into tiles, e.g. tile 1204, where each tile is processed and encoded independently. The compressed bit stream of a tile is packaged independently with a tile head. A tile is further transformed and quantized into a number of sub bands: I.fwdarw.S.sub.1,S.sub.2, . . . ,S.sub.n, where I is a tile, S.sub.i is a sub band of a certain resolution and color component. Each sub band is then segmented into fixed size blocks, S.sub.i.fwdarw.K.sub.i,1,K.sub.i,2, . . . K.sub.i,m, where K.sub.i,j is a block of sub band S.sub.i. Each block is encoded independently into a block bit stream, which may be further divided into bit stream segments: K.sub.i,j.fwdarw.B.sub.i,j,1,B.sub.i,j,2 . . . ,B.sub.i,j,L:, where B.sub.i,j,l is a compressed bit stream segment of block K.sub.i,j. The bit stream is segmented in such a way that all bit stream segments B.sub.i,j,l in the SNR layer l contribute to about the same distortion decrease per bit spent, i.e., they are on the same rate-distortion (R-D) slope. The bit stream segment B.sub.i,j,l may be accessed independently, and is the basic access unit of JPEG 2000. We assume that the JPEG 2000 bit stream is assembled in SNR progressive mode, as shown in FIG. 13. The bit stream segments of the same SNR layer l and of all blocks with the same resolution level and component are packaged together to form a packet, e.g. packet 1302. A header is attached to each packet which provides an index to the bit stream segments within the packet, e.g. how many bytes are there in bit stream segment B.sub.i,j,l. The JPEG 2000 bit stream is organized such that the file head is followed by the bit stream of tile 1, then the bit stream of tile 2, and so on. Within each tile, the bit stream of SNR layer 1 comes first, then the bit stream of SNR layer 2, etc. Within each SNR layer, the bit stream is organized from the coarsest resolution level to the finest resolution level. There are two components in the JPEG 2000 compressed bit stream: the bit stream segments B.sub.i,j,l that can be accessed independently, and the media structure that includes the file head, tile head, and the packet head. The media structure must be available before the individual bit stream segments can be accessed. A companion file is thus built to identify the location of the media structure, and is stored at the media server. The companion file can be eliminated if the design of the JPEG 2000 file format were to include such index in the file head. In the described example, a JPEG 2000 plug-in is implemented with the Vfile API. The plug-in wraps around a JPEG 2000 VM 5.2 decoder and provides Internet browsing functionality for a user-selected region and resolution. When the JPEG 2000 plug-in is launched, it contacts the Vfile API and establishes a connection with the media server where the JPEG 2000 bit stream is stored. Upon connection, the media structure is streamed in a synchronous mode to the Vfile API with the assistance of the companion file. In the described example, the file head, the tile heads and the packet heads are all streamed at the connection stage in the synchronous mode, which accounts for around 1% of the compressed bit stream. However, it is possible to pre-stream only the file head, the tile heads and a few low SNR layer packet heads. The rest of the high SNR layer packet heads can be loaded only when its media content is accessed. Such strategy is useful to reduce the connection time when a huge image is accessed. The exemplary JPEG 2000 plug-in allows a user to selectively browse a specific area of the image with a certain resolution. For simplicity, we call the accessed area with the specific resolution the "current region". The user may zoom and pan around to a new region. To decode the current region, all wavelet coefficient blocks above the current resolution within an area slightly larger than the current area (due to the extension of the wavelet filter) have to be decoded. Such wavelet coefficient blocks are said to "cover" the current region. The plug-in accesses all the needed coefficient blocks and their bit stream segments, but with different priorities and importance. Since the priority determines the streaming order, the priority assignment is determined by the SNR layer and the resolution level of the bit stream segment, and is disregarded relative to the component and the location of the coefficient block. The lower SNR layer is assigned a higher priority than the higher SNR layer. Within the same SNR layer, the coarser resolution level is assigned a higher priority than the finer resolution level. The bit stream of the current region is thus streamed with an SNR progressive order. The visual quality of the streaming is further improved by selecting a visual progressive order that determines the priority of the bit stream segment by a combination of the SNR layer and the visual weight of the sub band. The importance assignment correlates with the cache management strategy. In this example, the importance of the bit stream segment is only tied to its resolution level, and is irrelevant of the component and SNR layer. The bit stream segment of a coarse resolution level is assigned with a higher importance and thus will not be swapped out of the Vfile cache as often as the finer resolution coefficients. The JPEG 2000 plug-in also places a temporary lock on the bit stream segments of the current region and raises their importance to ensure that they will not be swapped out of the Vfile cache. To decode a view of the current region with the available data, the JPEG 2000 plug-in first assembles the bit stream segments B.sub.i,j,l, l=1, . . . ,L of each coefficient block covering the current region. If a bit stream segment B.sub.i,j,l is not available or is only partially available from the Vfile cache, all bit stream segments afterward for that block will be discarded. The available bit stream segments B.sub.i,j,1, B.sub.i,j,2, . . . , B.sub.i,j,l are assembled together and form the current available bit stream of the coefficient block which is then progressively decoded by the JPEG 2000 VM 5.2. If the available bit stream of the coefficient block is empty, the coefficient block is decoded as all zeros. After all coefficient blocks covering the current region are decoded, the coefficients are inverse quantized and inverse wavelet transformed. The JPEG 2000 plug-in decodes and updates the current region at a regular interval. With more and more bit stream segments arriving, the image quality of the current region becomes progressively better. In essence, the JPEG 2000 plug-in accesses only the bit stream segments that are necessary to decode the current view in a SNR progressive fashion. This provides a very good browsing experience for the user. Whenever the user moves the current region by panning around and/or zooming, the JPEG 2000 plug-in first clears the media list and informs the server to abandon any unfulfilled SMU requests (through a VFClear call). The rational behind this is that, in the case where the current region changes, it is essential to immediately deliver the bit stream segments of the updated region. The previous SMU requests that are still pending in the Vfile client and server queue hinder the immediate delivery of the new bit stream segments, and thus should be removed from the queue. The importance values of the MUs covered in the previous region are restored with an unlock call to the Vfile API. The bit stream segments of the updated region are then accessed by the JPEG 2000 plug-in in a similar mode described above. Some of the file portions corresponding to the updated region are immediately available in the Vfile cache, e.g., the bit stream segments of the coarse resolution level blocks that cover a very large area. Their associated coefficient blocks are decoded or partially decoded based on the available bits. Some of the bit stream segments may not be available, and their associated coefficient blocks will be first decoded as all zeroes. The unavailable bit stream segments of the updated region are again streamed in a SNR progressive mode. The JPEG 2000 plug-in still provides the "best" browsing experience with the available downloaded bit stream. Conclusion The described methods and systems greatly facilitate random access to structured media content file portions. Aspects of the invention provide a novel cache structure that is useable to manage structured media content that is provided from one or more servers. Browsing is enhanced because the rendering operation is performed over currently available content, while needed content is contemporaneously acquired over a network. The novel cache structure and disclosed management techniques ensure that content that is needed or likely to be needed to render a particular view is maintained on the client side during rendering operations. File portions are prioritized so that the most important file portions are first provided by the server, followed by lesser important file portions. In addition, important file portions are assigned importance values so that they are not inadvertently removed from the novel cache structure during cache management of the randomly accessed file portions. Further, media content that is likely to be needed by a client can be preemptively loaded at the client end (as through a preemptive request sent by the client to the server) so that regions that are most likely to be next viewed or visited by a user are streamed to the client in advance. Although the invention has been described in language specific to structural features and/or methodological steps, it is to be understood that the invention defined in the appended claims is not necessarily limited to the specific features or steps described. Rather, the specific features and steps are disclosed as preferred forms of implementing the claimed invention.
|
Same subclass Same class Consider this |
||||||||||