APPLICATION PROGRAM INTERFACE (API)

Multimedia project processing systems and multimedia project processing matrix systems

6983466

Abstract

A method of representing a development project is presented comprising identifying a plurality of sources comprising the development project, determining whether any of the sources are required simultaneously and, if not, dynamically generating a filter graph representation of the development project utilizing a segment filter to couple a source to multiple processing threads.


Claims

What is claimed is:

1. A computer storage medium comprising a plurality of executable instruction, which when executed, implement a software object coupled to a source processing chain in a media processing filter graph comprising:

a software object input, coupled to a media source, to receive content from the media source;

a dynamically determined plurality of software object outputs, each responsive to the processing chain, to requested respective disparate media type source clips from a single instance of the media source in accordance With a user defined software object input and coupled to a plurality of source provide each of the source processing chains with their media processing project, wherein said object is configured to parse the media source content into the respective disparate media type source clips provided by said single instance of the media source;

the source processing chain comprising:

a scalable, dynamically reconfigurable matrix switch having a plurality of inputs and a plurality of outputs;

at least one matrix switch input being communicatively linked with a first processing chain portion;

at least one other matrix switch input being communicatively linked with a second processing chain portion; the matrix switch being configured to dynamically couple one or more of the matrix switch inputs to one or more of the matrix switch outputs.

2. The computer storage medium of claim 1, wherein the matrix switch is configured to dynamically couple said one or more matrix switch inputs to said one or more matrix switch outputs based, at least in part, on a media time associated with the user defined media processing project.

3. The computer storage medium of claim 1, wherein the matrix switch is configured to dynamically couple said one or more matrix switch inputs to said one or more matrix switch outputs based, at least in part, on a project time associated with the user defined media processing project.

4. The computer storage medium of claim 1, wherein the matrix switch is configured to dynamically couple said one or more matrix switch inputs to said one or more matrix switch outputs based, at least in part, on content of a matrix switch programming grid.

5. The computer storage medium of claim 1, wherein the matrix switch is configured to dynamically couple said one or more matrix switch inputs to said one or more matrix switch outputs based, at least in part, on a media time associated with the user defined media processing project, a project time associated with the user defined media processing project, and content of a matrix switch programming grid.

6. The computer storage medium of claim 1, wherein the software object alleviates each source processing chain from opening an independent instance of the source.

7. The computer storage medium of claim 1, wherein the number of software object outputs are dynamically determined by the number of independent processing chains required to process media content from the media source.

8. The computer storage medium of claim 1, wherein the object receives requests for media content from one or more of the source processing chains and satisfies said requests.

9. The computer storage medium of claim 8, wherein the object issues seek commands to the media source to satisfy the request(s) for media content.

10. The computer storage medium of claim 8, wherein the object serializes simultaneous requests for media from the source received from multiple source chains.

11. The computer storage medium of claim 10, wherein the object prioritizes the serialized requests based, at least in part, on a relative project time of each of the requested clips.

12. The computer storage medium of claim 1, wherein the software object is an instance of a segment filter exposed to a media processing system executing on a computer system through a render engine.

13. A computer storage medium comprising a plurality of executable instructions, which when executed, implement a system comprising:

means for coupling to a media source to receive content from the media source to provide an input;

means for dynamically determining a plurality of outputs, each responsive to the input and coupled to a plurality of source processing chains, to provide each of the source processing chains with their requested respective disparate media type source clips from a single instance of the media source in accordance with a user defined media processing project, wherein said means for dynamically determined is configured to parse the media source content into the respective disparate media type source clips provided by said single instance of the media source;

the source processing chain comprising:

a scalable, dynamically reconfigurable matrix switch having a plurality of inputs and a plurality of outputs;

at least one matrix switch input being conimunicatively linked with a first processing chain portion;

at least one other matrix switch input being communicatively linked with a second processing chain portion;

the matrix switch being configured to dynamically couple one or more of the matrix switch inputs to one or more of the matrix switch outputs.

14. The computer storage medium of claim 13, wherein the instructions implement a system in which the matrix switch is configured to dynamically couple said one or more matrix switch inputs to said one or more matrix switch outputs based, at least in part, on a media time associated with the user defined media processing project.

15. The computer storage medium of claim 13, wherein the instructions implement a system in which the matrix switch is configured to dynamically couple said one or more matrix switch inputs to said one or more matrix switch outputs based, at least in part, on a project time associated with the user defined media processing project.

16. The computer storage medium of claim 13, wherein the instructions implement a system in which the matrix switch is configured to dynamically couple said one or more matrix switch inputs to said one or more matrix switch outputs based, at least in part, on content of a matrix switch programming grid.

17. The computer storage medium of claim 13, wherein the instructions implement a system in which the matrix switch is configured to dynamically couple said one or more matrix switch inputs to said one or more matrix switch outputs based, at least in part, on a media time associated with the user defined media processing project, a project time associated with the user defined media processing project, and content of a matrix switch programming grid.

18. A computer storage medium comprising a plurality of executable instruction, which when executed, implement a software object used in a media processing filter graph, the software object comprising:

an input, coupled to a media source, to receive content from the media source; and

a dynamically determined plurality of outputs, each responsive to the input and coupled to a source processing chain, to provide each of the source processing chains with their requested respective disparate media type source clips from a single instance of the media source in accordance with a user defined media processing project, wherein said object is configured to parse the media source content into the respective disparate media type source clips provided by said single instance of the media source.

19. The computer storage medium of claim 18, wherein the software object alleviates each source processing chain from opening an independent instance of the source.

20. The computer storage medium of claim 18, wherein the number of outputs are dynamically determined by the number of independent processing chains required to process media content from the media source.

21. The computer storage medium of claim 18, wherein the source processing chains are comprised of filter graph filters which uniquely transform the media content in some way.

22. The computer storage medium of claim 18, wherein the object receives requests for media content from one or more of the source processing chains and satisfies said requests.

23. The computer storage medium of claim 22, wherein the object issues seek commands to the media source to satisfy the request(s) for media content.

24. The computer storage medium of claim 22, wherein the object serializes simultaneous requests for media from the source received from multiple source chains.

25. The computer storage medium of claim 24, wherein the object prioritizes the serialized requests based, at least in part, on a relative project time of each of the requested clips.

26. The computer storage medium of claim 23, wherein the object receives request for media content from a user through a higher-level application, and issues a seek command to satisfy the request.

27. The computer storage medium of claim 18, wherein multiple objects are invoked and coupled to an associated multiple instances of source filters to satisfy multiple simultaneous requests for content from the sources.

28. The computer storage medium of claim 18, wherein the object is exposed by an operating system executing on a computing system implementing a media processing system.

29. The computer storage medium of claim 18, wherein the object is an instance of a segment filter exposed to a media processing system executing on a computer system through a render engine.

30. A computing system comprising:

a storage medium having stored therein a plurality of executable instructions; and

an execution unit, coupled to the storage medium, to execute at least a subset of the plurality of executable instructions to implement the software object of claim 18.


Description

TECHNICAL FIELD

This invention generally relates to processing media content and, more particularly, to an interface and related methods for reducing source filter invocation during execution of a development project.

BACKGROUND

Recent advances in computing power and related technology have fostered the development of a new generation of powerful software applications. Gaming applications, communications applications, and multimedia applications have particularly benefited from increased processing power and clocking speeds. Indeed, once the province of dedicated, specialty workstations, many personal computing systems now have the capacity to receive, process and render multimedia objects (e.g., audio and video content). While the ability to display (receive, process and render) multimedia content has been around for a while, the ability for a standard computing system to support true multimedia editing applications is relatively new.

In an effort to satisfy this need, Microsoft Corporation introduced an innovative development system supporting advanced user-defined multimedia editing functions. An example of this architecture is presented in U.S. Pat. No. 5,913,038 issued to Griffiths and commonly owned by the assignee of the present invention, the disclosure of which is expressly incorporated herein by reference.

In the '038 patent, Griffiths introduced the an application program interface which, when exposed to higher-level development applications, enable a user to graphically construct a multimedia processing project by piecing together a collection of "filters" exposed by the interface. The interface described therein is referred to as a filter graph manager. The filter graph manager controls the data structure of the filter graph and the way data moves through the filter graph. The filter graph manager provides a set of software objects for communication between a filter graph and its application. According to one implementation, individual filters of a filter graph architecture are implemented as component object model (COM) objects of an operating system executing on a computer, each implementing one or more interfaces which contain a predefined set of functions, called methods. Methods are called by an application program or other component objects in order to communicate with the object exposing the interface. The application program can also call methods or interfaces exposed by the filter graph manager object.

Filter graphs work with data representing a variety of media (or non-media) data types, each type characterized by a data stream that is processed by the filter components comprising the filter graph. A filter positioned closer to the source of the data is referred to as an upstream filter, while those further down the processing chain is referred to as a downstream filter. For each data stream that the filter handles it exposes at least one virtual pin (i.e., distinguished from a physical pin such as one might find on an integrated circuit). A virtual pin can be implemented as a COM object that represents a point of connection for a unidirectional data stream on a filter. Input pins represent inputs and accept data into the filter, while output pins represent outputs and provide data to other filters. Each of the filters include at least one memory buffer, wherein communication of the media stream between filters is accomplished by a series of "copy" operations from one filter to another.

As introduced in Griffiths, a filter graph has three different types of filters: source filters, transform filters, and rendering filters. A source filter is used to load data from some source; a transform filter processes and passes data; and a rendering filter renders data to a hardware device or other locations (e.g., saved to a file, etc.). An example of a filter graph for a simplistic media rendering process is presented with reference to FIG. 1.

FIG. 1 graphically illustrates an example filter graph for rendering media content. As shown, the filter graph 100 is comprised of a plurality of filters 102-114, which read, process (e.g., transform) and render media content from a selected source file. As shown, the filter graph includes each of the types of filters described above, interconnected in a linear fashion.

Products utilizing the filter graph have been well received in the market as it has opened the door to multimedia editing using otherwise standard computing systems. It is to be appreciated, however, that the construction and implementation of the filter graphs are computationally intensive and expensive in terms of memory usage. Even the most simple of filter graphs requires and abundance of memory to facilitate the copy operations required to move data between filters. Thus, complex filter graphs can become unwieldy, due in part to the linear nature of conventional development system architecture. Moreover, it is to be appreciated that the filter graphs themselves consume memory resources, thereby compounding the issue introduced above.

Thus, what is required is a filter graph architecture which reduces the computational and memory resources required to support even the most complex of multimedia projects. Just such a solution is disclosed below.

SUMMARY

This invention concerns a system and related interfaces supporting the processing of media content. In accordance with one aspect of the present embodiment, a method of representing a development project is presented comprising identifying a plurality of sources comprising the development project, determining whether any of the sources are required simultaneously and, if not, dynamically generating a filter graph representation of the development project utilizing a segment filter to couple a source to multiple processing threads. In this way, the source and source filter, typically coupled to each of the multiple processing threads, is shared among multiple processing threads, reducing the computational and memory requirements necessary to support execution of the development project.

BRIEF DESCRIPTION OF THE DRAWINGS

The same reference numbers are used throughout the figures to reference like components and features.

FIG. 1 is a graphical representation of a conventional filter graph representing a user-defined development project.

FIG. 2 is a block diagram of a computing system incorporating the teachings of the described embodiment.

FIG. 3 is a block diagram of an example software architecture incorporating the teachings of the described embodiment.

FIG. 4 is a graphical illustration of an example software-enabled matrix switch, according to an exemplary embodiment.

FIG. 5 is a graphical representation of a data structure comprising a programming grid to selectively couple one or more of a scalable plurality of input pins to a scalable plurality of output pins of the matrix switch filter, in accordance with one aspect of the described embodiment.

FIG. 6 is a graphical illustration denoting shared buffer memory between filters, according to one aspect of the described embodiment.

FIG. 7 is a flow chart of an example method for generating a filter graph, in accordance with one aspect of the described embodiment.

FIG. 8 is a flow chart of an example method for negotiating buffer requirements between at least two adjacent filters, according to one aspect of the described embodiment.

FIG. 9 graphically illustrates an overview of a process that takes a user-defined editing project and composites a data structure that can be used to program the matrix switch.

FIG. 10 graphically illustrates the project of FIG. 9 in greater detail.

FIG. 11 shows an exemplary matrix switch dynamically generated in support of the project developed in FIGS. 9 and 10, according to one described embodiment.

FIG. 12 illustrates a graphic representation of an exemplary data structure that represents the project of FIG. 10, according to one described embodiment.

FIGS. 13-18 graphically illustrate various states of a matrix switch programming grid at select points in processing the project of FIGS. 9 and 10 through the matrix switch, in accordance with one described embodiment.

FIG. 19 is a flow chart of an example method for processing media content, in accordance with one described embodiment.

FIG. 20 illustrates an example project with a transition and an effect, in accordance with one described embodiment.

FIG. 21 shows an exemplary data structure in the form of a hierarchical tree that represents the project of FIG. 20.

FIGS. 22 and 23 graphically illustrate an example matrix switch programming grid associated with the project of FIG. 20 at select points in time, according to one described embodiment.

FIG. 24 shows an example matrix switch dynamically generated and configured as the grid of FIGS. 22 and 23 was being processed, in accordance with one described embodiment.

FIG. 25 shows an exemplary project in accordance with one described embodiment.

FIG. 26 graphically illustrates an example audio editing project, according to one described embodiment.

FIG. 27 depicts an example matrix switch programming grid associated with the project of FIG. 26.

FIG. 28 shows an example matrix switch dynamically generated and configured in accordance with the programming grid of FIG. 27 to perform the project of FIG. 26, according to one described embodiment.

FIG. 29 illustrates an exemplary media processing project incorporating another media processing project as a composite, according to yet another described embodiment.

FIG. 30 graphically illustrates an example data structure in the form of a hierarchical tree structure that represents the project of FIG. 29.

FIGS. 31-37 graphically illustrate various matrix switch programming grid states at select points in generating and configuring the matrix switch to implement the media processing of FIG. 29.

FIG. 38 illustrates an example matrix switch suitable for use in the media processing project of FIG. 29, according to one described embodiment.

FIG. 38a graphically illustrates an example data structure in the form of a hierarchical tree structure that represents a project that is useful in understanding composites in accordance with the described embodiments.

FIG. 39 is a flow diagram that describes steps in a method in accordance with one described embodiment.

FIG. 40 illustrates an example method of generating a filter graph, in accordance with one aspect of the present invention.

FIG. 41 graphically illustrates an example reuse list, according to one aspect of the present invention.

FIG. 42 illustrates an example method for source combining in support of the method introduced in FIG. 40, according to one embodiment of the present invention.

FIG. 43 graphically illustrates a timeline representation of source combining introduced in FIG. 42.

FIG. 44 illustrates a block diagram of an example render engine utilizing a segment filter in a filter graph, in accordance with one aspect of the present invention.

FIG. 45 illustrates a flow chart of an example method of generating a filter graph to reuse source filters, in accordance with one aspect of the present invention.

FIG. 46 illustrates a flow chart of an example method of executing a development project utilizing a segment filter, in accordance with one aspect of the present invention.

DETAILED DESCRIPTION

«Previous Next»
Method and system for providing rich media content over a computer network Application programming interface enabling application programs to group code and data to control allocation of physical memory in a virtual memory system