by Maria Ingold
X.7 STREAM PROGRAMMING INTERFACE
When a digital video or a digital audio file is played back, delays and lack of synchronization can be obvious to the user. For example, in a video of a woman singing, the woman’s words need to match the correct movement of her lips. If these do not match, it is said that the “lip-sync” is off. Like the lip-sync in the video, multimedia applications require that different types of data are synchronized together and that multimedia data is efficiently and reliably transported through the system. When an application requests that multimedia data be played back or recorded, the appropriate media driver uses the Stream Programming Interface (SPI) services to move and synchronize the multimedia data. The need for each media driver to come up with its own solution for these requirements is eliminated since the stream interface is independent of the type of data that is being streamed.
The Stream Programming Interface layer is composed of stream handlers, which provide a continuous flow of application data in a real-time manner, and the Synchronization and Streaming Manager (SSM), which coordinates the execution of stream handlers and centrally manages the data buffers and synchronization. Figure X.6 shows the synchronization and streaming architectural layer.
Figure X.6 Syncronization/Streaming Architecture
X.7.1 Stream Handlers
Stream handlers operate as pairs to transport data from a source to a target location. These stream handlers may provide a direct connection from or to a physical device driver (PDD), or from or to something other than a physical device, such as memory or a file. Due to their interface requirements, stream handlers that interface to a physical device are typically provided as device drivers (DD) at the kernel mode (Ring 0), while the others are packaged as dynamic link libraries (DLLs) at the user mode (Ring 3). For example, the file system stream handler is a DLL, because all of the system input/output functions are available at Ring 3.
OS/2 Multimedia provides several stream handlers to accommodate different types of data streaming. For a list of all the OS/2 Multimedia provided stream handlers see Table X.8. More stream handlers can be added to the system as the need for them arises.
|Stream Handler||Ring 3 DLL/Ring 0 DD||Source||Target||Filter|
|Audio||Ring 3/Ring 0 DD(stub)||X (Recording)||X (Playing)|
|File System||Ring 3||X||X|
|MIDI Mapper||Ring 3||X|
|System Memory||Ring 3||X||X|
|Video Capture||Ring 0 DD||X|
Table X.8 Stream Handlers
The stream handler can be a source, target, or filter, depending on its type and the requirement specified by the application. Typically, a stream handler is the source in one instance, and the target in another. For example, as shown in Figure X.3, the audio stream handler is the target during the playback of audio data from a file and the file system stream handler is the source. Conversely, when audio is recorded from an audio device to a file, the audio stream handler is the source and the file system stream handler is the target. The CD-DA stream handler is an example of a stream handler that can only be the source in a stream. This is true for now, as a CD-ROM can only be played from, not recorded onto. Filter stream handlers can act as both a source and a target within the same instance. A filter, such as the MIDI stream handler, receives data from one stream, transforms it, and puts the modified data into another stream to be received by another stream handler.
The file system stream handler uses the file system services to read or write data from any associated storage device. This is necessary for file playback or recording into a file. The file system stream handler uses the Multimedia I/O (MMIO) subsystem APIs to provide file format independence.
Data may also be streamed to or from system memory buffers. This type of streaming is accomplished by using the memory stream handler. While this stream handler does not support synchronization, it does support the concept of cuepoints within playlists. A playlis is a small program that specifies what memory objects from an application should be streamed, and in what order. The playlist contains control information to enable looping, branching and subroutines. When the playlist is “played”, the cuepoint is sent to the target stream handler.
X.7.2 Split Streaming
Some data streams contain interleaved data, such as audio and video. The interleaved data stream is split apart and sent to multiple targets using a process called split streaming. If this data is streamed from a CD-ROM XA device, then the CD-XA stream handler splits the streams. For other types of interleaved data streams, such as an AVI file which contains both audio and video tracks, the multi-track stream handler reads the data and splits each track into separate output buffers. The multi-track stream handler then passes these buffers to the appropriate stream handlers. In this example, the audio tracks are sent to the audio stream handler, and the video tracks are sent to the video stream handler. This process is reversed during recording when an interleaved file is created.
X.7.3 Stream Protocol Control Block
When an application needs to perform an action that requires data streaming, such as playing a file, the appropriate MCD typically issues two SpiGetHandler calls when it opens its device. One call specifies the source stream handler and the other one specifies the target. The names of these stream handlers are passed to SSM, so that SSM knows which stream handlers are going to stream the data. The behavior of the data stream is controlled by the parameters specified in the Stream Protocol Control Block (SPCB). Each stream handler supports one or more stream protocols. The SPCB is uniquely identified by the values of the stream data type and subtype. Stream protocols that have the same data type and subtype are differentiated by an internal key. For example, the data type may be waveform, and the subtype may be 16-bit stereo PCM. Data is also composed of atomic blocks. PCM digital audio data, for example, at a 44.1 kHz sampling rate and 16-bit stereo, would have an atomic data block size of 4 bytes. The SPCB includes data about buffers to be used for streaming, and information needed by both the stream handler and SSM to maintain synchronization. Buffers and synchronization are discussed in sections X.7.4.1 and X.7.4.2, respectively.
The parameters of the Stream Protocol Control Block are listed in Figure X.7. This information is provided to SSM when the stream is created during the SpiCreateStream call. Once the stream has been created, the MCD can start the data streaming can be started by specifying SpiStartStream.
|SPCB Length||Length of the structure|
|SPCB Key||Data stream type and internal key|
|Data Flag||Attributes of the data type||SPCBDATA_CUETIME SPCBDATA_CUEDATA SPCBDATA_NOSEEK|
|Number of Records||Maximum number of records per buffer. (Split streams only.)|
|Data Block Size||Size of an atomic block|
|Data Buffer Size||Size of buffer used while streaming. Maximum buffer size is 64KB|
|Min Number of Buffers||Minimum number of buffers to maintain a constant data stream|
|Max Number of Buffers||Maximum number of buffers needed. This is the ideal number|
|Source Start Number||Number of EMPTY buffers required to start the source stream handler|
|Target Start Number||Number of FULL buffers required to start the target stream handler|
|Buffer Flag||Buffer attributes||SPCBBUF_USERPROVIDED SPCBBUF_FIXEDBUF SPCBBUF_NONCONTIGUOUS SPCBBUF_INTERLEAVED SPCBBUF_MAXSIZE SPCBBUF_16MEG|
|Handler Flag||Stream handler flags||SPCBHAND_GENSYNC SPCBHAND_RCVSYNC SPCBHAND_NONSTREAM SPCBHAND_GENTIME SPCBHAND_NOPREROLL SPCBHAND_NOSYNC SPCBHAND_PHYS_SEEK|
|Resync Tolerance||Resync tolerance value|
|Sync Pulse Granularity||Time interval in MMTIME units between sync pulses|
|Bytes Per Unit||Bytes per unit of time. Used to seek on linear data that is not compressed or of variable length|
|MMTIME Per Unit||The amount of MMTIME each unit represents. Used for seek and “get time” functions of stream handler|
Figure X.7 The Stream Protocol Control Block
X.7.4 Synchronization and Streaming Manager
The Synchronization and Streaming Manager manages the data buffers and the synchronization data. This includes stream resource management (including cleaning up resources when the stream is ended), connecting the stream source to the stream target, and maintaining the data flow. SSM also controls the registration and activities of all the stream handlers.
The Synchronization Streaming Manager operates at both Ring 3 and Ring 0. At Ring 3, the SSM DLL exports functions to Ring 3 DLL stream handlers and higher level multimedia components. At Ring 0, the SSM device driver exports functions to Ring 0 device driver stream handlers. When a higher level multimedia component, such as a media driver, communicates with the SSM DLL it uses the SPI services provided by SSM. See Table X.9 for a complete list of these services.
|SPIGetHandler||Handler Ids for specified stream handler|
|SpiEnumerateHandlers||Lists stream handler names installed in SPI.INI|
|SpiCreateStream||Creates stream instance between source and target stream handler|
|SpiDestroyStream||Removes stream from the system|
|SpiAssociate||Associates data object with stream handler|
|SpiSeekStream||Seeks to point in the stream or sets current stream time|
|SpiStartStream||Starts data streaming for stream instance or group of streams|
|SpiStopStream||Stops data streaming for stream instance or group of streams|
|SpiEnableSync||Establishes a group of streams to synchronize|
|SpiDisableSync||Removes a grouping of streams|
|SpiEnableEvent||Enables event notification for a particular event|
|SpiDisableEvent||Disables event notification for a particular event|
|SpiDetermineSyncMaster||Selects best master stream to be used in a sync group|
|SpiGetTime||Queries current stream time|
|SpiGetProtocol||Queries stream handler for a specific stream protocol|
|SpiInstallProtocol||Installs or removes a specific stream protocol for a stream handler|
|SpiEnumerateProtocols||List of stream protocol keys for a specified stream handler|
|SpiSendMsg||Sends stream-handler-specific message|
Table X.9 SPI Services
Device driver and DLL stream handlers communicate with the SSM by using Stream Manager Helper (SMH) messages and Stream Handler Commands (SHCs). SMH messages allow the stream handlers to register with the SSM, report events and synchronization cues, and request or return buffers. These messages are passed through the SMHEntryPoint and are synchronous calls. They are available to the Ring 3 DLL stream handlers as a DLL call, and as an inter-device driver communication (IDC) call to Ring 0 device driver stream handlers.
Stream Handler Commands allow the SSM DLL to set up and control data streams for DLL stream handlers and the SSM device driver to send initialization and streaming function operations to the device driver stream handler. These commands are accessible through the SHCEntryPoint.
Device driver stream handlers have two additional entry points that allow them to communicate with OS/2 Physical Device Drivers (PDDs). They are the DDCMDEntryPoint and the SHDEntryPoint. The DDCMDEntryPoint handles device driver commands (DDCMDs) which allow a device driver stream handler to request a PDD to perform functions such as starting, stopping, or resuming a device. This interface uses IDC calls.
The SHDEntryPoint enables the device driver stream handler to receive stream handler device (SHD) helper commands from PDDs. This lets the PDDs report events and interrupts, such as reporting status, indicating that a buffer is full, or specifying that an additional buffer is required.
When data is streamed from a source stream handler to a target stream handler, the source stream handler fills in a set of empty buffers and sends them to the target stream handler. When the target stream handler empties the buffers, they are available for use again to the source stream handler. Both the source and target stream handlers specify the minimum and maximum buffers needed for streaming. The Sync/Stream Manager tries to allocate the maximum number requested. If it can not do so, but it can allocate the minimum number to maintain a constant data stream, then the stream is created. Otherwise, the stream creation fails. The size of each data buffer has to be a multiple of the block size. If the block size is not specified, it will default to one byte.
X.7.4.2 Real-time Event Synchronization
The Synchronization and Streaming Manager provides several methods for doing real-time event synchronization. These include specifying a master/slave relationship between several data streams, generating and processing sync pulses, sending and acting on synchronization events, and providing a null stream handler to enable synchronization between streaming and nonstreaming devices. These methods exploit OS/2’s multitasking capability to prevent specific regions of code and data in physical memory from being paged to disk. This ensures that data and requests can be processed in a real-time manner.
A master/slave relationship is where one data stream acts a synchronization master over one or more subordinate slave data streams. This relationship is set up using the SpiEnableSync function. The current time information, or a sync pulse, is sent from the master to all the slaves by the Synchronization and Streaming Manager. The time is specified in the MMTIME (1/3 millisecond units) time base.
The sync pulse enables each slave stream handler to adjust the activity of that stream. This helps to ensure that synchronization is maintained. When a slave fails to maintain synchronization, a sync overrun occurs. This happens when a stream handler receives a sync pulse from SSM and the previous sync pulse has not yet been processed. The stream is not stopped unless the application requests to be notified about sync overruns, through the SpiEnableEvent function, and elects to stop the stream.
Sync pulses are generated by the master on a regular basis. The slave only receives a sync pulse when the Synchronization and Streaming Manager determines that the slave is out of synchronization with its master. This assessment is made by comparing what the slave handler stream believes to be the current time against the master stream time and ensuring that the difference does not exceed a specified resync tolerance value. The slave stream time is updated regularly by the slave stream handler and is stored in the sync pulse event control block. Device driver stream handlers check this event control block to see if a sync pulse has been sent. DLL stream handlers can receive sync pulses by either using this method or by registering a semaphore with the Synchronization and Streaming Manager.
A synchronization and streaming event indicates that the state of either a master or a slave stream handler has changed. These events can be implicit or explicit. Implicit events, such as end of stream, must always be supported by all stream handlers. Explicit events are events that do not always occur, such as a sync overrun, or are only supported by some handlers, for instance a custom events unique to a particular type of data. Applications automatically receive notification about implicit events, but must enable the notification of explicit events by using the SpiEnableEvent function.
X.7.5 Null Stream Handler
Applications sometimes need to synchronize a nonstreaming device with a streaming device. For example, an application may record digital audio from a nonstreaming CD-DA device into a streaming audio adapter. (A nonstreaming CD-DA would use its own DAC to convert the digital audio to analog audio and then input the analog audio through the line in jack on an audio adapter. A streaming CD-DA would stream the digital audio data through the system and use the DAC on the audio adapter.) A nonstreaming device uses a null stream handler to maintain synchronization. The null stream is created and started, but no data flow is associated with it. It simply establishes a resync tolerance value and sets a frequency of event notification to the calling process. The resync tolerance value is set by calling SpiGetProtocol, setting the value, and then calling SpiInstallProtocol.
A null stream is notified of sync pulses the same way as real stream handlers. For this reason, the null stream handler must support some of the Stream Handler Commands that a real stream handler supports. This includes actions on streams, enabling and disabling events and synchronization, and querying and setting protocol information.