Working with Audio and Video

Storyboard includes a media plugin for Storyboard Engine that supports the playback of several different types of audio and video formats. The media plugin provides a common action interface for creating and controlling media playback using a variety of different backend media player services.

It is the backend services that perform the decoding of content. The display of that content may be to a Storyboard External Buffer render extension which will be composited into the Storyboard application, or the service may have the ability to render to discrete hardware layers available on the platform and composited to the Storyboard application directly in hardware.

External media backend services are provided for particular operating system and hardware based on their underlying media support. Currently Storyboard provides two external media backends, which are described in Media Backend Services

note

Audio and Video support is not universally available on all embedded target platforms. It is however available on MacOS, Linux and Windows simulation platforms.

Video and Audio playback and control are accomplished by using a set of defined actions. When creating a media application you must include these media actions in your Storyboard project. The media actions must be installed on a project by project basis and are defined as Action Templates. The media action template is located in your Storyboard installation root under the directory Samples/ActionDefinitions/media.sbat. Copy this file to the templates directory of your project in order to be able to use media actions in your project. Once you have copied the file over, you will have to close and re-open your application to see the media actions.

In order to create a Storyboard application that will play video, you must first add a control to your project containing an External Render Extension. This render extension will be the display target where the video content will be rendered, so size and position it appropriately in your application. In the properties of the external render extension you must set the Buffer Name and Object Path values. These values will be required to play the video properly and will be used to point to where the video content should be displayed.

To initiate video playback, you will use one of the media actions. You will need to add the predefined media action gra.media.new.video to your application, which will tell the media plugin to play your video. As discussed in Connecting Events to Actions this action can be configured to be a response to any event.

In the action parameters, you will specify the Channel_name parameter, which will be used as the name of the Storyboard IO channel that will be created for communication with the media plugin. The channel name should not conflict with the application name and should be relevant to its use, a good suggestion might be media. Specify the project relative name of the video file to play in the parameter Media_name, for example video/myvideo.webm. Additionally fill in the parameters that will link the media command to the external buffer where the playback is to occur. For this you configure the External_buffer_name with the same name as the Buffer Name used in the external render extension and the Object_name with the same name as the Object Path. The Output_width and Output_height should also match the dimensions of the external buffer. Finally the parameter Output_depth should be set to 4.

Other media actions allow you to control the video playback operation once it has been started, including providing pause, resume, stop and seek action commands. Storyboard Designer includes a media application in its samples named Media. For more information on loading samples, refer to New Project from a Storyboard Sample.