By default all of the .lua files in the script folder root are loaded and parsed by the Lua plugin early on during application initialisation and global functions are executed, data variables allocated etc such that everything is ready for the application to display the initial screen. This is perfectly acceptable but perhaps there are some better ways to structure the application.

Taking the time to review the application structure and potentially breaking up the Lua functionality to sub-modules can help with maintenance and development flow plus it can have other benefits that may not be quite so obvious.

The benefits of modularizing your Lua​

One common approach is to localise the Lua functions specific to each screen in a separate file and perhaps organise these into sub-folders within the project whilst keeping common re-usable utility functions at the top of the tree directly under the scripts folder:

When you break-up the Lua code for the project in to modules by creating directories under the scripts directory as shown above this will change the way that the Lua interpreter handles the contents:

• It will not automatically load those scripts not located in the root of the scripts until a require operand is issued for the script file
• Functionality and local data variables specific to a given screen that are only needed ‘just-in-time’ for that screen and can often be disposed of once you leave which helps to manage memory utilisation
• Delay loading will allow certain modules to be loaded at specific times throughout the execution of the UI and for example can help you optimise the startup time and external dependencies
• Larger project teams can work on developing individual screens without impacting the rest of the application by localising their code
• It helps when you need to write unit tests for individual screens by localising Lua scripting functionality

Lua Modules 101​

Lua modules can come in a number of different flavours from simple global function groupings to full object-oriented class implementations.

It first worth an overview of what the require() does in Lua and how to use it to control how Storyboard loads and parses all of the Lua files. Take a look at the details here (Note: the Storyboard Lua interpreter is currently based on standard Lua 5.1):

https://www.lua.org/manual/5.1/manual.html#5.3

http://lua-users.org/wiki/ModulesTutorial

The default case is that the runtime loads all of Lua the files located in the root of the scripts folder on startup. The links above explain the way that the Lua module loader scans for modules referenced in a require statement and the role of the package.cpath in directing Lua which path to search.

This is a common area where you may find issues with integrating 3rd party Lua libraries and so is worth reading and understanding.

Within the Storyboard application you can exploit this capability to explicitly control the loading and execution of any additional Lua sub modules by inserting the appropriate require statements to pull-in module dependencies.

Re-factoring the Lua functionality​

As outlined in the previous sections there are some of the benefits in taking the additional steps described to restructure the application and Lua components. It hopefully delivers a better software architecture and project organisation that is recommended as best practice by our own apps developers.

For example if there was a module for each specific screen that was in the UI, the following design pattern could be used to delay load and unload the Lua module (and any dependencies) on demand.

First you might use a screenshow.pre event trigger on that screen to load the Lua module using a Lua function call:

Locan main_screen = nil
function screenshow_pre()
  main_screen = require("modules.mainScreen")
end

Then on the screenhide.post event, run the following code to unload the Lua module:

function screenhide_post()
  main_screen = nil
  package.loaded["modules.mainScreen"] = nil
  collectgarbage("collect") --if you aren't using the gc option at engine level
end

The require is undone by the line main_screen = nil and the package.loaded[“screens/main_screen”] = nil will remove the code from the internal Lua table and enable that memory to be recycled if needed.

The runtime will do this recycling automatically and for most platforms this is transparent. However to force Lua to be more aggressive you can set the gc=1 option to the Lua plugin. This will cause the Lua interpreter to perform a garbage collection cycle after each Lua action that is performed. This can help with fragmentation as objects that were just allocated and discarded will be freed immediately rather than left to linger for an undetermined amount of time.

For performance however, it may be better to not use the (global) gc=1 option and instead clean-up when leaving the specific screen. This can be achieved programmatically using a call to collectgarbage("collect") at appropriate points in your application.

In our example this may slow the screen transition slightly, but should not hinder the execution once on the particular screen or impact other behaviours elsewhere in the system.

Improving application structure with template Lua module code​

The simplistic approach outlined above will suffice for applications with a few screens however you will see that there are some improvements that can be made once you factor in other common application requirements and look to make the code more reusable.

This is just a simple example and used by way of illustration rather than a reference standard for developing with Lua in Storyboard!

For a better structure to our application code that provides benefit for developing even simple Storyboard applications and our in-house developers recommend adopting some object oriented design practices when coding with Lua.

It is generally the case that each screen would have a common requirement for some initial logic and variable setup to perform ‘On Entry’ before the screen is shown plus there may be some clean-up to do such as disposing of data objects and tables or clearing down animation and timers ‘On Exit’.

The general idea is that each screen has a Lua module file of the same name (watch-out for case sensitivity!) under the ‘modules’ sub-folder and that each file contains as a minimum a couple of template functions for the init and tear-down specific to that screen.

Obviously this base Lua content can be used as a code template and copied for each screen in turn and become a basis for defining the application development guidelines for the project.

Lua makes heavy use of ‘tables’ at the core of the language, indexed by strings that are used for look-up and manipulation using the extensive string libraries. When we re-factor the Lua code into modules we can use the string library to access the table of functions contained in our module by updating the metatable for strings, where the __index field points to the string table, and adding to it our class namespace.

In practice within our example application as shown we will have only a single instance of the class created implicitly (a singleton) and we are not making use of the new member function. Hence it is not being invoked however it is useful good practice to add this to our module implementations.

So for example in our Lua 5.1 syntax employing object-oriented notation:

https://www.lua.org/manual/5.1/manual.html#5.4

modules/mainScreen.lua

---
-- @module mainScreen
--  This class contains functions to underpin the mainScreen functionality. 
mainScreen = {}
mainScreen.__index = mainScreen

print("loaded module mainScreen.lua")

-- Create a new module instance and update metatable 
function mainScreen.new(subsystem)
local self = setmetatable({}, mainScreen)
self:init(subsystem)
return self
end 

-- Perform any dynamic screen specific configuration here
function mainScreen:init(mapargs) 
print("mainScreen:init()")
end

-- Perform any dynamic screen specific tear-down here
function mainScreen:exit(mapargs) 
print("mainScreen:exit()" )
end

-- Callback for 'Press' button to be triggered by gre.press action
function mainScreen:CBHandlePress(mapargs) 
local data={}
data["mainLayer.status_text.text"] = "Activated"
gre.set_data(data)
end

-- Callback for 'Press' button to be triggered by gre.release and gre.outbound actions
--- @param gre#context mapargs
function mainScreen:CBHandleRelease(mapargs) 
local data={}
data["mainLayer.status_text.text"] = "Deactivated"
gre.set_data(data)
end

And for a second screen which in this example performs some data and screen initialisation by assigning a fairly large text string and then explicitly clearing this on exit so that the Lua garbage collection can recycle this memory once finished with.

modules/secondScreen.lua

---
-- @module secondScreen
--  This class contains functions to underpin the secondScreen funtionality. 
secondScreen = {}
secondScreen.__index = secondScreen

local screen_text = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum."

print("loaded module secondScreen.lua")

-- Create a new module instance and update metatable 
function secondScreen.new(subsystem)
local self = setmetatable({}, secondScreen)
self:init(subsystem)
return self
end 

-- Perform any dynamic screen configuration here
function secondScreen:init(mapargs) 
print("secondScreen:init()") 
local data = {}
data["navLayer.content_text.text"] = screen_text
gre.set_data(data)
end

-- Perform any dynamic screen specific tear-down here
function secondScreen:exit(mapargs) 
print("secondScreen:exit()")
screen_text = nil
end

Because these functions are now localised to a specific module namespace for each screen we must refer to them in Storyboard using their full path using module class notation eg : mainScreen::CBHandlePress and mainScreen::CBHandleRelease as below:

Putting it all together from the top down​

By exploiting the context metadata for the events passed via the mapargs parameter we can create some generic utility functions for the application to use when triggered by loading (screenshow.pre) and unloading (screenhide.pre) screen events on each screen. As this is common (global namespace) utility code it can be located at the scripts root folder in the file called callbacks.lua for this example, so that it is always loaded and present by default.

The require statement coupled with the module definition and metatable entry in each module file now means that when the module namespace is loaded it is added to the global table so we can now access the functions specific to each module via their class.

We can load the module via lookup of the module path (here we are using screen name as the name of the Lua module) using the event context for convenience. Note the use of a ‘.’ (dot) separator in the module name to instruct the Lua loader path:

-- Load sub module on demand
-- NOTE: this context strategy requires that the lua module for the screen
-- is named the same as the screen
local module_name = "modules." .. mapargs.context_screen 
require( module_name )

local module = _G[mapargs.context_screen]
module.CBMyFunc()

Pulling this together into the top-level callbacks.lua file completes our implementation.

callbacks.lua

---
-- @module callbacks
--  This class contains functions to underpin the general callbacks funtionality. 
--  It is intended to be an application global utility class
--  
callbacks = {}
callbacks.__index = callbacks

print("loaded callbacks.lua")
-- Load sub module on start-up
--require("modules/mainScreen")
--require("modules/secondScreen")
--print("Lua memory used:"..collectgarbage("count").."KB")

-- Create a new module instance and update metatable 
function callbacks.new(subsystem)
  local self = setmetatable({}, callbacks)
  self:init(subsystem)
  return self
end

-- Perform any dynamic module specific configuration here
function callbacks:init(mapargs) 
  print("callbacks:init()")
end

--- @param gre#context mapargs
function callbacks:CBHandleScreenShow(mapargs) 
  -- Load sub module on demand
  -- NOTE: this context strategy requires that the lua module for the screen
  -- is named the same as the screen
  local module_name = "modules." .. mapargs.context_screen 
  require( module_name )
  
  -- Invoke loaded screen module specific initialisation function
  local module = _G[mapargs.context_screen]
  module:init(mapargs)
  --print("Lua memory used:"..collectgarbage("count").."KB")
end

--- @param gre#context mapargs
function callbacks:CBHandleScreenHide(mapargs) 
  -- Unload sub module on exit
  -- NOTE: this context strategy requires that the lua module for the screen
  -- is named the same as the screen
  local module_name = "modules." .. mapargs.context_screen 
  local module = _G[mapargs.context_screen]
  print("unloading module : " .. module_name )
  
  -- Invoke loaded screen module specific tear-down function
  module:exit()
  
  -- Release and clean-up sub module on demand
  package.loaded[ module_name ] = nil
  collectgarbage("collect") --if you aren't using the gc option at engine level
  --print("Lua memory used:"..collectgarbage("count").."KB")
end

The Lua function call collectgarbage("count") can be used to query the memory retained by the Lua runtime for debug and monitoring purposes.

In Storyboard 6.2 there is now an ---ADVANCED--- option for the Function Name within the Lua callback action properties dialog which will enables you to drill down into to select functions within classes.

Note at present this dialog will only show the classes defined at the application global level.

Observing the new modules structure in action​

Now we can observe the start-up and early initialisation of our top-level callbacks.lua functions then the subsequent delay loading of the screen specific Lua modules ‘just-in-time’ for their associated screen. The utility functions manage their lifespan automatically on screenshow.pre and screenhide.pre.

The trace output snippet below launched in the Designer simulation environment shows the the behaviour of our application modules and functions as follows:

sbengine -vvvvvv -ogreio,channel=LuaModulesTest LuaModuleTest-480x272.gapp

INFO [0.070]:Initialize plugin [greio (6.2.0.36502)] [channel=LuaModulesTest]
INFO [0.070]:Initialize plugin [harfbuzz (6.2.0.36502)] []
INFO [0.070]:Initialize plugin [lua (6.2.0.36502)] []
<emphasis role="bold">ACTION [0.071]:Loading Lua script [callbacks.lua]
loaded callbacks.lua</emphasis>
INFO [0.071]:Initialize plugin [luagredom (6.2.0.36502)] []
INFO [0.071]:Initialize plugin [media (6.2.0.36502)] []
INFO [0.071]:Initialize plugin [metrics (6.2.0.36502)] []
INFO [0.072]:Initialize plugin [poly (6.2.0.36502)] []
INFO [0.072]:Initialize plugin [rext-external (6.2.0.120)] []
INFO [0.072]:Initialize plugin [rtext (6.2.0.36502)] []
INFO [0.072]:Initialize plugin [screen-dump (6.2.0.36502)] []
INFO [0.072]:Initialize plugin [screen-fade (6.2.0.36502)] []
INFO [0.072]:Initialize plugin [screen-path (6.2.0.36502)] []
INFO [0.072]:Initialize plugin [screen-rotate (6.2.0.36502)] []
INFO [0.072]:Initialize plugin [screen-scale (6.2.0.36502)] []
INFO [0.073]:Initialize plugin [system (6.2.0.36502)] []
INFO [0.073]:Initialize plugin [timer (6.2.0.36502)] []
EVENT [0.074]:IO: Dispatch [gre.internalinit]
EVENT [0.108]:IO: Queue [1] gre.init INFO [0.109]:gesture: not enabled, no multi-touch events used in application
EVENT [0.115]:SBIO: Receiver registered @ [LuaModulesTest]
EVENT [0.116]:IO: Dispatch [gre.init]
EVENT [0.116]:IO: Queue [2] gre.screenshow.pre DIAG [0.116]:CONTROL RENDER: background (0x0 479x271)
DIAG [0.116]:CONTROL RENDER: status_text (277x62 476x164)
DIAG [0.118]:CONTROL RENDER: button_next (395x206 469x261)
DIAG [0.119]:CONTROL RENDER: button_test (203x86 277x141)
DIAG [0.120]:CONTROL RENDER: header_text (111x7 350x26)
EVENT [0.120]:IO: Queue [3] gre.screenshow.post EVENT [0.121]:IO: Dispatch [gre.screenshow.pre]
ACTION [0.121]:ACTION: Invoke [gre.screenshow.pre]->[gra.lua] on screen [mainScreen]
<emphasis role="bold">ACTION [0.121]:Lua: [callbacks.CBHandleScreenShow]
loaded module mainScreen.lua
mainScreen:init()</emphasis>
EVENT [0.122]:IO: Dispatch [gre.screenshow.post]
ACTION [1.661]:ACTION: Invoke [gre.outbound]->[gra.datachange] on control [button_test]
ACTION [1.661]:ACTION: Invoke [gre.outbound]->[gra.lua] on control [button_test]
<emphasis role="bold">ACTION [1.661]:Lua: [mainScreen::CBHandleRelease]</emphasis>
DIAG [1.661]:CONTROL RENDER: background (277x62 476x164)
DIAG [1.662]:CONTROL RENDER: status_text (277x62 476x164)
DIAG [1.662]:CONTROL RENDER: button_test (277x86 277x141)
EVENT [2.200]:IO: Queue [1] gre.press EVENT [2.200]:IO: Dispatch [gre.press]
ACTION [2.200]:ACTION: Invoke [gre.press]->[gra.datachange] on control [button_test]
ACTION [2.201]:ACTION: Invoke [gre.press]->[gra.lua] on control [button_test]
<emphasis role="bold">ACTION [2.201]:Lua: [mainScreen::CBHandlePress]</emphasis>
DIAG [2.201]:CONTROL RENDER: background (203x62 476x164)
DIAG [2.201]:CONTROL RENDER: status_text (277x62 476x164)
DIAG [2.201]:CONTROL RENDER: button_test (203x86 277x141)
EVENT [2.404]:IO: Queue [1] gre.release EVENT [2.404]:IO: Dispatch [gre.release]
ACTION [2.405]:ACTION: Invoke [gre.release]->[gra.datachange] on control [button_test]
ACTION [2.405]:ACTION: Invoke [gre.release]->[gra.lua] on control [button_test]
<emphasis role="bold">ACTION [2.405]:Lua: [mainScreen::CBHandleRelease]</emphasis>
DIAG [2.405]:CONTROL RENDER: background (203x62 476x164)
DIAG [2.405]:CONTROL RENDER: status_text (277x62 476x164)
DIAG [2.405]:CONTROL RENDER: button_test (203x86 277x141)
ACTION [3.203]:ACTION: Invoke [gre.outbound]->[gra.datachange] on control [button_test]
ACTION [3.203]:ACTION: Invoke [gre.outbound]->[gra.lua] on control [button_test]
<emphasis role="bold">ACTION [3.203]:Lua: [mainScreen::CBHandleRelease]</emphasis>
ACTION [3.373]:ACTION: Invoke [gre.outbound]->[gra.datachange] on control [button_next]
EVENT [3.699]:IO: Queue [1] gre.press EVENT [3.699]:IO: Dispatch [gre.press]
ACTION [3.699]:ACTION: Invoke [gre.press]->[gra.datachange] on control [button_next]
DIAG [3.700]:CONTROL RENDER: background (395x206 469x261)
DIAG [3.700]:CONTROL RENDER: button_next (395x206 469x261)
EVENT [3.896]:IO: Queue [1] gre.release EVENT [3.896]:IO: Dispatch [gre.release]
ACTION [3.896]:ACTION: Invoke [gre.touch]->[gra.screen] on control [button_next]
EVENT [3.897]:IO: Queue [2] gre.screenshow.pre EVENT [3.897]:IO: Queue [3] gre.screenhide.pre ACTION [3.897]:ACTION: Invoke [gre.release]->[gra.datachange] on control [button_next]
EVENT [3.897]:IO: Dispatch [gre.screenshow.pre]
ACTION [3.897]:ACTION: Invoke [gre.screenshow.pre]->[gra.lua] on screen [secondScreen]
<emphasis role="bold">ACTION [3.897]:Lua: [callbacks.CBHandleScreenShow]
loaded module secondScreen.lua
secondScreen:init()</emphasis>
EVENT [3.898]:IO: Dispatch [gre.screenhide.pre]
ACTION [3.898]:ACTION: Invoke [gre.screenhide.pre]->[gra.lua] on screen [mainScreen]
<emphasis role="bold">ACTION [3.898]:Lua: [callbacks.CBHandleScreenHide]
unloading module : modules.mainScreen
mainScreen:exit()</emphasis>
:
ACTION [4.915]:ACTION: Invoke [gre.touch]->[gra.screen] on control [button_back]
EVENT [4.915]:IO: Queue [2] gre.screenshow.pre EVENT [4.915]:IO: Queue [3] gre.screenhide.pre ACTION [4.915]:ACTION: Invoke [gre.release]->[gra.datachange] on control [button_back]
EVENT [4.916]:IO: Dispatch [gre.screenshow.pre]
ACTION [4.916]:ACTION: Invoke [gre.screenshow.pre]->[gra.lua] on screen [mainScreen]
<emphasis role="bold">ACTION [4.916]:Lua: [callbacks.CBHandleScreenShow]
loaded module mainScreen.lua
mainScreen:init()</emphasis>
EVENT [4.916]:IO: Dispatch [gre.screenhide.pre]
ACTION [4.917]:ACTION: Invoke [gre.screenhide.pre]->[gra.lua] on screen [secondScreen]
<emphasis role="bold">ACTION [4.917]:Lua: [callbacks.CBHandleScreenHide]
unloading module : modules.secondScreen
secondScreen:exit()</emphasis>
DIAG [4.917]:CONTROL RENDER: background (0x0 479x271)
DIAG [4.917]:CONTROL RENDER: status_text (277x62 476x164)
DIAG [4.917]:CONTROL RENDER: button_next (395x206 469x261)
DIAG [4.917]:CONTROL RENDER: button_test (203x86 277x141)
DIAG [4.917]:CONTROL RENDER: header_text (111x7 350x26)
EVENT [4.918]:IO: Queue [2] gre.screenshow.post EVENT [4.918]:IO: Queue [3] gre.screenhide.post DIAG [4.918]:CONTROL RENDER: background (0x0 479x271)
DIAG [4.918]:CONTROL RENDER: status_text (277x62 476x164)
DIAG [4.918]:CONTROL RENDER: button_next (395x206 469x261)
DIAG [4.918]:CONTROL RENDER: button_test (203x86 277x141)
DIAG [4.919]:CONTROL RENDER: header_text (111x7 350x26)
EVENT [4.919]:IO: Dispatch [gre.screenshow.post]
EVENT [4.919]:IO: Dispatch [gre.screenhide.post]
ACTION [5.052]:ACTION: Invoke [gre.outbound]->[gra.datachange] on control [button_test]
ACTION [5.053]:ACTION: Invoke [gre.outbound]->[gra.lua] on control [button_test]
<emphasis role="bold">ACTION [5.053]:Lua: [mainScreen::CBHandleRelease]</emphasis>
EVENT [5.085]:IO: Queue [1] gre.release EVENT [5.085]:IO: Dispatch [gre.release]
EVENT [5.832]:IO: Queue [1] gre.quit EVENT [5.832]:IO: Dispatch [gre.quit]

Getting started with the LuasSqliteDatabase sample​

If we take a closer look at the LuaSqliteDatabase sample you will find an implementation that uses sqlite3 libraries and a simple database populated with some users and data values for controls:

The Lua modules underpinning the database functionality are required to be first loaded by the Lua engine.

Remember to check that these additional files and resources have been included in the deployment configuration. The Storyboard Application Export Configurations dialog and Export Resources tab allows you to add resources to your target filesystem:

Loading the external Lua modules​

The code that loads the library dependencies is as follows and can be seen in callbacks.lua:

-- This is where we do all of the Database interaction and initialization.
-- we connect, query, and update our target database table via the sqlite3 plugin

local myenv = gre.env({ "target_os", "target_cpu" })
if(myenv.target_os=="win32")then
  package.cpath = gre.SCRIPT_ROOT .. "\\" .. myenv.target_os .. "-" .. myenv.target_cpu .."\\luasql_sqlite3.dll;" .. package.cpath
else
  package.cpath = gre.SCRIPT_ROOT .. "/" .. myenv.target_os .. "-" .. myenv.target_cpu .."/luasql_sqlite3.so;" .. package.cpath
end
 
luasql = require("luasql_sqlite3")
 
local database = "usersettings.sqlite"
local env = assert(luasql.sqlite3())
local db = assert(env:connect(gre.SCRIPT_ROOT .. "/"..database, "Failed to connect to database"))
 
local users = {}
local currentUser = 1
:

You can see that we have used some OS specific environment values to automatically create the load path (package.cpath) needed for the luasql_sqlite3 libraries on x86 Linux, MacOS and Windows as part of this project so that you can run these samples in the desktop environment and simulator.

The Storyboard install only includes the appropriate DLLs or shared libraries for these targets but we do build them for most of our embedded targets.

Deploying Lua modules for the target board​

In the example screenshot above I have added the linux-armle folder and populated it with the cross-compiled luasql_sqlite3.so shared library matching the target toolchain and processor architecture.

This binary library is from our Lua libraries package.

We do not include the ‘Lua libraries’ package for every target runtime for installer size reasons however, they are generally available at no charge. You can request these via email to your account manager at sales@cranksoftware.com or through our support channels at support@cranksoftware.com .

Notes on building Lua modules for your target board​

It is recommended that you prepare and build your own Lua libraries packages where possible so that you can take advantage of updates and bug fixes as part of your product development processes and procedures.

This is generally straightforward as it aligns with your other build tasks. There are a few guidance notes which should make this trouble free:

• LuaRocks is awesome however this builds libraries for your desktop development environment by default which is great for simulation and testing but unfortunately probably NOT compatible with your target board
• Instead, pull the sources and build your own libraries: Cross-compile for the target architecture you need using the right toolchain and make
• Always link against the Storyboard Lua library (sblua) to ensure that the API is compatible
• Likewise, ensure that you use the same version of the Lua libraries as the Storyboard core (Lua 5.1)

Additional Resources​

VIDEO: How to run the LuaSqliteDatabase sample

The examples and source code are suitable for use with the Storyboard Lite and Storyboard Real-time Executive (RTExec) SDKs from Crank Software. These are available on request from sales@cranksoftware.com for a number of supported hardware platforms including the i.MXRT1050 and i.MXRT1060 family of crossover processors from NXP, the STM32F4xx and STM32F7xx families from ST Microelectonics, as well as the RA8D1 and RZ/A3M from Renesas.

Sending events to the backend from Storyboard​

When testing events with Storyboard the runtime command line utilities can be used to prototype sending the event with data in a desktop environment on an SBIO connection channel, in this example 'Outgoing'.

For example, using the iogen utility to send an event ‘outgoingDataChange’ with one single byte (unit8_t) data parameter ‘requestState’ in the payload to the channel:

c:\Program Files\Crank_Software\Storyboard_Engine\5.3.201807051156\win32-x86-opengles_2.0-obj\bin>iogen Outgoing no_target outgoingDataChange 1u1:requestState 2
Connecting to Storyboard IO channel [Outgoing]

To receive the event use the iorcv utility. You should use the following command line to listen on the same named channel ‘Outgoing’:

c:\Program Files\Crank_Software\Storyboard_Engine\5.3.201807051156\win32-x86-opengles_2.0-obj\bin>iorcv.exe Outgoing
Waiting on channel [Outgoing]
outgoingDataChange,no_target,1u1:requestState 2

Within the Storyboard application model we can initiate the same event with data to the back-end using a Lua function eg: RequestCalibration() as follows:

stateValue=2

--[ On calibration timer trigger (1 second interval) send event "outgoingDataChange" to back-end channel "Outgoing"
function RequestCalibration()
    local eventData = {}
    local format = "1u1 requestState"
    eventData["requestState"] = stateValue
    local success = gre.send_event_data("outgoingDataChange",format,eventData,"Outgoing")
    if(success == false) then
        print(error)
        --TODO : handle error
    end
end

You should see the same output using the iorcv utility as before but this time the event is sent from within a Storyboard application

Now within the Storyboard RTExec environment, several changes need to be made in order to link the 'front-end' Storyboard application with the 'back-end' FreeRTOS code for receiving UI events.

Configure the IO channel with the runtime engine​

In the desktop and OS runtime environment the IO 'channel' is normally communicated on the sbengine command line whereas for RTExec this channel name is passed in as a greio argument to the app launch API call.

In a typical RTExec project, the Storyboard task code is located in file sbengine_task.c . To enable the Storyboard runtime engine to access and use a named IO channel, add the name as an argument parameter ultimately via the API call gr_application_create_args().

We have again used the channel name 'Outgoing' as in the above example:

/**
 * This is the main runtime task for Storyboard.  It should be possible to call this
 * task directly from within your FreeRTOS core application.
 */
void
sbengine_main_task(void *arg) {
  char *args[20];
  int n = 0;

  args[n++] = "greio";
  args[n++] = "channel=Outgoing";

  // If using VFS, the Storyboard application model will be loaded via string data
  // which is generally present in an sbengine_model.h file named 'sb_model'. This
  // string data would be passed to gr_application_create_args with the GR_APP_LOAD_STRING
  // option flag.
  //
  // In the case a filesystem is present, then the Storyboard model will be loaded
  // from a file on the filesystem. The path to the file should be passed to
  // gr_application_create_args with the GR_APP_LOAD_FILE option flag.
  run_storyboard_app(sb_model, GR_APP_LOAD_STRING, args, n);

Add an event listener and handler for receiving events from Storyboard​

For the 'back-end' FreeRTOS code to receive event notifications from the ‘front-end’ Storyboard UI we need to add an event listener and associated call-back handler to receive each type of these asynchronous events.

This should normally be done after the app object has been initialized but before it has been launched so that the call-back is ready to receive the events on start-up.

Again in sbengine_task.c you can add a call-back (here we just dump the event data by example) and add the listener for it with a convenient place, here in run_storyboard_app() works perfectly.

You should be able to write a single generic handler for multiple received events but the listeners are specific to each event name and channel so you may need to register several if you have lots of events.

static void CrankEventListenerCallback(gr_application_t *app, gr_event_t *event, void *arg)
{
  uint8_t *pData;
  PRINTF("Received Event[%s]: %s, %d bytes;", event->name, event->format, event->nbytes);

  pData = (uint8_t *)event->data;
  for( int i=0; i< event->nbytes; i++)
  {
    PRINTF(" 0x%X", *pData++ );
  }
  PRINTF("\n\r");
  return;
}

static void
run_storyboard_app(const char *bundle, int flags, char * const *options, int option_count) {

  app = gr_application_create_args(bundle, flags, options, option_count);

  if(!app) {
    return;
  }

  // Sets the application logging verbosity. For more logging verbosities, refer to gre.h.
  gr_application_debug(app, GR_DEBUG_CMD_VERBOSITY, GR_LOG_TRACE2);

    // Register our debug console logger callback
    gr_register_log_cb(app, &logger_cb, NULL, NULL);

    // Add event listener for event 'outgoingDataChange' via channel 'Outgoing'
  gr_application_event_listener_t *mEventListener = NULL;
  mEventListener = gr_application_add_event_listener(app,        "outgoingDataChange", &CrankEventListenerCallback, NULL);
  if (mEventListener == NULL)
  {
    PRINTF("failed to add event listener\n");
  }

  // Start the application
  gr_application_run(app);

  // Free the application instance and close any resources.
  gr_application_free(app);
}

Sending an event with data to Storyboard ‘front-end’​

The mechanism is just the same for the FreeRTOS platform as you use with Storyboard on Linux or another high-level OS. If you want to use a named SBIO channel, instead of specifying the channel name on the sbengine launcher command line you need to pass this name as an argument to the run_storyboard_app() API call as described in the section Configure the IO channel with the runtime engine above.

With the FreeRTOS platform you can use a slightly different, more direct approach to send an event to the Storyboard ‘front-end’ UI by using the gr_application_send_event() API call from the SDK (check in include/gre/gre.h) which uses the application handle and app object.

Note: Be sure to first check that this app handle is valid before calling an API function eg: from an interrupt or another RTOS task.

From gre.h:

/**
 * Inject an event into the application event queue.  
 *
 * After making this call the arguments are copied into the internal event queue and 
 * the content can be freely modified without affecting the queue'ed event. 
 *
 * @param app The application handle
 * @param event_target The name of the event target, or NULL to send to the default target
 * @param event_name The name of the event to send, must not be NULL
 * @param event_format The format of the data (see <data_format.h>, or NULL if no data is being sent
 * @param event_data A pointer do the data to transmit, or NULL if no data is transmitted
 * @param event_nbytes The number of data bytes to transmit, or NULL if no data is transmitted
 * @return 0 on success otherwise an error.
 */
DLLExport int gr_application_send_event(gr_application_t *app, 
                     const char *event_target, 
                     const char *event_name,
                     const char *event_format,
                     const void *event_data, 
                     int event_nbytes);

If you have a copy of the Storyboard SDK document (Storyboard_Suite_SDK.pdf) there are more details of the various APIs.

Below is an example ‘back-end’ code snippet for FreeRTOS which posts an 'update_angle' event to the Storyboard UI model with a single floating point data parameter payload as event data 'dial_angle' using the API call gr_application_send_event().

gr_application_t *g_app = NULL;
float angle = 90.0f;
int status = 0;

if( g_app != NULL ) {
  status = gr_application_send_event(g_app, NULL, "update_angle", "4f1 dial_angle", &angle, sizeof(angle));
}

Generating user input with key events to the Storyboard ‘front-end’​

Below is some code that you can use with hardware resources to send a key press-release cycle triggered from for example an IO pin interrupt trigger, UART character received etc:

#include <gre/greio.h>
#include <gre/iodefs.h>

gr_application_t *g_app = NULL; 
gr_key_event_t              key_event_data;
memset(&key_event_data, 0, sizeof(key_event_data));

key_event_data.code = 0;
key_event_data.key = c; // Read key value to send from serial port, keyboard device or sample IO pin etc
key_event_data.modifiers = 0;

if( g_app != NULL ) {
    // key is pressed and it was not a repeat key so send new press event
    gr_application_send_event(g_app, NULL, GR_EVENT_KEY_DOWN, GR_EVENT_KEY_FMT, &key_event_data, sizeof(key_event_data));

    greal_nanosleep(&sleep_time, NULL);
    // and release
    gr_application_send_event(g_app, NULL, GR_EVENT_KEY_UP, GR_EVENT_KEY_FMT, &key_event_data, sizeof(key_event_data));
}

The key-presses and touch events are posted to the model event queue and are handled at the application scope. Check out the definitions in gre/greio.h and gre/iodefs.h for more details of the events and formats.

Interacting with Storyboard data variables​

Storyboard application model data variables can be accessed directly from the C code via the Storyboard data manager either by using the gr_application_set_data() API call to update the variables, triggering a data change in the UI directly or alternatively by using the gr_application_send_event() API call as above to post an event and handle it using a call-back in Lua to set and update the variables.

When a data variable is updated in this way Storyboard is able to trigger data change events and subsequent event actions that you have defined in the UI model automatically.

The following code example shows an RTOS task function which updates several Storyboard application data variables and types (mySignedInt32, and myUnsignedInt32) once every second:

void sbengine_gre_io_data(void *arg)
{
  updatedata_event_t evUpdateData;
  char myMessage[] = "Hello Storyboard!";
  //char myFormat[4];
  int32_t *p_myInt32 = NULL;
  uint32_t *p_myUInt32 = NULL;

  int status = 0;

  strncpy(evUpdateData.myString, myMessage, sizeof(myMessage));
  evUpdateData.myInt32 = -1000;
  evUpdateData.myUInt32 = 1234;

  while (1) {

    vTaskDelay( 1000 );

    if( app != NULL ) {

//Dispatch data to Storyboard by writing directly to variables using the data manager API
// Set 
p_myInt32 = malloc(sizeof(int32_t));
*p_myInt32 = evUpdateData.myInt32;
status = gr_application_set_data(app, "mySignedInt32", "4s1", p_myInt32 );

p_myUInt32 = malloc(sizeof(uint32_t));
*p_myUInt32 = evUpdateData.myUInt32;
status = gr_application_set_data(app, "myUnsignedInt32", "4u1", p_myUInt32 );

//Alternative method: Dispatch data to Storyboard using an event API call
//status = gr_application_send_event(app, NULL, UPDATEDATA_EVENT, UPDATEDATA_FMT, &evUpdateData, sizeof(evUpdateData) );

//Increment values to show change
evUpdateData.myInt32 += 10;
evUpdateData.myUInt32 += 1;
    }
  }
}

Here are some helper functions that you can use to simplify setting data variables and UI control parameters:

void setKeyToIntValue(gr_application_t *app, char *keyName, int32_t value){
    void *pdata;

    pdata = pvPortMalloc( sizeof(value) );
    *(int32_t *)pdata = value;
    // the API call will free() pdata when finished with..
    gr_application_set_data(app, keyName, "4s1", pdata);
}

void setKeyToUIntValue(gr_application_t *app, char *keyName, uint32_t value){
    void *pdata;

    pdata = pvPortMalloc( sizeof(value) );
    *(uint32_t *)pdata = value;
    // the API call will free() pdata when finished with..
    gr_application_set_data(app, keyName, "4u1", pdata);
}

void setKeyToStringValue(gr_application_t *app, char *keyName, char* string){
    void *pdata;
    size_t len;

    len = strlen(string);
    pdata = pvPortMalloc( len + 1 );
    memset(pdata, '\0', len + 1 );
    memcpy(pdata, string, len);
    // the API call will free() pdata when finished with..
    gr_application_set_data(app, keyName, "1s0", pdata);
}

examples:

 setKeyToIntValue( app, "mySignedInt32", mySignedInt-- );
 setKeyToUIntValue( app, "myUnsignedInt32", myUnsignedInt++ );
 setKeyToStringValue(app, "topLayerMain.SetPoint.Temperature.text", strbuf);

Note: The Storyboard data variables must be defined at the ‘application’ scope.

There are several different data variable formats supported in Storyboard that are selectable via the dialog at creation:

For some languages/scripts this can be done simply by adjusting the string in the translation database (or wherever these strings are stored, assuming this is for localization). But this isn’t exactly an elegant, reliable, and realistic solution. There are some libraries that we can use to solve this problem for us, let’s get started.

For the purpose of this article, the instructions are written from the Linux perspective, but the methodology is the same across Windows and Mac.

Files Needed:​

• UTF8 library: https://github.com/meepen/Lua-5.1-UTF-8
• BitOp library: http://bitop.luajit.org/download.html (either .tar.gz or .zip, both work)

If you are on Storyboard version 5.1 and above you do not have to build the BitOp library. Instead, all you have to do is download the utf8.lua file and edit to make the following change:

local bit = bit

to

local bit = bit32

You can then continue on with what's written in the "Reversing Character Order With The UTF8 Module:" section of this article.

Building BitOp library:​

The BitOp library can be compiled against our Lua library and then used as a Lua module. In order to do this, we just need to point it at the libsblua.so that is included in the runtime you are using and set the include directory to be the runtime’s include/lua directory.

• Extract the BitOp download and edit the Makefile
• Comment out the first INCLUDES statement and replace it with:

#INCLUDES= -I/usr/local/include
INCLUDES= -I/[installpath]/Crank_Storyboard/Storyboard_Engine/6.2.202003130919.36509/linux-x86_64-opengles_2.0-x11-obj/include/lua

LDFLAGS= -L/[installpath]/Crank_Storyboard/Storyboard_Engine/6.2.202003130919.36509/linux-x86_64-opengles_2.0-x11-obj/lib -lsblua

Make sure this is formatted for whichever specific runtime and version of Storyboard you are using. The example shown is for the Linux, opengl runtime for Storyboard v6.2

• Scroll down near the bottom of the file and move SOLDFLAGS to be after the -o arguments:

Before:

$(MODSO): $(MODNAME).o
	$(SOCC) $(SOLDFLAGS) -o $@ $<

After:

$(MODSO): $(MODNAME).o
	$(SOCC)  -o $@ $< $(SOLDFLAGS)

The arguments to the linker have to be placed after the -o directive otherwise it won’t bring in the sblua library.

• Save this and open a terminal session in this directory.
• Execute a ‘make’ command and it should build the library successfully:

Now let’s go ahead and move the newly generated bit.so into our project directory. For the purpose of this article I’ve moved it into my project’s scripts folder in a new folder titled “linux-x86_64”. This is just a nice organizational method in case you have to build for multiple platforms.

Setup UTF8 Library:​

Since the library is pure Lua, it can simply be dropped right into your project’s directory. For organizational sake, I’ll put this in the /scripts directory. The UTF8 library does have a dependency on the BitOp library, however, so we will need to add the following modifications:

• At the top of the file, we need to point the module to our bit.so. In this case, mine is located in [project root]/scripts/linux-x86_64/

local info = gre.env({ "target_cpu","target_os" })
local moddir = string.format("%s/%s-%s", gre.SCRIPT_ROOT, info.target_os, info.target_cpu)

package.cpath = string.format("%s/?.so;%s", moddir, package.cpath)

If done this way, if you have also built the BitOp lib for different platforms, it will source the correct one based on the platform, as long as you’ve labeled the folders appropriately.

• Add a require for our bit lib:

Before:

local bit = bit

After:

local bit = require “bit”

And that is all the setup that is required. The next step is to actually use what we’ve included thus far.

Reversing Character Order With The UTF8 Module:​

A simple demonstration of how to use the utf8 module to reverse character order is as follows:

-- Get the arabic string, in this case it’s just hard-written, but in your case it will likely be from a database
 local arabic_string = 'مرحبا'
 local r = ""
 -- Iterate through each character and append them to the end of the string to flip the order
 for p,c in utf8.codes(arabic_string) do
   r = utf8.char(utf8.codepoint(c))..r
 end
 -- Set the text of the control’s text variable
 gre.set_value("Layer.text_ctrl.text", r)

You will also need to ensure that you set the grd_text_shaper_attrs to the appropriate value:

gre.set_value("grd_text_shaper_attrs", "script=Arabic;lang=ar")

To read more about this option and to get the appropriate ISO codes for the script and language, please refer to the documentation found here: https://support.cranksoftware.com/hc/en-us/articles/360039998812-Script-Specific-Text-Shaping-and-Layout

And that’s it, from there everything should be rendered as intended.

A sample project with these libraries compiled can be found here: LINK

A video of the process can be found here.

• Runtime engine for a supported hardware platform, at the time of writing we’ve tested on the i.MX8 and i.MX6 range with OpenGL using Wayland linux-imx8yocto-armle-opengles_2.0-wayland-obj

• linux-imx6wayland-armle-opengles_2.0-wayland_GST01-obj

• The application you wish to use to test the container You can use one of the samples shipped with Storyboard like the “AddressBook” or look at our public SVN for a more comprehensive demo application.

• Storyboard launch script This script contains the parameters used to launch the Storyboard engine and follow the patterns described in our help section for running your application

Once you have these pieces in place you can connect your development platform and start to build your docker file. At this point, it’s assumed that you were able to successfully run our pre-built Docker container on your hardware. If you’ve haven’t done that yet I’d suggest doing that now so you can validate a working scenario on the platform before creating your own.

This is the Recipe file from the standard i.MX8 container.

<para xinfo:text="23143"/><para xinfo:text="23144"/><itemizedlist><listitem><para xinfo:text="23145"/><itemizedlist><listitem><para xinfo:text="23146"/><itemizedlist><listitem><para xinfo:text="23147"/></listitem></itemizedlist></listitem><listitem><para xinfo:text="23148"/><itemizedlist><listitem><para xinfo:text="23149"/></listitem></itemizedlist></listitem><listitem><para xinfo:text="23150"/></listitem></itemizedlist></listitem></itemizedlist><para xinfo:text="23151"/><para xinfo:text="23152"/><programlisting xml:space="preserve" xinfo:translate="no">#!/bin/bash
#Crank Software's Torizon demo

#Variable Setup
crank_dir=/usr/crank
scp_dir=$crank_dir/scp
sbengine_ext=.gapp
runtime=/usr/crank/runtimes/linux-imx8yocto-armle-opengles_2.0-wayland-obj
sb_binary=$runtime/bin/sbengine
sb_app=VitalsHD/1280x720.gapp
demo_app=$crank_dir/apps/$sb_app
sb_options='-orender_mgr,fullscreen -omtdev,device=/dev/input/event1'

#ENV Variables
export SB_PLUGINS=$runtime/plugins
export LD_LIBRARY_PATH=$runtime/lib:$LD_LIBRARY_PATH
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libwayland-egl.so.1

#Verify that SCP directory exists, and check if .gapp file is stored in it.
#If a .gapp file exists run sbengine on this app, otherwise execute demo
launcher
scp_app=$(find /usr/crank/scp -name *.gapp -print -quit)
echo $scp_app
if [ -z $scp_app ]; then
    echo "Starting Crank Demo Application..."
    echo $sb_app    $sb_binary $sb_options $demo_app
else
    echo "Starting Storyboard SCP Application..."
    echo $scp_app    $sb_binary $sb_options $scp_app
fi

The script isn’t overly complicated but it does a few things. As I mentioned earlier you should consult the running your application section in our help center to learn more. However, I’ve highlighted the line which declares the “sb_options” variable. This is where we set up any engine parameters required to configure or tune the engine to our platform. As of right now, we are specifying the application to run in full-screen mode and use the mtdev plugin to read the touch device exposed as /dev/input/event1. This device might be enumerated as a different device depending on your system or the display that you’ve connected so you should check the devices exposed on the host system and once you’ve confirmed you have the right one add it to the script. This device will be exposed to the container from the host hardware based on the container’s run command. One common thing you might want to do is to expose the engine to receiving commands on a particular port. To do this you can leverage the SBIO over TCP functionality. So add the following option to your line which will expose tcp port number 55555.

-ogreio,channel=tcp://55555

Your sb_options line should now look like this.

sb_options='-orender_mgr,fullscreen -omtdev,device=/dev/input/event1 -ogreio,channel=tcp://55555'

You can experiment with adding other options or configurations but now we should try to build and run our container. Connect to your development hardware and copy the crank folder with your runtime, demo application and launch script to the hardware. Beside the crank directory create a file called “Dockerfile” without an extension. Add the lines from our i.MX8 docker recipe to the file, save it and close it. You can then build the container with the following docker command.

docker build -t ':' .

Once you’ve built the container you can launch it with the following command.

docker run -it --privileged -p 55555:55555 -v /home/torizon/crank/scp:/usr/crank/scp -v /dev:/dev -v /tmp:/tmp :

The -p flag will expose port 55555 within the container as port 55555 on the host machine. Finally, make sure to update the final path to your built image in the launch command.

From there you container should launch and if you want you’ll be able to send Storyboard IO events over a TCP socket connection.

The best place to start when looking to perform any kind of feature on Android is to first figure out how it would be accomplished in a native Android setting. Today we’ll be looking at how to create a custom launcher using Storyboard.

First order of business is modifying your Android manifest file. You will have to export your Android manifest file from Storyboard. You can do this via:

Storyboard Application Export Configuration > Change Packaging Method to Native Android Application

Select the Manifest Tab > Manifest File (under Advanced Options) > Export current manifest settings to a file

Choose the location you wish to export your manifest file to and click Save.

Open this xml file in any text editor of your choice, you can even open it in Storyboard if you want.

The first thing we have to change is the content inside the activity tag. By default your manifest will contain:

<intent-filter>
    <action android:name="android.intent.action.MAIN"/>
    <category android:name="android.intent.category.LAUNCHER"/>
</intent-filter>

Simply replace with these two lines:

<category android:name="android.intent.category.HOME"/>
<category android:name="android.intent.category.DEFAULT"/>

What this does is ensures the Android OS identifies this application as a launcher. This means that it can be loaded when the system boots, and can be shown when the “home” button is pressed.

Additionally, it is a good idea to specify that this application is a “singleTask” activity.This means that there can only be one instance of the activity and it is always at the root of the activity stack.

To do this, add this line within your activity tag:

android:launchMode="singleTask"

And to finish up the modifications to your manifest, I would ensure that the manifest’s package name for your application matches the package name described in Storyboard. You can see and modify the package name in Storyboard within the Storyboard Application Export Configuration window.

Upon first exporting the default manifest, the package name does not match the Android Unique Application Identifier shown in the image above.

<manifest xmlns:android="http://schemas.android.com/apk/res/android" android:versionCode="1" android:versionName="1.0" package="android_launcher_demo">

So we just want to change it to reflect the unique identifier:

<manifest xmlns:android="http://schemas.android.com/apk/res/android" android:versionCode="1" android:versionName="1.0" package="com.storyboard.android_launcher_demo">

After all these changes you will have a manifest file that looks like:

<?xml version="1.0" encoding="utf-8" standalone="no"?><manifest xmlns:android="http://schemas.android.com/apk/res/android" android:versionCode="1" android:versionName="1.0" package="com.storyboard.android_launcher_demo">
    <uses-sdk android:minSdkVersion="11"/>
    <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE"/>
    <supports-screens android:xlargeScreens="false"/>
    <application android:hasCode="true" android:icon="@drawable/icon" android:label="@string/app_name">
        <activity android:launchMode="singleTask" android:configChanges="orientation|keyboardHidden" android:label="@string/app_name" android:name="com.crank.android.native_activity.StoryboardNativeActivity" android:screenOrientation="landscape">
            <meta-data android:name="android.app.lib_name" android:value="sbandroid-activity"/>
            <intent-filter>
                <action android:name="android.intent.action.MAIN"/>
                <category android:name="android.intent.category.HOME" />
                <category android:name="android.intent.category.DEFAULT" />
</intent-filter>
</activity>
</application></manifest>

Now let’s take a look at actually launching an application.

Launching an application simply requires getting the launch intent for the specific application/package. This does mean you will need the package name/unique application identifier. In a native Java environment, launching chrome is as quick as:

Intent launchIntent = getPackageManager().getLaunchIntentForPackage("com.android.chrome");
startActivity(launchIntent);

All we have to do is convert that code into using the LuaJava module.

First step is to get an instance of the NativeActivity class. This is because Storyboard applications that are deployed to Android devices run as native activities. As a note, it is good practice to follow any LuaJava calls with nil checks.

local na = luajava.nativeActivity()
if (na == nil) then
    print("no native activity")
    return
end

Next step is to get an instance of the PackageManager class. The PackageManager class can get information related to the applications/packages installed on the device.

local package_manager = na:getPackageManager()
if (package_manager == nil) then
   print("No Package Manager Class")
   return
end

Using the package manager instance, we can now get the launch intent for any package/application as long as we know the package name. Let’s continue with launching Chrome.

local launch_intent = package_manager:getLaunchIntentForPackage(“com.android.chrome”)
if (launch_intent == nil) then
   print("No Launch Intent")
   return
end

Finally, all that is left to do is launch the application/activity.

na:startActivity(launch_intent)

That’s all the setup required to create a custom Android launcher application using Storyboard.In this example, we’ve launched a native Android application (Chrome), but you can also use this to launch other Storyboard applications. As long as they’ve been installed on your device and you know their package name, simply swap the “com.android.chrome” package name with your Storyboard package name. Again, ensure the Android Manifest package name matches the Android Unique Application Identifier in the Export Configuration.

Exporting and Installation​

When exporting, ensure that the apk is being exported using the Android Manifest file that we modified earlier. In the Storyboard Application Export Configuration dialog, uncheck “Use default manifest file” and browse to the custom manifest file.

Click Run and you will have successfully packaged your launcher application.

Move the apk file over to your Android device and install it. You will have to set your default Home application to be your application that you’ve just installed. This can be done by pressing the home button, which will open a dialog asking you to choose which Home launcher you wish to use.

It can also be set by navigating to Settings > Apps and Notifications > Default Apps > Home App

Click that and set it to your Storyboard application.

That’s it! Congratulations! Many cool interactive experiences can be made with this, so be sure to experiment.

Creating a Project​

A project with multiple application files starts the same as a single application file project. Select File > New > Storyboard Application or create a new project using the Photoshop Import or Sketch import features.

A project can accommodate multiple standalone applications that share project images and script resources. You can add an additional application file to an existing project in multiple ways:

• Create a new file within an existing project by selecting File > New > Storyboard Application and choose to create a New Model in Existing Project.
• Create a new file using the Photoshop Import feature and select the option to import into an Existing Project as a New File.
• Create a new file using the Sketch Import feature and select the option to import into an Existing Project as a New File.
• In the navigator view select and copy an existing application, then paste and rename the copied application file.

For a multiple application project to function as a whole, application files need to reference one another. Select the application from the Application Model View and in the Properties View, select Add external model resource button. In the next dialog select the .gde file that was created in Step 2 and press OK.

To add layers from an external model to a local application:

• Select the Import External Layer tab and then select Import Selected Layers.
• Choose the layer(s) to add to the current screen. After adding an external layer, Storyboard will recognize the external content and incorporate it to function like any other layer.

To create a transition from a source application screen to an external application screen, add actions to an application that perform a screen change.

Before launching a multi-file application with the simulator, the external .gde files have to be referenced in the Simulator Configurations dialog.

To manage the runtime configurations, select Run > Storyboard Simulator Configurations. A list of available models that can be included in the runtime export used with Storyboard Engine is in the selected project folder. To apply changes, select Apply and then Run.

If no conflicts occur within the selected applications, they are merged and converted into a single unified application at runtime. If conflicts exist, they must be resolved before the application can merge.

Resolving Conflicts​

The application properties page provides an action to synchronize source content with referenced external content. Any differences are flagged as a conflict and the user is prompted to resolve the conflict based on the issue. Conflict types include:

Layers: If two or more layers have the same name their content needs to be identical.

Variables: Any application/global variables with the same name must have values that are the same.

Animations: If two or more animations use the same name then the animation needs to be identical.

Screens: Screens from all applications are compared. Two or more screens with the same name prompt the user to resolve differences between the two.

What is gstreamer/gstreamer-backend?​

Gstreamer is an open-source library that supports audio and video playback/streaming/mixing. It is a tool that Storyboard utilizes, but it is not a Crank Software product. Gstreamer-backend is a media service which uses the gstreamer framework to control audio and video. While media can be played using just the base gstreamer framework via gst-launch, it’s usually better to use the gstreamer-backend service because it allows Media Control features like Pause, Resume, and Seek.

Before delving into any gstreamer work in Storyboard, if you have never worked with gstreamer before, please take some time to read their documentation and tutorials. Specifically, I would ensure to read this page on pipelines:

https://gstreamer.freedesktop.org/documentation/tutorials/basic/gstreamer-tools.html?gi-language=c

Having good foundational knowledge will only make the next steps easier.

How do I implement gstreamer-backend?​

Okay, so now that we’ve decided we want to use gstreamer-backend, how do we actually implement it?

Note: it’s generally a good idea to get your media successfully playing using basic gstreamer via ‘gst-launch-1.0 [pipeline]’ before attempting to integrate into Storyboard. This will help you identify potential issues with your video, platform, pipeline, etc. before introducing it into the gstreamer-backend/Storyboard environment.

If your Storyboard Engine’s plugin directory contains the ffmpeg plugin (libgre-plugin-ffmpeg.so), this will either need to be disabled or removed. This is because the ffmpeg plugin is loaded automatically when it is present. If it is not disabled/removed, it will compete with gstreamer-backend to service media requests.

Now all media requests will be serviced by gstreamer-backend. However, gstreamer-backend needs to be launched. Gstreamer-backend should be launched before your application is.

Here is an example of what a launch script might look like at this point:

Now that gstreamer-backend has been successfully set-up, let’s look at two ways we can render media playback in our application: using an external render extension, and the dual framebuffer approach. In a nutshell, if you wish to play low-fidelity videos, it’s quicker and more convenient to use external render extensions. If your video is medium to high fidelity or your main concern is performance, use the dual framebuffer approach. The following sections discuss these two approaches in detail.

Please ensure to read both, as they both have some crossover information.

External Render Extension​

The external render extension creates a buffer for other system applications or tasks to render into, things such as video players and web browsers. The MediaPlayer sample that comes with Storyboard uses an external render extension, so let’s use that sample as our reference. To import a sample, go to File > Import > Storyboard Development > Storyboard Sample > select MediaPlayer > Finish

Once the MediaPlayer application is open, select the external_buf control.

As you can see in the Properties tab on the right-hand side, this control contains an external render extension. In Designer, external render extensions will appear as a magenta square. Since external render extensions are a built in part of your application, they are easier to work with. You can overlay controls simply via re-arranging layer/control order in the application model, which adds a level of convenience when using external render extensions.

These don’t have any strict naming convention, but you do need to keep what you’ve set these to in mind as they will come into play momentarily.

Buffer Name: this is simply an identifier for your buffer.

Object Path: this is the path to a shared memory object.

Again, these can be named whatever you wish, as long as whatever platform you are running on does not currently have a conflicting shared memory object at that path, etc.

Let’s take a look at how to trigger video playback. Video playback is triggered via the gra.media.new.video action, so let’s find where that’s being used in this sample.In this case, the gra.media.new.video action can be found on the application level being triggered by an event called MediaPlay.

The Properties of this event are where the majority of the work for successful playback is done.

For greater detail on what each individual option here controls, please refer to our Plugin Action Definitions page, under gra.media.new.video:

(https://support.cranksoftware.com/hc/en-us/articles/360040001012-Plugin-Action-Definitions)

For now, we’ll focus on the most important ones.

Right away, one of the first things you will want to do is populate the External Buffer Name and Object Name. You must make sure these fields contain the same information as the Buffer Name and Object Path within the external render extension properties.

Channel Name: The channel name the new video is to be played on

Media Name: The name of the media to play, full path to a video file

Extra Data: Any extra data that should be passed to the backend (very important)

The Extra Data field is where you will specify the gstreamer pipeline to be used.The pipeline specified here can be similar to the pipeline used with gst-launch with only some minor changes.

Most notably, a pipeline that might’ve looked like:

gst-launch-1.0 [pipeline]

Translated for use by gstreamer-backend would look like:

pipeline: [pipeline]

There are a couple of extra flags that we will have to add/modify and pass into the pipeline for Storyboard usage though:

• use_external: This is only to be used if you are writing your pipeline within Storyboard, i.e. the Extra Data field. Otherwise use “-e”. If you are using an external render extension like we are in this example, you need to add the “use_external” flag at the end of your pipeline, separated by a semi-colon. “pipeline: [pipeline];use_external;”
• -e: This has the same functionality as the use_external flag, the difference in when to use which one simply comes down to where you are specifying the pipeline. Use this one if you are writing your pipeline from the command line or launch script. $ENGINE/bin/gstreamer-backend -e -p [pipeline]
• -p: use the following defined pipeline to play the gstreamer content.
• When it comes to specifying which video you wish to play, there are a couple ways to go about it. You could hard-code the path to the video you wish to play: “pipeline:filesrc location=/path/video.mov” Or you could tell gstreamer-backend to look for whatever you’ve specified in the Media Name field of the gra.media.new.video properties: “pipeline:filesrc name=media-src”
• appsink name=video-sink: In order to use gstreamer-backend with an external render extension, appsink needs to be used as opposed to specifying any other kind of video sink. The external render extension options we’ve specified (use_external/-e) work by looking for an ‘appsink’ element named ‘video-sink’ in which to render the Storyboard application to. appsink does not inherently know the format you wish to display your content with. So you will have to specify this. If I wanted my content to be 800x480 using BGRA color space I would write: video/x-raw,height=480,width=800,format=BGRA You will also need to add videoconvert and videoscale into your pipeline before adjusting the dimensions and color format, as these arguments are what allows the scaling, positioning, etc. All together this would look like: videoconvert ! videoscale ! video/x-raw,height=480,width=800,format=BGRA

Here’s an example pipeline with these modifications:

Pipeline:filesrc name=media-src ! videoconvert ! videoscale ! video/x-raw,height=480,width=800,format=BGRA ! appsink name=video-sink; use_external;

At this point these are the only modifications to your pipeline that are required in order to get your media to play on an external render extension. However, it is not uncommon to experience performance issues depending on your video or platform. Using an external render extension is slower. There are a few reasons for this. For every frame of a video, a memcopy is performed in order to copy the buffer data into the external render extension. Additionally, because we are using appsink and have to scale the video, both of these operations can be an intensive process for certain platforms.

If your platform has hardware acceleration capabilities, then you can add these elements into your pipeline to boost performance. If you are unsure what this pipeline element is called or what other elements you might have access to, perform gst-inspect-1.0 and it will spew out a list of elements that your platform has.

If after all of your tweaking for optimization, you still find performance lacking then you may have to switch implementation to take the dual framebuffer approach.

Dual Framebuffer Approach​

The dual frame-buffer approach is simple in principle. In general, this is the ideal approach when it comes to performance. Since we won’t be using an external render extension, what we do instead is render the Storyboard application to one framebuffer and gstreamer-backend to a separate framebuffer.

You are going to want to render gstreamer-backend to the framebuffer below your application’s framebuffer.

What this usually means is that your Storyboard application will need to have a “hole” punched out of it to allow the video to peak through.

Luckily the MediaSample already does this, if you simply remove the external_buf control you will see that the background has a transparent section.

An example of “punching a hole” in your application.

For gstreamer, It’s actually quite simple to specify the framebuffer to be used. Video sink elements should have a parameter to specify the framebuffer. I’ll take imxipuvideosink for example:

“Imxipuvideosink window-width=880 window-height=520 window-x-coord=10 window-y-coord=75 framebuffer=/dev/fb0”

In this case, all you need to do is add the “framebuffer” element and provide the path to the desired frame buffer. Note that I am also specifying the window’s dimensions and coordinates that the video is to be displayed at.

It is also simple to specify the framebuffer that Storyboard is to use. The render manager has options to specify the framebuffer, but they will vary depending on your platform. Take a look at our plugin option documentation, under “render_mgr” to determine which one is right for you:

https://support.cranksoftware.com/hc/en-us/articles/360040000752-Storyboard-Engine-Plugin-Options

You will either need to use ‘display=[x]’ or ‘fb=[x]’. Again these will vary from platform to platform.One of the key differences between these two options is that ‘display’ takes an index while ‘fb’ takes a path.

This just means that when you launch Storyboard, simply include something like: -orender_mgr,display=1 As part of your launch command line.

You may find that you will have to set up your secondary framebuffer via fbset before you are able to see content that you are rendering to that specific framebuffer. For example, that may look something like:

fbset -fb /dev/fb1 -g 1024 600 1024 600 32

Simply execute this command before trying to render to that buffer.

If at this point you are encountering some visual artifacts like a buffer having half-transparency, then you will have to modify the way alpha is handled in that buffer. Luckily, we do have a utility called fbalpha that can handle this for you.

The fbalpha utility provides added control over rendering content on the secondary framebuffer.

Installing fbalpha

Download fbalpha here

When installing fbalpha, ensure to install it in your runtime’s ‘bin’ directory. Add a call to start up the fbalpha utility and direct it to the top-most framebuffer.

$ENGINE/bin/fbalpha -l1 -f /dev/fb1

At this point your launch script may look like:

$ENGINE/bin/fbalpha -l1 -f /dev/fb1

$ENGINE/bin/gstreamer-backend -v ...etc.

$ENGINE/bin/sbengine -v ...etc.

The next time you launch your application, the alpha issues should be resolved. The download provided also includes the source code. So if you wish to adjust the utility to suit your needs, you can.

How do I stream a video instead of play a video file?​

This is purely gstreamer pipeline work. If your stream uses rtsp then instead of filesrc you will use rtspsource, which ends up looking like:

"rtspsrc location=[address]”

There are many pipeline parameters unique to rtsp streaming which you can familiarize yourself with here: (https://gstreamer.freedesktop.org/documentation/rtsp/rtspsrc.html?gi-language=c)

We use Subclipse as our SVN client. Click on Add and enter a name (we use "Subclipse" but you can use whatever name you want). For Location, enter https://subclipse.github.io/updates to get the latest version and click OK.

There is no need for Maven so you can uncheck the option and click Next.

Review the details and click Next.

Accept the license agreements and click Finish.

Review the warning about unsigned software and click OK.

Restart Storyboard Designer to complete the Subclipse installation.

Now that Subclipse is installed we need to go to that perspective to add a repository.

Select SVN Repository Exploring and click OK.

To add a repository, right-click the SVN perspective window and select New > Repository Location.

Enter the Crank public code repository URL http://svn.cranksoftware.com/repo/storyboard/public and click Finish. When you are prompted for login credentials use:

Username: storyboard

Password: crankrocks

You are now connected to the Crank Public Repository. By expanding the directories you can see the various demos available for checkout.

To checkout from the repository, right-click on the demo and select Checkout. After checkout in complete, click the Storyboard Development tab to see the application in your workspace.

The Discovery Kit features an 800x480 capacitive touch screen to complement the powerful STM32MP157 Arm®-based dual Cortex®-A7 32 bits + Cortex®-M4 32 bits MPU.

https://www.st.com/en/evaluation-tools/stm32mp157c-dk2.html

The instructions for unboxing and creating the SD Card Image using STM instructions for the Starter Package can be found on the ST Microelectronics website here:

https://wiki.st.com/stm32mpu/wiki/STM32MP15_Discovery_kits_-_Starter_Package

I used a Linux virtual machine for this but any supported Windows or Linux desktop environment should enable you to follow the step-by-step guide above and use the STM32CubeProgrammer tool to commission a micro SD card image.

This bootable SD Card image should launch into the ST Microelectronics out-of-box demo and run the base Wayland-based graphical environment.

The initial boot of the board may trigger some maintenance scripts and processes which will automatically redimension and expand the filesystem partitions one by one. This takes some minutes so be patient!

Once booted there is a serial console device created and available if you connect a network cable. On boot, if this is attached the board should get a DHCP address and register a default hostname.

You can SSH into the board (I prefer TeraTerm on Windows) using the hostname stm32mp1 and user root (blank password by default).

With the network link established we can take a first look around and begin to commission the board with Storyboard….

The OpenSTLinux system base image from the STM website resources used for this application note is en.FLASH-stm32mp1-openstlinux-5-4-dunfell-mp1-20-11-12.tar.xz which was the latest available at the time of writing.

root@stm32mp1:~# uname -a

Linux stm32mp1 5.4.56 #1 SMP PREEMPT Wed Aug 5 07:59:52 UTC 2020 armv7l armv7l armv7l GNU/Linux

The filesystem is created by the STM32CubeProgrammer tool such that the ‘user’ applications are located in a separate partition on the SD Card. This is mapped to the /usr/local tree in the root filesystem.

Step 1: Copy across the Storyboard runtime (to the target board)​

Create a Crank folder under /usr/local along with a runtimes folder and apps folder where we will deploy our Storyboard applications:

root@stm32mp1:~# mkdir /usr/local/Crank

root@stm32mp1:~# mkdir /usr/local/Crank/runtimes

root@stm32mp1:~# mkdir /usr/local/Crank/apps

The Storyboard runtime for the STM32MP1 boards is linux-stmA5-armle-opengles_2.0-wayland-obj and located under the installation folder for Storyboard on the desktop.

Copy this folder and contents to /usr/local/Crank/runtimes on the target filesystem which can simply be done using a utility such as WinSCP for Windows or FileZilla.

And once copied grant the runtimes folder contents execute permissions (recursively) to keep it simple..

root@stm32mp1:~# chmod -R 0755 /usr/local/Crank/runtimes

root@stm32mp1:~# ls -la /usr/local/Crank

total 4

drwxr-xr-x 4 root root 1024 Feb 7 16:23 .

drwxr-xr-x 8 root root 1024 Feb 7 16:22 ..

drwxr-xr-x 7 root root 1024 Feb 7 16:23 apps

drwxr-xr-x 3 root root 1024 Feb 7 16:22 runtimes

Step 2: Create a launcher script for Storyboard​

In the root of the Crank folder (/usr/local/Crank) on the target, we can create a script that sets up the execution environment for the Storyboard runtime and options for the application model we want to run at startup.

This is where we can use your own Storyboard application or perhaps one of the samples which ship with Storyboard. As the STM32MP157C-DK2 development board that I am using has an 800x480 LCD panel with touch screen and the processor variant features a GPU supporting OpenGL then we can use the splendid OpenGL3DModel sample which ships with Storyboard:

In this sample, we deploy the Storyboard application model and all associated assets have been exported as a GAPP file (OpenGL3DModel.gapp) and deployed to a folder OpenGL3DModel under the location /usr/local/Crank/apps

storyboard_launcher.sh:

#!/bin/sh

\export RUNTIME_ROOT=/usr/local/Crank/runtimes/linux-stmA5-armle-opengles_2.0-wayland-obj

\export SB_PLUGINS=$RUNTIME_ROOT/plugins

\export LD_LIBRARY_PATH=$RUNTIME_ROOT/lib

\export SB_DEMO=1

\export SB_APP=/usr/local/Crank/apps/OpenGL3DModel/OpenGL3DModel.gapp

killall sbengine

$RUNTIME_ROOT/bin/sbengine -vvvv -orender_mgr,multisample=0 $SB_APP

You can see in the command line options for sbengine we have added the -vvvv to the config. This will emit more verbose trace output to the Linux console and help us in diagnosing any issues. You can remove this trace verbosity option to minimize runtime overhead once you have everything set up and working.

Once the launcher script has been created we will again grant it execute permissions and we should be able to launch and run the application.

root@stm32mp1:~# chmod 0755 /usr/local/Crank/storyboard_launcher.sh

root@stm32mp1:~# ls -la /usr/local/Crank

total 5

drwxr-xr-x 4 root root 1024 Feb 7 16:23 .

drwxr-xr-x 8 root root 1024 Feb 7 16:22 ..

drwxr-xr-x 7 root root 1024 Feb 7 16:23 apps

drwxr-xr-x 3 root root 1024 Feb 7 16:22 runtimes

-rwxr-xr-x 1 root root 328 Dec 18 2020 storyboard_launcher.sh

root@stm32mp1:~# cd /usr/local/Crank/

root@stm32mp1:/usr/local/Crank# ./storyboard_launcher.sh

Step 3: Setup the board to auto-start the Storyboard application​

In vanilla configuration, the filesystem is set up to load and run a demo application to show a simple desktop window and a launcher menu for various utilities and sample applications bundled with the board.

The default demo is built upon the Wayland display manager using a Weston shell and its behaviour and appearance are configured through the file /etc/xdg/weston/weston.ini.

We want to disable the launcher and also hide the bottom control panel so that Storyboard has the full-screen surface to work with.

weston.ini:

[core]
#modules=cms-colord.so
#xwayland=true
shell=desktop-shell.so
backend=drm-backend.so
idle-time=0
repaint-window=100
require-input=false

[keyboard]
keymap_layout=us

[shell]
### Crank - disable background
#background-image=/usr/share/weston/backgrounds/ST13028_Linux_picto_11_1366x768.png
background-color=0xff002244
background-type=scale-crop
clock-format=minutes
panel-color=0x90a1a1a1
### Crank - disable panel
#panel-position=bottom
panel-position=none
locking=false
animation=none
startup-animation=none
close-animation=none
focus-animation=none
binding-modifier=ctrl
#num-workspaces=6
#cursor-theme=whiteglass
#cursor-size=24
:
:

### Crank - disable launcher
#[launcher]
#icon=/usr/share/weston/icon/ST13345_Products_light_blue_24x24.png
#path=/usr/local/demo/demo_launcher.py

[launcher]
icon=/usr/share/weston/icon/space.png
path=/bin/true

[input-method]
path=/usr/libexec/weston-keyboard

# HDMI connector
# If the hdmi cable is plugged, weston uses the hdmi output (else dsi output).
# Use the command "systemctl restart weston@root" after pluging/unpluging the hdmi cable.
[output]
name=HDMI-A-1
mode=1280x720

# DSI connector
[output]
name=DSI-1
mode=preferred
transform=270
:
:

Finally, we can modify the startup script to run our storyboard_launcher.sh script instead of the existing Weston demo_launcher.py script.

The configuration is found in the file:

/usr/local/weston-start-at-startup/start_up_demo_launcher.sh

start_up_demo_launcher.sh:

#!/bin/sh
# found card
cards=`pacmd list-cards |  egrep -i 'alsa.card_name' | sed 's/ //g'| sed 's/alsa.card_name=\"//g'| sed 's/\"//g'`
index=0
for i in $cards;
do
	found=`echo $i | grep -n STM32MP | wc -l`
	if [ $found -eq 0  ];
	then
		su.util-linux -l root -c 'pacmd set-card-profile $index output:analog-stereo'
	fi
	index=$((index+1))
done

#/usr/local/demo/demo_launcher.py
/usr/local/Crank/storyboard_launcher.sh

Upon the next reboot, the board should automatically launch the Storyboard application specified in the storyboard_launcher.sh script file.

root@stm32mp1:~# reboot now

Step 4: Exporting from Designer and running on the target using SCP​

Once the standalone launcher is up and running we can now progress to configuring your development environment so that Storyboard GUI application changes can be deployed and run on the target board directly from Storyboard Designer using the SCP secure file transfer protocol over the SSH connection.

Create a new modified version of the launcher script storyboard_scp.sh which configures the interactive user environment for our default user (root) and which instead accepts as an argument ($1) the application to run rather than a hard-coded path.

This argument is passed by Storyboard automatically and now enables us to deploy and run this script created in the root of the Crank folder (/usr/local/Crank) .

storyboard_scp.sh:

#!/bin/sh

# XDG_RUNTIME_DIR env var must be set so that the Wayland graphical 
# sub-system can locate the launching (root) user's specific data and 
# config plus a place to store temporary files
export XDG_RUNTIME_DIR=/run/user/0

export RUNTIME_ROOT=/usr/local/Crank/runtimes/linux-stmA5-armle-opengles_2.0-wayland-obj
export SB_PLUGINS=$RUNTIME_ROOT/plugins
export LD_LIBRARY_PATH=$RUNTIME_ROOT/lib
export SB_DEMO=1

killall sbengine

$RUNTIME_ROOT/bin/sbengine -vvvv -orender_mgr,multisample=0 $1

Again this script needs first to be granted execute permissions before it can run and we can test that it functions correctly by manually (re-)launching the 3D Demo:

root@stm32mp1:~# chmod 0755 /usr/local/Crank/storyboard_scp.sh
root@stm32mp1:~# ls -la /usr/local/Crank
total 6
drwxr-xr-x 4 root root 1024 Feb  7 16:23 .
drwxr-xr-x 8 root root 1024 Feb  7 16:22 ..
drwxr-xr-x 7 root root 1024 Feb  7 16:23 apps
drwxr-xr-x 3 root root 1024 Feb  7 16:22 runtimes
-rwxr-xr-x 1 root root  328 Dec 18  2020 storyboard_launcher.sh
-rwxr-xr-x 1 root root  504 Dec 18  2020 storyboard_scp.sh
root@stm32mp1:~# cd /usr/local/Crank/
root@stm32mp1:/usr/local/Crank# ./storyboard_scp.sh apps/OpenGL3DModel/OpenGL3DModel.gapp

With a working script file we are now able to deploy the model and assets and Run the script directly from Storyboard using the Export Configuration dialog and the Post Transfer Script option:

The HMI design can be iterated then the application can be modified and run in seconds!

Create an Arc Control​

Create a new project and add an arc control to a layer. You can do this by right-clicking on the layer and selecting Add > Control, or by dragging in an arc control from the Palette.

Change the colour of the arc to whatever you desire, and change the style from fill to Line. You can also adjust the stroke if you would like something thicker than the default.

At this point, you will want to decide where the arc will start. By default, the Start Angle is 0 degrees, which is the point on the right side of the circle, as seen in the above screenshot. If you would like it to start at a different angle, set the Start Angle to a different value and remember that you will now have to change the End Angle relative to that.

Set all angles to 0 so they do not show before the animation begins.

Create the Animation​

Create a new animation and create an animation step for the end angle. Set it to change from a specific value of 0 (or whatever start angle you want) to 360 degrees relative to the start value.

You can keep the Rate linear if you would like, or use some easing to make the wheel seem smoother.

Preview the animation to see if you are happy with how this simple animation looks. If you would like to add a little more flair, you can have the tail of the loading wheel trail behind and catch up with the head.

Add another animation step, this time for the Start Angle. Set it to change from the current value to a specific value of 0. Set the offset and duration both to 0, as well. This will ensure the start angle is reset every time the animation replays, since the animation will likely replay a few times while your app is in a loading state.

Create another animation step for the Start Angle, setting it to change from 0 to 360, mimicking the End Angle animation. Set it to the same duration as the End Angle animation, but stagger it so that it begins approximately 75% of the way through the End Angle animation. This way, it will begin just as the end angle is reaching its destination.

Preview the animation to see if this looks good to you. You can play with the easing graph or staggering of the animation steps if you would like something different.

Alternative Animation Using Rotation Angle​

If you would prefer a loading wheel that is more like a pinwheel, you can use the Rotation angle for the animation instead of the End and Start Angles.

Change the arc style to a fill and set the End Angle to 45 degrees from the Start Angle.

Now remove the animation steps from the loading_wheel animation and create a new one for the variable that is tied to the arc’s Rotation. Set it to go from 0 to 360. In this case, you probably want to use a linear Rate. Test the animation and alter the progression values as desired.

Playing and Repeating the Animation​

The loading animation will be triggered in very specific situations, but for the purpose of this tutorial, it will be triggered on screenshow. Right click on the Screen in the Application Model and create a new action that is triggered on screenshow and triggers the start of the loading_wheel animation.

The important aspect of a loading wheel animation is that it should repeat continuously until it is stopped by another event in the system. To achieve this, create another event on the Screen, this time triggered by the completion of the loading_wheel animation and triggering the start of the same animation.

Now if you start the app, the animation should begin playing and repeat until you send a gra.animate.stop event. Because this last event is on the Screen, it will not be triggered if the animation completes when the user moves to a different screen. This helps to clean up the lifecycle of the animation, so that you don’t have animations running in the background causing performance issues. Keep in mind that if you put this animation complete event at the app level (the top of the Application Model), the animation will continue to be re-triggered continuously, regardless of what screen the user is on, until you send it a stop event.

Create the Gauge​

The gauge will consist of 2 arcs. The bottom arc will be used as the shell of the gauge, to show the total amount of data that the gauge represents. The second arc will sit on top and be situated so that it sits within the outer shell. This is the arc that will be updated using the incoming data, to show the gauge’s current level.

Create a new control by right clicking on your layer and selecting Add > Control > Arc, or by dragging in an arc control from the Palette. Change the size of the control to make the arc as large as you want it.

Now add another arc render extension to the control by pressing the + button. Shrink the size and adjust the position of the render extension so that it sits centered within the first one. You can adjust the end angle to see what it will look like at different levels.

Tie the end angle to a variable, since that is what we will be adjusting, using the incoming data.

Create an Update Event​

This event can just be temporary for testing, or you can create the final event for your application now. The important thing is to first plan out what kind of data you will be sending to update the gauge. You may choose to send the exact level of whatever you are measuring, be that speed or gas level or air pressure. Since it is better to keep most calculations on the backend in order to keep the UI as simple as possible, we are going to send a direct percentage. This will be sent as an unsigned integer, between 0 and 100.

Now that the event is created, add a listener for it at the app level (right click on the app name at the top of the Application Model). Set it to trigger a Lua function.

Define your event channel if you have not done that yet. Click on the Simulator Configurations icon.

Select greio in the plugin options and set the Channel Name.

Close the dialog to save the setting.

Massage the Data and Update the Arc​

Open up the script editor and create the function that you named earlier. You can do this by finding the event in the Actions view for the app level (Click on the app in the Application Model), or by right-clicking in the Scripts view.

Gather the level from the incoming mapargs event data, and print it out to ensure that the level is coming through properly.

--- @param gre#context mapargs
function cb_gauge_update(mapargs) 
  local ev_data = mapargs.context_event_data
  local level = ev_data.level
  
  print(level)
end

Run the app and open the Storyboard IO tab. Select your event from the dropdown menu and slide the slider to ensure that the proper values are being printed in the Console.

Now set some constants for your min and max gauge angles. In this case, they would be 180 and 360.

local MIN_ANGLE = 180
local MAX_ANGLE = 360

Now comes the math to determine the angle of the gauge. Since the percentage is being sent through directly, the math becomes fairly simple.

local angle = ((level/100) * (MAX_ANGLE - MIN_ANGLE)) + MIN_ANGLE

Use gre.set_value to update the End Angle variable.

gre.set_value("Layer.arc_base.end_angle", angle)

Run the app and use the slider in Storyboard IO to update the gauge. You should now see the arc update in the app, staying within the bounds of the gauge.

1 - Create a Translation Database

Open your app and open the text translation view. This can be accomplished by clicking Window > Show View > Text Translation.

You will see there is a dropdown to select a language, as well as an “Extract Text” button. These will be explained shortly, but for now they are unavailable because there is currently no translation database in the project. To add a translation database, click the “Create Translation Database” button.

In the popup, there are options to create a new translation database or import an existing one. We will create a new one for this tutorial.

You will also see that there is a dropdown menu to select the database format. For now the only option is a single CSV file. This format was chosen because it is supported by programs such as Excel, which is often the preferred tool among those actually entering the translations. CSV formatting options make it easy to organize the different values being entered. Other formats, such as json, will be supported in the future.

Click Next

The attributes file will be explained in further detail later on. It is used for variables that will affect the appearance of the app depending on the current language. For example, the width of a button when a translation is longer in a specific language.

For now, just name the attributes file and click Finish.

Enter the name of your initial language and click OK.

2 - Getting to Know the Interface

The first tab of the translation view will show a list of all of the controls with text render extensions that can be translated, as well as the current text being shown and an indication of whether the text is currently clipped or not.

The filter box on the bottom allows you to search by control name. The “Static” and “Variable” checkboxes at the bottom can be used to filter by text that is currently linked to a variable or not, and the “Translated” checkbox can be used to filter out text that is already linked to a translation id.

The second tab of the translation view shows attribute variables, which, as explained previously, can be used to set certain visual aspects of the app depending on the language. For example, if a text area in the previous tab was showing that the text is clipped in the current language, you may want to create an attribute variable for the width of the text area to increase it so the text will fit. We will show the attribute variables in action later on.

The final tab of the translation view will show all existing translation ids, the text for that ID in the current language, and an optional description to indicate the variable’s use within the app. Currently we have no translation IDs in our database, so let’s translate some text.

3 - Extracting Text To Be Translated

In order to begin adding text to your database for translation, return to the Text References tab of the translation view and click “Extract Text” next to the language dropdown.

This view allows you to see all text fields in the project and tie translation variables to them individually. Click on one of the text fields in the “Model Text References” box. If an ID with a similar value to the one in the selected text reference exists, the ID will appear in the “Suggested Translation Variables” box and you can select it if you would like to use it. If no ID exists, click Create New ID. Select the new ID from the “Suggested Translation Variables” box to link this text reference to the ID. Do this for each text field you would like to translate and then click OK.

If your text is dynamic (maybe you have “Good Morning” text that changes to “Good Afternoon” or “Good Evening”), you can add multiple translation IDs for a field and switch between them at runtime depending on context. Use the checkbox in the “Suggested Translation Variable” box to select the default id to show.

A potentially more convenient way of tying each field to a translation variable is by doing it as you create the text fields in the app. After you have placed your text, right click on the icon to the left of the text field and select Translate. This will tie the text to a translation variable that will now appear in the translation view.

4 - Adding Another Language

Now that you have linked your text fields to translation IDs, you will have a database of initial values for your primary language ready to go. Before you send this database to the translator, you will have to add your secondary languages so that they can enter the proper values.

Click the Language dropdown at the top of the translation view, where you see the name of your initial language. Click the option to “Create New Language”. Enter the name of your second language.

If you would like to copy the initial values from your primary language, check the “copy values from” checkbox. If you would like to leave the values as empty strings until the translator can fill them, leave this unchecked.

Click OK to create your new language. Do this for all languages you would like to add to the database.

5 - Fill the Translation Database

This is the point where you give the CSV file to whomever will be filling in the translations for the secondary languages. The file can be found in the translations folder in the project directory. The translator can open it into Excel or another editor and fill in the values in the proper columns.

When they are finished, return the file to the translations directory and re-open the project in Storyboard.

There is an issue with refreshing in Storyboard 6.0, which requires the gde to be closed and reopened for the changes to appear in the translations view. This will be fixed in a future version of Storyboard.

6 - Viewing the Changes

Now that you have imported the updated translation database, you can easily preview each language in Designer by using the language dropdown in the translations view:

7 - Changing the Language at Runtime

Storyboard currently does not include a built-in action to change the language at runtime, however this can be achieved using the VariableLoader script (accompanied by the csv script) in the TranslationUsingID sample.

To use it, initialize the VariableLoader at application init (change the translation and attributes path to match whatever you named the files):

VarLoader = require("VariableLoader")

--- @param gre#context mapargs
function CBInit(mapargs) 
  -- English is the application's base design language so we don't have
  -- to perform any loading initially. If we start with a different language
  -- then we should use loadOnInit to set those initial values.
  local attrs = {}
  attrs.language = "english"
  --attrs.loadOnInit = true
  attrs.textDB = gre.APP_ROOT .. "/translations/translations.csv"
  attrs.attributeDB = gre.APP_ROOT .. "/translations/attributes.csv"

  Translation = VarLoader.CreateLoader(attrs)
end

And then you can simply call setLanguage and pass in the desired language when you would like to switch (make sure the language name you are passing in is identical to the name you gave it when you created it. This includes upper/lowercase):

--- @param gre#context mapargs
function CBLoadLanguage(mapargs)
  Translation:setLanguage(mapargs.language)
end

8 - Using Attributes to Adjust Your App Based on Language

There may be situations where your app will have to look a little different depending on the current language. For instance, some strings may be longer in different languages, and the formatting of the text field will have to be adjusted for it to fit. Storyboard’s translation tool provides attributes for this exact purpose. In this example, we will imagine that the English language uses a different font than the French language.

First, tie the font for all text fields to a variable. In our case, all the fonts could be tied to the same variable because they will share the same font in each language.

Now click on the attributes tab in the translations view. Right click in the empty variable list and click “Add Variable to Translation”. Select your variable and click finish.

Take the attributes csv file into an editor and fill in your desired values for each language. Re-import the csv into Storyboard and switch between languages in the translations view. You should see the fonts change as you move between languages. Try this out for any other variables you may want to change depending on language.

There is an issue with refreshing in Storyboard 6.0, which requires the gde to be closed and reopened for the changes to appear in the translations view. This will be fixed in a future version of Storyboard.

Getting started with user interface table variables​

Before getting started, download the example that we've started for you and import it into Storyboard. You should see something that looks like this:

Adding User Interface Table Variables​

If you were to run the application at this point, it wouldn't do too much. All of the cells in the table are the same. They all have the same text and there aren't enough of them to make up an entire week. To create a table variable:

• Click the table cell.
• Select the Text render extension.
• Bind a variable to the text value.

Creating the new table variable​

In the Table Variable Selection dialog:

• Click the New Variable button.
• Type "text" in the Name field.
• Select the Create a table cell variable checkbox to indicate that we are creating a table variable.
• Extend it to the 5th row (we could do all 7 here but we may not know how many rows there will be).
• Finish and select that variable.

Loading data​

In the Application Model, double-click callbacks.lua to open it up for editing. Let's add code to iterate through the weekdays table to populate the table cells that we have so far.

• Copy and paste the code below to fill in the load_table function.
• Run the program and press Load Table.

function load_table(mapargs)
	local data={}
	for i=1,#weekdays do --The "#weekdays here returns the number of entries in the table."
                --When changing table variables, you specify them by layer_name.control_name.variable_name.row.col
		data["TableLayer.weekdays.text."..i..".1"] = weekdays[i]
 	end
	gre.set_data(data)
 end

Expanding the table rows to include selected data​

A week isn't complete with only six days, so we will need to add Sunday to the table. But right now, our table is only six rows long. There are several ways to change the number of rows in a table, but for our purposes, the Lua function gre.set_table_attrs() is most appropriate. That function expects a table of tags with their values, as well as the name of a table. Paste this function underneath the gre.set_data(data) we just added:

local tag_table={}
tag_table["rows"] = #weekdays
gre.set_table_attrs("TableLayer.weekdays.text.",tag_table)
end

Now, let's run the application again and check our progress.

Adding a new variable for Fill​

Let's add another table variable but this time to the Fill render extension. To start, you'll notice we've only included 4 of the rows.

• Click the table cell.
• Select the Fill render extension.
• Switch the render extension from a static to dynamic variable.
• Click the New Variable button.
• Type "fill" in the Name field.
• Select the Create a table cell variable checkbox to indicate that we are creating a table variable.
• Extend it to the 4th row as you see in the screenshot below.
• Click Finish.

Run this in the simulator now and you'll find that the fills seem to be black in all the rows from 4 onward. This is because there is no data to go on when those cells are being filled out. To fix this, add the following under our text variable change in the load table:

data["TableLayer.weekdays.fill."..i..".1"] = 0x6D8F3A

Run that to see this fixed.

Adding interactivity to the table​

Next, we're going to add a few actions on the user interface table cell. We'll add gre.cell.gotfocus with a Data Change action, gre.cell.lostfocus with a Data Change action, and gre.press with a Lua Script action. gotfocus will change the table fill to #A7C445 and lostfocus will change it to #6D8F3A. The press event will call the Lua function cb_cell_press. You can add those parameters below the trigger events on the Actions tab, as you can see below:

Understanding which row was pressed​

Now that we have some highlighted cells and some dynamic data, let's see which row is pressed. We've already attached the Lua function cb_cell_press, so now we can add the following function and code to our callbacks.lua script:

function cb_cell_press(mapargs)

	local cell=mapargs.context_row

	print(cell)
 end

Run it and press on a couple rows.

Ensuring a meaningful response​

What we see printed out in the Storyboard console is the row number of each cell we press. While that's a step forward, we'd probably want more than just the row number. We could print weekdays[cell], but what if that data has gone through changes since it was put in the table? What we want is the text variable of that specific cell at that specific point in time. So let's add the last two lines below to our cb_cell_press function:

function cb_cell_press(mapargs)

	local cell=mapargs.context_row

	local data=gre.get_data("TableLayer.weekdays.text."..cell..".1")

	print(data["TableLayer.weekdays.text."..cell..".1"])
 end

Now we're getting meaningful data from that row that we can use elsewhere if required.

Playing with other data to make it your own​

We're now left with this as a final callbacks.lua file:

local weekdays={"Monday","Tuesday","Wednesday","Thursday","Friday","Saturday","Sunday"}
 function load_table(mapargs)

	local data={}

	for i=1,#weekdays do --The "#weekdays here returns the number of entries in the table."
                --When changing table variables, you specify them by layer_name.control_name.variable_name.row.col
		data["TableLayer.weekdays.text."..i..".1"] = weekdays[i] 
		data["TableLayer.weekdays.fill."..i..".1"] = 0x6D8F3A
	end
gre.set_data(data)

local tag_table = {}
tag_table["rows"] = #weekdays
gre.set_table_attrs("TableLayer.weekdays",tag_table)
end
function cb_cell_press(mapargs)
local cell=mapargs.context_row
local data=gre.get_data("TableLayer.weekdays.text."..cell..".1")
print(data["TableLayer.weekdays.text."..cell..".1"])
end

Now that we've play around with the different table settings to see what they do, try adding another variable and render extension, or add an image to the right of the table.

To help out further, take a look at our Lua documentation to see what other functionality can be added to tables in your user interfaces.

There are a few resources and references that you may find useful:

https://www.nxp.com/docs/en/user-guide/LPCXpresso_IDE_SWO_Trace.pdf

https://www.nxp.com/docs/en/quick-reference-guide/MCUXpresso_IDE_SWO_Trace.pdf

https://community.nxp.com/community/mcuxpresso/mcuxpresso-ide/blog/2019/06/10/swo-with-nxp-imx-rt1064-evk-board

Configuring the SWO hardware​

The first thing that needs to be done is to bring the SWO pin to the outside world and make this available to the onboard JTAG adapter. This requires a change to the pin MUX configuration for the single wire debug function [Pad G13] on the RT1060 device.

You can do this one of two ways:

Option 1: Using the MCUExpresso Pins configuration tool and then use the update option to insert the code automatically as follows:

* TEXT BELOW IS USED AS SETTING FOR TOOLS *************************************

CopiedFunction:
- options: {}
- pin_list:
- pinItem: {pin_num: G13, peripheral: ARM, signal: arm_trace_swo, pin_signal: GPIO_AD_B0_10, pull_up_down_config: Pull_Down_100K_Ohm, pull_keeper_select: Keeper,
pull_keeper_enable: Disable, open_drain: Disable, speed: MHZ_200, drive_strength: R0_7}

* BE CAREFUL MODIFYING THIS COMMENT - IT IS YAML SETTINGS FOR TOOLS ***********

Option 2: Manually insert the code below into the BOARD_InitPins() function within the pin_mux.c file to set the pin MUX and route the SWO TRACE function to the pin:

IOMUXC_SetPinMux(
    IOMUXC_GPIO_AD_B0_10_ARM_TRACE_SWO,     /* GPIO_AD_B0_10 is configured as ARM_TRACE_SWO */
    0U);                                    /* Software Input On Field: Input Path is determined by functionality */
 
    IOMUXC_SetPinConfig(
    IOMUXC_GPIO_AD_B0_10_ARM_TRACE_SWO,     /* GPIO_AD_B0_10 PAD functional properties : */
    0xF9U);                                 /* Slew Rate Field: Fast Slew Rate
                                                 Drive Strength Field: R0/7
                                                 Speed Field: max(200MHz)
                                                 Open Drain Enable Field: Open Drain Disabled
                                                 Pull / Keep Enable Field: Pull/Keeper Disabled
                                                 Pull / Keep Select Field: Keeper
                                                 Pull Up / Down Config. Field: 100K Ohm Pull Down
                                                 Hyst. Enable Field: Hysteresis Disabled */

Next the TRACE clock settings must be checked and or modified in clock_config.c within function BOARD_BootClockRUN() to be the following:

/* Set TRACE_PODF. */
CLOCK_SetDiv(kCLOCK_TraceDiv, 0);

/* Set Trace clock source. */
CLOCK_SetMux(kCLOCK_TraceMux, 3);

Finally the TRACE clock must be enabled on boot so insert the following in main() after BOARD_BootClockRUN():

// Enable SWO TRACE clock
*((uint32_t *)(0x400E0600)) = (1 << 11); /* enable TPIU clock */
CLOCK_EnableClock(kCLOCK_Trace);

Adding SWO tracing utility functions​

Once the hardware is configured we can now make use of the SWO pin to act as a JTAG back-channel for trace output instead of the much slower UART method. The utility functions enable you to send strings out of the SWO debug channel that can be read using the SWO ITM Console.

SWO.h:

/*
 * SWO.h
 *
 *  Created on: 27 Mar 2020
 */
 
#ifndef SWO_H_
#define SWO_H_
 
void SWO_PrintChar  (char c);
void SWO_PrintString(const char *s);
 
 
#endif /* SWO_H_ */

SWO.c:

/*
 * SWO.c
 *
 *  Created on: 27 Mar 2020
 */
 
#include "SWO.h"
 
/*******************************************************************************
 * Code
 ******************************************************************************/
 
 
/*********************************************************************
*
* Defines for Cortex-M debug unit
*/
 
/*********************************************************************
*
*       Defines for Cortex-M debug unit
*/
#define ITM_STIM_U32 (*(volatile unsigned int*)0xE0000000)    // Stimulus Port Register word acces
#define ITM_STIM_U8  (*(volatile         char*)0xE0000000)    // Stimulus Port Register byte acces
#define ITM_ENA      (*(volatile unsigned int*)0xE0000E00)    // Trace Enable Ports Register
#define ITM_TCR      (*(volatile unsigned int*)0xE0000E80)    // Trace control register
 
/*********************************************************************
*
*       SWO_PrintChar()
*
* Function description
*   Checks if SWO is set up. If it is not, return,
*    to avoid program hangs if no debugger is connected.
*   If it is set up, print a character to the ITM_STIM register
*    in order to provide data for SWO.
* Parameters
*   c:    The Character to be printed.
* Notes
*   Additional checks for device specific registers can be added.
*/
void SWO_PrintChar(char c) {
  volatile int timeout;
  //
  // Check if ITM_TCR.ITMENA is set
  //
  if ((ITM_TCR & 1) == 0) {
    return;
  }
  //
  // Check if stimulus port is enabled
  //
  if ((ITM_ENA & 1) == 0) {
    return;
  }
  //
  // Wait until STIMx is ready,
  // then send data
  //
  timeout = 5000; /* arbitrary timeout value */
  while ((ITM_STIM_U8 & 1) == 0)
  {
	/* Wait until STIMx is ready, then send data */
	timeout--;
	if (timeout==0) {
		return; /* not able to send */
	}
  }
  ITM_STIM_U8 = c;
}
 
/*********************************************************************
*
*       SWO_PrintString()
*
* Function description
*   Print a string via SWO.
*
*/
void SWO_PrintString(const char *s) {
  //
  // Print out character per character
  //
  while (*s) {
    SWO_PrintChar(*s++);
  }
}

To test the SWO output with your application include the SWO.h header file in your source and call the utility function to emit some sample trace output:

:
#include "board.h"
#include "peripherals.h"
#include "SWO.h"
:
SWO_PrintString("Hello world from SWO!");

Build the application and flash this to the board. When the board starts-up and halts with a breakpoint on main() then move on to the next step.

Configuring MCUXpresso for SWO ITM Console trace output​

With MCUXpresso the Analysis menu gives access to the advanced trace windows for the IDE:

First the SWO Trace Config needs to be configured to detect the link speed and add the ITM Console. You may need to detect the SWO link by clicking Change -> Detect :

If you select the SWO ITM Console and start the ITM Trace then allow the debugger to run then you should see the trace output appear:

Note you can stop and start the SWO trace without impacting the performance of the target application. You can even capture and save the ITM Trace Console contents to a file.

Similarly if you are using J-Link with IAR then the i.MXRT1060 setup below can be used to emit the SWO trace data in real time and be viewed through the Terminal IO window and also logged to the local file ITM.log. Note the default CPU CPU clock speed of 528MHz.

J-Link -> SWO Configuration:

Using SWO Tracing for Storyboard Performance logging:​

Now that the SWO back-channel is proven we can reconfigure the project to use the SWO as the default route for the STDOUT and STDERR pipes so that application printf() content will appear out of on SWO ITM Console.

With NEWLIB this can be achieved by re-implementing the _write() and re-entrant _write_r() library functions. Inserting the code snippet below will override the default library behaviour:

#ifdef __NEWLIB__
/* For GCC compiler revise _write() and _write_r() functions for printf functionality 
 * which enables the output from STDOUT and STDERR */
int _write(int file, char *ptr, int len)
{
    int i;
	file = file;
 
	//
	// Print out character per character
	//
    for( i=0; i < len; i++) {
    	SWO_PrintChar( ptr[i] );
    }
    return len;
}
 
_ssize_t _write_r (struct _reent *reent, int fd, const void *ptr, size_t len)
{
	portENTER_CRITICAL();
        _write(fd, ptr,len);
	portEXIT_CRITICAL();
	return (_ssize_t)len;
}
#endif

Now it will be possible to enable performance tracing from a Storyboard application which emits PERF data to STDOUT and now have this streamed out of the SWO ITM Console:

void
sbengine_main_task(void *arg) {
	char *args[20];
	int n = 0;
 
	// args[n++] = "screen_mgr";
	// args[n++] = "fps";
            
        args[n++] = "logger";
        args[n++] = "perf=1";

The same can be done for Storyboard debug output by elevating the trace verbosity:

static void
run_storyboard_app(const char *bundle, int flags, char * const *options, int option_count) {
	
	app = gr_application_create_args(bundle, flags, options, option_count);
 
	if(!app) {
		return;
	}
 
	// Sets the application logging verbosity. For more logging verbosities, refer to gre.h.
	gr_application_debug(app, GR_DEBUG_CMD_VERBOSITY, GR_LOG_EVENT2 );//GR_LOG_ERROR);

Issues:

There is a slight issue with MCUXpresso where the trace line output gets truncated every 100ms or so which will need some investigation as to the cause.