Skip to content

Introduction

Codethink has been developing a number of tools and processes in order to automate as much of the testing pipeline for embedded systems as possible. This includes everything from virtual hardware using QEMU, to testing on the hardware itself.

We've used a number of different technologies on our journey including:

  • Gitlab CI to run tests on daily builds or upon new merges.
  • OpenQA to run UI and graphics tests.
  • QAD to simulate keyboard, touch and gesture interactions.
  • CAN interfaces to simulate a vehicle on the road.
  • USB switches to control multiple peripherals to the hardware under test.

Typically this has taken considerable time to set up, wading through the red tape of client's IT departments to gain access to client infrastructure, understand their build, CI and Test setup and integrating the tools required to provide useful data from testing.

This is where Testing In A Box (TIAB) comes in. We've put everything into one box with minimal set up. So that setting up these testing frameworks is less painful and more time efficient.

TIAB hardware image

Testing-in-a-Box Setup Diagram

TIAB is a fully loaded toolbox that bridges between CI/CD pipeline and the hardware being tested. It can be set up as GitLab runner that is triggered by a build pipeline, and an OpenQA worker to execute UI test. Different features of the hardware can be tested and verified with different TIAB modules. Below is a diagram showing areas of the system can be tested with specific TIAB modules.

Software
SUT Control
Gitlab runner, OpenQA
Can simulation
Hardware
GPIO, I2C, serial, ssh
Can Interface
USB Switch
Bluetooth Interface
Audio Interface
Gitlab Pipeline
Testing-in-a-box
System Under Test
System Features
Hardware state
Software state, update
QAD
Display, KVM and touch screen interactions
Can signals
USB devices (storage, phone)
Bluetooth features
Audio input/output

Technologies used

Ansible

Ansible scripts are used to provision the host to run tests on OpenQA and run Continuous Integration/Continuous Deployment(CI/CD) tasks on GitLab.

  • OpenQA
    • OpenSUSE's automated testing tool.
  • QAD
    • Quality Assurance Daemon.
  • ser2net
    • Serial to Network proxy. A program for allowing connections between gensio accepters and gensio connector.
  • GitLab
    • GitLab runner for Continuous Integration and Deployment.
    • There is also a script to setup a self-hosted GitLab instance if required.

Hardware

This is to house all the hardware needed into one box. This includes:

  • USB hub
  • USB switch
    • A bidirectional USB-C switcher. It can be pragmatically controlled to switch connection between USB devices and hosts.
  • USB HID emulation
    • Emulating HID input from host to device to test against.
  • CAN interface
    • Communication with CAN devices.

Open Source Projects

Here are the following Open Source repositories for TIAB:

  • Ansible scripts
    • All the Ansible scripts needed to provision a TIAB host hardware.
  • Hardware
    • CAD models and Schematics of TIAB case and I/O PCB board.
  • ESP32-C3 firmware
    • The firmware to be flashed onto the ESP32-C3 chip on the I/O board.
  • TIAB Tools
    • Tools to assist in flashing and configuring the TIAB I/O board.