Skip to content

Testing In A Box

flowchart LR
    A(Server: hosts GitLab) -->|triggers job on runner| B(GitLab Runner: hosts GitLab CI)
    B -->|deploys OpenQA worker and OS| C(System Under Test)
    C -->|returns test results to GitLab CI and OpenQA| B
    B -->|returns test results to OpenQA| A

The above works by having a server which will host GitLab, and then deploy a runner. This runner will host the GitLab CI, and deploy an OpenQA worker as well as the OS to be put on the system under test. The runner interacts with the system under test, and the runner will calculate the test results based upon the systems behaviour, and returned data. The runner will return the results to the OpenQA and GitLab server so they can report the test results to anyone logged into the server.

To learn how to deploy the playbooks, visit deployment.md.

Introduction

Testing in a box, or T.I.A.B., is Codethink's is a solution for automated testing. This repository holds all the Ansible scripts to provision the box or device to be ready for use in testing.

T.I.A.B. Software Components

All components used in this project are FOSS.

T.I.A.B. integrates the following softwares:

  • Gitlab server and runner: so you can run a CI locally to cover white-box testing for your product code.
  • OpenQA webUI and worker: aims for end-user black-box software tests, it's ideal for making sure your software GUI is working as expected.
  • Quality Assurance Daemon: Codethink's contribution to extend OpenQA's ability to hardware testing, it eliminates the need for physical interactions with your hardware under testing, e.g, a touch screen.
  • ser2net: Automatically start certain tasks when some USB devices are connected to the box.
  • libvirt: Allows you to run virtual machines inside the box.
  • CLI tool: Allow easy management of the box power supply

User documentation

  • Gitlab Instance: You can use it just like a normal Gitlab, with the full ability to create/push repos and use its CI for white box testing.

  • OpenQA: The case for OpenQA is a bit complicated. To properly use it, you will need an OS image file an create your own tests and needles. A good starting point would be this youtube tutorial. If you have all of them ready, to trigger the actual tests, you can docker exec -it <name> bash into OpenQA worker container and execute:

openqa-cli api -X POST isos \
    --param-file SCENARIO_DEFINITIONS_YAML=</path/to/scenario_definitions.yaml> \
    ISO=<os.iso> \
    CASEDIR=<path/to/tests/folder> \
    NEEDLES_DIR=<path/to/needle/folder> \
    VERSION=<version> \
    DISTRI=<distri> \
    FLAVOR=<flavor> \
    ARCH=<arch> \
    --host http://<box_ip>

  • Q.A.D.: It's best to use Q.A.D. alongside with OpenQA to maximize its ability. Codethink has developed a Q.A.D. backend for os-autoinst, make sure you follow the steps here to first set the backend up and then you can use Q.A.D. within OpenQA context for automated hardware testing.

  • libvirt host: The deployment script sets up a virt pool and a virt bridge nat network for you. As a user, if you decide to start a virtual machine, then you should first download the corresponding iso file, then in the box shell do:

sudo virt-install \
    --name=<virtual machine name> \
    --memory=<memory in MiB> \
    --vcpus=<number of virtual cpus> \
    --cpu=<host|host-passthrough|coreduo,+2apic> \
    -c <path/to/downloaded/iso/file> \
    --os-type=<os type> \
    --os-variant=<os variant> \
    --disk=pool=<pool name>,size=<size in GiB>,format=qcow2 \
    -w networ=<network name> \
    --graphics=vnc,port=<vnc port>,listen=0.0.0.0

A summary of default ports used by T.I.A.B.

  • 80 and 443: OpenQA
  • 81: Gitlab
  • 6001: ser2net