Tools and template to generate Linaro Test Plans and Test Reports

Python dependencies
===================

Both tools use Jinja2 and Pytextile, as such it is recommended to
install these tools with:

$ pip install jinja2 textile

Test plan
=========

To generate a test plan, we use the tool testplan2html.py from the
Linaro test-definitions project, which needs to be installed first:

$ git clone https://github.com/Linaro/test-definitions.git
$ export PATH=<path to Linaro test-definitions/plans>:$PATH

To generate a test plan, you can run:

$ testplan2html.py -s \
  -f openembedded/boards/rb5/Testplan.yaml \
  --testplan-template-name Testplan.textile \
  --templates-directory openembedded/boards/rb5/

Test Report
===========

The tool scripts/testreport.py can be used to generate a test
report. It reads tests data from a Squad server, such as Linaro
qa-reports.linaro.org, and provides tests data to a Jinja2 template.

This tool requires Linaro Squad_client Python library. To install locally:

$ git clone https://github.com/Linaro/squad-client.git
$ cd squad-client
$ pip install .

Squad-client will connect to a Squad server, and retrieve data. An API
token is required to connect, you will need to create your token in
the Settings of your Squad server account.

Here is a example to run testreport.py:

$ ./scripts/testreport.py \
  --group qcomlt \
  --project openembedded-rpb-dunfell \
  --build 163 \
  --token <TOKEN> \
  --template openembedded/boards/rb5/Testreport.textile

You can reduce the tests data to show only the tests for a specific
'environment' with --environment argument.

Upload Test results
===================

Tests executed locally/manually, can be published to the Linaro Squad
server, so that they can be later included in a Test report. This
would allow to combine tests run from multiple users, or automated
tests run in LAVA and manual tests.

Squad-client can be used to upload test results, either directly, or
using its Python API. Assuming we have a .csv results file as input
file, we can use scripts/sendresults.py to upload them (assuming the
user is allow to push data into the Squad server project).

$ ./scripts/sendresults.py \
  --group <GROUP> \
  --project <PROJECT> \
  --build <BUILD> \
  --environment <ENV> \
  --token <TOKEN> \
  --results result.csv

Only results in .csv format are supported with the following CSV
format: "name,test_case_id,result,measurement".