This tutorial shows how you can connect software component / unit test cases to SystemWeaver requirements and generate requirement coverage test reports and store them on a git repository. In the example we are implementing a simple software component performing basic mathematical operations in Python, although the methodology and some of the scripts can be reused. Figure 1 below describes the overall process.
Figure 1 Overall process
In a Python file called unit_test.py (see attachments) we have created four test cases, verifying four requirements for the basic math operations, in the version one of the Math operations component (Figure 2). The connection between the Python test cases and requirements in SystemWeaver are made through referring to the SystemWeaverID of the requirements in the following format:
We then made a second version of the software component in SystemWeaver and added two new requirements, removed one requirement and took a new version of another requirement, as seen in Figure 3. The entire process is also described in the video at the bottom of the page.
Figure 2 Math operations software component Version 1 in SystemWeaver
Figure 3 Math operations software component Version 2 in SystemWeaver
At this point we check in the code again, which triggers the build process in the Azure build machine as seen in Figure 4. The build script (configuration of Azure is attached in azure-pipelines.yml) executes all test cases and generates a json file called converted_unit_test.json (see attachments).
As you might notice in the Azure configuration, the test results are initially exported to a file called unit_test.json and then transformed to converted_unit_test.json using the Python script convert_json.py to the standard SystemWeaver format.
Figure 4 The build script in Azure
In the Git repository, we also define a file called config.json (see attachments) which points to the current software component ID in SystemWeaver (math operation (v2)). The config file also specifies the address of the SystemWeaver REST API server and credentials to log into the server. Another script in the build process (read_sw_requirements.py - attached) uses this data to read requirements from SystemWeaver and generates a file called reqs.json – attached. A last script (create_pdf.py - attached) then utilized reqs.json and converted_unit_test.json to generate a pdf report which list all requirements under the component and shows the traceability to the test cases on each requirement and also presents the test results (testReport.pdf - attached) which looks like Figure 5.
Figure 5 Requirements Test Pdf Report
In the report,
- Green signifies that a requirement under the component has been tested.
- Yellow signifies that a version of the requirement that is not under the current component has been tested. The developer should then check the updated requirement and do the necessary changes in the test script and update the reference to the correct version of the requirement.
- Red signifies that a requirement has been tested that is not in the scope of the component. The link should be removed from the test case or changed to a correct requirement ID.
- White signifies that the requirement has no coverage on test side. A test method should be specified and linked to the requirement to assure full requirements test coverage.
The same principle can be used in any build environment and some of the scripts shall be reused. This principle also works for code to requirements coverage.
All these step are described in the video below.