Here are the details of my testing so far.
Current Test: 3x Test files, each with multiple run blocks. Two test files crafted to deliberately error. 12 ‘tests’ in total across the 3 files.
XML Output:
<?xml version="1.0" encoding="UTF-8"?><testsuites>
<testsuite name="tests\api_connection_azureblob.tftest.hcl" tests="5" skipped="0" failures="1" errors="0">
<testcase name="setup"></testcase>
<testcase name="create_storage_account"></testcase>
<testcase name="azureblob-managed_id"></testcase>
<testcase name="azureblob-service_principal"></testcase>
<testcase name="azureblob-access_key">
<failure message="Test run failed"></failure>
<system-err><![CDATA[
Error: Test assertion failed
on tests\api_connection_azureblob.tftest.hcl line 87:
(source code not available)
api type name did not match expected
]]></system-err>
</testcase>
</testsuite>
<testsuite name="tests\api_connection_azurefile.tftest.hcl" tests="3" skipped="0" failures="0" errors="0">
<testcase name="setup"></testcase>
<testcase name="create_storage_account"></testcase>
<testcase name="azurefile-access_key"></testcase>
</testsuite>
<testsuite name="tests\api_connection_keyvault.tftest.hcl" tests="4" skipped="0" failures="1" errors="0">
<testcase name="setup"></testcase>
<testcase name="create_keyvault"></testcase>
<testcase name="keyvault-managed_id">
<failure message="Test run failed"></failure>
<system-err><![CDATA[
Error: Test assertion failed
on tests\api_connection_keyvault.tftest.hcl line 52:
(source code not available)
[{"error":{"code":"ConfigurationNeeded","message":"Parameter value
missing."},"status":"Error","target":"vaultName"}]
]]></system-err>
</testcase>
<testcase name="keyvault-service_principal"></testcase>
</testsuite>
</testsuites>
This was rendered in the Azure Devops Tests page as follows:
Overall test page showing all tests and detail of the high-level test run
Showing detail of a successful test:
Showing detail of a failed test :
Detail of attached log showing error for the same test:
Some observations based upon the above and also the Azure Devops Documentation for result formats mapping
The separate test suites (The individual test files) appear to be ignored when in a single XML file and all tests grouped together under the test-run level only - I will do some manual splitting of the same XMLfile into an XML file per test suite (.tftest) and run through a dummy pipeline to see what occurs and report back.
The start & completed datetime and durations, as not logged in the XML are all reported as 0 (duration) or as the completion time of all of the tests in the pipeline (start/end) - Based upon the documentation, for these to be displayed correctly the following are required:
For the ‘test run’
- Date started
/testsuites/testsuite.Attributes["timestamp"].Value
- Date completed
/testsuites/testsuite.Attributes["timestamp"].Value + SUM(/testsuites/testsuite/testcase.Attributes["time"].Value) for all test cases in the test suite
For each test result:
- Date started
/testsuites/testsuite.Attributes["timestamp"].Value
- Date completed
/testsuites/testsuite.Attributes["timestamp"].Value + /testsuites/testsuite/testcase.Attributes["time"].Value
- Duration
/testsuites/testsuite/testcase/.Attributes["time"].Value
The reported error message Test Run Failed should probably contain the returned error from the actual test. Perhaps (taking from the output log and reviewing the Machine Readable UI output to see if these are separate elements available): Test assertion failed: api type name did not match expected this appears to be needed in the following XML elements:
Error message:
/Testsuites/testsuite/testcase/failure.Attributes["message"].Value Or /Testsuites/testsuite/testcase/error.Attributes["message"].Value Or /Testsuites/testsuite/testcase/skipped.Attributes["message"].Value
The Stack Trace field in ADO could be populated with the current value used for the <system-err> node via the following:
Stack trace /Testsuites/testsuite/testcase/failure.InnerText Or /Testsuites/testsuite/testcase/error.InnerText
And would contain:
Error: Test assertion failed
on tests\api_connection_azureblob.tftest.hcl line 87:
(source code not available)
api type name did not match expected
The content of the <system-err> should contain the 'console error log:
Console error log: /Testsuites/testsuite/testcase/system-err
And if appropriate the normal console log output should be populated via:
Console log: /Testsuites/testsuite/testcase/system-out
I realise there is a lot here - happy to discuss item by item if needed. I will submit, in a separate reply, details of what occurs if each .tftest maps to a single test.xml file as opposed to them being within a single file.
Cheers!
If this gets stabilized then it’ll have better test coverage to catch silly mistakes like this, but our experiment dev process is optimized more for faster iteration.
