| Title: | Convert 'tinytest' Output to JUnit XML |
|---|---|
| Description: | Unit testing is a solid component of automated CI/CD pipelines. 'tinytest' - a lightweight, zero-dependency alternative to 'testthat' was developed. To be able to integrate 'tinytests' results into common CI/CD systems the 'tinytests'-object is converted to JUnit XML format. 'tinytest2JUnit' enables this conversion while staying lightweight, having only 'tinytest' as its dependency. |
| Authors: | Anne-Katrin Hess [aut], Lennart Tuijnder [aut, cre] |
| Maintainer: | Lennart Tuijnder <[email protected]> |
| License: | GPL-3 |
| Version: | 1.1.2-9000 |
| Built: | 2025-01-30 03:06:04 UTC |
| Source: | https://github.com/openanalytics/tinytest2junit |
An object of class tinytests2JUnit. Note the plurar. A subclass of tinytest::tinytests()
containing extra info recordings that are used in the export to JUnit.
## S3 method for class 'tinytests2JUnit' x[i]## S3 method for class 'tinytests2JUnit' x[i]
x |
|
i |
object to subset the |
Following details are recorded when running the tests files and stored as additional attributes to the object:
fileDurations: named-numeric(n). Names = filename of tests files, value = duration
in seconds on how long the test file took to run.
fileTimestamps: named-character(n). Names = filename of tests files, value = timestamp
when the test was invoked.
fileHostnames: named-character(n). Names = filename of tests files, value = The hostname
of the system that ran the tests. (Usefull in combination with clusters).
disabled: character. A character vector of filenames where no tests were ran.
They are flagged as disabled tests.
Convert any will character vector to a single length character vector
charVecToSingleLength(x)charVecToSingleLength(x)
x |
a |
x a single-length character vector Non-NA
tinytest2JUnit:::charVecToSingleLength(c("Hello", "World")) # -> "HelloWorld" tinytest2JUnit:::charVecToSingleLength(c("Hello", NA_character_)) # -> "HelloNA" tinytest2JUnit:::charVecToSingleLength(character(0L)) # -> ""tinytest2JUnit:::charVecToSingleLength(c("Hello", "World")) # -> "HelloWorld" tinytest2JUnit:::charVecToSingleLength(c("Hello", NA_character_)) # -> "HelloNA" tinytest2JUnit:::charVecToSingleLength(character(0L)) # -> ""
Helper function specifying the 'classname' attribute of the testcase tag. Currently equal to the fileName. The classname is already xml escaped.
classnameTestcase(tinytest)classnameTestcase(tinytest)
tinytest |
a |
character(1) being the 'classname'
Helper function generating the body of a failure description tag! Attempts to mimic the print behaviour of a tinytest object.
constructFailureDescription(tinytest)constructFailureDescription(tinytest)
tinytest |
A |
character(1) being the failure tag description body. This string is already propery
xml escaped.
</testcase> tagConstruct JUnit </testcase> tag based on a single tinytest result.
constructTestcaseTag(tinytest)constructTestcaseTag(tinytest)
tinytest |
a |
XMLtag: with tag-name = tinytest and contains the test result per test.
</testsuites> tagConvert the tinytests2Junit or tinytests-object containing test across
possibly multiple files into a JUnit </testsuites> tag. More details are reported to the
JUnit if a tinytests2JUnit object compared to the native tinytests object.
constructTestsuitesTag(testResults)constructTestsuitesTag(testResults)
testResults |
|
Reference for JUnit XML format: https://llg.cubic.org/docs/junit/
See details runTestDir() which additional info is recorded.
XMLtag: with tag-name = </testsuites>. This is the root of the JUnit XML document.
</testsuite> tagConstruct the </testsuite> tag of a tinytest, given all the tinytest results
from a single test file.
constructTestsuiteTag(testsFile, id)constructTestsuiteTag(testsFile, id)
testsFile |
|
id |
|
In case a tinytest2JUnit is provided following additional info can be reported:
testsuite duration.
timestamp when the testsuite was performed.
hostname where the testsuite was ran.
XMLtag: with tag-name = </testsuite>
that contains all the test results per test file.
Construct a testcase-tag for an error test.
errorTestcaseTag(tinytest)errorTestcaseTag(tinytest)
tinytest |
a |
a testase XMLtag
Escape the characters &,",',<,>
escapeXml(x)escapeXml(x)
x |
a |
The same character vector x but xml escaped.
https://stackoverflow.com/a/1091953/10415129
Escape the characters '<' and & in a character vector meant to be xml-text content.
escapeXmlText(x)escapeXmlText(x)
x |
a |
The same character vector x but xml text escaped.
Construct a testcase-tag for a failed test.
failureTestcaseTag(tinytest)failureTestcaseTag(tinytest)
tinytest |
a |
a testase XMLtag
Format S3 method for the XMLtag-class
## S3 method for class 'XMLtag' format(x, level = 0, ...)## S3 method for class 'XMLtag' format(x, level = 0, ...)
x |
an |
level |
print depth level. For each level 2 spaces are added to the left. The content of a tag is automatically indented with 1 level. Except for text-content (see details). |
... |
to ignore |
Note, text content does not get indented or put on a new line, since whites space characters are of relevance.
character(1) vector of the formatted XML tag.
Help function to generate the formatted string for a single stack frame.
formattedFrame(framecall, frameN, hasSrcInfo, dirName, fileName, lineNr)formattedFrame(framecall, frameN, hasSrcInfo, dirName, fileName, lineNr)
framecall |
|
frameN |
|
hasSrcInfo |
|
dirName |
|
fileName |
|
lineNr |
|
For a given frame in the stack the string is formatted as follows (substitute the arguments
between the curly braces)
{frameN}| {call[1]}
{frameN}| {call[2]}
{frameN}| {call[3]}
{frameN}| {call[3]}
---> at File={dirName/fileName} Line={line}:
For example for only a single line error:
1: stop("This is a crash")
---> at File=R/my_r_code_file.R Line=234
Currently all call lines are printed for a given stack. The last line with source file info only printed if hasSrcInfo=TRUE. Else it is ommited.
characer(1) the formatted character string containing info of a single frame in the
stackstrace
getFormattedStacktrace is a helper function that formats stacktrace for uncaught errors from a tinytest run file.
getFormattedStacktrace()getFormattedStacktrace()
This function assumes that it directly called from a withCallingHandler error handling function! This fact is then used to remove the calling handler info from the stack such that stack directly starts from where the error was thrown.
The function also removes the calls from the stack that involve
executing the test_file. The internals of runTestDir and tinytest are not of intererst.
And the highest level of the stack to consider is the top level of the test_file.
Note, this does mean that errors that occur on the top-level of the test file will not have a a stacktrace! For example: "Error: object 'x' not found" where x is attempted to be resolved at the root levels
character(1) a single length character string suitable to be printed to the end-user.
In case of no stakctrace (eg the error occured at root level of the script) NA_character_
is returned!
Test if single length character non NA.
isSingleLengthCharNonNA(x)isSingleLengthCharNonNA(x)
x |
object to test. |
logical(1)
ltuijnder
Helper function to construct the name of a testcase. Note, the charater is already xml escaped.
nameTestcase(tinytest)nameTestcase(tinytest)
tinytest |
a |
character(1) the testcase name to use for this tinytest object.
Construct a testcase-tag for a passed test.
passedTestcaseTag(tinytest)passedTestcaseTag(tinytest)
tinytest |
a |
a testase XMLtag
Print method for XMLtag class.
## S3 method for class 'XMLtag' print(x, ...)## S3 method for class 'XMLtag' print(x, ...)
x |
a |
... |
to be ignored |
invisibly the string that was printed to stdout.
runTestDir() is a drop in replacement for tinytest::run_test_dir() with the key
difference that errors thrown from within a test file are caught and get reported with a
a stacktrace in the JUnit report. In addition, some extra metrics are recored for the JUnit
report, such as: timestamp, test duration, hostname and if tests are disabled (see details for
more info).
runTestDir( dir = "inst/tinytest", at_home = FALSE, pattern = "^test.*\\.[rR]$", cluster = NULL, lc_collate = getOption("tt.collate", NA), ... )runTestDir( dir = "inst/tinytest", at_home = FALSE, pattern = "^test.*\\.[rR]$", cluster = NULL, lc_collate = getOption("tt.collate", NA), ... )
dir |
|
at_home |
|
pattern |
|
cluster |
A |
lc_collate |
|
... |
Arguments passed on to |
runTestDir() is meant as a CI-friendly alternative to the native tinytest::run_test_dir().
It catches errors that are raised in the tests files and adds them as a "failed" tinytest
in the output.
tinytest::run_test_dir() would have let the error bubble up, stop the testing process and
not report any failures from other tests. One is then also forced to look into the
logs of the CI to see what the error was. The output of runTestDir() in combination with
writeJUnit() will present you the error in the JUnit togheter with a stack trace. Next to the
test results of the other files that ran without a problem.
If you prefer the behaviour of tinytest::run_test_dir() you can still use it in combination
writeJUnit().
Caught errors are returned in the output as as sub-class of tinytest object. This is however
considred implemenation detail and can be subject to change.
Note, function arguments explicilty listed in tinytest::run_test_dir() but not here can still
still be provided via ...
A tinytests2Junit object to be provided to the writeJUnit() function.
The returned object is a tinytests2JUnit object (note the plural). This object
contains additional info compared to a tinytests object that is used in the JUnit report.
The following additional info will get reported:
The timestamp per test file on when it got invoked.
The test duration per test file.
The system hostname per test file on where it got invoked. This is mainly of interests for
different clusters.
If a test file is disabled. A test file is considered disabled if no tests occur with in the file. It is then assumed that at the top of file some conditional statement made the test file exist early.
tinytest::run_test_dir() for how the function is inteded to behave.
writeJUnit() where it is expected that the output of this function to be provided to.
testPackage() for an higher-level function to simply test a package.
# Run tests with `tinytest` dirWithTests <- system.file("example_tests/multiple_files",package = "tinytest2JUnit") testresults <- runTestDir(dirWithTests) writeJUnit(testresults) # Writes content to stdout# Run tests with `tinytest` dirWithTests <- system.file("example_tests/multiple_files",package = "tinytest2JUnit") testresults <- runTestDir(dirWithTests) writeJUnit(testresults) # Writes content to stdout
Internal wrapper arround tinytest::run_test_file() that records the test duration and
catches uncaught errors and logs the stacktrace of where the error occured.
runTestFile(file, ...)runTestFile(file, ...)
file |
|
... |
arguments passed on to |
The response is a subclass of the tinytests object called: tinytests2Junit
object which captures additional info for the reporting to JUnit:
Duration to run the file.
Timestamp when the test was run.
hostname of the computer where it was ran on.
The caught error is turned into a subclass uncaught-error of tinytest. This is implementation detail and only to be understood by constructJUnitTag.
If an error occured it is captured and uncaught-error object (subclass of tinytest) is
returned in the tinytests object.
This tinytest object represents a "failed" tests that will get reported as an Error in the
JUnit. Various aspects of the error are also captured like the the stacktrace.
a tinytests2JUnit object (being a subclass of tinytest object).
Construct a testcase-tag for a side-effect test.
sideeffectTestcaseTag(tinytest)sideeffectTestcaseTag(tinytest)
tinytest |
a |
a testase XMLtag
Create a list object that roughly mimics the behaviour of a simplistic XML tag element. Supported are XML tag-name, tag-attributes and tag-content.
tag(name, attributes = list(), content = list())tag(name, attributes = list(), content = list())
name |
|
attributes |
|
content |
|
If a character vector is in the content it is converted to a single-length character vector.
See charVecToSingleLength()
Mixed content eg. a text string and a child xml tag next to each other is syntaxtically allows. In practices it does not occur for XML that is schema formatted with XSD (like JUnit). So for simplicity it is not supported here.
a XMLtag-object.
Run all tests of a package and report the results as JUnit xml. This function
can be seen as a drop in replacement for tinytest::test_package() but with a
key difference that uncaught errors will be catched and reported JUnit!
This function is intended to be used in a test stage of a CI build.
testPackage( pkgname, file = stdout(), errorOnFailure = TRUE, testdir = "tinytest", lib.loc = NULL, at_home = FALSE, ncpu = NULL, ... )testPackage( pkgname, file = stdout(), errorOnFailure = TRUE, testdir = "tinytest", lib.loc = NULL, at_home = FALSE, ncpu = NULL, ... )
pkgname |
|
file |
|
errorOnFailure |
|
testdir |
|
lib.loc |
|
at_home |
|
ncpu |
|
... |
Extra arguments passed on to |
testPackage() is meant as a CI-friendly alternative to the native tinytest::test_package().
Next to directly reporting the tests results in a JUnit xml format, it also catches errors that
are raised in the tests files and reports them as "error" in the JUnit.
tinytest::test_package() would have let the error bubble up, stop the testing
process and not report any failures from other test files. One is then also forced
to look into the logs of the CI to see what the error was. testPackage() presents you that
error in the JUnit with a stacktrace. Next to all the test results of the other files that
ran without a problem.
If you prefer the behaviour from tinytest::test_package(), you can still use it in
combination with writeJUnit() if all tests results pass.
Just like tinytest::test_package() an error is raised if at least one failure occured during
testing. Obviously catched errors are also seen as failures. This error is raised
after the test results have been written away to the file, such that your CI can still pick
it up and report the failure.
The error raising is done as a convenience to stop the CI from continue if test-failure occured.
You can opt-out of this behaviour by setting the errorOnFailure parameter to FALSE. Then a
case tinytests2JUnit object is returned (a sub-class of tinytests object containing addition
info for the JUnit).
Caught errors are also captured in this object as tinytest-objects. They
actually have a special sub-class but this is considered an internal implemenation detail.
testPackage() is NOT meant to be called from within your tests/tinytests.R file! Tests
invoked by R CMD Check or on CRAN should still make use of tinytest::test_package().
This function is only meant to be called from within a testing step in your CI to
report the test results in an JUnit xml format.
If errorOnFailure = FALSE, a tinytests2JUnit object (a subclass of tinytests
object that captures more info for export to JUnit). Else, an error is raised if at least
on failure occurs. Meant as convenience to automatically stop the CI build.
Side effects are registered as 'passed' tests in the JUnit output and have been given a status "SIDE-EFFECT". The call and diff is also returned in the standard-output of the testcase tag.
They are not considred failures and would thus not stop a pipeline.
To comply the the JUnit specification the tests results are adapted as follows:
A single test run tinytests is mapped to a <testsuites> tag.
All tinytest results from a single file are mapped to a single <testsuite> tag.
The name of the testsuite is equal to the test file name (without the file suffix)
An individual tinytest object (eg. a single except_* exception test) is mapped to a
<testcase> tag.
The name of the testcase is equal to the fileName + Line specification of where the expect statement is performed + the info.
For reference: https://llg.cubic.org/docs/junit/
runTestDir() and tinytest::test_package().
tmpFile <- tempfile(fileext = ".xml") testPackage("tinytest", file = tmpFile, verbose = 0)tmpFile <- tempfile(fileext = ".xml") testPackage("tinytest", file = tmpFile, verbose = 0)
tinytests-object into JUnit xml report.Write the tinytests-object to a JUnit XML reporting file. If a tinytests2JUnit is provided
(returned by runTestDir()) more info will get reported.
writeJUnit(tinytests, file = stdout(), overwrite = TRUE)writeJUnit(tinytests, file = stdout(), overwrite = TRUE)
tinytests |
|
file |
|
overwrite |
|
invisible(TRUE) Might get another use in the future.
In case of overwrite = FALSE and the file already exists an error is thrown.
Side effects are registered as 'passed' tests in the JUnit output and have been given a status "SIDE-EFFECT". The call and diff is also returned in the standard-output of the testcase tag.
They are not considred failures and would thus not stop a pipeline.
To comply the the JUnit specification the tests results are adapted as follows:
A single test run tinytests is mapped to a <testsuites> tag.
All tinytest results from a single file are mapped to a single <testsuite> tag.
The name of the testsuite is equal to the test file name (without the file suffix)
An individual tinytest object (eg. a single except_* exception test) is mapped to a
<testcase> tag.
The name of the testcase is equal to the fileName + Line specification of where the expect statement is performed + the info.
For reference: https://llg.cubic.org/docs/junit/
The JUnit XML report format: https://llg.cubic.org/docs/junit/
# Run tests with `tinytest` dirWithTests <- system.file("example_tests/multiple_files",package = "tinytest2JUnit") testresults <- runTestDir(dirWithTests) writeJUnit(testresults) # Writes content to stdout tmpFile <- tempfile(fileext = ".xml") writeJUnit(tinytests = testresults, file = tmpFile)# Run tests with `tinytest` dirWithTests <- system.file("example_tests/multiple_files",package = "tinytest2JUnit") testresults <- runTestDir(dirWithTests) writeJUnit(testresults) # Writes content to stdout tmpFile <- tempfile(fileext = ".xml") writeJUnit(tinytests = testresults, file = tmpFile)