This ongoing series of articles will explain how to create a code coverage report for Android device testing running the Firebase Test Lab.
In Part 1 of this series, we saw how:
- Generate XML and HTML reports to cover testing code on the device
- Run on API 28, API 29, and API 30s in the Firebase testing lab
- Support for Android apps targeting API 28, API 29 and API 30
In Part 2, we will expand this with:
- Supports Android Test Orchestrator
- Combining results with unit tests outside the device
- Support for multi-module applications
Android Test Orchestra Is a tool that allows us to run tests on the device in independent processes from each other, so that a crash or incorrect test can not contaminate the results of other tests.
To better illustrate this, we will first give our classes a few more methods, and then add another test case so we can see the orchestra at work.
To run Orchestrator for our tests, we need to make some modifications to our module-level build.gradle
Finally, we will build the app (just like in the previous article)
./gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest
And perform our tests with our gcloud command by taking it one from the previous article and adding the --use-orchestrator
option.
gcloud firebase test android run
--type instrumentation
--use-orchestrator
--no-performance-metrics
--no-record-video
--app app/build/outputs/apk/debug/app-debug.apk
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
--device model=Pixel2,version=29,locale=en,orientation=portrait
--environment-variables coverage=true,coverageFile=/sdcard/Download/coverage.ec
--directories-to-pull /sdcard/Download
Then download the created cover file
mkdir app/build/outputs/coveragegsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/coverage.ec app/build/outputs/coverage
And produce a report
./gradlew jacocoReport
Opening the report, we see
Wait … why do we only see the coverage results from one of our tests?
The problem is that each test runs in its own process, and each process writes its coverage file to the same location specified in coverageFile
The parameter was passed to the gcloud command.
So we’ll replace coverageFile
With coverageFilePath
(Note: coverageFilePath
Must end in /
but directories-to-pull
Forbidden) so that each individual cover file will be written to the same directory. Our updated gcloud command is now
gcloud firebase test android run
--type instrumentation
--use-orchestrator
--no-performance-metrics
--no-record-video
--app app/build/outputs/apk/debug/app-debug.apk
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
--device model=Pixel2,version=29,locale=en,orientation=portrait
--environment-variables coverage=true,coverageFilePath=/sdcard/Download/
--directories-to-pull /sdcard/Download
Then, when we look at the objects folder we see each cover file stored separately.
We will need to update our download command to download all .ecs files:
mkdir app/build/outputs/coveragegsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/*.ec app/build/outputs/coverage
But the task of creating our reports can remain the same, since it is already searching all the .ecs files. running gradlew jacocoReport
, We now get the coverage we expect:
Adding unit tests outside the device to the mixture is relatively simple. We will first add an off-device test app/src/test/java/com/github128/coverage1/OffDeviceTests.kt
:
We can easily run it from the command line:
./gradlew testDebugUnitTest
And a cover file is created in app/build/jacoco/testDebugUnitTest.exec
.
If you pass the -Pcoverage
Option (thus defining testCoverageEnabled
To true
, As described in the previous article) then it will generate the .exc file in another location (app/build/outputs/unit_test_code_coverage/debugUnitTest/testDebugUnitTest.exec
) But more importantly it will not produce the file properly for library modules, which will cause problems once we get to the multi-module projects of Android. (This topic is being tracked in https://issuetracker.google.com/issues/210500600)
Anyway, now we just have to tweak the getExecutionData().setFrom
In ours jacocoReport
Work at our module level build.gradle
To pull cover files from this new location.
getExecutionData().setFrom(
fileTree(dir: "${buildDir}/outputs/coverage",
includes: ['*.ec']),
fileTree(dir: "${buildDir}/jacoco",
includes: ['*.exec'])
)
Our coverage report will now include the calls from our unit tests combined with the calls from our Firebase Test Lab. All our execution is now
./gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTestgcloud firebase test android run
--type instrumentation
--use-orchestrator
--no-performance-metrics
--no-record-video
--app app/build/outputs/apk/debug/app-debug.apk
--test app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
--device model=Pixel2,version=29,locale=en,orientation=portrait
--environment-variables coverage=true,coverageFilePath=/sdcard/Download/
--directories-to-pull /sdcard/Downloadmkdir app/build/outputs/coveragegsutil cp gs://<project-id>/<timestamp>/Pixel2-30-en-portrait/artifacts/*.ec app/build/outputs/coverage./gradle testDebugUnitTest./gradlew jacocoReport
Which will create the desired report of
There are two sections to support code coverage for a multi-module application: coverage reports for each individual module and a consolidated report that integrates the coverage of all modules.
First we will add a second module to our app. Like our application module, this new library module will include a pair of classes with several functions as well as in-device and out-of-device tests.
One approach is basically to duplicate what we did before, adding dependency on the jacoco plugin and a new task to the new module build.gradle
. A better solution, however, is to resume the dependency and task of the plugin into a separate Gradle script and then import it into build.gradle for both our main module and our new module. The separate script will then be
So we just need to change the build.gradle of our new library module by adding
apply from: '../module-jacoco.gradle'
As well as our standard
buildTypes {
debug {
testCoverageEnabled (project.hasProperty('coverage'))
}
}
Similarly, we can edit the build.gradle of our original module by removing the dependency on the jacoco plugin, deleting the jacocoReport
Work completely, adding the dependence on module-jacoco.gradle
.
At this point, we can activate the jacocoReport
Task in each module (after running the unit tests or downloading coverage reports from the Firebase test lab). Keep in mind that when running Firebase Test Lab, the APK was transferred to gcloud with the --app
The parameter should still be the APK generated from the application module. (That is app/build/outputs/apk/debug/app-debug.apk
)
cd app
../gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest
<gcloud to run tests, gsutil to download *.ec files>
../gradlew testDebugUnitTest
../gradlew jacocoReport
<view build/reports/jacoco/jacocoReport/html/index.html>cd ../library
../gradlew -Pcoverage clean assembleDebug assembleDebugAndroidTest
<gcloud to run tests, gsutil to download *.ec files>
../gradlew testDebugUnitTest
../gradlew jacocoReport
<view build/reports/jacoco/jacocoReport/html/index.html>
And the code coverage reports received for each module will accurately reflect the coverage of the unit tests combined with the coverage of the tests on the device activated in the Firebase testing laboratory.
We can put all of this into build.gradle at the project level, but as with individual module scripts, it’s a little cleaner to put it into its gradual script and then import it into build.gradle at the project level. We can follow the collection / flattening pattern to build the desired list of files or directories. The separate script at the project level will look like this
And we will only need to add a
apply from: 'project-jacoco.gradle'
To our build.gradle at the root of our project. Once it exists, after going through the steps mentioned above to create the .exc files for off-device testing and downloading the .ecc files for the Firebase Test Lab, we just need to go to the root of the project and run
./gradlew jacocoUnifiedReport
And then a report covering the entire project is seen in build/reports/jacoco/jacocoUnifiedReport/html/index.html
.
We have now achieved the requirements for this article. Looking back, these were the steps we needed to take:
- Make a few small changes to build.gradle for each module to run Android Test Orchestrator, and then move on
--use-orchestrator
Togcloud
Instruct the Firebase Test Lab to use it. - Because Android Test Orchestrator creates multiple .ec files, we’ll also need to change it
coverageFile
Parameter tocoverageFilePath
And then change our download phase to pull*.ec
and not onlycoverage.ec
. - To create the cover files for off-device unit testing, all we have to do is make sure that the jacoco plugin is included (already done) and do not pass the coverage when running the actual unit tests. (Because https://issuetracker.google.com/issues/210500600)
- To cleanly support the production of reports per module for multi-module applications, we have re-executed the
jacocoReport
A task from the build.gradle of each module and entered into the commonmodule-jacoco.gradle
file. - To support a uniform project-wide coverage report, we’ve added a new report
jacocoUnifiedReport
Task noproject-jacoco.gradle
File included in build.gradle at project level.
You can see a solution that combines all of these changes by observing https://github.com/Aidan128/CoverageExample1 And checks the git tag part_two
.
We now have the ability to generate code coverage reports for unit tests and Firebase Test Lab tests, as well as integrate them across the entire project. However, this process is cumbersome and requires many steps. In the third and final article in this series, we will see how to write this process, integrate it with Firebase Test Lab automation tools like flank, and finally integrate it with CI / CD systems such as gitlab.
Saravana Thiyargaraj wrote a great article Article (And build a great example on that github) Which were very useful. Thanks.