Running CTS from A to Z
CTS stands for Compatibility Test Suite. It is an automated tool that results decide if a particular device will get Google certification. An approval is required to use Play Store and related GMS services (Google apps like Hangouts) and have them pre-installed as part of their OS and use it as part of marketing strategy (icons on the box).
You could simply follow the steps from CTS Introduction website to get it up and running on your device based on this CTS manual, however, I prefer more automated approach and running it from Jenkins.
1 . First lets script downloading all necessary files:
# adding links to variables MEDIA="https://dl.google.com/dl/android/cts/android-cts-media-1.0.zip" # as an example will download latest version of CTS for android 4.4 running x86 CPU CTS_PACK="https://dl.google.com/dl/android/cts/android-cts-4.4_r3-linux_x86-x86.zip" # now lets download the media files mkdir ./android-cts-media wget -O ./android-cts-media/mediafiles.zip $MEDIA # download CTS files wget -O ./cts_pack.zip $CTS_PACK
2. Unzipping the files:
# extracting media files to a destination folder unzip ./android-cts-media/mediafiles.zip -d ./android-cts-media/ # removing *.zip file as its no longer required rm ./android-cts-media/mediafiles.zip # extracting CTS files (make new folder, remove *.zip) mkdir ./android-cts-kitkat-x86/ unzip ./cts_pack.zip -d ./android-cts-kitkat-x86/ rm ./cts_pack.zip
3. Copying required files on a device:
# installing APK on a device # $Device - use S/N of the device to target this particular one if multiple android devices are plugged into Jenkins LOCATION="./android-cts-kitkat-x86/android-cts/repository/testcases/" adb -s $Device install -r $LOCATION/CtsDeviceAdmin.apk # for devices 4.2 and below we need additional adb -s $Device install -r $LOCATION/CtsDelegatingAccessibilityService.apk # copy media files on a device - this can be done in two ways # 1st official recommended (using a script): sh ./android-cts-media/copy_media.sh all # 2nd copying using alternative method (adb push) as 1st not always working adb -s $Device push ./android-cts-media/bbb_short/1920x1080/ /mnt/sdcard/test/bbb_short/1920x1080/ adb -s $Device push ./android-cts-media/bbb_full/1920x1080/ /mnt/sdcard/test/bbb_full/1920x1080/ adb -s $Device push ./android-cts-media/bbb_short/1280x720/ /mnt/sdcard/test/bbb_short/1280x720/ adb -s $Device push ./android-cts-media/bbb_full/1280x720/ /mnt/sdcard/test/bbb_full/1280x720/ adb -s $Device push ./android-cts-media/bbb_short/720x480/ /mnt/sdcard/test/bbb_short/720x480/ adb -s $Device push ./android-cts-media/bbb_full/720x480/ /mnt/sdcard/test/bbb_full/720x480/ adb -s $Device push ./android-cts-media/bbb_short/176x144/ /mnt/sdcard/test/bbb_short/176x144/ adb -s $Device push ./android-cts-media/bbb_full/176x144/ /mnt/sdcard/test/bbb_full/176x144/ adb -s $Device push ./android-cts-media/bbb_short/480x360/ /mnt/sdcard/test/bbb_short/480x360/ adb -s $Device push ./android-cts-media/bbb_full/480x360/ /mnt/sdcard/test/bbb_full/480x360/
4. Manual device preparation:
In order for CTS to run without any problems following settings on a target device have to be changed:
- make sure that device is charged to 100%
- enable USB debugging, connect the device to execution slave and make sure RSA keys are accepted (with the box ticked always allow this machine)
- in Settings -> Developer options -> select the following:
- Stay awake
- Allow mock locations
- in Settings -> Date & time change the timezone to Azores GMT +0:00
- in Settings -> Language & input change the language to English (United states)
- in Settings -> Security change the Screen lock to ‘None’
- insert an empty SD Card into a device
- connect the device to WiFi with IPv6 (if not available use normal instead)
- suggested change (optional) – change the device brightness to minimum (use less battery during the test)
This preparation can be done in parallel of copying media files on the device via the script.
This is not the end – Jenkins will install:
- adb install -r android-cts/repository/testcases/CtsDeviceAdmin.apk (you will have to confirm the installation on a device)
- Manual process: After the APK is installed you will have to enable the two android.deviceadmin.cts.CtsDeviceAdminReceiver* device administrators in Settings -> Security -> Device administrators
Note: Make sure the android.deviceadmin.cts.CtsDeviceAdminDeactivatedReceiver stays disabled in the same menu.
5. Time to run the CTS tests:
# running the CTS chmod 777 ./android-cts-kitkat-x86/android-cts/tools/cts-tradefed ./android-cts-kitkat-x86/android-cts/tools/cts-tradefed run cts --plan CTS --serial $Device -l debug
–plan CTS -> name of the plan that will be run on a device (full)
–serial $Device -> target device
-l debug -> enable debug console
CTS will now start running over 24,000 tests on a device.
6. Re-running failed test cases:
Google allows to run failed test cases twice and there is an easy method of how it can be achieved. We can create a new test plan using only failed tests.
First, lets use:
l r -> list the results of the test runs
./android-cts-kitkat-x86/android-cts/tools/cts-tradefed l r
Which should give the following result:
Non-interactive mode: Running initial command then exiting. Using commandline arguments as starting command: [l, r] Session Pass Fail Not Executed Start time Plan name 0 24842 55 0 2014.10.02_17.22.48 CTS
Based on the output we have 55 failed tests, we can create a new plan using:
add derivedplan --p <new_plan> --s <session_id> --r <fail/pass/notexecuted>
–p <new_plan> -> name of the new plan created
–s <session_id> -> session source for our new plan (e.g 0)
–r <fail/pass/notexecuted> -> which test cases should be used
Now let’s add and run a new test plan with only failures from the previous run:
./android-cts-kitkat-x86/android-cts/tools/cts-tradefed add derivedplan --p CTS_failures_1 --s 0 --r fail ./android-cts-kitkat-x86/android-cts/tools/cts-tradefed run cts --plan CTS_failures_1 --serial $Device -l debug
So we accomplished the following:
- download all source files
- copy necessary files onto the device
- run the CTS tests
- re-run the failures
The only problem is device preparation which has to be done manually. Example script should not be difficult to integrate with Jenkins (I got my version running with no issues).
Good luck and see you next time.