1
0
mirror of https://github.com/ARM-software/workload-automation.git synced 2024-10-06 19:01:15 +01:00
Commit Graph

796 Commits

Author SHA1 Message Date
Marc Bonnici
27b08bade0 PeacekeeperWorkload: Updated to work with new APK versions
Updated peacekeeper UI automation to work with newer version of
chrome and firefox.
2016-12-12 16:49:54 +00:00
Marc Bonnici
1477a89ee4 AndroidDevice: Renamed 'supported_eabis' property to 'supported_abis'
Renamed android device property from 'supported_eabis' to 'supported_abis' to be consistent with linux device.

Updated dex2oat workload to use new property name.
2016-12-09 15:23:58 +00:00
Marc Bonnici
0dfbbae7b6 Renamed 'check_abi' parameter to 'exact_abi' 2016-12-09 14:11:16 +00:00
Marc Bonnici
a8a8d21de6 AndroidWorkload: Modified workload to properly check for an apks abi
Previously when retrieving apks only it's name would be used to choose
an apk. Now the native code reported by the apk is used to determine
the correct version to run for the specific device. It tries to
match the primary abi of device with native code before falling back to
using a compatible apk.

When using the check_abi parameter it no longer relies on naming convention
and only allows apks with native code supporting a devices primary abi to be
used.

Updated the relevant documentation.
2016-12-09 14:11:16 +00:00
Marc Bonnici
f467f6f991 AndroidDevice: Added method to retrive primary abi of installed package
Tries to retireve the primary abi of a currently installed package on
device.
2016-12-09 14:11:16 +00:00
Marc Bonnici
4352e02806 APK Info: Added property to extract native code from an apk
Looks for any native code present in an apk and stores the
mapped abi(s) for use when selecting appropriate apks.
2016-12-09 14:11:16 +00:00
Marc Bonnici
693d0544a3 ABI_MAP: Added armeabi-v7a as armeabi 2016-12-09 14:11:16 +00:00
Marc Bonnici
eb239c65d0 LinuxDevice: Added supported eabi property.
Added property to linux device to return list of supported abis
to be consistent with android devices. Currently only returns a
list containing the primary abi.
2016-12-09 14:11:16 +00:00
Michael McGeagh
8cbf189029 skype: longer wait time for signin and search 2016-12-09 13:45:35 +00:00
Marc Bonnici
b6f770cfc5 AdobeReader: Updated workload for new app version
Updated the workload to work with the latest version (v16.4)
of Adobe Reader app.
2016-12-08 18:14:16 +00:00
setrofim
c7de8cabd6 Merge pull request #294 from jimboatarm/improve-geekbench-timeouts
geekbench: Improve UiAutomator timeouts for slower devices
2016-12-02 17:04:01 +00:00
setrofim
fd7df36a5a Merge pull request #301 from jimboatarm/skype-hangup
skype: now hangs up at the end. Better duration handling
2016-12-02 17:03:13 +00:00
setrofim
607187ad5b Merge pull request #306 from marcbonnici/revent_BC
Revent: Fixed backwards compatibility with v1
2016-12-02 17:02:06 +00:00
setrofim
b4036c5f15 Merge pull request #304 from jimboatarm/uxperf-fix
UXPERF: handle error when only one timestamp is logged
2016-12-02 16:58:33 +00:00
Marc Bonnici
b6e077c06b Revent: Fixed backwards compatibility with v1.
Fixed new gamemode not being set for compatibility mode.
Fixed input buffer not being flushed causing replay to fail.
Fixed infinte loop if revent recording did not end correctly.
Updated revent binaries.
2016-12-02 16:58:09 +00:00
Michael McGeagh
8b82451230 util-fps: handle divide by zero possibility 2016-12-02 16:34:32 +00:00
Michael McGeagh
64c352fab6 UXPERF: handle error when only one timestamp is logged 2016-12-02 16:31:45 +00:00
Michael McGeagh
ae4ae3da5e skype: now hangs up at the end. Better duration handling 2016-12-02 11:30:30 +00:00
Michael McGeagh
393abc267f GooglePhotos: Navigate up only when needed. Use baseclass' selectGalleryFolder 2016-12-02 11:28:38 +00:00
setrofim
751970f991 Merge pull request #302 from marcbonnici/master
Revent: Added updated revent binaries.
2016-12-02 10:31:03 +00:00
Marc Bonnici
254e9fff38 Revent: Added updated revent binaries.
Added updated revent binaries for armabi and arm64 for the fixes
in commit 7a19046645
2016-12-02 09:55:19 +00:00
James Hartley
088709f290 geekbench: Improve UiAutomator timeouts for slow devices, fixes for rooted devices
Tested by running Mate 8 on the lowest cluster at the lowest available frequency

On rooted devices, skip attempting to pull log files from device.  This allows
the benchmark to run to completion without failure.

GB log files are stored in /data/data/com.primatelabs.geekbench/files and
not accessible without root.  On Chromebooks ( which have no adb root
rights) it is possible to copy from this folder manually after the test run.
2016-12-02 07:23:44 +00:00
Sergei Trofimov
850fcb24ab pylint/pep8 fixes 2016-12-01 15:01:45 +00:00
Michael McGeagh
cb53fe9ec8 Gmail: A workload to perform standard productivity tasks within Gmail. The workload carries out various tasks, such as creating new emails, attaching images and sending them.
Moved broadcast to super. Mandatory and Default are XOR

Added a longer wait for sync to finish. Increases reliability on certain phones

Changed recipient to not mandatory and a default set

Wait for sync when launching gmail from the sharing feature

Fix: cornercase where image viewer already points to working directory. Refactored code due to duplication

Added new function to BaseUiAutomation class to find a folder in the gallery
2016-12-01 13:21:49 +00:00
setrofim
4e161127e1 Merge pull request #300 from setrofim/master
A couple of fixes for trace CPU power state post-processing and energy model scripts
2016-11-30 09:38:05 +00:00
setrofim
bf43bf93bc Merge pull request #298 from marcbonnici/revent_fixes
Revent: Fixes
2016-11-29 17:34:17 +00:00
Marc Bonnici
7a19046645 Revent: Fixes
Changed termination signal to interrupt signal to prevent code exiting too early.
Added exit handler to ensure revent exits correctly as previously was
crashing and therefore not running final code.
Fixed error in writing input event where half of timestamp seconds was missing.
Fixed typo in documentation for revent file structure.
2016-11-29 17:29:33 +00:00
Sergei Trofimov
5db11462be utils/power: handle devlib frequency reporting
When collecting ftrace events, the instrument will insert frequencies
into the trace to make it possible reconstruct power states when there
were no frequency transitions during the measured period.

The format in which frequencies are inserted is different in devlib.
Since post-processing scripts may be run on traces collected by devlib
as well as WA, it needs to support both formats.
2016-11-29 10:23:32 +00:00
Sergei Trofimov
7bf0e3c344 energy_model: only set "ui" parameter for ChomeOS
Fixed a bug, where "ui" runtime parameter was being set regardless of
the underlying platforms for "freq" iterations, causing the to fail on
non-ChomeOS devices.
2016-11-29 10:23:32 +00:00
setrofim
d16d8bf62d Merge pull request #297 from marcbonnici/octaned
ResourceGetters: Updated executable discovery to use http_getter
2016-11-28 11:45:46 +00:00
Marc Bonnici
c93cc81aac ResourceGetters: Updated executable discovery to use http_getter
Executables were not previously automatically downloaded from a
provided remote_assets_url. The http_getter is now called to attempt to
find and download executable files.
2016-11-28 11:38:29 +00:00
setrofim
a172c8f624 Merge pull request #295 from jimboatarm/fix-octaned8-log-appending
Fix octaned8 results capture
2016-11-28 11:04:42 +00:00
James Hartley
f69e4c5b18 Fix octaned8 results capture
Ensure the device log file is deleted between runs and that the host file is not kept as an
artefact.  Both can lead to errors appending data from the next iteration/run.
2016-11-24 14:54:19 +00:00
James Hartley
5b543d2edf Fix exact_apk_version check 2016-11-24 12:47:57 +00:00
setrofim
40d281b336 Merge pull request #283 from jimboatarm/octaned8
Add Octane d8 workload
2016-11-11 17:32:53 +00:00
Michael McGeagh
8edce40301 FPS percentiles: Ignore ValueError when NaN
Fixed: if index isnt zero based, drop(0) will fail
2016-11-11 17:21:47 +00:00
James Hartley
642d757066 Add Octane d8 workload
A workload to run the binary (non-browser) version of Octane
2016-11-11 17:04:25 +00:00
setrofim
969201968e Merge pull request #282 from jimboatarm/min-apk-version-fix
Support for Geekbench 4
2016-11-11 11:23:07 +00:00
James Hartley
46b78d35be Geekbench 4 is similar in terms of operation but runs significantly different tests and hence
produces a different results output.

- Add UiAutomator and results parsing support for GB4
- Remove the version checking in init.py and rely instead on the ApkWorkload version
  checking.  The required version in the agenda is checked against that on the host or target

Tested with GB3.4.1 and GB4.0.0 on a Galaxy S7.  Older variants of the APK were not available.
2016-11-10 18:56:22 +00:00
James Hartley
ba34b973ac ApkWorkload: add check for an exact APK version
Checks for an APK version on the host or target which matches a specified value
2016-11-10 18:46:31 +00:00
Michael McGeagh
6d173f2f3f Change min apk ver for reader 2016-11-10 15:48:50 +00:00
Leo Yan
d3c59e2f74 rt-app: add user case for camera
Add camera-long.json and camera-short.json files to simulate camera
 recorder.

Signed-off-by: Leo Yan <leo.yan@linaro.org>
2016-11-10 12:08:02 +08:00
setrofim
fce04d2938 Merge pull request #281 from jimboatarm/uxperf-newfps
uxperf result_processor: updated for changes in FPS instrument
2016-11-08 13:20:31 +00:00
setrofim
27df426c0d Merge pull request #263 from jimboatarm/books-codetidy
GooglePlayBooks: Code tidy to conform.
2016-11-08 13:19:27 +00:00
Michael McGeagh
f8966bf324 GooglePlayBooks: Code tidy to conform. Additional parameter added for when the search name differs to the library name. Defaults changed due to Hamlet dissapearing from the store.
Added cornercase of accidentially triggering a card popup

New version includes a new dialogue and name for search. Added support for this

If device has more than one account associated, a new popup appears. Handle that with an optional parameter listing specific account to select, if ommitted the first account in the list is selected
2016-11-08 12:31:27 +00:00
Michael McGeagh
e076d47a7b uxperf result_processor: updated for changes in FPS instrument 2016-11-08 12:20:47 +00:00
Michael McGeagh
dcf13f8c2c FPS - fix regex and handle empty frames 2016-11-07 17:39:51 +00:00
James Hartley
359d9d3e5f Fix incorrect min_apk_version setting in adobereader and googleplaybooks 2016-11-07 15:42:38 +00:00
setrofim
d56f581a0a Merge pull request #277 from jimboatarm/fps-gfxinfo
FPS. Added gfxinfo methods of obtaining fps stats.
2016-11-04 17:11:59 +00:00
Sergei Trofimov
f12cf6d557 statedetect: pylint and dependencies fixes
- Corrected the depency for opencv bindings. Even though the library is
  imported as "cv2", the package containing it is called
  "opencv-python".
- pylint: ignore no-member warnings; cv2 pulls it's memebers dynamically
  from the underlying libopencv, so pylint can't see them.
2016-11-04 17:05:39 +00:00
Michael McGeagh
311c4e419f FPS. Added gfxinfo methods of obtaining fps stats. Auto detects and uses appropriate method via android version of device
Output 90th, 95th and 99th percentile metrics. This happens for both SurfaceFlinger and Gfxinfo methods.
2016-11-04 17:05:00 +00:00
Marc Bonnici
1daa7f97c0 ReventWorkload: Modified statedection to vary scale of images
Due to inaccuracies in revent playback, the resultant state does not
always match the templates precisely causing state detection to fail.
To help this images are now scaled to different sizes before being
matched to the templates to compensate for slight variations in size.
2016-11-04 15:25:31 +00:00
Marc Bonnici
2e1ce49170 android: Fixed issue using single quoted command with adb_shell
When using 'check_exit_code' and 'as_root' options for adb_shell with
a command containing single quotes, the provided command was escaped
twice which has now been avoided.
2016-11-02 14:52:47 +00:00
Sergei Trofimov
edfef444fb revent: Updated WA to use the new revent
- Updated the revent parser to handle the new revent format.
- Updated the revent parser to expose device info and recording
  metatdata.
- The parser can now be used in streaming and non-streaming mode (stream
  events from the file as they are being accessed, or read them all in
  at once).
- -g option added to "record" command to expose the gamepad recording
  mode.
2016-10-26 13:33:54 +01:00
Sergei Trofimov
3a7a5276e4 revent: added gamepad recording + major refactor
- Added support for gamepad recording. This type of recording contains
  only the events from a gamepad device (which is automatically
  identified). The details of this devices are collected and stored as
  part of the recording. On playback, uinput is used to create a virtual
  gamepad and replay into it.
- A "mode" field has been added to the recording format to help
  distinguish between the normal and gamepad recording types.
- A field for the total number of input events has been added before the
  start of the event stream (after the device description).
- The structure of revent code has undergone a major overhaul to improve
  maintainability and robustness.
- More detailed "info" command output.
- Updated Makefile to support debug/production builds.
2016-10-26 13:17:07 +01:00
Marc Bonnici
620fbfdd2a ReventWorkload: Corrected mistake in screen capture argument 2016-10-26 11:56:03 +01:00
Marc Bonnici
9fffa7958a ReventWorkload: State detection fixes
- Corrected code highlighting and phase names in documentation
- Fixed check_states paremeter not being honoured
- Moved state dependencies check to happen earlier in execution
  and to be a user facing error
2016-10-26 11:16:49 +01:00
Marc Bonnici
0f2bc17eca ReventWorkload: Added argument to take screen capture with revent record command
In order to use state detection with revent, a screen capture
is required for comparison. Added an argument to revent record command to
automatically record a screen capture after the recording is
complete.
2016-10-26 11:16:49 +01:00
Sergei Trofimov
558e40698b AndroidDevice: fixing UI hierarchy capture
- renamed the capture method from "capture_view_hierachy" to
  "capture_ui_hierarchy" to fix typo and make more descriptive.
- Change the file extension of the cature file to ".uix" because this is
  the extension uiautomatorviewer looks for.
2016-10-17 13:49:19 +01:00
Sergei Trofimov
41b52178bb pylint fixes. 2016-10-17 11:02:41 +01:00
Sergei Trofimov
8aa1bdc63d AndroidDevice: correctly handle None output on get_pids_of
It is possible that the command executed by get_pids_of() will return
None (in cases where there are no running processes with the specified
name and the grep call didn't find anything). If that happens, then the
subsequent call to split() failed (as that is not a method of None). To
avoid this, substitute an empty string instead.
2016-10-17 10:54:10 +01:00
Sergei Trofimov
8355fcf886 adb_shell: handle zero stdout on error
It looks like on recent systems, adb has started to correctly forward
stderr from the target device to stderr on the host (wereas in the past,
it got output to stdout on the host). This commit makes sure that
stderr gets correctly forwarded to the coller in cases where return code
checking was not enabled.
2016-10-17 10:49:53 +01:00
Michael McGeagh
4649fa13db AdobeReader: Fails on S7 due to search icons being a different class. Removed the class specifier 2016-10-11 13:25:35 +01:00
Sergei Trofimov
10dd2b304e ApkWorkload: ensure that the APK is always replaced and that a downgrade is allowed. 2016-10-10 13:15:24 +01:00
Sergei Trofimov
93fbb7282a AndroidDevice: added support for downgrade when installing an APK
Adds support for passing -d option to "pm install", which allows
installing an APK when a newer version of the package is already present
on the device.
2016-10-10 12:55:14 +01:00
Sergei Trofimov
edc26fe75c vellamo: fixing capitalization in automation
Capitalization for the "LET'S ROLL" button text was incorrect in the
automation code. This has not caused issues up to this point, but it
seems the button is no longer being found in the latest AOSP. This
commit corrects the capitalization.
2016-10-07 17:21:56 +01:00
setrofim
6aecf1b35e Merge pull request #264 from jimboatarm/reader-workload
Reader Workload: The Adobe Reader workflow carries out typical…
2016-10-05 08:30:57 +01:00
Michael McGeagh
4c5e008609 Reader Workload: The Adobe Reader workflow carries out typical productivity tasks.
Conform to code style of other workloads.

Changed search strings to be a list. Allows user the ability to search for 1 or more search terms.

Renamed Reader to AdobeReader. Removed sleep and now detect and wait for searchicon
2016-10-04 16:59:22 +01:00
setrofim
5bed658045 Merge pull request #262 from jimboatarm/youtube-codetidy
Youtube: Codetidy to conform with the other workloads
2016-10-03 14:33:08 +01:00
setrofim
8ac5657993 Merge pull request #260 from jimboatarm/slides-codetidy
GoogleSlides: Code tidy, to conform with the other workloads.
2016-10-03 14:09:42 +01:00
setrofim
fc26daecfc Merge pull request #261 from jimboatarm/skype-codetidy
Skype CodeTidy to conform with the other workloads
2016-10-03 13:44:02 +01:00
setrofim
0232341445 Merge pull request #252 from jimboatarm/googlephotos-cropspeedchange
GooglePhotos: Change step value for crop operation to speed up workload
2016-10-03 13:25:45 +01:00
setrofim
3b052cc619 Merge pull request #266 from marcbonnici/revent
revent: Removed redundant android code
2016-09-30 14:33:52 +01:00
Marc Bonnici
3a9505d54e revent: Removed redundant android code 2016-09-30 14:25:11 +01:00
Michael McGeagh
4e94ff9ed7 Skype CodeTidy to conform with the other workloads
Dont double click if the confirm icon appears
2016-09-29 17:29:21 +01:00
Michael McGeagh
c47ae5cfcf Youtube: Codetidy to conform with the other workloads 2016-09-28 14:39:53 +01:00
Michael McGeagh
ce11b94f28 GoogleSlides: Code tidy, to conform with the other workloads. 2016-09-28 13:02:49 +01:00
Michael McGeagh
765fdd7cbb GooglePhotos: Change step value for crop operation to speed up workload. 500 is too slow and not realistic of real user behaviour.
Image order not guaranteed. Changed workflow to use subfolders to guarantee ordering and have same action be performed on the same image. As a result, swipe gestures are no longer applicable

Colour change was using clicks. Now uses drags. This is more reliable across devices
2016-09-28 09:58:22 +01:00
setrofim
77d724efa3 Merge pull request #258 from ep1cman/install_fix
ApkWorkload: Fixed replacing apps on a device
2016-09-26 17:00:39 +01:00
Sebastian Goscik
acb9dd61e7 ApkWorkload: Fixed replacing apps on a device
Previously if trying to downgrade an app using the `-r` option of
`adb install` the `INSTALL_FAILED_VERSION_DOWNGRADE` error would occur.
The app is now uninstalled first to prevent this.
2016-09-26 16:53:03 +01:00
Marc Bonnici
e69aea4e69 Skype: Added check for update popup.
Since there has been an update to the skype app, there was now a
update prompt preventing test from running. Now checks for the
prompt on older versions of skype and dismisses it if present.
2016-09-23 11:18:40 +01:00
Michael McGeagh
5738d19114 Skype: Codetidy, no functional changes. Make voice action default in code, move private function to bottom. 2016-09-22 16:52:39 +01:00
Marc Bonnici
1c8e18bf36 ReventWorkload:Fixed revent workloads with multiple iterations
Since initilize only runs once per workload, we can't set variables required
for each iteration in initilize. Revent workload was setting its file paths
in initilize, now moved to setup.
2016-09-22 11:26:25 +01:00
Sebastian Goscik
5dbf7e7d38 Merge pull request #251 from jimboatarm/googlephotos-codetidy
GooglePhotos: Reorder functions and relabel ActionLogger tags
2016-09-20 12:48:45 +01:00
Michael McGeagh
a2945d58cb Reorder functions and relabel ActionLogger tags. No functional changes made to actual workload run 2016-09-20 11:15:40 +01:00
Sebastian Goscik
8727fe514a Merge pull request #249 from jimboatarm/googleplaybooks-gesturefix
GooglePlayBooks: Reorder so that the gesture test happens after selec…
2016-09-20 10:55:57 +01:00
setrofim
6465e732fd Merge pull request #250 from ep1cman/apk_ver_fixes
Redone APK file resolution
2016-09-20 10:50:29 +01:00
Sebastian Goscik
f9ec869c7b Revert "Updated workload versions to match APK files"
This reverts commit b426e00f2f.
2016-09-20 10:47:57 +01:00
Sebastian Goscik
1bbd3ef87a Revert "workloads: Fixed versions to be backward compatible"
This reverts commit 8608c3b747.
2016-09-20 10:47:57 +01:00
Sebastian Goscik
486ade6499 ApkWorkload: Reworked APK Resolution.
APK Resolution is now handled a bit differently to try maximise the likelyhood
of a workload running.

Like before `force_install` will always try to install the host version, if it
is not present or is not a correct version, it will error.

`check_apk` has changed so that when it is `True` it will prefer to use the host
side APK. If it is not there, or not a suitable version and/or abi and the target
already has a correct version of the app, the target app will be used. When it is
to `False` WA will prefer the target version of the app so long as it is a valid
version, if it is not then it will fallback to the host side APK.
2016-09-20 10:47:57 +01:00
Sebastian Goscik
1a23bd03a2 runner: Some types of exceptions will no longer take screenshots or dump UI hiarchy
For resource or Host errors (error which are not caused by the target).
the runner will no longer take screenshots ect as these are meaningless.
2016-09-20 10:13:14 +01:00
Michael McGeagh
5018a1ec94 GooglePlayBooks: Reorder so that the gesture test happens after selecting a known chapter in the book. Previously the gestures could happen on any random page thus making the results not neccessarily comparable to another run. 2016-09-20 10:10:21 +01:00
Sebastian Goscik
fe58245843 Merge pull request #248 from jimboatarm/youtube-actionlogger-modified
Youtube: Changed actionlogger to measure different more relevant acti…
2016-09-19 17:36:39 +01:00
Michael McGeagh
19569816d3 Youtube: Changed actionlogger to measure different more relevant actions. Also rearranged if statement for video_source so that the default is at the top. 2016-09-19 16:46:03 +01:00
Michael McGeagh
1dfbaf4ebe GooglePlayBooks: Modified ActionLogger tags and capture points 2016-09-19 14:21:03 +01:00
Michael McGeagh
b5dc5b8648 GoogleSlides workload did not do slide_count number of swipes in edit mode. This fixes that. 2016-09-19 13:26:21 +01:00
Sebastian Goscik
8608c3b747 workloads: Fixed versions to be backward compatible
In a recent commit workload versions were changed to match their APK versions.
This commit adds the old versions to the allowed versions and automatically
maps them onto the new values.
2016-09-13 11:07:18 +01:00
Sebastian Goscik
9afe084f2c Merge pull request #243 from setrofim/master
Miscellaneous fixes
2016-09-13 10:26:51 +01:00
Sergei Trofimov
ea1d13c37f common/android: pep8 fixes
- added missing space between global definitions and a class
- added missing space for inline comment
2016-09-13 10:22:47 +01:00
Sergei Trofimov
20996e9a58 core: changing the time of constraint validation for params
Constraints and allowed values of Extension Parameters will now be check
when the Parameter value is set, rather than when validating the
extension. Mandatory status of a Parameter is still checked during
valudation.
2016-09-13 10:20:52 +01:00