produces a different results output.
- Add UiAutomator and results parsing support for GB4
- Remove the version checking in init.py and rely instead on the ApkWorkload version
checking. The required version in the agenda is checked against that on the host or target
Tested with GB3.4.1 and GB4.0.0 on a Galaxy S7. Older variants of the APK were not available.
Added cornercase of accidentially triggering a card popup
New version includes a new dialogue and name for search. Added support for this
If device has more than one account associated, a new popup appears. Handle that with an optional parameter listing specific account to select, if ommitted the first account in the list is selected
- Corrected the depency for opencv bindings. Even though the library is
imported as "cv2", the package containing it is called
"opencv-python".
- pylint: ignore no-member warnings; cv2 pulls it's memebers dynamically
from the underlying libopencv, so pylint can't see them.
Due to inaccuracies in revent playback, the resultant state does not
always match the templates precisely causing state detection to fail.
To help this images are now scaled to different sizes before being
matched to the templates to compensate for slight variations in size.
When using 'check_exit_code' and 'as_root' options for adb_shell with
a command containing single quotes, the provided command was escaped
twice which has now been avoided.
- Updated the revent parser to handle the new revent format.
- Updated the revent parser to expose device info and recording
metatdata.
- The parser can now be used in streaming and non-streaming mode (stream
events from the file as they are being accessed, or read them all in
at once).
- -g option added to "record" command to expose the gamepad recording
mode.
- Added support for gamepad recording. This type of recording contains
only the events from a gamepad device (which is automatically
identified). The details of this devices are collected and stored as
part of the recording. On playback, uinput is used to create a virtual
gamepad and replay into it.
- A "mode" field has been added to the recording format to help
distinguish between the normal and gamepad recording types.
- A field for the total number of input events has been added before the
start of the event stream (after the device description).
- The structure of revent code has undergone a major overhaul to improve
maintainability and robustness.
- More detailed "info" command output.
- Updated Makefile to support debug/production builds.
- Corrected code highlighting and phase names in documentation
- Fixed check_states paremeter not being honoured
- Moved state dependencies check to happen earlier in execution
and to be a user facing error
In order to use state detection with revent, a screen capture
is required for comparison. Added an argument to revent record command to
automatically record a screen capture after the recording is
complete.
- renamed the capture method from "capture_view_hierachy" to
"capture_ui_hierarchy" to fix typo and make more descriptive.
- Change the file extension of the cature file to ".uix" because this is
the extension uiautomatorviewer looks for.
It is possible that the command executed by get_pids_of() will return
None (in cases where there are no running processes with the specified
name and the grep call didn't find anything). If that happens, then the
subsequent call to split() failed (as that is not a method of None). To
avoid this, substitute an empty string instead.
It looks like on recent systems, adb has started to correctly forward
stderr from the target device to stderr on the host (wereas in the past,
it got output to stdout on the host). This commit makes sure that
stderr gets correctly forwarded to the coller in cases where return code
checking was not enabled.
Adds support for passing -d option to "pm install", which allows
installing an APK when a newer version of the package is already present
on the device.
Capitalization for the "LET'S ROLL" button text was incorrect in the
automation code. This has not caused issues up to this point, but it
seems the button is no longer being found in the latest AOSP. This
commit corrects the capitalization.
Conform to code style of other workloads.
Changed search strings to be a list. Allows user the ability to search for 1 or more search terms.
Renamed Reader to AdobeReader. Removed sleep and now detect and wait for searchicon
Image order not guaranteed. Changed workflow to use subfolders to guarantee ordering and have same action be performed on the same image. As a result, swipe gestures are no longer applicable
Colour change was using clicks. Now uses drags. This is more reliable across devices
Previously if trying to downgrade an app using the `-r` option of
`adb install` the `INSTALL_FAILED_VERSION_DOWNGRADE` error would occur.
The app is now uninstalled first to prevent this.
Since there has been an update to the skype app, there was now a
update prompt preventing test from running. Now checks for the
prompt on older versions of skype and dismisses it if present.
Since initilize only runs once per workload, we can't set variables required
for each iteration in initilize. Revent workload was setting its file paths
in initilize, now moved to setup.
APK Resolution is now handled a bit differently to try maximise the likelyhood
of a workload running.
Like before `force_install` will always try to install the host version, if it
is not present or is not a correct version, it will error.
`check_apk` has changed so that when it is `True` it will prefer to use the host
side APK. If it is not there, or not a suitable version and/or abi and the target
already has a correct version of the app, the target app will be used. When it is
to `False` WA will prefer the target version of the app so long as it is a valid
version, if it is not then it will fallback to the host side APK.
In a recent commit workload versions were changed to match their APK versions.
This commit adds the old versions to the allowed versions and automatically
maps them onto the new values.
Constraints and allowed values of Extension Parameters will now be check
when the Parameter value is set, rather than when validating the
extension. Mandatory status of a Parameter is still checked during
valudation.
When running the stand-alone cpustates script and specifying a timeline
file (which causes the corresponding reporter to be enabled), a
timeline report is generated in addition to the usual cpustates and
parallelism reports.
Up to this point, the main() of the stand-alone script was expecting
exactly two reports and so it crashing when running with the timeline
option. This commit fixes this case.
Super works by having a iterator of parent classes to ensure each is called
once and only once. WA calls some parent methods in classes with multiple
inheritenceconditionally so calls them directly instread. This breaks super
which ends up calling some methods multiple times.
To work around this until workloads are reworked to use composition rather than
inheritance, all classes that that subclass `Workload` directly no longer use
`super`.
Implement new method to enforce that a valid apk version is used to run
the workload. Based on a min and max range of apk versions tested
during development.
- Device subclasses should provide their own implementation.
Default behaviour is to raise a `NotImplementedError`
- Workload subclasses can set `requires_network` to `True` and
network connectivity check will be performed during `setup()`
New method to force a re-index of the mediaserver cache for the
specified directory. Used in workloads that require external media files
as dependencies.
It has been observed on some devices that the FINE_LOCATION premissions is required
for antutu to run without asking for permissions at run time but this was not listed
in the APK manifest. This caused issues on devices were only the permissions in the
manifest can be granted. This commit sliences any error when trying to set this permission
as well as only trying only on Android 6+
CalledProcessErrorWithStderr is a subclass of CalledProcessError that
also takes stderr output. Both that and normal output must be passed as
keyword arguments. They were being passed as keyword arguments inside
_scp() of SshConnection, causing cryptic errors to appear.
Additionally, "output" was not being properly poped off before invoking
super's init.
Add method to automatically check against a dictionary of known package
versions that don't work well with AndroidUiAutoBenchmark workloads.
Raises an exception if found.
- Added statedetect.py in utils which is a standalone module that
contains all the methods needed for state detection
- Modified the setup() and run() methods of the GameWorkload class
in common/android/workload.py to have a parameter that enables
state checks and run the check after setup and run if requested.
State detection uses the template matching method available in
OpenCV to determine the state of the workload by detecting
predefined unique elements on a screenshot from the device.
Dump window hierarchy view from uiautomator to a file when WA fails
during execution. Note: the xml file are pre-formatted after dump.
Implementation specific to android.device.
Issue: On some devices, _grant_requested_permissions may throw an
error when trying to grant permissions that were already granted
at install time using 'adb install -g'.
Fix: Surround permission grant method for API 23+ with try/except
Issue: Currently, _grant_requested_permissions skips all lines after
the first line not starting with 'android.permission'. This causes
a problem for apps where some required permissions appear after a
non-matching line e.g. Google Slides.
Fix: Don't break on non-matching line until end of section is reached
The uxperf result processor now provides warnings for unmatched UX_PERF
markers when running the fps instrument. Previously unmatched markers
resulted in an exception being thrown. Includes additional logging for
debugging purposes.
Add a general check that pings a IP address rather than checking the
status of dumpsys wifi due to the fact that not all devices are
connected via wifi. Intended for workloads that require an internet
connection to operate.
Add a new parameter to fps instrument to specify the time period between
calls to ``dumpsys SurfaceFlinger --latency`` in seconds when collecting
frame data. A lower value improves the granularity of timings when
recording actions for UX Performance metrics.
revent:
- Fixed 32-bit/64-bit compatibility by no longer "long" for timestamps
- Removed superfluous code
- SIGTERM is now handled only while waiting for a file not while processing one
- Added '-s' to docs
- Fixed path_buff size
Record Command:
- Removed timeout in command as -s is specified.
- Previously the command would send SIGTERM to revent but not wait for it to terminate.
This would result in the pulled recording missing its send. This has now been fixed.
Replay Command:
- Added more logging
- Implement a new Marker API in BaseUiAutomation so workload can
generate start and end markers with string name. Outputs to logcat.
- Document the Marker output log format in the WA documentation
- Create a results processor to take existing instrument fps logs and
parse them based on the workload markers. Produce per-action fps
metrics.
- Add simple timing results based on the workload markers
Processing logic for frame statistics can be moved out of fps instrument
to a new utility file. This will allow result processors to use the same
logic to produce frame statistics on a subsection of the data
produced by the fps instrument.
- Check the APK's versionName property against the workload's
expected version if specified
- If workload specifies check_abi param, try to get APK from
ABI-specific path on the host
- Add variant_name param to APK resource-getter for backwards
compatibility of dex2oat and peacekeeper
Issue: For certain installation errors, it is possible for WA to
incorrectly report that an APK was reinstalled while it actually
wasn't, leading to bugs later on in the run.
Fix:
- Add the '-r' flag to adb install when reinstalling, to make sure
APK is replaced.
- Add '-g' flag for API 23 and higher, to grant all permissions that
would otherwise be requested at runtime (similar to pre-API 23)
- Updated poller binary to propery check for errors (e.g. on attempting
to open a file) and report them to stderr
- Updated the file_poller instrument to collect poller stderr output
into a log file and to check the log for errors or warnings on
completion of an iteration.
- device.is_rooted check was being perfromed inside validate() which
gets invoked before the device is connected. This worked for most
Android devices, because connections is a no-op for them, however
failed for Linux targets. The check is now performed inside
initialize().
- Added _is_ready() check inside is_rooted to catch similar problems
quicker in the future.
After running the benchmark, when collecting the output files the execution
fails because the split('\n') call, used for creating a list of output files,
returns an empty string as last element. The empty string makes the pull command
fail because file '' doesn't exist on the target device.
Signed-off-by: Michele Di Giorgio <michele.digiorgio@arm.com>
Standard string representation of a subprocess.CalledProcessError does
not include the output of the command, so it was not previsouly included
in the resulting DeviceError. This commit ensures that the output is
propagated, regardless of whether it came from stdout or stderr of the
underlying process.
Moved ``working_directory`` parameter to BaseLinuxDevice.
Changed the default ``binaries_directory`` for AndroidDevice to allow
WA binaries to be easily separated.
energy_model instrument generates job specs during the run. One of the
things it does is set "ui" runtime parameter to "off". This parameter
only exists for ChromeOS devices. This commit ensure that the parameter
is not set when running on any other device.
If the tar.gz already existed on the target device the instruments would
fail. This fix adds the "-f" option to gzip to force it to overwrite the
file.
Previously, the workload was invoking the executable via its name,
assuming that it will be in PATH. As WA's executables directory is not
in path, the invocation was failing. This commit saves the full path to
the installed executable and uses that instead.
Inconsistently, while install() for Android devices automatically
handles both APKs and executables appropriately, uninstall() only works
for packages. Changing to use uninstall_executable() for the scripts
deployed by recentfling.
- Scripts are now deployed during via install() ensuring that they are
executable.
- Handle the case where the PID file is delted before getting to
process_results.
- Exposed the option to not start any apps before flinging via a
parameter.
- make sure results_list is always instatiated even if no metrics found;
this would previously cause a "used before declaration" error
- Detect that no metrics were extracted from the log and raise a
WorkloadError
cpustates can now handle the lack of a start marker in three ways:
- try: If the start marker is present only the correct section of the trace
will be used, if its not the whole trace will be used.
- error: An error will be raised if the start marker is missing
- ignore: The markers are ignored and the whole trace is always used.
If an exception is raised inside a generator it cannot be continued.
To get around this exceptions are now caught and later output via the
logger.
Also added logger setup when running cpustates as a standalone script
This allows the return value of a function to be cached so that
when it is called in the future the function does not need to
run.
Borrowed from: https://github.com/ARM-software/devlib
Before WA would raise a error message that wasn't very clear.
Now when cpustates tries to nudge cores and and error occurs it
will only show a warning (which promts users to check if the cpu is
hot plugged out) and keep going with the reset of the run without
causing errors in other WA extensions.
Previously the WA device name was used when searching for revent files.
Since most were `generic_android` this made it difficult to keep revent
files for multiple android devices. Now it the device model is used instead.
If a file with the device model is not found it will fall back to the WA
device name.
kick_off has been changed to behave the same as AndroidDevice.
Said changes caused kill all to fail on rooted devices. Killall will now
behave in the same way as kick_off, if specifically told to (or not to)
run as root it will. Otherwise it will run as root if the device is rooted
Servo is a debug board used for Chromium OS test and development. Among other uses, it allows
access to the built in power monitors (if present) of a Chrome OS device. More information on
Servo board can be found in the link bellow:
https://www.chromium.org/chromium-os/servo
based on: 03ede10739
and: 9a0dc55b55
These are now gathered via `getprop` rather than trying to parse the
build.prop file directly.
This fixes issues with build.prop files that have imports.
Blogbench is a portable filesystem benchmark that tries to reproduce the
load of a real-world busy file server.
Signed-off-by: Chase Qi <chase.qi@linaro.org>
The stock camera app as of Android M has changed. This commit updates
the ui automation to work with this new app. As part of this change
it was required to bump the API level of the ui automation to 18.
Also made the teardown of the capture workload close the app like the
record workload.
Added splitVersion and compareVersions functions allow versions strings
like "3.2.045" to be compared.
Also fixed the build script to now copy to the correct folder
Some characters would be interpreted by the shell thus breaking the
command. Adding quotes around the parameters solved this.
N.B Space still needs to be replaced.
stress-ng will stress test a computer system in various selectable ways.
It was designed to exercise various physical subsystems of a computer as
well as the various operating system kernel interfaces.
Signed-off-by: Chase Qi <chase.qi@linaro.org>
Previously the sensor name was just appeneded to the end of the
previous sensors name.
Now the hwmon name is added as a classifier of the metric.
If the hwmon sensor has a label, the metric will use this for its name,
if it does not then the sensors kind and ID will be used e.g. temp3
Due to the previous commits, this command no longer works properly.
It turns out there is an issue with using multiple levels of escaping.
It seems that bash handles the backslashes and single quotes separately
incorrectly processing our escaping. To get around this we are writing the
sqlite command to a shell script file and running that.
This seems to be the only case in WA at the moment that requires this,
if more show up/when WA moves to devlib it should use the devlib shutil
mechanism.
They way we were attempting to get return codes before always gave
us a return code of the previous echo, therefore always `0`.
This commit adds the newline into the last echo.
Previously, only reports that were generated were returned. With this
commit, there will be an entry for each active reporter in the returned
list. If a reporter did not produce a valid report, the entry will be
None.
This ensures consistent output, even if a run time issue causes a
reporter not to produce a report (e.g. if cpufreq events were not
enabled).
It is not possible to read frequencies from a core that has been hotplugged.
The code will now set the current and max frequencies of hotplugged cores
to None.
This still doesn't work for devices that have dynamic hotplug enabled
Commit 724f6e590e changed sysfile_getter
behavior to first tar up copied files and then gzip them. Tarball name
needs to be updated to not include '.gz' extension.
Previously Linaro U-Boot releases had a bug where they used \n\r
as the line ending. This has now been fixed which caused
issues with WA. WA now detects the U-Boot version and uses the
coresponding line ending.
The default boot args have been removed since these cause issues with
the latest Linaro builds, which boot correctly without any bootargs.
Also made a regex string a raw-string.
If uninstall fails, "pm clear" should be called to make sure that the
next time the app is launched it starts from a known state (which would
normally be ensured by the uninstall).
It's possible that there is already a version of an app on target that
differs form the version of the apk on the host. In such cases, WA will
usually try to uninstall the target version and install the host
version.
It's possible that the uninstall may fail. If that happens, it will be
reported as a warning but workload exectuion will proceed with the
target version. In this case, apk_version would have already been set to
that of the host apk. This change ensures that the APK version is
correctly set to the target version (the one that actually ran).
