- Added statedetect.py in utils which is a standalone module that
contains all the methods needed for state detection
- Modified the setup() and run() methods of the GameWorkload class
in common/android/workload.py to have a parameter that enables
state checks and run the check after setup and run if requested.
State detection uses the template matching method available in
OpenCV to determine the state of the workload by detecting
predefined unique elements on a screenshot from the device.
Dump window hierarchy view from uiautomator to a file when WA fails
during execution. Note: the xml file are pre-formatted after dump.
Implementation specific to android.device.
Issue: On some devices, _grant_requested_permissions may throw an
error when trying to grant permissions that were already granted
at install time using 'adb install -g'.
Fix: Surround permission grant method for API 23+ with try/except
Issue: Currently, _grant_requested_permissions skips all lines after
the first line not starting with 'android.permission'. This causes
a problem for apps where some required permissions appear after a
non-matching line e.g. Google Slides.
Fix: Don't break on non-matching line until end of section is reached
The uxperf result processor now provides warnings for unmatched UX_PERF
markers when running the fps instrument. Previously unmatched markers
resulted in an exception being thrown. Includes additional logging for
debugging purposes.
Add a general check that pings a IP address rather than checking the
status of dumpsys wifi due to the fact that not all devices are
connected via wifi. Intended for workloads that require an internet
connection to operate.
Add a new parameter to fps instrument to specify the time period between
calls to ``dumpsys SurfaceFlinger --latency`` in seconds when collecting
frame data. A lower value improves the granularity of timings when
recording actions for UX Performance metrics.
revent:
- Fixed 32-bit/64-bit compatibility by no longer "long" for timestamps
- Removed superfluous code
- SIGTERM is now handled only while waiting for a file not while processing one
- Added '-s' to docs
- Fixed path_buff size
Record Command:
- Removed timeout in command as -s is specified.
- Previously the command would send SIGTERM to revent but not wait for it to terminate.
This would result in the pulled recording missing its send. This has now been fixed.
Replay Command:
- Added more logging
- Implement a new Marker API in BaseUiAutomation so workload can
generate start and end markers with string name. Outputs to logcat.
- Document the Marker output log format in the WA documentation
- Create a results processor to take existing instrument fps logs and
parse them based on the workload markers. Produce per-action fps
metrics.
- Add simple timing results based on the workload markers
Processing logic for frame statistics can be moved out of fps instrument
to a new utility file. This will allow result processors to use the same
logic to produce frame statistics on a subsection of the data
produced by the fps instrument.
- Check the APK's versionName property against the workload's
expected version if specified
- If workload specifies check_abi param, try to get APK from
ABI-specific path on the host
- Add variant_name param to APK resource-getter for backwards
compatibility of dex2oat and peacekeeper
Issue: For certain installation errors, it is possible for WA to
incorrectly report that an APK was reinstalled while it actually
wasn't, leading to bugs later on in the run.
Fix:
- Add the '-r' flag to adb install when reinstalling, to make sure
APK is replaced.
- Add '-g' flag for API 23 and higher, to grant all permissions that
would otherwise be requested at runtime (similar to pre-API 23)
- Updated poller binary to propery check for errors (e.g. on attempting
to open a file) and report them to stderr
- Updated the file_poller instrument to collect poller stderr output
into a log file and to check the log for errors or warnings on
completion of an iteration.
- device.is_rooted check was being perfromed inside validate() which
gets invoked before the device is connected. This worked for most
Android devices, because connections is a no-op for them, however
failed for Linux targets. The check is now performed inside
initialize().
- Added _is_ready() check inside is_rooted to catch similar problems
quicker in the future.
After running the benchmark, when collecting the output files the execution
fails because the split('\n') call, used for creating a list of output files,
returns an empty string as last element. The empty string makes the pull command
fail because file '' doesn't exist on the target device.
Signed-off-by: Michele Di Giorgio <michele.digiorgio@arm.com>
Standard string representation of a subprocess.CalledProcessError does
not include the output of the command, so it was not previsouly included
in the resulting DeviceError. This commit ensures that the output is
propagated, regardless of whether it came from stdout or stderr of the
underlying process.
Moved ``working_directory`` parameter to BaseLinuxDevice.
Changed the default ``binaries_directory`` for AndroidDevice to allow
WA binaries to be easily separated.
energy_model instrument generates job specs during the run. One of the
things it does is set "ui" runtime parameter to "off". This parameter
only exists for ChromeOS devices. This commit ensure that the parameter
is not set when running on any other device.
If the tar.gz already existed on the target device the instruments would
fail. This fix adds the "-f" option to gzip to force it to overwrite the
file.
Previously, the workload was invoking the executable via its name,
assuming that it will be in PATH. As WA's executables directory is not
in path, the invocation was failing. This commit saves the full path to
the installed executable and uses that instead.
Inconsistently, while install() for Android devices automatically
handles both APKs and executables appropriately, uninstall() only works
for packages. Changing to use uninstall_executable() for the scripts
deployed by recentfling.
- Scripts are now deployed during via install() ensuring that they are
executable.
- Handle the case where the PID file is delted before getting to
process_results.
- Exposed the option to not start any apps before flinging via a
parameter.
- make sure results_list is always instatiated even if no metrics found;
this would previously cause a "used before declaration" error
- Detect that no metrics were extracted from the log and raise a
WorkloadError
cpustates can now handle the lack of a start marker in three ways:
- try: If the start marker is present only the correct section of the trace
will be used, if its not the whole trace will be used.
- error: An error will be raised if the start marker is missing
- ignore: The markers are ignored and the whole trace is always used.
If an exception is raised inside a generator it cannot be continued.
To get around this exceptions are now caught and later output via the
logger.
Also added logger setup when running cpustates as a standalone script
This allows the return value of a function to be cached so that
when it is called in the future the function does not need to
run.
Borrowed from: https://github.com/ARM-software/devlib
Before WA would raise a error message that wasn't very clear.
Now when cpustates tries to nudge cores and and error occurs it
will only show a warning (which promts users to check if the cpu is
hot plugged out) and keep going with the reset of the run without
causing errors in other WA extensions.
Previously the WA device name was used when searching for revent files.
Since most were `generic_android` this made it difficult to keep revent
files for multiple android devices. Now it the device model is used instead.
If a file with the device model is not found it will fall back to the WA
device name.
kick_off has been changed to behave the same as AndroidDevice.
Said changes caused kill all to fail on rooted devices. Killall will now
behave in the same way as kick_off, if specifically told to (or not to)
run as root it will. Otherwise it will run as root if the device is rooted
Servo is a debug board used for Chromium OS test and development. Among other uses, it allows
access to the built in power monitors (if present) of a Chrome OS device. More information on
Servo board can be found in the link bellow:
https://www.chromium.org/chromium-os/servo
based on: 03ede10739
and: 9a0dc55b55
These are now gathered via `getprop` rather than trying to parse the
build.prop file directly.
This fixes issues with build.prop files that have imports.
Blogbench is a portable filesystem benchmark that tries to reproduce the
load of a real-world busy file server.
Signed-off-by: Chase Qi <chase.qi@linaro.org>
The stock camera app as of Android M has changed. This commit updates
the ui automation to work with this new app. As part of this change
it was required to bump the API level of the ui automation to 18.
Also made the teardown of the capture workload close the app like the
record workload.
Added splitVersion and compareVersions functions allow versions strings
like "3.2.045" to be compared.
Also fixed the build script to now copy to the correct folder
Some characters would be interpreted by the shell thus breaking the
command. Adding quotes around the parameters solved this.
N.B Space still needs to be replaced.
stress-ng will stress test a computer system in various selectable ways.
It was designed to exercise various physical subsystems of a computer as
well as the various operating system kernel interfaces.
Signed-off-by: Chase Qi <chase.qi@linaro.org>
Previously the sensor name was just appeneded to the end of the
previous sensors name.
Now the hwmon name is added as a classifier of the metric.
If the hwmon sensor has a label, the metric will use this for its name,
if it does not then the sensors kind and ID will be used e.g. temp3
Due to the previous commits, this command no longer works properly.
It turns out there is an issue with using multiple levels of escaping.
It seems that bash handles the backslashes and single quotes separately
incorrectly processing our escaping. To get around this we are writing the
sqlite command to a shell script file and running that.
This seems to be the only case in WA at the moment that requires this,
if more show up/when WA moves to devlib it should use the devlib shutil
mechanism.
They way we were attempting to get return codes before always gave
us a return code of the previous echo, therefore always `0`.
This commit adds the newline into the last echo.
Previously, only reports that were generated were returned. With this
commit, there will be an entry for each active reporter in the returned
list. If a reporter did not produce a valid report, the entry will be
None.
This ensures consistent output, even if a run time issue causes a
reporter not to produce a report (e.g. if cpufreq events were not
enabled).
It is not possible to read frequencies from a core that has been hotplugged.
The code will now set the current and max frequencies of hotplugged cores
to None.
This still doesn't work for devices that have dynamic hotplug enabled
Commit 724f6e590e changed sysfile_getter
behavior to first tar up copied files and then gzip them. Tarball name
needs to be updated to not include '.gz' extension.
Previously Linaro U-Boot releases had a bug where they used \n\r
as the line ending. This has now been fixed which caused
issues with WA. WA now detects the U-Boot version and uses the
coresponding line ending.
The default boot args have been removed since these cause issues with
the latest Linaro builds, which boot correctly without any bootargs.
Also made a regex string a raw-string.
If uninstall fails, "pm clear" should be called to make sure that the
next time the app is launched it starts from a known state (which would
normally be ensured by the uninstall).
It's possible that there is already a version of an app on target that
differs form the version of the apk on the host. In such cases, WA will
usually try to uninstall the target version and install the host
version.
It's possible that the uninstall may fail. If that happens, it will be
reported as a warning but workload exectuion will proceed with the
target version. In this case, apk_version would have already been set to
that of the host apk. This change ensures that the APK version is
correctly set to the target version (the one that actually ran).
Fixed error in percentage when frequency is 'None'. Now default to 0 in these cases
cpu_utilisation is now a separate parameter in cpustate. Now generates a floating point number representing the utilisation based on the maximum frequency of the capture. No longer performs averaging of values, this can be done as a post-processing step
cpu utilisation now based on the max cpu freq per core, not max captured freq overall
As of ADB 1.0.35/Android N, it will return the exit code of the command that it runs
This code handles this scenario as before WA treated a return code from ADB as an
error with ADB.
- android/workload: emoved an extra bank line between methods
- trace_cmd: define member attribute inside __init__
- adb_shell: ignore pylint warning about too many branches in this case
The old results looked like:
I/TfwActivity(30824): "description": "",
I/TfwActivity(30824): "elapsed_time": 62070,
I/TfwActivity(30824): "error": "NOERROR",
The new format is:
04-04 11:38:04.144 1410 1410 I TfwActivity: "description": "",
04-04 11:38:04.144 1410 1410 I TfwActivity: "elapsed_time": 62009,
04-04 11:38:04.144 1410 1410 I TfwActivity: "error": "NOERROR",
The timeout for the pulling the trace file after the run is being set
based on the time for which the trace was collected. For workloads with
short execution time, but large number of events, the resulting timeout
might be too short. To deal with this, do not let the timout be shorter
than 1 minute.
Depending on the kernel, sched_switch events may be formatted one of two
different ways in the text output. Previously, we've only handled the
"old" format. This commit updates the parser to handle the new format as
well.
This patch tidies up the benchmark code to bring it in line with the
style used in Workload Automation in general. Additionally, the
results from sub-benchmarks are now directly written to a file on the
device as opposed to processing the standard output/error from the
benchmark, which was error prone.
applaunch creates and deploys an auxilary script in order to collect
precise timings. This script invoked busybox with the assumption that it
is in PATH.
Since recent changes mean that it is no longer deployed to /system/bin,
the busybox in not found. With this commit, the full path to busybox
will be passed into the script's template.
If the specified agenda argument is not found in the file system, WA
assumes it is the name of a workload and would then raise an "extension
not found error", which may be confusing if the user's intension was to
specify a path.
Now, WA will first check that neither path separator, nor a '.' are
present in the agenda argument before assuming it is a workload name, and
will provide a less confusing error in that case.
- Deal with the dropped connection on issuing "reboot"
- Introduced a fixed initial delay before polling for connection to
avoid re-connecting to adevice that is still in the process of
shutting down.
cpustate result processor generates a view of the cpu subsystem power
state during execution of a workload from cpu_idle and cpu_frequency
ftraces. There exists a period before the first events in those
categories are seen where the state of the cpu subsystem is (partially)
unknown and it is reported as such by the result processor.
Unknown state usually exists for a relatively short period of time and
is generally not a big deal. For certain kinds of workloads, however, it
may constitude a significant protion of the trace.
Changes in this commit attempt to deal with this by a) reading starting
cpu frequencies and writing them into the trace, and b) nudging each
core to bring it out of idle; this happens before the start marker, so
that the system state between the markers should be completely known.
Previously you could have `override` set to True on parameters that
only existed in the current scope.
Now if you try to override a parameter that doesn't exist higher up
in the hiarchy you will get a ValueError.
Previously if parameters with the same names and override set to True
were added to an extension at the same level one would silently
override the other.
This is no longer the case and an error will be show instead.
Also added tests to check that this is handeled correctly
- Two different parameters may now have the same global alias as long as
their types match
- `extension_asset` resource getter now picks up the path to the mouted
filer from ``remote_assets_path`` global setting.
Added two commands to WA to record and replay input events using revent.
As part of this also added the ability to get a device model from
android and linux device. This may need to be improved in the future.
Added:
get_binary_path: Checks binary_directory for the wanted binary, if
if its not there, it will use which to find a
system one. returns the full path
install_if_needed: will install a binary only if it is not present.
Changes:
- Busybox is now deployed to non-rooted devices
- is_installed has now been removed as the new functions supersede it
- binaries will now always be installed to `binaries_directory` and
not system folders.
- updated workloads to use these new functions
- rt-app and sysbench might still need work
When report_on_target option is set, binary trace will be "reported"
into a text version on the target device. This removes the need for
trace-cmd to be installed on the host, in which case that should not be
reported as an error.
When IO is heavily loaded, the impact of IO schedulers on App launch
time varies. To measure the impact, added io_stress and io_scheduler two
parameters and related jinja2 blocks.
Signed-off-by: Chase Qi <chase.qi@linaro.org>
Since cleanup test block is defined in the device_script.template, the
value of cleanup is needed to render the template properly.
Signed-off-by: Chase Qi <chase.qi@linaro.org>
Juno's bootargs parameter specifies the kernel boot arguments as a
sigle string. However, when it is passed into _boot_via_uboot, it was
expanded as a mapping, causing an error. This fixes that boneheaded
mistake...
As of Android 6.0, apps can request permissions at runtime. If the
target device is running Android 6.0+ these permissions are now automatically
granted.
- Summary metrics are now calculated from the contents of energy.csv and
added to the overall results.
- Added a new "strict" parameter. If this is set to False, the device
check during validation is omitted.
In order to execute as root, the command string gets echo'd into so;
previusly, double quotes were used in echo, which caused any veriables
in the command string to be expanded _before_ it was echoed.
If the value of a classifier was zero (or any other value that
interprets as boolean False), it used to be coverted to an empty entry.
This makes sure that the value gets correctly ropagated.
Previously swipe_to_unlock was not used and conflicted with a method
of the same name.
- swipe_to_unlock() renamed perform_unlock_swipe()
- swipe_to_unlock parameter now takes a direction, this allows swipe unlocking on Android M devices
- ensure_screen_is_on() will now also unlock the screen if swipe_to_unlock is set
- Compiled regular expressions in EVENT_PARSER_MAP now get handled
correctly.
- regex_body_parser now attemts to convert field values to ints,
bringing it inline with the default parser behavior.
- There is now a regex for sched_switch events.
Added two parameters which allow the user to change the strings used
to match the login prompt and the following password prompt to match
their device configurations.
Adjust the wait_for_boot method of Gem5AndroidDevice to no longer wait
for the boot animation to finish if the animation has been
disabled. The service.bootanim.exit property is only set (to 0) when
the animation starts, and is set to 1 when the animation finishes. If
the animation never starts, then the property is not set at
all. Hence, we assume the boot animation has finished, unless the
property has been set.
Moved the commands to resize the shell to their own method. They are
now executed twice. Once as soon as the shell is connected, and a
second time as part of initialize. This latter call takes place after
the isntallation of busybox.
This is a fix to the previous fix
(2510329cdf) that updated get_properties
to store "dumpsys window" output relative to the working_directory. That
commit constructed the path using self.device, which is wrong, as in
this case self itself is the device.
- adb protcol uses "\r\n" for line breaks. This is not handled by
Python's line break translation, as not a file. So spliting on '\n'
when extracting the exit code resulted in stray '\r' in the output.
- adb_shell expects exit code to be echoed on the same line. This may
not have been the case if latest output from executed command was not
a complete line. An extra echo will now ensure that the exit code will
be on its own line even in that case.
- This patch addresses an issue with the arguments passed the
BaseGem5Device __init__. With this patch these are no longer passed
in as they are not required in the base device implementation.
- The streamline instrument can now run on linux clients
because it no longer relies on adb port forwarding
- Updated description.
- Cleaned up code
- Now check for streamline not caiman
- Gatord is now only run once instead of restarted every
job
Commit 95f17702d7 redirected output of
"dumpsys window" to a file (needed for Gem5 support). However, the file
was created in $PWD, which breaks on production devices, as that
location is not writable. This moves the file under the working
directory on the device.
- Refactored the Gem5Device to avoid duplicated code. There is now a
BaseGem5Device which includes all of the shared functionality. The
Gem5LinuxDevice and the Gem5AndroidDevice both inherit from
BaseGem5Device, and LinuxDevice or AndroidDevice, respectively.
If the parameter merge_channels is set to true (false by default).
DAQ channels that have consecutive letters on the end of their names
will be summed together. E.g: A7, A15a, A15b becomes A7, A15
You can also manually specify a channel mapping by setting
merge_channels to a dict. .
- Remove the unbind and rebind for the VirtIO 9P mount method as gem5
now checkpoints the basic state of the device. This allows us to
just mount it assuming that checkpoint have been created correctly.
- Move the processing of Android properties to an internal
method. This allows the Android properties to be extracted without
extracting those of the Linux device.
- Redirect the output from 'dumpsys window' to a file and pull the
file as opposed to extracting the output from the terminal. This is
more reliable in the event that another process writes to the shell.
