- make sure results_list is always instatiated even if no metrics found;
this would previously cause a "used before declaration" error
- Detect that no metrics were extracted from the log and raise a
WorkloadError
cpustates can now handle the lack of a start marker in three ways:
- try: If the start marker is present only the correct section of the trace
will be used, if its not the whole trace will be used.
- error: An error will be raised if the start marker is missing
- ignore: The markers are ignored and the whole trace is always used.
If an exception is raised inside a generator it cannot be continued.
To get around this exceptions are now caught and later output via the
logger.
Also added logger setup when running cpustates as a standalone script
This allows the return value of a function to be cached so that
when it is called in the future the function does not need to
run.
Borrowed from: https://github.com/ARM-software/devlib
Before WA would raise a error message that wasn't very clear.
Now when cpustates tries to nudge cores and and error occurs it
will only show a warning (which promts users to check if the cpu is
hot plugged out) and keep going with the reset of the run without
causing errors in other WA extensions.
Previously the WA device name was used when searching for revent files.
Since most were `generic_android` this made it difficult to keep revent
files for multiple android devices. Now it the device model is used instead.
If a file with the device model is not found it will fall back to the WA
device name.
kick_off has been changed to behave the same as AndroidDevice.
Said changes caused kill all to fail on rooted devices. Killall will now
behave in the same way as kick_off, if specifically told to (or not to)
run as root it will. Otherwise it will run as root if the device is rooted
Servo is a debug board used for Chromium OS test and development. Among other uses, it allows
access to the built in power monitors (if present) of a Chrome OS device. More information on
Servo board can be found in the link bellow:
https://www.chromium.org/chromium-os/servo
based on: 03ede10739
and: 9a0dc55b55
These are now gathered via `getprop` rather than trying to parse the
build.prop file directly.
This fixes issues with build.prop files that have imports.
Blogbench is a portable filesystem benchmark that tries to reproduce the
load of a real-world busy file server.
Signed-off-by: Chase Qi <chase.qi@linaro.org>
The stock camera app as of Android M has changed. This commit updates
the ui automation to work with this new app. As part of this change
it was required to bump the API level of the ui automation to 18.
Also made the teardown of the capture workload close the app like the
record workload.
Added splitVersion and compareVersions functions allow versions strings
like "3.2.045" to be compared.
Also fixed the build script to now copy to the correct folder
Some characters would be interpreted by the shell thus breaking the
command. Adding quotes around the parameters solved this.
N.B Space still needs to be replaced.
stress-ng will stress test a computer system in various selectable ways.
It was designed to exercise various physical subsystems of a computer as
well as the various operating system kernel interfaces.
Signed-off-by: Chase Qi <chase.qi@linaro.org>
Previously the sensor name was just appeneded to the end of the
previous sensors name.
Now the hwmon name is added as a classifier of the metric.
If the hwmon sensor has a label, the metric will use this for its name,
if it does not then the sensors kind and ID will be used e.g. temp3
Due to the previous commits, this command no longer works properly.
It turns out there is an issue with using multiple levels of escaping.
It seems that bash handles the backslashes and single quotes separately
incorrectly processing our escaping. To get around this we are writing the
sqlite command to a shell script file and running that.
This seems to be the only case in WA at the moment that requires this,
if more show up/when WA moves to devlib it should use the devlib shutil
mechanism.
They way we were attempting to get return codes before always gave
us a return code of the previous echo, therefore always `0`.
This commit adds the newline into the last echo.
Previously, only reports that were generated were returned. With this
commit, there will be an entry for each active reporter in the returned
list. If a reporter did not produce a valid report, the entry will be
None.
This ensures consistent output, even if a run time issue causes a
reporter not to produce a report (e.g. if cpufreq events were not
enabled).
It is not possible to read frequencies from a core that has been hotplugged.
The code will now set the current and max frequencies of hotplugged cores
to None.
This still doesn't work for devices that have dynamic hotplug enabled
Commit 724f6e590e changed sysfile_getter
behavior to first tar up copied files and then gzip them. Tarball name
needs to be updated to not include '.gz' extension.
Previously Linaro U-Boot releases had a bug where they used \n\r
as the line ending. This has now been fixed which caused
issues with WA. WA now detects the U-Boot version and uses the
coresponding line ending.
The default boot args have been removed since these cause issues with
the latest Linaro builds, which boot correctly without any bootargs.
Also made a regex string a raw-string.
If uninstall fails, "pm clear" should be called to make sure that the
next time the app is launched it starts from a known state (which would
normally be ensured by the uninstall).
It's possible that there is already a version of an app on target that
differs form the version of the apk on the host. In such cases, WA will
usually try to uninstall the target version and install the host
version.
It's possible that the uninstall may fail. If that happens, it will be
reported as a warning but workload exectuion will proceed with the
target version. In this case, apk_version would have already been set to
that of the host apk. This change ensures that the APK version is
correctly set to the target version (the one that actually ran).
Fixed error in percentage when frequency is 'None'. Now default to 0 in these cases
cpu_utilisation is now a separate parameter in cpustate. Now generates a floating point number representing the utilisation based on the maximum frequency of the capture. No longer performs averaging of values, this can be done as a post-processing step
cpu utilisation now based on the max cpu freq per core, not max captured freq overall
As of ADB 1.0.35/Android N, it will return the exit code of the command that it runs
This code handles this scenario as before WA treated a return code from ADB as an
error with ADB.
- android/workload: emoved an extra bank line between methods
- trace_cmd: define member attribute inside __init__
- adb_shell: ignore pylint warning about too many branches in this case
The old results looked like:
I/TfwActivity(30824): "description": "",
I/TfwActivity(30824): "elapsed_time": 62070,
I/TfwActivity(30824): "error": "NOERROR",
The new format is:
04-04 11:38:04.144 1410 1410 I TfwActivity: "description": "",
04-04 11:38:04.144 1410 1410 I TfwActivity: "elapsed_time": 62009,
04-04 11:38:04.144 1410 1410 I TfwActivity: "error": "NOERROR",
The timeout for the pulling the trace file after the run is being set
based on the time for which the trace was collected. For workloads with
short execution time, but large number of events, the resulting timeout
might be too short. To deal with this, do not let the timout be shorter
than 1 minute.
Depending on the kernel, sched_switch events may be formatted one of two
different ways in the text output. Previously, we've only handled the
"old" format. This commit updates the parser to handle the new format as
well.
This patch tidies up the benchmark code to bring it in line with the
style used in Workload Automation in general. Additionally, the
results from sub-benchmarks are now directly written to a file on the
device as opposed to processing the standard output/error from the
benchmark, which was error prone.
applaunch creates and deploys an auxilary script in order to collect
precise timings. This script invoked busybox with the assumption that it
is in PATH.
Since recent changes mean that it is no longer deployed to /system/bin,
the busybox in not found. With this commit, the full path to busybox
will be passed into the script's template.
If the specified agenda argument is not found in the file system, WA
assumes it is the name of a workload and would then raise an "extension
not found error", which may be confusing if the user's intension was to
specify a path.
Now, WA will first check that neither path separator, nor a '.' are
present in the agenda argument before assuming it is a workload name, and
will provide a less confusing error in that case.
- Deal with the dropped connection on issuing "reboot"
- Introduced a fixed initial delay before polling for connection to
avoid re-connecting to adevice that is still in the process of
shutting down.
cpustate result processor generates a view of the cpu subsystem power
state during execution of a workload from cpu_idle and cpu_frequency
ftraces. There exists a period before the first events in those
categories are seen where the state of the cpu subsystem is (partially)
unknown and it is reported as such by the result processor.
Unknown state usually exists for a relatively short period of time and
is generally not a big deal. For certain kinds of workloads, however, it
may constitude a significant protion of the trace.
Changes in this commit attempt to deal with this by a) reading starting
cpu frequencies and writing them into the trace, and b) nudging each
core to bring it out of idle; this happens before the start marker, so
that the system state between the markers should be completely known.
Previously you could have `override` set to True on parameters that
only existed in the current scope.
Now if you try to override a parameter that doesn't exist higher up
in the hiarchy you will get a ValueError.
Previously if parameters with the same names and override set to True
were added to an extension at the same level one would silently
override the other.
This is no longer the case and an error will be show instead.
Also added tests to check that this is handeled correctly
- Two different parameters may now have the same global alias as long as
their types match
- `extension_asset` resource getter now picks up the path to the mouted
filer from ``remote_assets_path`` global setting.
Added two commands to WA to record and replay input events using revent.
As part of this also added the ability to get a device model from
android and linux device. This may need to be improved in the future.
Added:
get_binary_path: Checks binary_directory for the wanted binary, if
if its not there, it will use which to find a
system one. returns the full path
install_if_needed: will install a binary only if it is not present.
Changes:
- Busybox is now deployed to non-rooted devices
- is_installed has now been removed as the new functions supersede it
- binaries will now always be installed to `binaries_directory` and
not system folders.
- updated workloads to use these new functions
- rt-app and sysbench might still need work
When report_on_target option is set, binary trace will be "reported"
into a text version on the target device. This removes the need for
trace-cmd to be installed on the host, in which case that should not be
reported as an error.
When IO is heavily loaded, the impact of IO schedulers on App launch
time varies. To measure the impact, added io_stress and io_scheduler two
parameters and related jinja2 blocks.
Signed-off-by: Chase Qi <chase.qi@linaro.org>
Since cleanup test block is defined in the device_script.template, the
value of cleanup is needed to render the template properly.
Signed-off-by: Chase Qi <chase.qi@linaro.org>
Juno's bootargs parameter specifies the kernel boot arguments as a
sigle string. However, when it is passed into _boot_via_uboot, it was
expanded as a mapping, causing an error. This fixes that boneheaded
mistake...
As of Android 6.0, apps can request permissions at runtime. If the
target device is running Android 6.0+ these permissions are now automatically
granted.
- Summary metrics are now calculated from the contents of energy.csv and
added to the overall results.
- Added a new "strict" parameter. If this is set to False, the device
check during validation is omitted.
In order to execute as root, the command string gets echo'd into so;
previusly, double quotes were used in echo, which caused any veriables
in the command string to be expanded _before_ it was echoed.
If the value of a classifier was zero (or any other value that
interprets as boolean False), it used to be coverted to an empty entry.
This makes sure that the value gets correctly ropagated.
Previously swipe_to_unlock was not used and conflicted with a method
of the same name.
- swipe_to_unlock() renamed perform_unlock_swipe()
- swipe_to_unlock parameter now takes a direction, this allows swipe unlocking on Android M devices
- ensure_screen_is_on() will now also unlock the screen if swipe_to_unlock is set
- Compiled regular expressions in EVENT_PARSER_MAP now get handled
correctly.
- regex_body_parser now attemts to convert field values to ints,
bringing it inline with the default parser behavior.
- There is now a regex for sched_switch events.
Added two parameters which allow the user to change the strings used
to match the login prompt and the following password prompt to match
their device configurations.
Adjust the wait_for_boot method of Gem5AndroidDevice to no longer wait
for the boot animation to finish if the animation has been
disabled. The service.bootanim.exit property is only set (to 0) when
the animation starts, and is set to 1 when the animation finishes. If
the animation never starts, then the property is not set at
all. Hence, we assume the boot animation has finished, unless the
property has been set.
Moved the commands to resize the shell to their own method. They are
now executed twice. Once as soon as the shell is connected, and a
second time as part of initialize. This latter call takes place after
the isntallation of busybox.
This is a fix to the previous fix
(2510329cdf) that updated get_properties
to store "dumpsys window" output relative to the working_directory. That
commit constructed the path using self.device, which is wrong, as in
this case self itself is the device.
- adb protcol uses "\r\n" for line breaks. This is not handled by
Python's line break translation, as not a file. So spliting on '\n'
when extracting the exit code resulted in stray '\r' in the output.
- adb_shell expects exit code to be echoed on the same line. This may
not have been the case if latest output from executed command was not
a complete line. An extra echo will now ensure that the exit code will
be on its own line even in that case.
- This patch addresses an issue with the arguments passed the
BaseGem5Device __init__. With this patch these are no longer passed
in as they are not required in the base device implementation.
- The streamline instrument can now run on linux clients
because it no longer relies on adb port forwarding
- Updated description.
- Cleaned up code
- Now check for streamline not caiman
- Gatord is now only run once instead of restarted every
job
Commit 95f17702d7 redirected output of
"dumpsys window" to a file (needed for Gem5 support). However, the file
was created in $PWD, which breaks on production devices, as that
location is not writable. This moves the file under the working
directory on the device.
- Refactored the Gem5Device to avoid duplicated code. There is now a
BaseGem5Device which includes all of the shared functionality. The
Gem5LinuxDevice and the Gem5AndroidDevice both inherit from
BaseGem5Device, and LinuxDevice or AndroidDevice, respectively.
If the parameter merge_channels is set to true (false by default).
DAQ channels that have consecutive letters on the end of their names
will be summed together. E.g: A7, A15a, A15b becomes A7, A15
You can also manually specify a channel mapping by setting
merge_channels to a dict. .
- Remove the unbind and rebind for the VirtIO 9P mount method as gem5
now checkpoints the basic state of the device. This allows us to
just mount it assuming that checkpoint have been created correctly.
- Move the processing of Android properties to an internal
method. This allows the Android properties to be extracted without
extracting those of the Linux device.
- Redirect the output from 'dumpsys window' to a file and pull the
file as opposed to extracting the output from the terminal. This is
more reliable in the event that another process writes to the shell.
- Replaced hard-coded pexpect expect string with UNIQUE_PROMPT.
- Changed the capture_screen debug to a warning to make sure that the
user knows when it happens.
- Fixed the logic for checking when a file exists. Previously, if the
output could not correctly be processed (ValueError) then we just
assumed that the file existed if there was any output at all. This
is clearly not a good default. Changed to default to False if it was
not able to process the output as this seems to be the safest
option.
- Changed ad hoc filename extraction to use os.path.basename.
- Removed the processing of some kwargs and defaults that are handled
by the parent class.
- Stopped overriding some paramaters which were purely defined in the
Gem5Device.
- Added the gem5_binary option to the agenda which allows a different
gem5 binary to be specified. This allows WA to be used with
different levels of gem5 debugging. as well as allowing non-standard
gem5 binary names and locations.
