Don't assume the first job iteration is already in the workload cache.
This may not always be the case, for example with the random execution
order a later iteration can be processed first.
Instead check to see if the job id is present or not.
If the device does not have the hoptplug module installed, avoid
unnecessary querying of the device to check the number of cpus which
can cause issues with some devices.
On systems that do not have git installed WA will currently fail
to install with a FileNotFound Exception. If git is not present then
we will not have a commit hash so just ignore this error.
With the perf instrument configured as:
perf:
perf_type: simpleperf
command: stat
optionstring: '-a --interval-only-values --csv'
WA fails to parse simpleperf's output:
INFO Extracting reports from target...
ERROR Error in instrument perf
ERROR File "/work/workload_automation/workload-automation/wa/framework/instrument.py", line 272, in __call__
ERROR self.callback(context)
ERROR File "/work/workload_automation/workload-automation/wa/instruments/perf.py", line 142, in update_output
ERROR self._process_simpleperf_output(context)
ERROR File "/work/workload_automation/workload-automation/wa/instruments/perf.py", line 155, in _process_simpleperf_output
ERROR self._process_simpleperf_stat_output(context)
ERROR File "/work/workload_automation/workload-automation/wa/instruments/perf.py", line 233, in _process_simpleperf_stat_output
ERROR self._process_simpleperf_stat_from_csv(stat_file, context, label)
ERROR File "/work/workload_automation/workload-automation/wa/instruments/perf.py", line 245, in _process_simpleperf_stat_from_csv
ERROR context.add_metric('{}_{}'.format(label, row[1]), row[0], 'count', classifiers=classifiers)
ERROR File "/work/workload_automation/workload-automation/wa/framework/execution.py", line 222, in add_metric
ERROR self.output.add_metric(name, value, units, lower_is_better, classifiers)
ERROR File "/work/workload_automation/workload-automation/wa/framework/output.py", line 142, in add_metric
ERROR self.result.add_metric(name, value, units, lower_is_better, classifiers)
ERROR File "/work/workload_automation/workload-automation/wa/framework/output.py", line 390, in add_metric
ERROR metric = Metric(name, value, units, lower_is_better, classifiers)
ERROR File "/work/workload_automation/workload-automation/wa/framework/output.py", line 653, in __init__
ERROR self.value = numeric(value)
ERROR File "/work/workload_automation/devlib/devlib/utils/types.py", line 88, in numeric
ERROR raise ValueError('Not numeric: {}'.format(value))
ERROR
ERROR ValueError(Not numeric: Performance counter statistics)
With the above options, the csv that simpleperf produces looks like
this:
Performance counter statistics,
123456789,raw-l1-dtlb,,(60%),
42424242,raw-l1-itlb,,(60%),
Total test time,1.001079,seconds,
Performance counter statistics,
123456789,raw-l1-dtlb,,(60%),
42424242,raw-l1-itlb,,(60%),
Total test time,2.001178,seconds,
Performance counter statistics,
[...]
That is, with "--interval-only-values", the "Performance counter
statistics," header is repeated every interval. WA current expects
it only in the first line. Modify the condition so that it is ignored
every time we find it in the file and not just the first time.
On the latest version of android (currently Q) for applications that
are designed to run on older versions of android, an additional
screen asking to confirm the required permissions can popup.
Enable confirming of the granted permissions.
Add an additional `run_completed` reboot policy for when a run
has finished.
This complements the `initial` reboot policy and aims to leave
the device in a fresh state after WA has finished executing.
Resourceid and classes have been modified so update the
workload to handle these cases.
Additionally on some devices regex matches appear to fail
so workaround to match separately.
Add support for newer version of the apk.
Also add support for differing screen sizes, on larger devices
the direction of swipe to change slide differs, perform both
horizontal and vertical swipes to satisfy both layouts.
The WA default `sudo_cmd` is out of date compared to devlib.
Update the parameter to use the value directly from devlib
to prevent these from being out of sync in the future.
Adds parameters needed for WA to support file transfer polling.
``poll_transfers`` of type ``bool``, default ``True`` sets whether
transfers should be polled
``transfer_wait_no_poll`` controls the initial time in seconds that the
poller should wait for the transfer to complete before polling its
progress.
Ensure that runtime and workload parameters specified across multiple
config files and the config section of the agenda are merged rather than
overwritten.
Update our version of pylint to use the latest version and update the
codebase to comply with the majority of the updates.
For now disable the additional checks for `super-with-arguments`,
`useless-object-inheritance`, `raise-missing-from`, `no-else-raise`,
`no-else-break`, `no-else-continue` to be consistent with the existing
codebase.
The PCMark Work2.0 data-set is cleared and downloaded before each run. This
operation is time-consuming and pollutes the benchmark instrumentation.
Disabling clear_data_on_reset for the PCMark workload bypass this per-run
download.
This version replaces the previous uiauto version of Speedometer with a new
version.
* Supports both chrome and chromium again, this is selected with the
chrome_package parameter.
* No longer needs internet access.
* Version 1.0 of Speedometer is no longer supported.
* Requires root:
- sometimes uiautomator dump doesn't capture the score if not run as root
- need to modify the browser's XML preferences file to bypass T&C acceptance
screen
Add a global configuration parameter ``bail_on_job_failure`` that
allows all remaining jobs in a run to be skipped should a job fail its
initial execution and its retries. This is by default disabled.
Avoid recursing into subdirectory of folders containing __meta, since
they are not of interest and recursing can take a very large amount of
time if there are lot of files, like if there is a sysfs dump.
Add cascading deletes to foreign keys as well as a rule to delete large
objects when artifacts are deleted.
Deleting a run entry should delete all dependent data of that run.
As WA currently supports either a single logcat dump after each job,
or fixed rate polling of logcat, it is possible that the fixed size
logcat buffer wraps around and overwrites data between each dump or
poll. This data may be used by output processors that should be
notified of the loss.
This change allows the detection of buffer wrapping by inserting a
known log entry into the buffer, although it cannot say how much data
was lost, and only applies to the "main" logcat buffer.
If buffer wrap is detected, a warning is logged by WA.
To prevent long timeouts occurring during to file locking on
both reads and writes replace locking with
atomic writes.
While this may results in cache entries being overwritten,
the amount of time used in duplicated retrievals will likely
be saved with the prevention of stalls due to waiting to
acquire the file lock.
To simulate atomic writes, use a context manager to write to
a temporary file location and then rename over the original
file.
This is performed using the `safe_move` method which performs
this operation and handles cases where the source and destination
are on separate file systems.
While the Louie system operated on weakrefs for the callback
functions, the priority list wrapper did not. This difference led to
weakrefs to callback functions being compared to strong references in
list element operations within Louie's disconnect method, so that
handler methods were not disconnected from signals.
Converting the receiver to a weakref then allowed Louie to operate as
normal, which may include deleting and re-appending the handler method
to the receivers list. As ``append`` is a dummy method that allows the
priority list implementation, the handler method is then never added
back to the list of connected functions, so we must ``add`` it after
``connect`` is called.
Also included is a testcase to confirm the proper disconnection of
signals.
report provides a summary of a run and an optional list of all
jobs in the run, with any events that might have occurred during each
job and their current status.
report allows an output directory to be specified or will attempt to
discover possible output directories within the current directory.
JobState, previously handled by RunState, is now held in the
Job.
Changes and accesses to a Job's status access the Job's
JobState directly, so that there is only one place now that each Job's
state data is tracked.
This also means there is no use for update_job in RunState.
When setting the job status through ExecutionContext, this change
should be accompanied by an update to the state file, so that the state
file accurately reflects execution state.
As Jobs should not be aware of the output, this method is added to
ExecutionContext, and couples setting job state with writing to the
state file.
Pandas removed .ix as a way to iterate the index, .loc is the replacement
in most cases. Jankbench as a workload fails on a clean install due to
this call.
Replacing this works for me on a native install of Lisa with Ubuntu 20.04
Signed-off-by: Chris Redpath <chris.redpath@arm.com>
Alter the element to check popups have closed to be present
on any main screen and ensure we switch to the homescreen of the
app before performing setup steps.
Provide warning to user when attempting to set a depreciated
parameter instead of during validation and only raise the warning
if a value has been explicitly provided.
Rework how the result matching is performed. Some tests from
gfxbench provide more than 1 score per test and
some provide their output in a different format to others.
Update the matching to perform more flexible matching as well
as dealing with entries that do not fit on a single results screen.
Use the original file path to create a lock file in the system temp
directory. This prevents issues where we are attempting to lock a file
where wa does not have permission to create new files.
* Pcmark sometimes auto installs without need for clicking button,
in such cases workload throws UiObjectNotFound exception.
* Added logic to check for installation button existence.
* Increased install wait time to 5 mins.
* Updated regex to reflect correct test name.
* Enabling/disabling tests on setup was missing tesselation on some devices
so changed order of toggle.
* Sometimes whilst collecting scores the workload grabs the wrong score.
Updated to check the name of the test before grabbing the score.
* Tested on mate20, xperia, s9-exynos, s10-exynos, pixel-4
Make sure we only override parameters that are present in the current
config. This allows for connection parameters to be supplied for a
platform but only overridden if required for the connection.
Allow for overriding connection parameters on a per platform basis, and
make the `host` parameter for `Juno` optional as this can be auto
detected via the serial connection.
Instead of supplying only the parameter name and value to be set as a
default, allow for replacing the entire parameter object as this allow
more control over what needs overriding for a particular platform.
Some devices print non standard characters to logcat. If an error
occurs when parsing the output, replace the offending character instead
of raising an error.
* Workload was failing when chrome was not set as default broser so
altered to use chrome every time.
* Changed workload to an ApkuiAutoWorkload since chrome is now a
dependency.
* Refactored opening speedometer to new method.
* Added wait time for scores to show up when test finished.
Allow specifying a ConfigutationPoint is deprecated. This means that any
supplied configuration will not be used however execution will continue
with a warning displayed to the user.
Correctly handle the presence of both an element and its toggle in the
input, and handle them base on order, e.g.
toggle_set(['x', 'y', '~x']) --> {'y', '~x'}
toggle_set(['~x', 'y', 'x']) --> {'y', 'x'}
Move cleanup_assets from Workload up into TargetedPlugin. This way,
Instruments may also utilize it if they deploy assets.
More generally, it makes sense for it to be inside TargetedPlugin, as
any plugin that interacts with the target may conceivably need to clean
up.
