Implements support for dynamically loading additional RuntimeConfig
and associated RuntimeParameter that are defined in a plugin.
Currently, the various RuntimeConfig's are hard coded in a list within
WA. This patch extends RuntimeParameterManager to use PluginLoader to
load RuntimeConfig classes and append to the hard coded list.
The implementation, as written, does not allow loading RuntimeConfig
from a plugin if it has the same name as one of the hard coded
RuntimeConfig. This is meant to prevent conflicts and unexpected
behavior.
Currently, when processing the output of 'simpleperf stat', wa does not
skip the header and tries to process part of it as a number, leading
to type errors. This change skips the header (line starting with '#').
Furthermore, some events (e.g. cpu-clock or task-clock) include "(ms)"
in their count value and are floats instead of integers. Because of
this, when either of those is included, processing metrics fails due to
assuming every metric is an integer. Then another error happens when the
code tries to split the line on '(' assuming that there's only one set
of those around the percentage.
This change removes "(ms)" from the line
before it's processed and properly determines whether 'count' is an
integer or a float before attempting to convert it.
The non corporate version of geekbench5 didn't work although the code
had everything needed, except for a number of tiny required tweaks:
1. Add '5' in the supported versions in __init__.py
2. Fix the name of the android package in__init__.py and
UiAutomation.java
3. Improve handling of minorVersion to fix potential exception when we
don't specify the minorVersion number in the yaml file. Launching
geekbench5 works fine when it's the only one installed. But if you
have multiple versions, then using the version string in the yaml
agenda didn't like specifying '5' as the version and threw exception
out of bound because we assume '5.X' as input. No reason I'm aware of
to force support for a specific version of geekbench5. So keep it
relaxed until we know for sure it breaks with a specific version.
Signed-off-by: Qais Yousef <qais.yousef@arm.com>
Which works as the corporate version except for a different in package
name and a set of Fixed Time Tests that don't exist on free version.
Only support v 4.X and v5.X as that's what's available.
Note there's a clash with glbenchmark package name. glbenchmark is an
ancient version provided by the same developers but was superseded by
gfxbench. The version checks in both workloads should ensure we get the
right one in the unlikely case both are installed.
Signed-off-by: Qais Yousef <qais.yousef@arm.com>
The test was hanging for a long time waiting for RUN text. Checking for
description first prevents that.
Signed-off-by: Qais Yousef <qais.yousef@arm.com>
The code for reading the results is almost the same as for Antutu 8,
but it needs to be adjusted to account for a slightly different set
of benchmarks.
At least on my device, Antutu 9 takes over 10 minutes to run, so increase
the timeout to 20 minutes.
The older versions of the plugin caused problems building with newer
NDK versions due to a lack of MIPS support.
This also required upgrading to a version of Gradle that knows about
the Google Maven repository.
When running perf/simpleperf stat, the report_option_string,
report_sample_options and run_report_sample configuration parameters
don't make sense. Instead of quietly ignoring them, raise a
ConfigError so that the user can fix the agenda.
devlib learnt to use report-sample in
ARM-software/devlib@fe2fe3ae04 ("collector/perf: run simpleperf
report-sample in the target if requested"). Adapt the perf instrument
to use the new parameters of the PerfCollector.
We need to make those attributes class-attributes, to make sure they are still
defined in subsequent jobs. We still access them through 'self', however.
Don't assume the first job iteration is already in the workload cache.
This may not always be the case, for example with the random execution
order a later iteration can be processed first.
Instead check to see if the job id is present or not.
If the device does not have the hoptplug module installed, avoid
unnecessary querying of the device to check the number of cpus which
can cause issues with some devices.
On systems that do not have git installed WA will currently fail
to install with a FileNotFound Exception. If git is not present then
we will not have a commit hash so just ignore this error.
With the perf instrument configured as:
perf:
perf_type: simpleperf
command: stat
optionstring: '-a --interval-only-values --csv'
WA fails to parse simpleperf's output:
INFO Extracting reports from target...
ERROR Error in instrument perf
ERROR File "/work/workload_automation/workload-automation/wa/framework/instrument.py", line 272, in __call__
ERROR self.callback(context)
ERROR File "/work/workload_automation/workload-automation/wa/instruments/perf.py", line 142, in update_output
ERROR self._process_simpleperf_output(context)
ERROR File "/work/workload_automation/workload-automation/wa/instruments/perf.py", line 155, in _process_simpleperf_output
ERROR self._process_simpleperf_stat_output(context)
ERROR File "/work/workload_automation/workload-automation/wa/instruments/perf.py", line 233, in _process_simpleperf_stat_output
ERROR self._process_simpleperf_stat_from_csv(stat_file, context, label)
ERROR File "/work/workload_automation/workload-automation/wa/instruments/perf.py", line 245, in _process_simpleperf_stat_from_csv
ERROR context.add_metric('{}_{}'.format(label, row[1]), row[0], 'count', classifiers=classifiers)
ERROR File "/work/workload_automation/workload-automation/wa/framework/execution.py", line 222, in add_metric
ERROR self.output.add_metric(name, value, units, lower_is_better, classifiers)
ERROR File "/work/workload_automation/workload-automation/wa/framework/output.py", line 142, in add_metric
ERROR self.result.add_metric(name, value, units, lower_is_better, classifiers)
ERROR File "/work/workload_automation/workload-automation/wa/framework/output.py", line 390, in add_metric
ERROR metric = Metric(name, value, units, lower_is_better, classifiers)
ERROR File "/work/workload_automation/workload-automation/wa/framework/output.py", line 653, in __init__
ERROR self.value = numeric(value)
ERROR File "/work/workload_automation/devlib/devlib/utils/types.py", line 88, in numeric
ERROR raise ValueError('Not numeric: {}'.format(value))
ERROR
ERROR ValueError(Not numeric: Performance counter statistics)
With the above options, the csv that simpleperf produces looks like
this:
Performance counter statistics,
123456789,raw-l1-dtlb,,(60%),
42424242,raw-l1-itlb,,(60%),
Total test time,1.001079,seconds,
Performance counter statistics,
123456789,raw-l1-dtlb,,(60%),
42424242,raw-l1-itlb,,(60%),
Total test time,2.001178,seconds,
Performance counter statistics,
[...]
That is, with "--interval-only-values", the "Performance counter
statistics," header is repeated every interval. WA current expects
it only in the first line. Modify the condition so that it is ignored
every time we find it in the file and not just the first time.
On the latest version of android (currently Q) for applications that
are designed to run on older versions of android, an additional
screen asking to confirm the required permissions can popup.
Enable confirming of the granted permissions.
Add an additional `run_completed` reboot policy for when a run
has finished.
This complements the `initial` reboot policy and aims to leave
the device in a fresh state after WA has finished executing.
Resourceid and classes have been modified so update the
workload to handle these cases.
Additionally on some devices regex matches appear to fail
so workaround to match separately.
Add support for newer version of the apk.
Also add support for differing screen sizes, on larger devices
the direction of swipe to change slide differs, perform both
horizontal and vertical swipes to satisfy both layouts.
The WA default `sudo_cmd` is out of date compared to devlib.
Update the parameter to use the value directly from devlib
to prevent these from being out of sync in the future.
Adds parameters needed for WA to support file transfer polling.
``poll_transfers`` of type ``bool``, default ``True`` sets whether
transfers should be polled
``transfer_wait_no_poll`` controls the initial time in seconds that the
poller should wait for the transfer to complete before polling its
progress.
Ensure that runtime and workload parameters specified across multiple
config files and the config section of the agenda are merged rather than
overwritten.
Update our version of pylint to use the latest version and update the
codebase to comply with the majority of the updates.
For now disable the additional checks for `super-with-arguments`,
`useless-object-inheritance`, `raise-missing-from`, `no-else-raise`,
`no-else-break`, `no-else-continue` to be consistent with the existing
codebase.
The PCMark Work2.0 data-set is cleared and downloaded before each run. This
operation is time-consuming and pollutes the benchmark instrumentation.
Disabling clear_data_on_reset for the PCMark workload bypass this per-run
download.
