Create two new classes to deal with workloads for which we want to
collect frame metrics/jank percentages.
ApkUiautoJankTestWorkload inherits from ApkUiautoWorkload and instantiates
a UiAutomatorJankTestGUI gui member.
UiAutomatorJankTestGUI (inherits from UiAutomatorGUI) handles most of the
new logic for jank tests, including using a regular expression to parse the
final output.
Jank testing involves using a different library (JankTestBase) that our
testing class needs to extend. Since we already make the base testing
class UiAutomation extend BaseUiAutomation, we cannot use the UiAutomation
class for our tests.
The uiauto_class property is hardcoded to "UiAutomation" though, so this
patch adds a uiauto_jank_class property for the Jank test class.
While at it, we also make the uiauto_runner configurable (currently hardcoded
to the old deprecated android.support version of it) so that our workloads
can pass whatever runner they need to use.
In terms of stages for testing, their current names are the following:
stages = ['setup', 'runWorkload', 'extractResults', 'teardown']
Those won't work for jank tests. For some reason the jank test library wants
to see the testing function name contain or start with a "test" string
pattern on it.
This behavior doesn't seem to be documented, but at least with the current
version of the JankTestBase library we have this restriction.
To make things work despite this restriction, I've added separate jank
test stages that can be provided by the workload. The generic workload
code will use these stages to generate the instrumentation invocation
command, and the provided jank test stages (essentially jank test
functions) will be invoked one by one.
Finally, we make the UiAutomatorGUI's _execute method return the output
of the execution so the workload can freely process and pattern-match
the results, like the UiBenchJankTests workload.
Currently the uiauto library is built on top of the deprecated android.support
libraries. Create an updated uiauto library (uiauto-androidx) using androidx
and keep it alongside the old one for compatibility reasons.
As we update workload-automation going forward, we should update the workloads
to use the androidx version of uiauto.
This has been tested with the latest gradle version available at the time
(8.8).
The key for 'include#' can now be either a scalar or a list.
A scalar triggers the same behaviour as before.
If the value is a list it must be a list of scalars (filepaths).
The paths will be loaded and merged in order, and finally the resulting
dict is included into the current scope.
Introduce 'unlock_screen' option in order to help in automating Android
tests by unlocking device screen automatically. Surely this works only
if no passcode is set.
'unlock_screen' option implicitly requires turning on the screen. IOW,
it will override value of 'screen_on' option.
'diagonal', 'vertical' and 'horizontal' are valid values for
'unlock_screen' option as of now.
Note that this patch depends on
https://github.com/ARM-software/devlib/pull/659 in devlib repo.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
"--user <USER_ID" (current user: 0) option is added to
activity manager (am) command because of "Invalid userId" command.
Tested with other benchmarks (geekbench) as well.
WA_PLUGIN_PATHS currently contains a list of filesystem paths to scan
for plugins. This is appropriate for end-user plugins, but this is
problematic for plugins distributed by a 3rd party, such as a plugin
installed from PyPI.
In those cases, the path to the sources is unknown and typically depends
on the specify Python version, local setup etc. What is constant is
Python name of the package, e.g. "lisa.wa.plugins".
Extend the input allowed in WA_PLUGIN_PATHS by trying to load entries as
a Python package name if:
* There is no filesystem path with that name
* The entry is a "relative path" (from an fs point of view)
In some workloads it is beneficial to be able to provide arguments
when launching the required APK. Add a new `arpk_arguments` property
to allow a workload to provide a dict of parameter names and values
that should be used when launching the APK.
The python types of the parameters are used to determine the data
type provided to the APK. Currently supported types are string, bool,
int and float.
Implements support for dynamically loading additional RuntimeConfig
and associated RuntimeParameter that are defined in a plugin.
Currently, the various RuntimeConfig's are hard coded in a list within
WA. This patch extends RuntimeParameterManager to use PluginLoader to
load RuntimeConfig classes and append to the hard coded list.
The implementation, as written, does not allow loading RuntimeConfig
from a plugin if it has the same name as one of the hard coded
RuntimeConfig. This is meant to prevent conflicts and unexpected
behavior.
The older versions of the plugin caused problems building with newer
NDK versions due to a lack of MIPS support.
This also required upgrading to a version of Gradle that knows about
the Google Maven repository.
Don't assume the first job iteration is already in the workload cache.
This may not always be the case, for example with the random execution
order a later iteration can be processed first.
Instead check to see if the job id is present or not.
If the device does not have the hoptplug module installed, avoid
unnecessary querying of the device to check the number of cpus which
can cause issues with some devices.
On systems that do not have git installed WA will currently fail
to install with a FileNotFound Exception. If git is not present then
we will not have a commit hash so just ignore this error.
On the latest version of android (currently Q) for applications that
are designed to run on older versions of android, an additional
screen asking to confirm the required permissions can popup.
Enable confirming of the granted permissions.
Add an additional `run_completed` reboot policy for when a run
has finished.
This complements the `initial` reboot policy and aims to leave
the device in a fresh state after WA has finished executing.
The WA default `sudo_cmd` is out of date compared to devlib.
Update the parameter to use the value directly from devlib
to prevent these from being out of sync in the future.
Adds parameters needed for WA to support file transfer polling.
``poll_transfers`` of type ``bool``, default ``True`` sets whether
transfers should be polled
``transfer_wait_no_poll`` controls the initial time in seconds that the
poller should wait for the transfer to complete before polling its
progress.
Ensure that runtime and workload parameters specified across multiple
config files and the config section of the agenda are merged rather than
overwritten.
Update our version of pylint to use the latest version and update the
codebase to comply with the majority of the updates.
For now disable the additional checks for `super-with-arguments`,
`useless-object-inheritance`, `raise-missing-from`, `no-else-raise`,
`no-else-break`, `no-else-continue` to be consistent with the existing
codebase.
Add a global configuration parameter ``bail_on_job_failure`` that
allows all remaining jobs in a run to be skipped should a job fail its
initial execution and its retries. This is by default disabled.
Avoid recursing into subdirectory of folders containing __meta, since
they are not of interest and recursing can take a very large amount of
time if there are lot of files, like if there is a sysfs dump.
