API level 27 (Android 8.1/Oreo) doesn't report a frame metric called
"deadline missed", but the JankTest library expects to be able to find
data for that metric. If it is not found, an assertion is reported and
the run fails.
Given Android Oreo is reasonably old at this point, set the minimum SDK
version to 28 so we can gather all the metrics we need.
Also bump the targetSdkVersion to make it clear what version was used
for putting this code together.
I noticed we were sometimes getting stale object exceptions. To help prevent
that situation from happening, pass around resource id's for the objects and
only fetch an instance of the objects whenever we need to click/scroll.
This means the time between fetching the object and using it is short, and
hopefully the object won't go stale or get recycled in that brief moment,
preventing the stale exceptions.
On tablets we have a landscape view by default, so we need to account for that
in our tests. On smaller phones, the landscape mode doesn't show the article
preview.
On both tablets and small screen phones, we need to be mindful of scrolling
too fast and missing an element we're looking for, either because a bigger
screen means we scroll faster or because a smaller screen means only a few
items are shown at one time.
Modify the JetNews workload to use the new jank testing classes. This allows
simplification of the code and also makes it more straightforward to implement
new workloads that also need to gather frame metrics.
Create two new classes to deal with workloads for which we want to
collect frame metrics/jank percentages.
ApkUiautoJankTestWorkload inherits from ApkUiautoWorkload and instantiates
a UiAutomatorJankTestGUI gui member.
UiAutomatorJankTestGUI (inherits from UiAutomatorGUI) handles most of the
new logic for jank tests, including using a regular expression to parse the
final output.
This patch adds the JetNews jank-testing workload. This is accomplished
through the uiauto helper library and the jank test classes.
This workload requires a JetNews app APK to be available. We plan to
make it available through the workload-automation-assets repo.
At the end of the run, users should end up with a json file containing all
the frame/jank metrics.
There are 3 parameters for this workload:
- tests: Specifies which of the 3 available tests to run (default is to run
all of them)
- flingspeed: The speed of the fling interactions.
- repeat: How many times each of the selected tests is to be executed in a
single measuring session.
Jank testing involves using a different library (JankTestBase) that our
testing class needs to extend. Since we already make the base testing
class UiAutomation extend BaseUiAutomation, we cannot use the UiAutomation
class for our tests.
The uiauto_class property is hardcoded to "UiAutomation" though, so this
patch adds a uiauto_jank_class property for the Jank test class.
While at it, we also make the uiauto_runner configurable (currently hardcoded
to the old deprecated android.support version of it) so that our workloads
can pass whatever runner they need to use.
In terms of stages for testing, their current names are the following:
stages = ['setup', 'runWorkload', 'extractResults', 'teardown']
Those won't work for jank tests. For some reason the jank test library wants
to see the testing function name contain or start with a "test" string
pattern on it.
This behavior doesn't seem to be documented, but at least with the current
version of the JankTestBase library we have this restriction.
To make things work despite this restriction, I've added separate jank
test stages that can be provided by the workload. The generic workload
code will use these stages to generate the instrumentation invocation
command, and the provided jank test stages (essentially jank test
functions) will be invoked one by one.
Finally, we make the UiAutomatorGUI's _execute method return the output
of the execution so the workload can freely process and pattern-match
the results, like the UiBenchJankTests workload.
Currently the uiauto library is built on top of the deprecated android.support
libraries. Create an updated uiauto library (uiauto-androidx) using androidx
and keep it alongside the old one for compatibility reasons.
As we update workload-automation going forward, we should update the workloads
to use the androidx version of uiauto.
This has been tested with the latest gradle version available at the time
(8.8).
When speedometer is running on a Chrome package on Android 14, a pop-up
window was showing on the screen. Chrome preferences file is modified
to dismiss the window.
The key for 'include#' can now be either a scalar or a list.
A scalar triggers the same behaviour as before.
If the value is a list it must be a list of scalars (filepaths).
The paths will be loaded and merged in order, and finally the resulting
dict is included into the current scope.
- ``load_struct_from_yaml()`` has been moved to devlib [1].
- ``LoadSyntaxError()`` is already implemented in devlib.
- Remove ``load_struct_from_file()`` and ``RAND_MOD_NAME_LEN`` since
they are not used at all.
[1] https://github.com/ARM-software/devlib/commit/591825834028
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Use importlib.import_module instead of __import__ as per Python doc
recommendation.
This will also fix the case where the class is in a
package's submodule (since __import__ returns the top-level package),
e.g. "foo.bar.Class".
Introduce 'unlock_screen' option in order to help in automating Android
tests by unlocking device screen automatically. Surely this works only
if no passcode is set.
'unlock_screen' option implicitly requires turning on the screen. IOW,
it will override value of 'screen_on' option.
'diagonal', 'vertical' and 'horizontal' are valid values for
'unlock_screen' option as of now.
Note that this patch depends on
https://github.com/ARM-software/devlib/pull/659 in devlib repo.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
"--user <USER_ID" (current user: 0) option is added to
activity manager (am) command because of "Invalid userId" command.
Tested with other benchmarks (geekbench) as well.
Add an instrument that uses devlib's PerfettoCollector to collect a
Perfetto trace during the execution of a workload.
The instrument takes a path to a Perfetto config file which specifies
how Perfetto should be configured for the tracing.
Signed-off-by: Kajetan Puchalski <kajetan.puchalski@arm.com>
- Bump python version used when building documentation.
- `setuptools` is a deprecated installation method on RTD so switch to `pip`.
- Add additional dependency on devlib master branch as RTD env is not
respecting dependency_links during installation.
- Bump copyright year.
Bromite is a fork of Chromium that's easily available for Android. Apart
from small changes it works the same as Chromium and works with this
speedometer workload. Add it to the 'package_names' list to allow using
it as an option.
https://www.bromite.org/
Signed-off-by: Kajetan Puchalski <kajetan.puchalski@arm.com>
Update rt-app binaries to the latest version of the "lisa" branch in
douglas-raillard-arm GitHub fork. This tracks the upstream master branch
with a number of critical patches required notably to work with uclamp.
Allow optionally setting the top level ftrace buffer size separately
from the devlib buffer size. The parameter will be passed to the devlib
FtraceCollector and take effect there.
Signed-off-by: Kajetan Puchalski <kajetan.puchalski@arm.com>
WA_PLUGIN_PATHS currently contains a list of filesystem paths to scan
for plugins. This is appropriate for end-user plugins, but this is
problematic for plugins distributed by a 3rd party, such as a plugin
installed from PyPI.
In those cases, the path to the sources is unknown and typically depends
on the specify Python version, local setup etc. What is constant is
Python name of the package, e.g. "lisa.wa.plugins".
Extend the input allowed in WA_PLUGIN_PATHS by trying to load entries as
a Python package name if:
* There is no filesystem path with that name
* The entry is a "relative path" (from an fs point of view)
Set the view parameter so that the FPS instrument can collect frame data
from the workload.
Signed-off-by: Kajetan Puchalski <kajetan.puchalski@arm.com>
