Add a workload that launches UIBenchJankTests. This differs from the
UIBench application as it adds automation and instrumentation to that
APK. This therefore requires a different implementation than classical
ApkWorkloads as 2 APKs are required (UIBench and UIBenchJankTests) and
the main APK is invoked through `am instrument` (as opposed to `am
start`).
Add support for Android applications that are invoked through `am
instrument` (as opposed to `am start`) _i.e._ that have been
instrumented. See AOSP `/platform_testing/tests/` for examples of such
applications.
If no jobs have been generated that are available for running then WA
will crash when trying to access the job queue. Add an explicit check to
ensure that a sensible error is raised in this case, for example if
attempting to run a specific job ID that is not found.
If the activity field of an instance of ApkWorkload does not the '.'
character, it is assumed that it is in the Java namespace of the
application. This is similar to how activities can be referred to with
relative paths:
com.domain.app/.activity
instead of
com.domain.app/com.domain.app.activity
Include missing `flatten_mapping` call in our implementation of
`construct_mapping`. This is performed by a subclass in the default
implementation which was missing in our previous fix.
The workload could attempt to use the version attribute before it was
discovered to assess the workload activity causing an error however the
whole process can be simplified using newer discovery features.
When attempting to access the message of a exception check not only that
e.args is populated, but also that e.args[0] actually contains
something, before defaulting to str(e).
Update checking of attributes to allow for empty structures as they can
be set to empty lists etc. and therefore should not be checking if
explicitly `None`.
In newer versions of PyYAML we need to manually specify the `Loader` to
be used as per https://msg.pyyaml.org/load.
`FullLoader` is now the default loader which attempts to avoid arbitrary
code execution, however if we are running an older version where this is
not available default back to the original Loader.
Due to mismatches in WA and devlib versions this previous upgrade method
could have been trigger before it was needed and would not be called a
second time. Now we can be sure that WA and devlib are updated together
bump the version number again to ensure the upgrade method is called a
second time to ensure the POD is upgraded correctly.
To ensure that a compatible version of devlib is installed on the system
keep track of the version of devlib that is required by WA and provide a
more useful error message if this is not satisfied.
Some tablet devices use an alternate tab switching method due to the
larger screen space. Add support for adding new tabs via the menu
instead of via the tab switcher.
No longer specify a default version to allow any available apks to be
detected and then choose the appropriate automation based on the
detected version.
Refactor to support new supported_versions attribute and since APK
resolution needs to have happened before setting uiauto parameter
move assignments to ``initialize``.
Allow for specifying a list of supported APK versions for a workload. If
a specific version is no specified then attempt to a resolve any valid
version for the workload.
If a workloads `version` attribute is not set, and an APK file is
found, use this as the version number. This allows for workloads to not
specify a default version via parameters and for an available APK to be
automatically chosen.
The Devlib KernelConfig object was refactored in commit
f65130b7c7
therefore update the way KernelConfig objects are deserialized to reflect the new
implementation and provide a conversion for PODs.
Allow for easy configuring of a view for a particular workload as this
can vary depending on the device which can be used when using certain
instruments for example `fps`.
- cross-compiled revent binaries to match latest version (with recording timestamp fix f64aaf6 on 12 Oct 2018)
toolchains used:
gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabi
gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu
- fixes error in utils/revent.py when reading timestamps from recordings made with previous wa revent binaries
- Move the slide editing test into the main runWorkload instead of
setup.
- On some devices the folder picker has changed layout so add support for
navigating.
- Add support for differently capitalized splash buttons.
- Add workaround for adding a new slide if click the button doesn't work
the first time.
- Fix formatting
- Skip teardown automation if elements are not present on some devices
instead of failing the workload.
- Give extra time for start button to appear as some devices can be slow
to load.
Add additional check for text based directions button as id can be
missing on some devices and allow for skipping the view steps stage for
large screen devices which do not require this step.
In commit bb282eb19c devlibs
`escape_double_quotes` method was retired in favour of the `pipes.quote`
method however this does not format correctly for this purpose therefore
revert back to the original escaping method.
Previously the `Status` Enum was converted to a string as part of
serialization however now use the Enum `to_pod` method and make the
respective changes for de-serialization.
Update the postgres database schema:
- Rename "resourcegetters" schema to "resource_getters" for
consistency
- Rename "retreies" colum to "retry" to better relflect it purpose
- Store additional information including:
- POD serialization data
- Missing target information
- JSON formatted runstate
Ensure that Run structures now have serialization versions.
Also fix serialization/de-serialization of `Status` type as previously this
was formatted as a String instead a pod.
Add a new mix-in class for classes that are serialized to PODs, the aim
of this class is to provide a way to ensure that both the original data
version and the current serialization version are known. When attempting
to de-serialize a POD the serialization version will be compared to the
latest version in WA if not matching will call the appropriate method to
upgrade the pod to a known structure state populating any missing fields
with a sensible default or converting the existing data to the new
format.
On some devices the score string obtained can contain extra characters.
Only use the numerical values from the score when converting, otherwise
if not found set the result to 'NaN'.
By clearing the application data each time the workload is run this
forces the required assets to be re-installed each time. As the
workload is not affected by persistent state do not perform the
clearing.
Add format_version class attribute to TargetInfo to track format
changes. This is checked when deserializing from POD to catch format
changes between cached and obtained TargetInfo's.
Previously run level artifacts would be added with a particular job_id,
and updated artifacts would be stored as new objects each time. Refactor
to remove unnecessary instance variables, only provide a job_id when
required and add an update capability for largeobjects to ensure this
does not happen.
In Android Q a popup will be displayed warning if the application has
not been designed for the latest version of android. This has currently been
dealt with on a per workload basis however this is a common popup so
add a method to dismiss the popup if present to the base class.
While attempting to run vellamo on Android Q, a popup warning with
the message, "This app was built for an older version of Android and may not
work properly. Try checking for updates, or contact the developer." would
appear, causing the workload to halt.
Close the popup warning before dismissing EULA and executing the remaining
steps to run vellamo.
Tested with vellamo apk version 3.2.4.
Signed-off-by: Lisa Nguyen <lisa.nguyen@linaro.org>
- force cast start/end timestamps to uint64_t to correct recording format issue on 32bit devices (i.e. 4 bytes timespec tv_sec written on 8 bytes memory slot)
Previously the LogcatRunMonitor left the logcat process running in the
background causing issues with concurrent accesses. Now ensure the thread
terminates correctly.
The RunInfo object in the run output is initally created before the
config has been fully parsed therefore attributes for the project and
run name are never updated, once the config has been finalized make sure
to update the relavant information.
- change magic string literal to a b'' string so that the comparison
works in python 3
- expand timestamp tuples (struct.unpack always returns a tuple) before
attempting to cast to float.
For workloads that support Linux targets do not use
`uninstall_executable` as this is not available, instead use `uninstall` as
other targets should be able to determine the appropriate uninstallation
method.
Previously the workload assumed that all the scores were visible on a
single screen however for devices with smaller displays the results need
to scrolled.
A new popup appears when running pcmark on android Q that complains
about the app being built for an older version of android.
Since this popup will be temporary, the fix has to make sure not to
break in the future when this popup disappears or when the test is ran
on a compatible version of android.
To achieve this, we attempt to dismiss the popup and if we timeout we
silently carry on with the test assuming no popup will appear.
Signed-off-by: Qais Yousef <qais.yousef@arm.com>
Creating a new workload to execute the following tests on GFXBench.
* Car Chase
* Car Chase Offscreen
* Manhattan 3.1
* 1080p Manhattan 3.1 Offscreen
* 1440p Manhattan 3.1 Offscreen
* Tessellation
* Tessellation Offscreen
Now allows for specifying a `group` value for each section which will
cross product the sections within that group with the sections in each
other group. Additionally classifiers will automatically be added to
each job spec with the relevant group information.
In the case of a failure in the initialization of one output_processor the
remaining `initialize` methods may not get called causing variables to
not be initialized correctly.
The 'jsonB' datatype was only added in v9.4 so ensure that the Postgres
server to is running this or later and inform the user if this is not
the case.
The Output processor which is used to upload the results found in the
wa_output folder to a Postgres database, whose schema is defined by the
WA Create Database command.
Add a command to create a PostgreSQL database with supplied parameters
which draws its structure from the supplied schema (Version 1.1). This
database is of a format intended to be used with the forthcoming WA
Postgres output processor.
Leftover code from WA2 meant that logcat was cleared on the device by
the workload directly instead of using devlib, this caused issues if logcat was
still being cleared from other areas of the code.
Expose the context to the initialize and finalize functions for Output
Processors. This was found to be necessary for the upcoming PostgreSQL
Output Processor.
Previously when using the create command for adding
EnergyInstruemntBackends they were treated like any other plugin and
generated incorrect configuration. Now automatically add the
`energy_measurement` instrument and populate it's configuration with the
relevant defaults for the specified Backend.
In commit d2ece we are now tracking augmentations which are used during a
run in the run_config via the context when installing augmentations.
Update the Process command and its ProcessContext with a dummy method
to relect this change.
Ensure that a plug-ins config entry is converted to an identifier before being
stored in the PluginCache so that the relevant configuration can be
retrieved appropriately. For example this allows for both 'trace-cmd' and
'trace_cmd' to be used as config entries to provide configuration for the
'trace-cmd' plugin.
In commit 8da911 the initialization of the target was split into a
separate method of the TargetManger. Ensure that we call the relevant
method after creating the manager.
Add Context.write_config() to write the combined config into run output
__meta. Use it after instruments and result processors get installed to
make sure their configuration gets serialized in the output.
- Rename "instrument_name" to "instrument" inside do_execute(), as
ConfigManger.get_instrument() returns a list of Instrument objects,
not names.
- To avoid name clash, rename the imported instrument module to
"instrumentation".
- Change how "source" and "dest" are handled inside merge() to be more
sane and less confusing, ensuring that disabling toggles are merged
correctly.
- Do not drop disabling values during merge, to ensure that merging
is a transitive operation.
- Add unit tests for the above fixes.
On some devices running Android 8.1 the start benchmark button was
failing to be clicked, this is a workaround to click on the coordinates
of the button instead of the UiObject iteslf.
- Add copyright headers to files that did not already have them
- Update the year to the last year the file was modified in files with
existing headers.
Remove unused ``platform`` attributes and add ``supported_platforms``
along with check so that workloads can fail early when attempting to run
on an incorrect platform.
Mongoperf currently has a non-ASCII character in the file that causes a failure.
This commit rewords the description to remove the incriminating character.
