When setting the job status through ExecutionContext, this change
should be accompanied by an update to the state file, so that the state
file accurately reflects execution state.
As Jobs should not be aware of the output, this method is added to
ExecutionContext, and couples setting job state with writing to the
state file.
Pandas removed .ix as a way to iterate the index, .loc is the replacement
in most cases. Jankbench as a workload fails on a clean install due to
this call.
Replacing this works for me on a native install of Lisa with Ubuntu 20.04
Signed-off-by: Chris Redpath <chris.redpath@arm.com>
Alter the element to check popups have closed to be present
on any main screen and ensure we switch to the homescreen of the
app before performing setup steps.
Provide warning to user when attempting to set a depreciated
parameter instead of during validation and only raise the warning
if a value has been explicitly provided.
Rework how the result matching is performed. Some tests from
gfxbench provide more than 1 score per test and
some provide their output in a different format to others.
Update the matching to perform more flexible matching as well
as dealing with entries that do not fit on a single results screen.
Use the original file path to create a lock file in the system temp
directory. This prevents issues where we are attempting to lock a file
where wa does not have permission to create new files.
* Pcmark sometimes auto installs without need for clicking button,
in such cases workload throws UiObjectNotFound exception.
* Added logic to check for installation button existence.
* Increased install wait time to 5 mins.
* Updated regex to reflect correct test name.
* Enabling/disabling tests on setup was missing tesselation on some devices
so changed order of toggle.
* Sometimes whilst collecting scores the workload grabs the wrong score.
Updated to check the name of the test before grabbing the score.
* Tested on mate20, xperia, s9-exynos, s10-exynos, pixel-4
Make sure we only override parameters that are present in the current
config. This allows for connection parameters to be supplied for a
platform but only overridden if required for the connection.
Allow for overriding connection parameters on a per platform basis, and
make the `host` parameter for `Juno` optional as this can be auto
detected via the serial connection.
Instead of supplying only the parameter name and value to be set as a
default, allow for replacing the entire parameter object as this allow
more control over what needs overriding for a particular platform.
Some devices print non standard characters to logcat. If an error
occurs when parsing the output, replace the offending character instead
of raising an error.
* Workload was failing when chrome was not set as default broser so
altered to use chrome every time.
* Changed workload to an ApkuiAutoWorkload since chrome is now a
dependency.
* Refactored opening speedometer to new method.
* Added wait time for scores to show up when test finished.
Allow specifying a ConfigutationPoint is deprecated. This means that any
supplied configuration will not be used however execution will continue
with a warning displayed to the user.
Correctly handle the presence of both an element and its toggle in the
input, and handle them base on order, e.g.
toggle_set(['x', 'y', '~x']) --> {'y', '~x'}
toggle_set(['~x', 'y', 'x']) --> {'y', 'x'}
Move cleanup_assets from Workload up into TargetedPlugin. This way,
Instruments may also utilize it if they deploy assets.
More generally, it makes sense for it to be inside TargetedPlugin, as
any plugin that interacts with the target may conceivably need to clean
up.
Previously, when pulling an apk from the target to the host, the default
package name was used for both regular apks and test apks. This could
result in one overwriting the other. To prevent this ensure
`TestPackages` have the "test_" prefixed to their filename.
Now that classifiers may be added to the job during execution, its
classifiers dict should be unique to each job rather than just returning
them form spec (which may be shared between multiple jobs.)
Add add_classifier() method to context. Allow plugins to add classifiers
to the current job, or the run as a whole. This will ensure that the new
classifiers are propagated to all relevant current and future artifacts
and metrics.
- Correct handling of skipped jobs -- the output directory would not
have been generated, so do not try to write it.
- Do not attempt to process runs that are in progress, unless forced,
and do not try to process jobs that have not completed yet.
- Write the run info as well as the result, allowing output processors
to modify it (e.g. adjusting run names).
The list of modules retrieved from a `Target` may include configuration
as a dictionary. This helper function will produce a set of only the
module names allowing for comparison.
The update to Antutu major version 8 has changed a lot of element names.
There have also been changes to the tests run in three of the four categories.
This commit handles those updates while also retaining backwards compatibility
with major version 7.
Use Executable.match() rather than just checking the path inside
get_from_location(); this allows for alternative matching semantics
(e.g. globbing) inside derived implementations.
When updating a ParameterDict with another ParameterDict the unencoded
values were being merged. Ensure consistent behaviour by implicitally
iterating via `__iter__` which will cause ParameterDict values to be
decoded before being re-endcoded as expected.
As part of our continous integration system it has become
clear that gathering the app package data as well as the
version name can provide useful.
Adding this functionality to mainline as it could prove
useful to other developers.
Fix warnings reported when running unit tests via pytest.
- Rename TestDevice to MockDevice, so that it is not interpreted as a
test case.
- Fix collections abstract base class imports.
- Add an ini file to ignore the same from "past".
Android 10 has introduced a new permission warning and a
seperate warning about the APK being built for an older version
of the Android OS. Have added two checks to accept these
permissions and continue with the workload and the given APK
rather than attempting to update.
Some devices have proved to have a natural orientation
that does not lend itself well to this workload. Therefore
I have edited the orientation lock to portrait instead of
natural.
Adding support for Geekbench 4.3.4 and 4.4.0.
Adding support for Geekbench Corporate 5.0.1 and 5.0.3.
There are no changes required to the functional workload.
Improve checking of whether cpu idle state information is available for
processing.
Add debug message to inform user if the cpuidle module is not detected
on the target.
On devices running android 9 with google play services, PlayProtect
blocks the installation of our automation apks due to targeting a lower
SDK version. Update all apk builds to target SDK version 28 (Android 9)
however do not change the minimum version to maintain backwards
compatibility.
The workload is clicking the run button in the centre
of the element but this is no longer starting the
run operation.
Refactoring the code to click in the topleft of the
object seems to rectify the issue.
* Added simpleperf type to perf instrument as it's more stable
on Android devices.
* Added record command to instrument
* Added output processing for simpleperf
- Add modules entry to TargetInfo
- When retrieving TargetInfo from cache, make sure info modules match
those for the current target, otherwise mark info as stale and
re-generate.
Before exporting output to ensure that we are still connected to the
database. The connection may be dropped so reconnect if necessary, this
is a more of an issue with longer running jobs.
To provide the same user experience of accessing a directory
artifact from a standard `wa_output` when attempting to retrieve the
path of the artifact extract the stored tar file and extract it to a
temporary location on the host returning the path.
Reflecting the addition to being able to store directories as Artifacts
enable uploading of a directory as a compressed tar file rather than
storing the file directly.
In the case of an empty iterable an empty string would be returned
however this was not an valid value so ensure that the brackets are
always inserted into the output.
Check what the default encoding for the system is set to. If this is not
configured to use 'UTF-8', log a warning to the user as this is known
to cause issues when attempting to parse none ascii files during operation.
Previously the `setScreenOrientation` function only accepted relative
orientations, this causes issue when attempt to use across tablets and
phones with different natural orientations. Now take into account the
current orientation and screen resolution to allow specifying portrait vs
landscape across different types of devices.
Ensure that the appropriate error message is returned to the user to
outline what caused the version matching to fail.
Additionally fix the case where if specifying a package name directly
the version matching result would be ignored.
`ApkRevent` workloads should be able to use the same Apk selection
criteria as `ApkWorkloads` therefore rely on the superclass to
instantiate the `PackageHandler`.
In additional to being able to specify whether the APK should be
uninstalled as part of a `APKWorkload`s teardown add the `uninstall`
parameter to the base `workload` class in order to specify whether any
binaries installed for a workload should be uninstalled again.
Add a workload that launches UIBenchJankTests. This differs from the
UIBench application as it adds automation and instrumentation to that
APK. This therefore requires a different implementation than classical
ApkWorkloads as 2 APKs are required (UIBench and UIBenchJankTests) and
the main APK is invoked through `am instrument` (as opposed to `am
start`).
Add support for Android applications that are invoked through `am
instrument` (as opposed to `am start`) _i.e._ that have been
instrumented. See AOSP `/platform_testing/tests/` for examples of such
applications.
If no jobs have been generated that are available for running then WA
will crash when trying to access the job queue. Add an explicit check to
ensure that a sensible error is raised in this case, for example if
attempting to run a specific job ID that is not found.
If the activity field of an instance of ApkWorkload does not the '.'
character, it is assumed that it is in the Java namespace of the
application. This is similar to how activities can be referred to with
relative paths:
com.domain.app/.activity
instead of
com.domain.app/com.domain.app.activity
Include missing `flatten_mapping` call in our implementation of
`construct_mapping`. This is performed by a subclass in the default
implementation which was missing in our previous fix.
The workload could attempt to use the version attribute before it was
discovered to assess the workload activity causing an error however the
whole process can be simplified using newer discovery features.
When attempting to access the message of a exception check not only that
e.args is populated, but also that e.args[0] actually contains
something, before defaulting to str(e).
