1
0
mirror of https://github.com/ARM-software/workload-automation.git synced 2024-10-06 10:51:13 +01:00
Commit Graph

1035 Commits

Author SHA1 Message Date
scojac01
da0ceab027 PCMark: Updating to handle new Android 10 permission warnings
Android 10 has introduced a new permission warning and a
seperate warning about the APK being built for an older version
of the Android OS. Have added two checks to accept these
permissions and continue with the workload and the given APK
rather than attempting to update.
2019-11-28 16:02:20 +00:00
scojac01
683eec2377 PCMark: Editing the screen orientation
Some devices have proved to have a natural orientation
that does not lend itself well to this workload. Therefore
I have edited the orientation lock to portrait instead of
natural.
2019-11-22 16:29:59 +00:00
scojac01
07e47de807 Geekbench: Updating supported versions
Adding support for Geekbench 4.3.4 and 4.4.0.

Adding support for Geekbench Corporate 5.0.1 and 5.0.3.

There are no changes required to the functional workload.
2019-11-22 12:07:50 +00:00
Marc Bonnici
5906bca6b3 instruments/acme_cape: Fix missing parameter to get_instruments
The signature of `get_instruments` was missing the `keep_raw` parameter
so fix this and use it as part of the subsequent common invocation.
2019-11-18 16:09:09 +00:00
Robert Freeman
796dfb1de6 Update googlephotos workload to work against most recent version
* Updated googlephotos workload to work against version 4.28.0
2019-10-29 16:17:20 +00:00
Robert Freeman
f3e7b14b28 Update googlemaps workload to work against most recent version
* Updated google maps workload to work against apk version 10.19.1
2019-10-29 13:34:08 +00:00
Marc Bonnici
e9839d52c4 output_processor/postgres: Fix out of range for hostid
Change the field type of `hostid` as part of `TargetInfo` from `Int` to
Bigint to prevent some ids from exceeding the maximum value.
2019-10-23 15:45:56 +01:00
Marc Bonnici
7ebbb05934 target/info: Fix missing return statement
Add missing return statement when upgrading a `TargetInfo` POD to v5.
2019-10-23 15:45:56 +01:00
Robert Freeman
13166f66d1 Update gmail workload to work against most recent version
* Updated gmail to work against 2019.05.26.252424914.release.
2019-10-17 14:17:56 +01:00
Marc Bonnici
ab5d12be72 output_processors/cpustates: Improve handling of missing cpuinfo data
Improve checking of whether cpu idle state information is available for
processing.
Add debug message to inform user if the cpuidle module is not detected
on the target.
2019-10-15 15:17:02 +01:00
Marc Bonnici
298bc3a7f3 output_processors/cpustates: Deal with cpufreq data being unavailable
If the `cpufreq` module is not detected as present then warn the user
and processes the remaining data instead of crashing.
2019-10-15 15:17:02 +01:00
Marc Bonnici
09d6f4dea1 uilts/cpustates: Fix inverted no_idle check
If there is no information about idle states then `no_idle` should be
set to `True` instead of `False`.
2019-10-15 15:17:02 +01:00
Marc Bonnici
d7c95fa844 instrument/energy_measurement: Fix typo and formatting 2019-10-03 12:48:24 +01:00
Marc Bonnici
0efd20cf59 uiauto: Update all applications to target SDK version 28
On devices running android 9 with google play services, PlayProtect
blocks the installation of our automation apks due to targeting a lower
SDK version. Update all apk builds to target SDK version 28 (Android 9)
however do not change the minimum version to maintain backwards
compatibility.
2019-10-03 12:18:48 +01:00
Marc Bonnici
e41aa3c967 instruments/energy_measurement: Add a keep_raw parameter
Add a `keep_raw` parameter to control whether raw files should be
deleted during teardown.
2019-10-03 11:38:29 +01:00
Marc Bonnici
3bef4fc92d instrument/energy_measurement: Invoke teardown method for backends
Forward the teardown method invocation to the instrument backend.
2019-10-03 08:39:53 +01:00
Scott Jackson
0166180f30 PCMark: Fixing click co-ordinates
The workload is clicking the run button in the centre
of the element but this is no longer starting the
run operation.

Refactoring the code to click in the topleft of the
object seems to rectify the issue.
2019-09-24 16:01:37 +01:00
Robert Freeman
a9f3ee9752 Update adobe reader workload to work with most recent version
Updated adobe reader workload to work with version 19.7.1.10709
2019-09-24 12:57:26 +01:00
Scott Jackson
35ce87068c Antutu: Updating to handle the new shortcut popup
Added logic to dismiss the new popup message which asks
to add a shortcut to the android homescreen.
2019-09-19 14:07:50 +01:00
Robert Freeman
6beac11ee2 Add simpleperf type to perf instrument
* Added simpleperf type to perf instrument as it's more stable
  on Android devices.
* Added record command to instrument
* Added output processing for simpleperf
2019-09-18 12:55:59 +01:00
Sergei Trofimov
2f231b5ce5 fw/target: detect module variations in TargetInfo
- Add modules entry to TargetInfo
- When retrieving TargetInfo from cache, make sure info modules match
  those for the current target, otherwise mark info as stale and
  re-generate.
2019-09-12 15:27:23 +01:00
Marc Bonnici
75878e2f27 uiauto/build_scripts: Update to use python3
Ensure we are invoking python3 when attempting to import `wa` and
update the printing syntax to be compatible.
2019-09-06 11:02:13 +01:00
Marc Bonnici
023cb88ab1 templates/setup: Update package setup template to specify python3 2019-09-06 11:02:13 +01:00
Marc Bonnici
d27443deb5 workloads/rt_app: Update to use python3
Update the workload generation script to be python3 compatible and
invoke with python3.
2019-09-06 11:02:13 +01:00
Scott Jackson
1a15f5c761 Geekbench: Adding support for Version 5 of Geekbench Corporate 2019-09-06 10:34:41 +01:00
Marc Bonnici
062be6d544 output_processors/postgresql: Don't reconnect if not initialised
Update check to clarify that we should not attempt to reconnect to
the database if the initialisation of the output processor failed.
2019-08-28 11:33:46 +01:00
Marc Bonnici
c1e095be51 output_processors/postgresql: Ensure still connected to the database
Before exporting output to ensure that we are still connected to the
database. The connection may be dropped so reconnect if necessary, this
is a more of an issue with longer running jobs.
2019-08-28 11:33:46 +01:00
Marc Bonnici
eeebd010b9 output_processors/postgresql: Group database connection operations
Refactors connection operations into the `connect_to_database`
method.
2019-08-28 11:33:46 +01:00
Marc Bonnici
6042fa374a fw/version: Version Bump 2019-07-19 17:07:46 +01:00
Marc Bonnici
050329a5ee fw/version: Update version for WA and required devlib 2019-07-19 16:37:00 +01:00
Marc Bonnici
12bb21045e instruments/SysfsExtractor: Add extracted directories as artifacts
Add the directories that have been extracted by the `SysfsExtractor` and
derived instruments as artifacts.
2019-07-19 16:36:11 +01:00
Marc Bonnici
4bb1f4988f fw/DatabaseOutput: Only attempt to extract config if avaliable
Do not try to parse `kernel_config` if no data is present.
2019-07-19 16:36:11 +01:00
Marc Bonnici
0ff6b4842a fw/DatabaseOutput: Fix the retrieval of job level artifacts 2019-07-19 16:36:11 +01:00
Marc Bonnici
98b787e326 fw/DatabaseOutput: Enabled retrieving of directory artifacts
To provide the same user experience of accessing a directory
artifact from a standard `wa_output` when attempting to retrieve the
path of the artifact extract the stored tar file and extract it to a
temporary location on the host returning the path.
2019-07-19 16:36:11 +01:00
Marc Bonnici
e915436661 commands/postgres: Upgrade the data base schema to v1.3
Upgrade the database schema to reflect the additions of directory
artifacts and the missing TargetInfo property.
2019-07-19 16:36:11 +01:00
Marc Bonnici
68e1806c07 output_processors/postgresql: Add support for new directory Artifacts
Reflecting the addition to being able to store directories as Artifacts
enable uploading of a directory as a compressed tar file rather than
storing the file directly.
2019-07-19 16:36:11 +01:00
Marc Bonnici
f19ebb79ee output_processors/postgresql: Add missing system_id field
When storing the `TargetInfo` the `system_id` attribute was ommited.
2019-07-19 16:36:11 +01:00
Marc Bonnici
c950f5ec8f utils/postgres: Fix formatting 2019-07-19 16:36:11 +01:00
Marc Bonnici
6aaa28781b fw/Artifact: Allows adding directories as artifacts
Adds a `is_dir` property to an `Artifact` to indicate that the
artifact represents a directory rather than an individual file.
2019-07-19 16:36:11 +01:00
Marc Bonnici
d87025ad3a output_processors/postgres: Fix empty iterable
In the case of an empty iterable an empty string would be returned
however this was not an valid value so ensure that the brackets are
always inserted into the output.
2019-07-19 16:36:11 +01:00
Marc Bonnici
ac5819da8e output_processors/postgres: Fix incorrect dict keys 2019-07-19 16:36:11 +01:00
Marc Bonnici
31e08a6477 instruments/interrupts: Add interrupt files as artifacts
Ensure that the interrupt files pulled and diffed from the device are
added as artifacts.
2019-07-19 16:36:11 +01:00
scott
47769cf28d Add a workload for Motionmark tests 2019-07-19 14:31:04 +01:00
Marc Bonnici
564738a2ad workloads/monoperf: Fix typos 2019-07-18 15:17:24 +01:00
Marc Bonnici
c092128e94 workloads: Sets requires_network attribute for workloads
Both speedometer and aitutu require internet to function however this
attribute was missing from the workloads.
2019-07-18 15:17:24 +01:00
Marc Bonnici
a6f0ab31e4 fw/entrypoint: Add check for system default encoding
Check what the default encoding for the system is set to. If this is not
configured to use 'UTF-8', log a warning to the user as this is known
to cause issues when attempting to parse none ascii files during operation.
2019-07-12 13:32:28 +01:00
Marc Bonnici
766bb4da1a fw/uiauto: Allow specifying landscape and portrait orientation
Previously the `setScreenOrientation` function only accepted relative
orientations, this causes issue when attempt to use across tablets and
phones with different natural orientations. Now take into account the
current orientation and screen resolution to allow specifying portrait vs
landscape across different types of devices.
2019-07-04 13:18:48 +01:00
Marc Bonnici
a5f0521353 utils/types: Fix typos 2019-06-28 17:56:13 +01:00
Marc Bonnici
3435c36b98 fw/workload: Improve version matching and error propergation
Ensure that the appropriate error message is returned to the user to
outline what caused the version matching to fail.
Additionally fix the case where if specifying a package name directly
the version matching result would be ignored.
2019-06-28 17:56:13 +01:00
Marc Bonnici
bd252a6471 fw/workload: Introduce max / min versions for apks
Allow specifying a maximum and minimum version of an APK to be used for
a workload.
2019-06-28 17:56:13 +01:00
Marc Bonnici
f46851a3b4 utils/types: Add version_tuple
Allow for `version_tuple` to be used more generically to enable
natural comparing of versions encoded as strings.
2019-06-28 17:56:13 +01:00
Marc Bonnici
8910234448 fw/workload: Don't override the package manager for ApkRevent workloads
`ApkRevent` workloads should be able to use the same Apk selection
criteria as `ApkWorkloads` therefore rely on the superclass to
instantiate the `PackageHandler`.
2019-06-28 17:56:13 +01:00
Marc Bonnici
1108c5701e workloads: Update to better utilize cleanup_assets and uninstall
Update the workload classes to attempt and standardize the use of the
`cleanup_assets` parameter and the newly added `uninstall` parameter
2019-06-28 17:54:04 +01:00
Marc Bonnici
f5d1a9e94a fw/workload: Add the uninstall parameter to the base workload class
In additional to being able to specify whether the APK should be
uninstalled as part of a `APKWorkload`s teardown add the `uninstall`
parameter to the base `workload` class in order to specify whether any
binaries installed for a workload should be uninstalled again.
2019-06-28 17:54:04 +01:00
Marc Bonnici
959106d61b fw/workload: Update description of cleanup_assests parameter
Improve the description of the parameter as the parameter may be used in
other places aside from the teardown method.
2019-06-28 17:54:04 +01:00
Pierre-Clement Tosi
0aea3abcaf workloads: Add support for UIBench Jank Tests
Add a workload that launches UIBenchJankTests. This differs from the
UIBench application as it adds automation and instrumentation to that
APK. This therefore requires a different implementation than classical
ApkWorkloads as 2 APKs are required (UIBench and UIBenchJankTests) and
the main APK is invoked through `am instrument` (as opposed to `am
start`).
2019-06-28 09:27:56 +01:00
Pierre-Clement Tosi
24ccc024f8 framework.workload: am instrument APK manager
Add support for Android applications that are invoked through `am
instrument` (as opposed to `am start`) _i.e._ that have been
instrumented. See AOSP `/platform_testing/tests/` for examples of such
applications.
2019-06-28 09:27:56 +01:00
Marc Bonnici
42ab811032 workloads/lmbench: Fix missing run method declaration 2019-06-19 11:28:28 +01:00
Marc Bonnici
832ed797e1 fw/config/execution: Raise error if no jobs are available for running
If no jobs have been generated that are available for running then WA
will crash when trying to access the job queue. Add an explicit check to
ensure that a sensible error is raised in this case, for example if
attempting to run a specific job ID that is not found.
2019-06-06 15:17:42 +01:00
Marc Bonnici
9aec4850c2 workloads/uibench: Pylint Fix 2019-05-28 09:33:15 +01:00
scott
bdaa26d772 Geekbench: Updating supported versions to include 4.3.2 2019-05-24 17:47:49 +01:00
Pierre-Clement Tosi
d7aedae69c workloads/uibench: Initial commit
Add support for Android's UIBench suite of tests as a WA workload.
2019-05-24 17:47:35 +01:00
Pierre-Clément Tosi
45af8c69b8 ApkWorkload: Support implicit activity path
If the activity field of an instance of ApkWorkload does not the '.'
character, it is assumed that it is in the Java namespace of the
application. This is similar to how activities can be referred to with
relative paths:
    com.domain.app/.activity
instead of
    com.domain.app/com.domain.app.activity
2019-05-24 17:47:35 +01:00
scott
e398083f6e PCMark: Removing hard coded index to make the workload more robust 2019-05-22 11:07:43 +01:00
Marc Bonnici
83f826d6fe utils/serializser: Re-fix support for YAML anchors
Include missing `flatten_mapping` call in our implementation of
`construct_mapping`. This is performed by a subclass in the default
implementation which was missing in our previous fix.
2019-05-17 20:04:33 +01:00
scott
1599b59770 workloads: add aitutu
Add a workload to execute the Aitutu benchmark.
2019-05-17 13:26:36 +01:00
Marc Bonnici
8cd9862e32 workloads/geekbench: Clean up version handling
The workload could attempt to use the version attribute before it was
discovered to assess the workload activity causing an error however the
whole process can be simplified using newer discovery features.
2019-05-17 09:15:23 +01:00
Marc Bonnici
76e6f14212 utils/serializer: pylint fixes 2019-05-15 19:48:39 +01:00
Marc Bonnici
ce59318e66 utils/serializer: Fix using incorrect loader and imports
- Ensure that the new loader is used when opening files to ensure that our
custom constructors are used.
- Fix missing imports
2019-05-15 19:48:39 +01:00
Sergei Trofimov
5652057adb utils/serializer: fix support for YAML anchors.
Change the way maps get processed by YAML constructor to support YAML
features, such as anchors, while still maintaining dict ordering.
2019-05-15 09:59:14 +01:00
Sergei Trofimov
e9f5577237 utils/serializer: fix error reporting for YAML
When attempting to access the message of a exception check not only that
e.args is populated, but also that e.args[0] actually contains
something, before defaulting to str(e).
2019-05-15 09:57:52 +01:00
Marc Bonnici
e3406bdb74 instruments/delay: Convert module name to identifier
- Ensure cooling module name is converted to identifier when resolving
- Fix typo
2019-04-26 08:04:45 +01:00
Marc Bonnici
55d983ecaf workloads/vellamo: Fix initialization order
Ensure that uiauto parameters are set before calling the super method.
2019-04-26 08:04:45 +01:00
Marc Bonnici
753786a45c fw/workload: Add activity attribute to APK workloads
Allow specifying an `activity` attribute for an APK based workload which
will override the automatically detected activity from the resolved APK.
2019-04-18 10:44:50 +01:00
Marc Bonnici
8647ceafd8 workloads/meabo: Fix incorrect add_metric call 2019-04-03 11:33:27 +01:00
Marc Bonnici
2c2118ad23 fw/resource: Fix attempting to match against empty values
Update checking of attributes to allow for empty structures as they can
be set to empty lists etc. and therefore should not be checking if
explicitly `None`.
2019-04-02 07:54:05 +01:00
Marc Bonnici
0ec8427d05 fw/output: Implement retriving "augmentations" for JobDatabaseOutputs
Enable retriving augmentations on a per job basis when using a Postgres
database backend.
2019-03-18 15:26:19 +00:00
Marc Bonnici
cf5c3a2723 fw/output: Add missing "augmentation" attribute to JobOutput
Add attribute to `JobOutput` to allow easy listing of enabled augmentations
for individual jobs rather than just the overall run level.
2019-03-18 15:26:19 +00:00
Marc Bonnici
8ddc1c1eba fw/version: Bump to development version 2019-03-04 15:50:13 +00:00
Marc Bonnici
b5db4afc05 fw/version: Version Bump
Bump to the next revision release.
2019-03-04 15:50:13 +00:00
Marc Bonnici
769aae3047 utils/serializer: Explicitly state yaml loader
In newer versions of PyYAML we need to manually specify the `Loader` to
be used as per https://msg.pyyaml.org/load.
`FullLoader` is now the default loader which attempts to avoid arbitrary
code execution, however if we are running an older version where this is
not available default back to the original Loader.
2019-03-04 15:50:13 +00:00
Marc Bonnici
a1ba3c6f69 utils/misc: Update load structure to use WA's yaml wrapper 2019-03-04 15:50:13 +00:00
Marc Bonnici
536fc7eb92 workloads/stress_ng: Update to use WA's yaml wrapper 2019-03-04 15:50:13 +00:00
Marc Bonnici
de36dacb82 fw/version: Bump to development versions 2019-03-04 10:37:39 +00:00
Marc Bonnici
637bf57cbc fw/version: Bump revison versions
Bump the revision version for WA and the required version for
devlib.
2019-03-04 10:37:39 +00:00
Marc Bonnici
ba0cd7f842 fw/target/info: Bump target info version
Due to mismatches in WA and devlib versions this previous upgrade method
could have been trigger before it was needed and would not be called a
second time. Now we can be sure that WA and devlib are updated together
bump the version number again to ensure the upgrade method is called a
second time to ensure the POD is upgraded correctly.
2019-03-04 10:37:39 +00:00
Marc Bonnici
adb3ffa6aa fw/version: Introduce required version for devlib
To ensure that a compatible version of devlib is installed on the system
keep track of the version of devlib that is required by WA and provide a
more useful error message if this is not satisfied.
2019-03-04 10:37:39 +00:00
Marc Bonnici
2ce8d6fc95 output_processors/postgresql: Add missing default
In the case of no screen resolution being present ensure that a default
is used instead of `None`.
2019-02-14 10:51:38 +00:00
Marc Bonnici
1415f61e36 workloads/chrome: Fix for tablet devices
Some tablet devices use an alternate tab switching method due to the
larger screen space. Add support for adding new tabs via the menu
instead of via the tab switcher.
2019-02-08 14:32:58 +00:00
Marc Bonnici
6ab1ae74a6 wa/apk_workloads: Update to not specify a default apk version.
No longer specify a default version to allow any available apks to be
detected and then choose the appropriate automation based on the
detected version.
Refactor to support new supported_versions attribute and since APK
resolution needs to have happened before setting uiauto parameter
move assignments to ``initialize``.
2019-02-08 13:56:55 +00:00
Marc Bonnici
a1cecc0002 fw/workload: Add "support_versions" attribute to workloads
Allow for specifying a list of supported APK versions for a workload. If
a specific version is no specified then attempt to a resolve any valid
version for the workload.
2019-02-08 13:56:55 +00:00
Marc Bonnici
0cba3c68dc fw/resource: Support matching APKs on multiple versions.
In the case where a range of apk versions are valid allow for the matching
process to accommodate a list of versions instead of a single value.
2019-02-08 13:56:55 +00:00
Marc Bonnici
f267fc9277 fw/workload: Use apk version for workload if not set.
If a workloads `version` attribute is not set, and an APK file is
found, use this as the version number. This allows for workloads to not
specify a default version via parameters and for an available APK to be
automatically chosen.
2019-02-08 13:56:55 +00:00
Sergei Trofimov
462a5b651a fw/output: add label property to Metric
Add a "label" property to Metric that combines its name with its
classifiers into a single string.
2019-02-05 10:27:06 +00:00
pablololo12
7cd7b73f58 Fixed an error emptying the reading buffer of the poller
Fixed identation

Fixed identation
2019-02-04 09:46:13 +00:00
Marc Bonnici
4a9a2ad105 fw/target/info: Fix for KernelConfig refactor
The Devlib KernelConfig object was refactored in commit
f65130b7c7
therefore update the way KernelConfig objects are deserialized to reflect the new
implementation and provide a conversion for PODs.
2019-01-31 09:44:30 +00:00
Marc Bonnici
9f88459f56 fw/workload: Fix Typo 2019-01-30 15:46:54 +00:00
Marc Bonnici
a2087ea467 workloads/manual: Fix incorrect attribute used to access target 2019-01-30 15:46:54 +00:00
Marc Bonnici
31a5a95803 output_processors/postgresql: Ensure screen resolution is a list
Ensure that the screen resolution is converted to a list to prevent
casting errors.
2019-01-30 15:46:54 +00:00
Marc Bonnici
ce7720b26d instruments/fps: Fix Typo 2019-01-28 12:45:10 +00:00