1
0
mirror of https://github.com/ARM-software/workload-automation.git synced 2024-10-06 19:01:15 +01:00
Commit Graph

1191 Commits

Author SHA1 Message Date
Sergei Trofimov
72f2f82594 fw/config: better error when merging augs
If one or more entries for augmentations in configuration contains an
invalid value, raise ConfigError with the entry name.
2018-03-22 12:28:48 +00:00
Sergei Trofimov
3d7984412a fw/config: better message on config setting error
Catch exceptions raised when attempting to set value of a config point,
and re-raise as ConfigError with name associated with the value in the
config file.
2018-03-22 12:28:48 +00:00
Sergei Trofimov
67ea7c8ee1 utils/types: toggle_set: TypeError on string
Strings are iterable, so can be used to instantiate sets (resulting in a
set of chars). This is never what we want for toggle_set's though, and
may result in difficult-to-interpret errors when parsing configuration,
so raise a TypeError if attempting to create a toggle_set with a string.
2018-03-22 12:28:48 +00:00
Marc Bonnici
7464010677 workloads/geekbench: Fix formatting
Add missing newline to description in order to display correctly in
the show command and remove extra spaces.
2018-03-21 15:37:55 +00:00
Marc Bonnici
2371364ce4 utils/doc: Adds support for showing aliases when formatting parameters
Now displays all available local and global aliases when generating the
rst for a parameter.
2018-03-21 13:27:31 +00:00
Marc Bonnici
daa1959fd1 framework/workload: Refactor apk resolution
Now allows a `prefer_host_package` parameter to be set to control
whether a package located on the host system (`True`) or on the target
(`False`) should be favoured during resolution.
2018-03-20 16:17:16 +00:00
Marc Bonnici
1cb8f1670a framework/workload: Return path of pulled apk
`pull_apk` now returns the path to the apk it pulled from the device so
WA can keep track of it and avoid conflicts with other apks present
on the host.
2018-03-20 16:17:16 +00:00
Marc Bonnici
c1cc18df2f instruments/DAQBackend: Add global aliases 2018-03-20 08:24:58 +00:00
Marc Bonnici
1b9205f94a wa/framework: Add support for converting WA2 configuration to WA3 style
Upon initilisation WA3 will now also check if a WA3 config file
exisits, rather than just the exsistance of the config directory.
If a `config.yaml` file is not found WA will look for a WA2 config.py
file and convert it to the new format, otherwise a default config file
will be generated.
2018-03-20 08:24:58 +00:00
Marc Bonnici
f5ed4e317a utils/misc: Fix missing import 2018-03-20 08:24:58 +00:00
Marc Bonnici
bf8c4a856a framework/workload: Add 'cleanup_assets' to global aliases
Allow specifying whether to clean up deployed assets as a global config
option as in WA2.
2018-03-20 08:24:58 +00:00
Marc Bonnici
31b46fb3b7 framework/configuration: Refactor code into seperate method 2018-03-20 08:24:58 +00:00
Sergei Trofimov
f3bb8e135a fw/execution: record UI state on error
Record UI state if an error occurs during setup, run, and output
processing stages (for other stages, the UI state is unlikely to be
relevant as they typically would not include UI manipulation).
2018-03-19 09:06:21 +00:00
Sergei Trofimov
e93199a0a0 fw/execution: add record_ui_state() to context
Add a method to record UI state (take a screen shot, and, for Android
devices, dump the UI tree structure) to ExecutionContext.
2018-03-19 09:06:21 +00:00
Sergei Trofimov
c0049d0a86 fw/job: only finalize if initialized
Only run finalize() for a job if initialize has succeed. finalize()
should be able to assume that initialize() has succeed, without needing
to check that that file have been created, variables set, etc.
2018-03-19 09:06:21 +00:00
Sergei Trofimov
f946f13838 fw/job: add workload to INFO logging
Add workload label to the INFO-level log messages produced by the job.
2018-03-16 15:38:09 +00:00
Sergei Trofimov
5b15b115a7 framework/output: add list_workloads to RunOutput
Add a convenience method to list which workloads (as per their label)
were involved in a run.
2018-03-16 15:38:09 +00:00
Marc Bonnici
2b0b6ed3db commands/show: Adds support for showing global settings 2018-03-16 14:05:19 +00:00
Marc Bonnici
5e4a9311f2 workloads/gmail: Fix attaching files
Ensure the workload uses the passed working directory for attaching
images instead of a hard coded name and allow for finding of the attach
button by description as well as name.
2018-03-16 07:57:00 +00:00
Sergei Trofimov
2b7d1de9f1 workloads: add hwuitest
Port automation for hwuitest (Android's libhwui microbenchmark) from
WA2.
2018-03-15 17:42:25 +00:00
Sascha Bischoff
dd7ff641f3 fw/workload: Fix installed apk info
Fix an issue where AndroidWorkload.pull_apk would sometimes get the
wrong package if the desired package name is a substring of another
package name. Rather than using a regex to match the package name, use
the new get_package_info method to match the name exactly.
2018-03-15 17:06:23 +00:00
Marc Bonnici
758abedd1f workloads/antutu: Fix location of AndroidManifest.xml 2018-03-15 12:33:17 +00:00
Marc Bonnici
6742c5c8ad workloads/antutu: Allow for unsupported tests
Not all benchmarks are available on all phones and will report "Not
supported" To allow for this update the regexes and if not a valid score
set any unsupported results to 'NaN'
2018-03-15 12:33:17 +00:00
Marc Bonnici
811dfbe5af workloads/exoplayer: Fix splitting of format in incorrect location
Split the desired format outside of for loop as this may not be
executed.
2018-03-14 15:51:03 +00:00
scott
cfe4c18b71 Speedometer: Adding in the missing build files 2018-03-13 14:20:13 +00:00
scott
cb958d668e Androbench: Adding missing build files 2018-03-13 14:20:03 +00:00
scott
f5a00bf4fd Antutu: Adding the missing build files 2018-03-13 14:19:47 +00:00
Sergei Trofimov
5ed5b6f819 workloads: add jankbench
Add automation for Google's jankbench workload.
2018-03-12 17:46:14 +00:00
Sergei Trofimov
e6f7d3fcde output_processors: add sqlite processor
This is WA2 sqlite result processor ported to WA3 output processor API.
It is schema-compatible with the WA2 version (in fact, there are *no*
schema changes).

Note: There is a medium-term goal of adding a more generic SQL database
exporter with a more complete schema sitting on top of something like
sqlalchemy to make it agnostic of DBMS implementations. At point, this
will likely be deprecated.
2018-03-12 17:12:53 +00:00
Sascha Bischoff
ae6d62da39 extras: Update README & reflow
Update README to contain information about the Dockerfile and
re-wrapped to 80 characters.
2018-03-09 17:00:45 +00:00
Sascha Bischoff
2743103e08 Docker: Add a Dockerfile for running WA
With this commit we add a Dockerfile which can be used to create a
Docker image for running WA and devlib. Users are able to use this
image in a Docker container to run workloads on devices, without
requiring WA on their host systems. Please be aware that udev rules
must still be correctly setup on the host.

NOTE: This Dockerfile will accept the Android SDK licenses
automatically.
2018-03-09 16:43:37 +00:00
Marc Bonnici
71f0452b73 commands/record: Add checking for android targets
Update the 'package' argument description to specify it is for android
packages and add error checking to ensure that if specified, the target
supports android.
2018-03-09 11:57:00 +00:00
Marc Bonnici
8729702ee5 commands/record: Make recording of run stage optional
Change the recording of the run stage during workload recording mode to be
optional as the user may want to use the command to re-record a specific
stage of a workload without affecting the workload recording.
2018-03-09 11:57:00 +00:00
Marc Bonnici
d253fd07e6 workloads/templerun2: Add missing package name 2018-03-08 11:46:40 +00:00
Sergei Trofimov
9689420cff fw/execution: fix Exception on abort
Hitting CTRL-C will abort execution of the current job, but will still
trigger run finalization, and possibly, post-processing and teardown of
the current job. If an exception is raised during this
post-process/teardown, the previous exception state (for the
KeybardInterrupt) will be clobbered. That means that, after the new
exception has been handled, WA would attempt to execute the next job,
rather than go to finalization of the run.

To avoid this, set a flag in the context upon catching KeybardInterrupt,
and check this flag before attempting to execute the next job in the
queue.
2018-03-08 11:18:12 +00:00
Sergei Trofimov
fda418093d fw/execution: change how error events get added
Rather than relying on a custom Logger with a context to add events when
they are logged, have the runner register hooks for corresponding
signals that do that.

The issue with the previous approach is that the event ended up
being added too late -- after the current job was already cleared, so
all events were added at run level.

With the new approach, the event will be added when it is first logged,
while there is still a current job being set. There will be no
duplication for Exceptions being re-raised and handled at different
levels because log_error() will ensure that each Exception is logged
only once.
2018-03-08 11:18:12 +00:00
Sergei Trofimov
d20422ac15 workloads/rt-app: only initialize once
Only initialize/finalize once per run to avoid reinstalling the binary
and recreating the working directory needlessly.
2018-03-08 11:18:12 +00:00
Sergei Trofimov
eacc319239 wa: export target descriptor API
- Add exports for target descriptor API.
- Arrange exports alphabetically
2018-03-08 11:18:12 +00:00
Sergei Trofimov
10f79c37c2 fw/target/descriptor: add usability functions
- Add SimpleTargetDescriptor that simply returns a global list of target
  descriptions.
- Add create_target_description function to populate that list with a
  description crated from the specified components.
- Add add_description_for_target helper function to create a description
  for the specified target class, and specified components. For
  components that are not explicitly specified, attempt to guess
  sensible defaults based on the target class' bases.
2018-03-08 11:18:12 +00:00
Sergei Trofimov
9303c70727 fw/target/descriptor: fixes
- fix TargetDescription._set to set lists rather than dicts for params
  to be consistent with how params are set after creation (should
  probably get rid of this entirely...)
- fix the comment describing the structure of TARGETS
2018-03-08 11:18:12 +00:00
Sergei Trofimov
879dfc7700 framework/output: output fixes
- Get events from the result
- Correctly handle pending jobs
2018-03-08 11:18:12 +00:00
Sergei Trofimov
79aaef9986 fw/execution: abort and unresponsive target handling
Make sure KeyboardInterrupt's and unresponsive target errors are caught
and handled at appropriate points in the execution.
2018-03-08 11:18:12 +00:00
Sergei Trofimov
04ab336afc fw: fix error logging
- Keep track of logged exceptions inside log_error itself.
- signal: log the exception, if there is one  in the finally clause of
  the  signal wrapper; this will ensure that the error will be logged
  closer to the command that originated.
- entrypoint: use log.log_error for top-level error logging, rather than
  the  entrypoint logger directly; this will ensure that errors are not
  repeated unnecessarily.
- Log CTRL-C message at zeroth indent level to make it easier to see in
  the  non-verbose output where it occurred.
2018-03-08 11:18:12 +00:00
Sergei Trofimov
6fe31d6cad fw/execution: Handle unresponsive targets
If a target error occurs, check whether the target is unresponsive. If
it is, attempt to hard reset it if possible, or gracefully terminate
execution if not.
2018-03-08 11:18:12 +00:00
Sergei Trofimov
fdb872d9cd fw/target/manager: Add verify_target_responsive()
Add a method to check that the target is still responsive, and cache
the value so that the target does not need to be repeatedly probed
during various stages of wrapping up execution.
2018-03-08 11:18:12 +00:00
Sergei Trofimov
a9329554d1 wa: fix exception exports 2018-03-08 11:18:12 +00:00
Sergei Trofimov
1b9ac9236a framework/exception: add ExecutionError
Add an exception class to represent an error from within the execution
framework (rather than a plugin).
2018-03-08 11:18:12 +00:00
Marc Bonnici
05b726db09 commands/create: Add support for creating packages to WA3 2018-03-08 10:36:29 +00:00
scott
b204e9c22a Antutu: New antutu workload for the 7.0.4 version 2018-03-08 10:17:13 +00:00
scott
7e0e7456a9 Androbench: Migrating to WA3 and modifying to extract results instead of taking a screenshot 2018-03-08 10:16:27 +00:00