1
0
mirror of https://github.com/ARM-software/workload-automation.git synced 2025-01-19 04:21:17 +00:00

Merge pull request #487 from setrofim/next

Various fixes.
This commit is contained in:
setrofim 2017-09-27 14:01:21 +01:00 committed by GitHub
commit 1df033fc22
18 changed files with 776 additions and 93 deletions

View File

@ -1,7 +1,10 @@
.. _revent_files_creation:
revent
======
++++++
Overview and Usage
==================
revent utility can be used to record and later play back a sequence of user
input events, such as key presses and touch screen taps. This is an alternative
@ -17,36 +20,47 @@ to Android UI Automator for providing automation for workloads. ::
info:shows info about each event char device
any additional parameters make it verbose
.. note:: There are now also WA commands that perform the below steps.
Please see ``wa show record/replay`` and ``wa record/replay --help``
for details.
Recording
---------
To record, transfer the revent binary to the device, then invoke ``revent
record``, giving it the time (in seconds) you want to record for, and the
file you want to record to (WA expects these files to have .revent
plugin)::
WA features a ``record`` command that will automatically deploy and start
revent on the target device::
host$ adb push revent /data/local/revent
host$ adb shell
device# cd /data/local
device# ./revent record 1000 my_recording.revent
wa record
INFO Connecting to device...
INFO Press Enter when you are ready to record...
[Pressed Enter]
INFO Press Enter when you have finished recording...
[Pressed Enter]
INFO Pulling files from device
Once started, you will need to get the target device ready to record (e.g.
unlock screen, navigate menus and launch an app) then press ``ENTER``.
The recording has now started and button presses, taps, etc you perform on
the device will go into the .revent file. To stop the recording simply press
``ENTER`` again.
Once you have finished recording the revent file will be pulled from the device
to the current directory. It will be named ``{device_model}.revent``. When
recording revent files for a ``GameWorkload`` you can use the ``-s`` option to
add ``run`` or ``setup`` suffixes.
From version 2.6 of WA onwards, a "gamepad" recording mode is also supported.
This mode requires a gamepad to be connected to the device when recoridng, but
the recordings produced in this mode should be portable across devices.
For more information run please read :ref:`record-command`
The recording has now started and button presses, taps, etc you perform on the
device will go into the .revent file. The recording will stop after the
specified time period, and you can also stop it by hitting return in the adb
shell.
Replaying
---------
To replay a recorded file, run ``revent replay`` on the device, giving it the
file you want to replay::
To replay a recorded file, run ``wa replay``, giving it the file you want to
replay::
device# ./revent replay my_recording.revent
wa replay my_recording.revent
For more information run please read :ref:`replay-command`
Using revent With Workloads
@ -100,3 +114,396 @@ where as UI Automator only works for Android UI elements (such as text boxes or
radio buttons), which makes the latter useless for things like games. Recording
revent sequence is also faster than writing automation code (on the other hand,
one would need maintain a different revent log for each screen resolution).
Using state detection with revent
=================================
State detection can be used to verify that a workload is executing as expected.
This utility, if enabled, and if state definitions are available for the
particular workload, takes a screenshot after the setup and the run revent
sequence, matches the screenshot to a state and compares with the expected
state. A WorkloadError is raised if an unexpected state is encountered.
To enable state detection, make sure a valid state definition file and
templates exist for your workload and set the check_states parameter to True.
State definition directory
--------------------------
State and phase definitions should be placed in a directory of the following
structure inside the dependencies directory of each workload (along with
revent files etc):
::
dependencies/
<workload_name>/
state_definitions/
definition.yaml
templates/
<oneTemplate>.png
<anotherTemplate>.png
...
definition.yaml file
--------------------
This defines each state of the workload and lists which templates are expected
to be found and how many are required to be detected for a conclusive match. It
also defines the expected state in each workload phase where a state detection
is run (currently those are setup_complete and run_complete).
Templates are picture elements to be matched in a screenshot. Each template
mentioned in the definition file should be placed as a file with the same name
and a .png extension inside the templates folder. Creating template png files
is as simple as taking a screenshot of the workload in a given state, cropping
out the relevant templates (eg. a button, label or other unique element that is
present in that state) and storing them in PNG format.
Please see the definition file for Angry Birds below as an example to
understand the format. Note that more than just two states (for the afterSetup
and afterRun phase) can be defined and this helps track the cause of errors in
case an unexpected state is encountered.
.. code-block:: yaml
workload_name: angrybirds
workload_states:
- state_name: titleScreen
templates:
- play_button
- logo
matches: 2
- state_name: worldSelection
templates:
- first_world_thumb
- second_world_thumb
- third_world_thumb
- fourth_world_thumb
matches: 3
- state_name: level_selection
templates:
- locked_level
- first_level
matches: 2
- state_name: gameplay
templates:
- pause_button
- score_label_text
matches: 2
- state_name: pause_screen
templates:
- replay_button
- menu_button
- resume_button
- help_button
matches: 4
- state_name: level_cleared_screen
templates:
- level_cleared_text
- menu_button
- replay_button
- fast_forward_button
matches: 4
workload_phases:
- phase_name: setup_complete
expected_state: gameplay
- phase_name: run_complete
expected_state: level_cleared_screen
File format of revent recordings
================================
You do not need to understand recording format in order to use revent. This
section is intended for those looking to extend revent in some way, or to
utilize revent recordings for other purposes.
Format Overview
---------------
Recordings are stored in a binary format. A recording consists of three
sections::
+-+-+-+-+-+-+-+-+-+-+-+
| Header |
+-+-+-+-+-+-+-+-+-+-+-+
| |
| Device Description |
| |
+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| Event Stream |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+
The header contains metadata describing the recording. The device description
contains information about input devices involved in this recording. Finally,
the event stream contains the recorded input events.
All fields are either fixed size or prefixed with their length or the number of
(fixed-sized) elements.
.. note:: All values below are little endian
Recording Header
----------------
An revent recoding header has the following structure
* It starts with the "magic" string ``REVENT`` to indicate that this is an
revent recording.
* The magic is followed by a 16 bit version number. This indicates the format
version of the recording that follows. Current version is ``2``.
* The next 16 bits indicate the type of the recording. This dictates the
structure of the Device Description section. Valid values are:
``0``
This is a general input event recording. The device description
contains a list of paths from which the events where recorded.
``1``
This a gamepad recording. The device description contains the
description of the gamepad used to create the recording.
* The header is zero-padded to 128 bits.
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 'R' | 'E' | 'V' | 'E' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 'N' | 'T' | Version |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Mode | PADDING |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PADDING |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Device Description
------------------
This section describes the input devices used in the recording. Its structure is
determined by the value of ``Mode`` field in the header.
general recording
~~~~~~~~~~~~~~~~~
.. note:: This is the only format supported prior to version ``2``.
The recording has been made from all available input devices. This section
contains the list of ``/dev/input`` paths for the devices, prefixed with total
number of the devices recorded.
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of devices |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Device paths +-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Similarly, each device path is a length-prefixed string. Unlike C strings, the
path is *not* NULL-terminated.
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of device path |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Device path |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
gamepad recording
~~~~~~~~~~~~~~~~~
The recording has been made from a specific gamepad. All events in the stream
will be for that device only. The section describes the device properties that
will be used to create a virtual input device using ``/dev/uinput``. Please
see ``linux/input.h`` header in the Linux kernel source for more information
about the fields in this section.
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| bustype | vendor |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| product | version |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| name_length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| name |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ev_bits |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| key_bits (96 bytes) |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| rel_bits (96 bytes) |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| abs_bits (96 bytes) |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| num_absinfo |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| |
| |
| absinfo entries |
| |
| |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Each ``absinfo`` entry consists of six 32 bit values. The number of entries is
determined by the ``abs_bits`` field.
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| value |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| minimum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| maximum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fuzz |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flat |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| resolution |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Event stream
------------
The majority of an revent recording will be made up of the input events that were
recorded. The event stream is prefixed with the number of events in the stream,
and start and end times for the recording.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of events |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of events (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Start Time Seconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Start Time Seconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Start Time Microseconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Start Time Microseconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| End Time Seconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| End Time Seconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| End Time Microseconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| End Time Microseconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| Events |
| |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Event structure
~~~~~~~~~~~~~~~
Each event entry structured as follows:
* An unsigned integer representing which device from the list of device paths
this event is for (zero indexed). E.g. Device ID = 3 would be the 4th
device in the list of device paths.
* A signed integer representing the number of seconds since "epoch" when the
event was recorded.
* A signed integer representing the microseconds part of the timestamp.
* An unsigned integer representing the event type
* An unsigned integer representing the event code
* An unsigned integer representing the event value
For more information about the event type, code and value please read:
https://www.kernel.org/doc/Documentation/input/event-codes.txt
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Device ID | Timestamp Seconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp Seconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp Seconds (cont.) | stamp Micoseconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp Micoseconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp Micoseconds (cont.) | Event Type |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Event Code | Event Value |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Event Value (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Parser
------
WA has a parser for revent recordings. This can be used to work with revent
recordings in scripts. Here is an example:
.. code:: python
from wlauto.utils.revent import ReventRecording
with ReventRecording('/path/to/recording.revent') as recording:
print "Recording: {}".format(recording.filepath)
print "There are {} input events".format(recording.num_events)
print "Over a total of {} seconds".format(recording.duration)

BIN
wa/assets/bin/arm64/revent Executable file

Binary file not shown.

BIN
wa/assets/bin/armeabi/revent Executable file

Binary file not shown.

View File

@ -178,7 +178,7 @@ class RecordCommand(Command):
self.logger.info('Deploying {}'.format(args.workload))
workload = pluginloader.get_workload(args.workload, self.target)
workload.apk.init_resources(context.resolver)
workload.apk.initialize(context)
workload.apk.setup(context)
sleep(workload.loading_time)

View File

@ -14,7 +14,7 @@
import os
import re
from copy import copy
from copy import copy, deepcopy
from collections import OrderedDict, defaultdict
from wa.framework.exception import ConfigError, NotFoundError
@ -34,7 +34,7 @@ KIND_MAP = {
Status = enum(['UNKNOWN', 'NEW', 'PENDING',
'STARTED', 'CONNECTED', 'INITIALIZED', 'RUNNING',
'SKIPPED', 'ABORTED', 'FAILED', 'PARTIAL', 'OK'])
'OK', 'PARTIAL', 'FAILED', 'ABORTED', 'SKIPPED'])
@ -272,12 +272,12 @@ class ConfigurationPoint(object):
value = merge_config_values(getattr(obj, self.name), value)
setattr(obj, self.name, value)
def validate(self, obj):
def validate(self, obj, check_mandatory=True):
value = getattr(obj, self.name, None)
if value is not None:
self.validate_value(obj.name, value)
else:
if self.mandatory:
if check_mandatory and self.mandatory:
msg = 'No value specified for mandatory parameter "{}" in {}.'
raise ConfigError(msg.format(self.name, obj.name))
@ -928,7 +928,8 @@ class JobSpec(Configuration):
def merge_workload_parameters(self, plugin_cache):
# merge global generic and specific config
workload_params = plugin_cache.get_plugin_config(self.workload_name,
generic_name="workload_parameters")
generic_name="workload_parameters",
is_final=False)
cfg_points = plugin_cache.get_plugin_parameters(self.workload_name)
for source in self._sources:
@ -1041,7 +1042,7 @@ class JobGenerator(object):
sections.insert(0, ancestor)
for workload_entry in workload_entries:
job_spec = create_job_spec(workload_entry, sections,
job_spec = create_job_spec(deepcopy(workload_entry), sections,
target_manager, self.plugin_cache,
self.disabled_instruments)
if self.ids_to_run:

View File

@ -99,7 +99,7 @@ class PluginCache(object):
def list_plugins(self, kind=None):
return self.loader.list_plugins(kind)
def get_plugin_config(self, plugin_name, generic_name=None):
def get_plugin_config(self, plugin_name, generic_name=None, is_final=True):
config = obj_dict(not_in_dict=['name'])
config.name = plugin_name
@ -120,7 +120,8 @@ class PluginCache(object):
else:
# A more complicated merge that involves priority of sources and
# specificity
self._merge_using_priority_specificity(plugin_name, generic_name, config)
self._merge_using_priority_specificity(plugin_name, generic_name,
config, is_final)
return config
@ -152,13 +153,13 @@ class PluginCache(object):
def _get_target_params(self, name):
td = self.targets[name]
params = {p.name: p for p in chain(td.target_params, td.platform_params)}
params = {p.name: p for p in chain(td.target_params, td.platform_params, td.conn_params)}
#params['connection_settings'] = {p.name: p for p in td.conn_params}
return params
# pylint: disable=too-many-nested-blocks, too-many-branches
def _merge_using_priority_specificity(self, specific_name,
generic_name, final_config):
generic_name, merged_config, is_final=True):
"""
WA configuration can come from various sources of increasing priority,
as well as being specified in a generic and specific manner (e.g
@ -175,13 +176,15 @@ class PluginCache(object):
In this situation it is not possible to know the end users intention
and WA will error.
:param generic_name: The name of the generic configuration
e.g ``device_config``
:param specific_name: The name of the specific configuration used
e.g ``nexus10``
:param cfg_point: A dict of ``ConfigurationPoint``s to be used when
merging configuration. keys=config point name,
values=config point
:param generic_name: The name of the generic configuration
e.g ``device_config``
:param merge_config: A dict of ``ConfigurationPoint``s to be used when
merging configuration. keys=config point name,
values=config point
:param is_final: if ``True`` (the default) make sure that mandatory
parameters are set.
:rtype: A fully merged and validated configuration in the form of a
obj_dict.
@ -197,18 +200,18 @@ class PluginCache(object):
# set_value uses the 'name' attribute of the passed object in it error
# messages, to ensure these messages make sense the name will have to be
# changed several times during this function.
final_config.name = specific_name
merged_config.name = specific_name
for source in sources:
try:
update_config_from_source(final_config, source, ms)
update_config_from_source(merged_config, source, ms)
except ConfigError as e:
raise ConfigError('Error in "{}":\n\t{}'.format(source, str(e)))
# Validate final configuration
final_config.name = specific_name
merged_config.name = specific_name
for cfg_point in ms.cfg_points.itervalues():
cfg_point.validate(final_config)
cfg_point.validate(merged_config, check_mandatory=is_final)
class MergeState(object):

View File

@ -32,6 +32,7 @@ from wa.framework.configuration.core import settings, Status
from wa.framework.exception import (WAError, ConfigError, TimeoutError,
InstrumentError, TargetError, HostError,
TargetNotRespondingError)
from wa.framework.job import Job
from wa.framework.output import init_job_output
from wa.framework.plugin import Artifact
from wa.framework.processor import ProcessorManager
@ -75,6 +76,11 @@ class ExecutionContext(object):
return True
return self.current_job.spec.id != self.next_job.spec.id
@property
def workload(self):
if self.current_job:
return self.current_job.workload
@property
def job_output(self):
if self.current_job:
@ -150,6 +156,11 @@ class ExecutionContext(object):
self.output.write_result()
self.current_job = None
def set_status(self, status, force=False):
if not self.current_job:
raise RuntimeError('No jobs in progress')
self.current_job.set_status(status, force)
def extract_results(self):
self.tm.extract_results(self)
@ -171,6 +182,14 @@ class ExecutionContext(object):
def write_state(self):
self.run_output.write_state()
def get_metric(self, name):
try:
return self.output.get_metric(name)
except HostError:
if not self.current_job:
raise
return self.run_output.get_metric(name)
def add_metric(self, name, value, units=None, lower_is_better=False,
classifiers=None):
if self.current_job:
@ -373,12 +392,12 @@ class Runner(object):
try:
log.indent()
self.do_run_job(job, context)
job.status = Status.OK
job.set_status(Status.OK)
except KeyboardInterrupt:
job.status = Status.ABORTED
job.set_status(Status.ABORTED)
raise
except Exception as e:
job.status = Status.FAILED
job.set_status(Status.FAILED)
context.add_event(e.message)
if not getattr(e, 'logged', None):
log.log_error(e, self.logger)
@ -392,7 +411,7 @@ class Runner(object):
self.check_job(job)
def do_run_job(self, job, context):
job.status = Status.RUNNING
job.set_status(Status.RUNNING)
self.send(signal.JOB_STARTED)
with signal.wrap('JOB_TARGET_CONFIG', self):
@ -411,15 +430,15 @@ class Runner(object):
self.pm.process_job_output(context)
self.pm.export_job_output(context)
except Exception:
job.status = Status.PARTIAL
job.set_status(Status.PARTIAL)
raise
except KeyboardInterrupt:
job.status = Status.ABORTED
job.set_status(Status.ABORTED)
self.logger.info('Got CTRL-C. Aborting.')
raise
except Exception as e:
job.status = Status.FAILED
job.set_status(Status.FAILED)
if not getattr(e, 'logged', None):
log.log_error(e, self.logger)
e.logged = True
@ -436,19 +455,24 @@ class Runner(object):
if job.retries < rc.max_retries:
msg = 'Job {} iteration {} completed with status {}. retrying...'
self.logger.error(msg.format(job.id, job.status, job.iteration))
self.retry_job(job)
self.context.move_failed(job)
job.retries += 1
job.status = Status.PENDING
self.context.job_queue.insert(0, job)
self.context.write_state()
else:
msg = 'Job {} iteration {} completed with status {}. '\
'Max retries exceeded.'
self.logger.error(msg.format(job.id, job.status, job.iteration))
self.logger.error(msg.format(job.id, job.iteration, job.status))
self.context.failed_jobs += 1
else: # status not in retry_on_status
self.logger.info('Job completed with status {}'.format(job.status))
self.context.successful_jobs += 1
def retry_job(self, job):
retry_job = Job(job.spec, job.iteration, self.context)
retry_job.workload = job.workload
retry_job.retries = job.retries + 1
retry_job.set_status(Status.PENDING)
self.context.job_queue.insert(0, retry_job)
def send(self, s):
signal.send(s, self, self.context)

View File

@ -104,7 +104,8 @@ from collections import OrderedDict
from wa.framework import signal
from wa.framework.plugin import Plugin
from wa.framework.exception import WAError, TargetNotRespondingError, TimeoutError
from wa.framework.exception import (WAError, TargetNotRespondingError, TimeoutError,
WorkloadError)
from wa.utils.log import log_error
from wa.utils.misc import get_traceback, isiterable
from wa.utils.types import identifier, enum, level
@ -250,7 +251,7 @@ def check_failures():
class ManagedCallback(object):
"""
This wraps instruments' callbacks to ensure that errors do interfer
This wraps instruments' callbacks to ensure that errors do not interfer
with run execution.
"""
@ -270,7 +271,14 @@ class ManagedCallback(object):
global failures_detected # pylint: disable=W0603
failures_detected = True
log_error(e, logger)
disable(self.instrument)
context.add_event(e.message)
if isinstance(e, WorkloadError):
context.set_status('FAILED')
else:
if context.current_job:
context.set_status('PARTIAL')
else:
raise
# Need this to keep track of callbacks, because the dispatcher only keeps

View File

@ -1,4 +1,5 @@
import logging
from datetime import datetime
from wa.framework import pluginloader, signal
from wa.framework.configuration.core import Status
@ -37,6 +38,7 @@ class Job(object):
self.context = context
self.workload = None
self.output = None
self.run_time = None
self.retries = 0
self._status = Status.NEW
@ -56,7 +58,7 @@ class Job(object):
self.logger.info('Initializing job {}'.format(self.id))
with signal.wrap('WORKLOAD_INITIALIZED', self, context):
self.workload.initialize(context)
self.status = Status.PENDING
self.set_status(Status.PENDING)
context.update_job_state(self)
def configure_target(self, context):
@ -71,7 +73,11 @@ class Job(object):
def run(self, context):
self.logger.info('Running job {}'.format(self.id))
with signal.wrap('WORKLOAD_EXECUTION', self, context):
self.workload.run(context)
start_time = datetime.utcnow()
try:
self.workload.run(context)
finally:
self.run_time = datetime.utcnow() - start_time
def process_output(self, context):
self.logger.info('Processing output for job {}'.format(self.id))
@ -90,3 +96,8 @@ class Job(object):
self.logger.info('Finalizing job {}'.format(self.id))
with signal.wrap('WORKLOAD_FINALIZED', self, context):
self.workload.finalize(context)
def set_status(self, status, force=False):
status = Status(status)
if force or self.status < status:
self.status = status

View File

@ -49,6 +49,18 @@ class Output(object):
def status(self, value):
self.result.status = value
@property
def metrics(self):
if self.result is None:
return []
return self.result.metrics
@property
def artifacts(self):
if self.result is None:
return []
return self.result.artifacts
def __init__(self, path):
self.basepath = path
self.result = None
@ -84,6 +96,9 @@ class Output(object):
def add_event(self, message):
self.result.add_event(message)
def get_metric(self, name):
return self.result.get_metric(name)
def get_artifact(self, name):
return self.result.get_artifact(name)
@ -241,6 +256,12 @@ class Result(object):
def add_event(self, message):
self.events.append(Event(message))
def get_metric(self, name):
for metric in self.metrics:
if metric.name == name:
return metric
return None
def get_artifact(self, name):
for artifact in self.artifacts:
if artifact.name == name:

View File

@ -110,9 +110,8 @@ class JobState(object):
@staticmethod
def from_pod(pod):
instance = JobState(pod['id'], pod['label'], Status(pod['status']))
instance = JobState(pod['id'], pod['label'], pod['iteration'], Status(pod['status']))
instance.retries = pod['retries']
instance.iteration = pod['iteration']
instance.timestamp = pod['timestamp']
return instance

View File

@ -2,7 +2,8 @@ from collections import OrderedDict
from copy import copy
from devlib import (LinuxTarget, AndroidTarget, LocalLinuxTarget,
Platform, Juno, TC2, Gem5SimulationPlatform)
Platform, Juno, TC2, Gem5SimulationPlatform,
AdbConnection, SshConnection, LocalConnection)
from wa.framework import pluginloader
from wa.framework.exception import PluginLoaderError
@ -248,16 +249,90 @@ GEM5_PLATFORM_PARAMS = [
'''),
]
# name --> (target_class, params_list, defaults, assistant_class)
CONNECTION_PARAMS = {
AdbConnection: [
Parameter('device', kind=str,
description="""
ADB device name
"""),
Parameter('adb_server', kind=str,
description="""
ADB server to connect to.
"""),
],
SshConnection: [
Parameter('host', kind=str, mandatory=True,
description="""
Host name or IP address of the target.
"""),
Parameter('username', kind=str, mandatory=True,
description="""
User name to connect with
"""),
Parameter('password', kind=str,
description="""
Password to use.
"""),
Parameter('keyfile', kind=str,
description="""
Key file to use
"""),
Parameter('port', kind=int,
description="""
The port SSH server is listening on on the target.
"""),
Parameter('telent', kind=bool, default=False,
description="""
If set to ``True``, a Telent connection, rather than
SSH will be used.
"""),
Parameter('password_prompt', kind=str,
description="""
Password prompt to expect
"""),
Parameter('original_prompt', kind=str,
description="""
Original shell prompt to expect.
"""),
Parameter('sudo_cmd', kind=str,
default="sudo -- sh -c '{}'",
description="""
Sudo command to use. Must have ``"{}"``` specified
somewher in the string it indicate where the command
to be run via sudo is to go.
"""),
],
LocalConnection: [
Parameter('password', kind=str,
description="""
Password to use for sudo. if not specified, the user will
be prompted during intialization.
"""),
Parameter('keep_password', kind=bool, default=True,
description="""
If ``True`` (the default), the password will be cached in
memory after it is first obtained from the user, so that the
user would not be prompted for it again.
"""),
Parameter('unrooted', kind=bool, default=False,
description="""
Indicate that the target should be considered unrooted; do not
attempt sudo or ask the user for their password.
"""),
],
}
# name --> ((target_class, conn_class), params_list, defaults, assistant_class)
TARGETS = {
'linux': (LinuxTarget, COMMON_TARGET_PARAMS, None),
'android': (AndroidTarget, COMMON_TARGET_PARAMS +
'linux': ((LinuxTarget, SshConnection), COMMON_TARGET_PARAMS, None),
'android': ((AndroidTarget, AdbConnection), COMMON_TARGET_PARAMS +
[Parameter('package_data_directory', kind=str, default='/data/data',
description='''
Directory containing Android data
'''),
], None),
'local': (LocalLinuxTarget, COMMON_TARGET_PARAMS, None),
'local': ((LocalLinuxTarget, LocalConnection), COMMON_TARGET_PARAMS, None),
}
# name --> assistant
@ -303,17 +378,19 @@ class DefaultTargetDescriptor(TargetDescriptor):
def get_descriptions(self):
result = []
for target_name, target_tuple in TARGETS.iteritems():
target, target_params = self._get_item(target_tuple)
(target, conn), target_params = self._get_item(target_tuple)
assistant = ASSISTANTS[target_name]
conn_params = CONNECTION_PARAMS[conn]
for platform_name, platform_tuple in PLATFORMS.iteritems():
platform, platform_params = self._get_item(platform_tuple)
name = '{}_{}'.format(platform_name, target_name)
td = TargetDescription(name, self)
td.target = target
td.conn = conn
td.platform = platform
td.assistant = assistant
td.target_params = target_params
td.conn_params = conn_params
td.platform_params = platform_params
td.assistant_params = assistant.parameters
result.append(td)

View File

@ -107,9 +107,9 @@ class ApkWorkload(Workload):
package_names = []
parameters = [
Parameter('package', kind=str,
Parameter('package_name', kind=str,
description="""
The pacakge name that can be used to specify
The package name that can be used to specify
the workload apk to use.
"""),
Parameter('install_timeout', kind=int,
@ -153,10 +153,14 @@ class ApkWorkload(Workload):
""")
]
@property
def package(self):
return self.apk.package
def __init__(self, target, **kwargs):
super(ApkWorkload, self).__init__(target, **kwargs)
self.apk = PackageHandler(self,
package=self.package,
package_name=self.package_name,
variant=self.variant,
strict=self.strict,
version=self.version,
@ -384,8 +388,9 @@ class ReventGUI(object):
def setup(self):
self._check_revent_files()
self.revent_recorder.replay(self.on_target_setup_revent,
timeout=self.setup_timeout)
if self.revent_setup_file:
self.revent_recorder.replay(self.on_target_setup_revent,
timeout=self.setup_timeout)
def run(self):
msg = 'Replaying {}'
@ -429,8 +434,14 @@ class ReventGUI(object):
class PackageHandler(object):
@property
def package(self):
if self.apk_info is None:
return None
return self.apk_info.package
def __init__(self, owner, install_timeout=300, version=None, variant=None,
package=None, strict=False, force_install=False, uninstall=False,
package_name=None, strict=False, force_install=False, uninstall=False,
exact_abi=False):
self.logger = logging.getLogger('apk')
self.owner = owner
@ -438,7 +449,7 @@ class PackageHandler(object):
self.install_timeout = install_timeout
self.version = version
self.variant = variant
self.package = package
self.package_name = package_name
self.strict = strict
self.force_install = force_install
self.uninstall = uninstall
@ -462,7 +473,7 @@ class PackageHandler(object):
self.apk_file = context.resolver.get(ApkFile(self.owner,
variant=self.variant,
version=self.version,
package=self.package,
package=self.package_name,
exact_abi=self.exact_abi,
supported_abi=self.supported_abi),
strict=self.strict)
@ -471,19 +482,19 @@ class PackageHandler(object):
if self.version:
installed_version = self.target.get_package_version(self.apk_info.package)
host_version = self.apk_info.version_name
if (installed_version != host_version and
if (installed_version and installed_version != host_version and
loose_version_matching(self.version, installed_version)):
msg = 'Multiple matching packages found for {}; host version: {}, device version: {}'
raise WorkloadError(msg.format(self.owner, host_version, installed_version))
else:
if not self.owner.package_names and not self.package:
if not self.owner.package_names and not self.package_name:
msg = 'No package name(s) specified and no matching APK file found on host'
raise WorkloadError(msg)
self.resolve_package_from_target(context)
def resolve_package_from_target(self, context):
if self.package:
if not self.target.package_is_installed(self.package):
if self.package_name:
if not self.target.package_is_installed(self.package_name):
msg = 'Package "{}" cannot be found on the host or device'
raise WorkloadError(msg.format(self.package_name))
else:
@ -496,23 +507,23 @@ class PackageHandler(object):
for package in installed_versions:
package_version = self.target.get_package_version(package)
if loose_version_matching(self.version, package_version):
self.package = package
self.package_name = package
break
else:
if len(installed_versions) == 1:
self.package = installed_versions[0]
self.package_name = installed_versions[0]
else:
msg = 'Package version not set and multiple versions found on device'
raise WorkloadError(msg)
if not self.package:
if not self.package_name:
raise WorkloadError('No matching package found')
self.pull_apk(self.package)
self.pull_apk(self.package_name)
self.apk_file = context.resolver.get(ApkFile(self.owner,
variant=self.variant,
version=self.version,
package=self.package),
package=self.package_name),
strict=self.strict)
self.apk_info = ApkInfo(self.apk_file)

View File

@ -18,12 +18,12 @@
from __future__ import division
import os
from devlib import DerivedEnergyMeasurements
from devlib.instrument import CONTINUOUS
from devlib.instrument.energy_probe import EnergyProbeInstrument
from devlib.instrument.daq import DaqInstrument
from devlib.instrument.acmecape import AcmeCapeInstrument
from devlib.utils.misc import which
from devlib.derived.derived_measurements import DerivedEnergyMeasurements
from wa import Instrument, Parameter
from wa.framework import pluginloader

View File

@ -22,6 +22,7 @@
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/stat.h>
@ -49,8 +50,10 @@
const char MAGIC[] = "REVENT";
// NOTE: This should be incremented if any changes are made to the file format
uint16_t FORMAT_VERSION = 2;
// NOTE: This should be incremented if any changes are made to the file format.
// Should that be the case, also make sure to update the format description
// in doc/source/revent.rst and the Python parser in wa/utils/revent.py.
uint16_t FORMAT_VERSION = 3;
typedef enum {
FALSE=0,
@ -117,6 +120,8 @@ typedef struct {
input_devices_t devices;
device_info_t *gamepad_info;
uint64_t num_events;
struct timeval start_time;
struct timeval end_time;
replay_event_t *events;
} revent_recording_t;
@ -268,14 +273,17 @@ void adjust_event_times(revent_recording_t *recording)
if (recording->num_events == 0)
return;
time_zero.tv_sec = recording->events[0].event.time.tv_sec;
time_zero.tv_usec = recording->events[0].event.time.tv_usec;
time_zero.tv_sec = recording->start_time.tv_sec;
time_zero.tv_usec = recording->start_time.tv_usec;
for(i = 0; i < recording->num_events; i++) {
timersub(&recording->events[i].event.time, &time_zero, &time_delta);
recording->events[i].event.time.tv_sec = time_delta.tv_sec;
recording->events[i].event.time.tv_usec = time_delta.tv_usec;
}
timersub(&recording->end_time, &time_zero, &time_delta);
recording->end_time.tv_sec = time_delta.tv_sec;
recording->end_time.tv_usec = time_delta.tv_usec;
}
int write_record_header(int fd, const revent_record_desc_t *desc)
@ -559,6 +567,28 @@ void print_device_info(device_info_t *info)
}
}
int read_record_timestamps(FILE *fin, revent_recording_t *recording)
{
int ret;
ret = fread(&recording->start_time.tv_sec, sizeof(uint64_t), 1, fin);
if (ret < 1)
return errno;
ret = fread(&recording->start_time.tv_usec, sizeof(uint64_t), 1, fin);
if (ret < 1)
return errno;
ret = fread(&recording->end_time.tv_sec, sizeof(uint64_t), 1, fin);
if (ret < 1)
return errno;
ret = fread(&recording->end_time.tv_usec, sizeof(uint64_t), 1, fin);
if (ret < 1)
return errno;
return 0;
}
int write_replay_event(FILE *fout, const replay_event_t *ev)
{
size_t ret;
@ -982,13 +1012,33 @@ inline void read_revent_recording_or_die(const char *filepath, revent_recording_
if (ret < 1)
die("Could not read the number of recorded events");
if (recording->desc.version > 2) {
ret = read_record_timestamps(fin, recording);
if (ret)
die("Could not read recroding timestamps.");
}
recording->events = malloc(sizeof(replay_event_t) * recording->num_events);
if (recording->events == NULL)
die("Not enough memory to allocate replay buffer");
for(i=0; i < recording->num_events; i++) {
// start/end times tracking for recording as a whole was added in version 3
// of recording format; for earlier recordings, use timestamps of the first and
// last events.
read_replay_event(fin, &recording->events[0]);
if (recording->desc.version <= 2) {
recording->start_time.tv_sec = recording->events[0].event.time.tv_sec;
recording->start_time.tv_usec = recording->events[0].event.time.tv_usec;
}
for(i=1; i < recording->num_events; i++) {
read_replay_event(fin, &recording->events[i]);
}
if (recording->desc.version <= 2) {
recording->end_time.tv_sec = recording->events[i].event.time.tv_sec;
recording->end_time.tv_usec = recording->events[i].event.time.tv_usec;
}
} else { // backwards compatibility
/* Prior to verion 2, the total number of recorded events was not being
* written as part of the recording. We will use the size of the file on
@ -1039,6 +1089,7 @@ void exitHandler(int z) {
void record(const char *filepath, int delay, recording_mode_t mode)
{
int ret;
struct timespec start_time, end_time;
FILE *fout = init_recording(filepath, mode);
if (fout == NULL)
die("Could not create recording \"%s\": %s", filepath, strerror(errno));
@ -1075,10 +1126,11 @@ void record(const char *filepath, int delay, recording_mode_t mode)
// Write the zero size as a place holder and remember the position in the
// file stream, so that it may be updated at the end with the actual event
// count.
// count. Reserving space for five uint64_t's -- the number of events and
// end time stamps.
uint64_t event_count = 0;
long size_pos = ftell(fout);
ret = fwrite(&event_count, sizeof(uint64_t), 1, fout);
ret = fwrite(&event_count, sizeof(uint64_t), 5, fout);
if (ret < 1)
die("Could not initialise event count: %s", strerror(errno));
@ -1096,6 +1148,7 @@ void record(const char *filepath, int delay, recording_mode_t mode)
errno = 0;
signal(SIGINT, exitHandler);
clock_gettime(CLOCK_MONOTONIC_RAW, &start_time);
while(1)
{
FD_ZERO(&readfds);
@ -1160,6 +1213,7 @@ void record(const char *filepath, int delay, recording_mode_t mode)
}
}
}
clock_gettime(CLOCK_MONOTONIC_RAW, &end_time);
dprintf("Writing event count...");
if ((ret = fseek(fout, size_pos, SEEK_SET)) == -1)
@ -1167,6 +1221,16 @@ void record(const char *filepath, int delay, recording_mode_t mode)
ret = fwrite(&event_count, sizeof(uint64_t), 1, fout);
if (ret < 1)
die("Could not write event count: %s", strerror(errno));
dprintf("Writing recording timestamps...");
uint64_t usecs;
fwrite(&start_time.tv_sec, sizeof(uint64_t), 1, fout);
usecs = start_time.tv_nsec / 1000;
fwrite(&usecs, sizeof(uint64_t), 1, fout);
fwrite(&end_time.tv_sec, sizeof(uint64_t), 1, fout);
usecs = end_time.tv_nsec / 1000;
ret = fwrite(&usecs, sizeof(uint64_t), 1, fout);
if (ret < 1)
die("Could not write recording timestamps: %s", strerror(errno));
fclose(fout);
@ -1190,6 +1254,8 @@ void dump(const char *filepath)
printf("recording version: %u\n", recording.desc.version);
printf("recording type: %i\n", recording.desc.mode);
printf("number of recorded events: %lu\n", recording.num_events);
printf("start time: %ld.%06ld \n", recording.start_time.tv_sec, recording.start_time.tv_usec);
printf("end time: %ld.%06ld \n", recording.end_time.tv_sec, recording.end_time.tv_usec);
printf("\n");
if (recording.desc.mode == GENERAL_MODE) {
@ -1264,18 +1330,35 @@ void replay(const char *filepath)
int32_t idx = (recording.events[i]).dev_idx;
struct input_event ev = (recording.events[i]).event;
while((i < recording.num_events) && !timercmp(&ev.time, &last_event_delta, !=)) {
while(!timercmp(&ev.time, &last_event_delta, !=)) {
ret = write(recording.devices.fds[idx], &ev, sizeof(ev));
if (ret != sizeof(ev))
die("Could not replay event");
dprintf("replayed event: type %d code %d value %d\n", ev.type, ev.code, ev.value);
i++;
if (i >= recording.num_events) {
break;
}
idx = recording.events[i].dev_idx;
ev = recording.events[i].event;
}
last_event_delta = ev.time;
}
timeradd(&start_time, &recording.end_time, &desired_time);
gettimeofday(&now, NULL);
if (timercmp(&desired_time, &now, >)) {
timersub(&desired_time, &now, &delta);
useconds_t d = (useconds_t)delta.tv_sec * 1000000 + delta.tv_usec;
dprintf("now %u.%u recording end time %u.%u sleeping %u uS\n",
(unsigned int)now.tv_sec,
(unsigned int)now.tv_usec,
(unsigned int)desired_time.tv_sec,
(unsigned int)desired_time.tv_usec,
d);
usleep(d);
}
if (recording.desc.mode == GAMEPAD_MODE)
destroy_replay_device(recording.devices.fds[0]);

View File

@ -165,6 +165,8 @@ def log_error(e, logger, critical=False):
log_func(tb)
log_func('{}({})'.format(e.__class__.__name__, e))
e.logged = True
class ErrorSignalHandler(logging.Handler):
"""

View File

@ -202,7 +202,7 @@ class ReventRecording(object):
raise ValueError(msg.format(self.filepath))
self.version = version
if self.version == 2:
if 3 >= self.version >= 2:
self.mode, = read_struct(fh, header_two_struct)
if self.mode == GENERAL_MODE:
self._read_devices(fh)
@ -211,6 +211,14 @@ class ReventRecording(object):
else:
raise ValueError('Unexpected recording mode: {}'.format(self.mode))
self.num_events, = read_struct(fh, u64_struct)
if self.version > 2:
ts_sec = read_struct(fh, u64_struct)
ts_usec = read_struct(fh, u64_struct)
self.start_time = datetime.fromtimestamp(ts_sec + float(ts_usec) / 1000000)
ts_sec = read_struct(fh, u64_struct)
ts_usec = read_struct(fh, u64_struct)
self.end_time = datetime.fromtimestamp(ts_sec + float(ts_usec) / 1000000)
elif 2 > self.version >= 0:
self.mode = GENERAL_MODE
self._read_devices(fh)

View File

@ -0,0 +1,28 @@
# Copyright 2013-2015 ARM Limited
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
from wa import ReventWorkload
class TempleRun2(ReventWorkload):
name = 'templerun2'
description = """
Temple Run 2 game.
Sequel to Temple Run. 3D on-the-rails racer.
"""
view = 'SurfaceView - com.imangi.templerun2/com.imangi.unityactivity.ImangiUnityNativeActivity'