mirror of
https://github.com/ARM-software/workload-automation.git
synced 2025-10-29 22:24:51 +00:00
Initial commit of open source Workload Automation.
This commit is contained in:
Sergei Trofimov
956
doc/source/writing_extensions.rst
Normal file
956
doc/source/writing_extensions.rst
Normal file
@@ -0,0 +1,956 @@
|
||||
==================
|
||||
Writing Extensions
|
||||
==================
|
||||
|
||||
Workload Automation offers several extension points (or plugin types).The most
|
||||
interesting of these are
|
||||
|
||||
:workloads: These are the tasks that get executed and measured on the device. These
|
||||
can be benchmarks, high-level use cases, or pretty much anything else.
|
||||
:devices: These are interfaces to the physical devices (development boards or end-user
|
||||
devices, such as smartphones) that use cases run on. Typically each model of a
|
||||
physical device would require it's own interface class (though some functionality
|
||||
may be reused by subclassing from an existing base).
|
||||
:instruments: Instruments allow collecting additional data from workload execution (e.g.
|
||||
system traces). Instruments are not specific to a particular Workload. Instruments
|
||||
can hook into any stage of workload execution.
|
||||
:result processors: These are used to format the results of workload execution once they have been
|
||||
collected. Depending on the callback used, these will run either after each
|
||||
iteration or at the end of the run, after all of the results have been
|
||||
collected.
|
||||
|
||||
You create an extension by subclassing the appropriate base class, defining
|
||||
appropriate methods and attributes, and putting the .py file with the class into
|
||||
an appropriate subdirectory under ``~/.workload_automation`` (there is one for
|
||||
each extension type).
|
||||
|
||||
|
||||
Extension Basics
|
||||
================
|
||||
|
||||
This sub-section covers things common to implementing extensions of all types.
|
||||
It is recommended you familiarize yourself with the information here before
|
||||
proceeding onto guidance for specific extension types.
|
||||
|
||||
To create an extension, you basically subclass an appropriate base class and them
|
||||
implement the appropriate methods
|
||||
|
||||
The Context
|
||||
-----------
|
||||
|
||||
The majority of methods in extensions accept a context argument. This is an
|
||||
instance of :class:`wlauto.core.execution.ExecutionContext`. If contains
|
||||
of information about current state of execution of WA and keeps track of things
|
||||
like which workload is currently running and the current iteration.
|
||||
|
||||
Notable attributes of the context are
|
||||
|
||||
context.spec
|
||||
the current workload specification being executed. This is an
|
||||
instance of :class:`wlauto.core.configuration.WorkloadRunSpec`
|
||||
and defines the workload and the parameters under which it is
|
||||
being executed.
|
||||
|
||||
context.workload
|
||||
``Workload`` object that is currently being executed.
|
||||
|
||||
context.current_iteration
|
||||
The current iteration of the spec that is being executed. Note that this
|
||||
is the iteration for that spec, i.e. the number of times that spec has
|
||||
been run, *not* the total number of all iterations have been executed so
|
||||
far.
|
||||
|
||||
context.result
|
||||
This is the result object for the current iteration. This is an instance
|
||||
of :class:`wlauto.core.result.IterationResult`. It contains the status
|
||||
of the iteration as well as the metrics and artifacts generated by the
|
||||
workload and enable instrumentation.
|
||||
|
||||
context.device
|
||||
The device interface object that can be used to interact with the
|
||||
device. Note that workloads and instruments have their own device
|
||||
attribute and they should be using that instead.
|
||||
|
||||
In addition to these, context also defines a few useful paths (see below).
|
||||
|
||||
|
||||
Paths
|
||||
-----
|
||||
|
||||
You should avoid using hard-coded absolute paths in your extensions whenever
|
||||
possible, as they make your code too dependent on a particular environment and
|
||||
may mean having to make adjustments when moving to new (host and/or device)
|
||||
platforms. To help avoid hard-coded absolute paths, WA automation defines
|
||||
a number of standard locations. You should strive to define your paths relative
|
||||
to one of those.
|
||||
|
||||
On the host
|
||||
~~~~~~~~~~~
|
||||
|
||||
Host paths are available through the context object, which is passed to most
|
||||
extension methods.
|
||||
|
||||
context.run_output_directory
|
||||
This is the top-level output directory for all WA results (by default,
|
||||
this will be "wa_output" in the directory in which WA was invoked.
|
||||
|
||||
context.output_directory
|
||||
This is the output directory for the current iteration. This will an
|
||||
iteration-specific subdirectory under the main results location. If
|
||||
there is no current iteration (e.g. when processing overall run results)
|
||||
this will point to the same location as ``root_output_directory``.
|
||||
|
||||
context.host_working_directory
|
||||
This an addition location that may be used by extensions to store
|
||||
non-iteration specific intermediate files (e.g. configuration).
|
||||
|
||||
Additionally, the global ``wlauto.settings`` object exposes on other location:
|
||||
|
||||
settings.dependency_directory
|
||||
this is the root directory for all extension dependencies (e.g. media
|
||||
files, assets etc) that are not included within the extension itself.
|
||||
|
||||
As per Python best practice, it is recommended that methods and values in
|
||||
``os.path`` standard library module are used for host path manipulation.
|
||||
|
||||
On the device
|
||||
~~~~~~~~~~~~~
|
||||
|
||||
Workloads and instruments have a ``device`` attribute, which is an interface to
|
||||
the device used by WA. It defines the following location:
|
||||
|
||||
device.working_directory
|
||||
This is the directory for all WA-related files on the device. All files
|
||||
deployed to the device should be pushed to somewhere under this location
|
||||
(the only exception being executables installed with ``device.install``
|
||||
method).
|
||||
|
||||
Since there could be a mismatch between path notation used by the host and the
|
||||
device, the ``os.path`` modules should *not* be used for on-device path
|
||||
manipulation. Instead device has an equipment module exposed through
|
||||
``device.path`` attribute. This has all the same attributes and behaves the
|
||||
same way as ``os.path``, but is guaranteed to produce valid paths for the device,
|
||||
irrespective of the host's path notation.
|
||||
|
||||
.. note:: result processors, unlike workloads and instruments, do not have their
|
||||
own device attribute; however they can access the device through the
|
||||
context.
|
||||
|
||||
|
||||
Parameters
|
||||
----------
|
||||
|
||||
All extensions can be parameterized. Parameters are specified using
|
||||
``parameters`` class attribute. This should be a list of
|
||||
:class:`wlauto.core.Parameter` instances. The following attributes can be
|
||||
specified on parameter creation:
|
||||
|
||||
name
|
||||
This is the only mandatory argument. The name will be used to create a
|
||||
corresponding attribute in the extension instance, so it must be a valid
|
||||
Python identifier.
|
||||
|
||||
kind
|
||||
This is the type of the value of the parameter. This could be an
|
||||
callable. Normally this should be a standard Python type, e.g. ``int`
|
||||
or ``float``, or one the types defined in :mod:`wlauto.utils.types`.
|
||||
If not explicitly specified, this will default to ``str``.
|
||||
|
||||
.. note:: Irrespective of the ``kind`` specified, ``None`` is always a
|
||||
valid value for a parameter. If you don't want to allow
|
||||
``None``, then set ``mandatory`` (see below) to ``True``.
|
||||
|
||||
allowed_values
|
||||
A list of the only allowed values for this parameter.
|
||||
|
||||
.. note:: For composite types, such as ``list_of_strings`` or
|
||||
``list_of_ints`` in :mod:`wlauto.utils.types`, each element of
|
||||
the value will be checked against ``allowed_values`` rather
|
||||
than the composite value itself.
|
||||
|
||||
default
|
||||
The default value to be used for this parameter if one has not been
|
||||
specified by the user. Defaults to ``None``.
|
||||
|
||||
mandatory
|
||||
A ``bool`` indicating whether this parameter is mandatory. Setting this
|
||||
to ``True`` will make ``None`` an illegal value for the parameter.
|
||||
Defaults to ``False``.
|
||||
|
||||
.. note:: Specifying a ``default`` will mean that this parameter will,
|
||||
effectively, be ignored (unless the user sets the param to ``None``).
|
||||
|
||||
.. note:: Mandatory parameters are *bad*. If at all possible, you should
|
||||
strive to provide a sensible ``default`` or to make do without
|
||||
the parameter. Only when the param is absolutely necessary,
|
||||
and there really is no sensible default that could be given
|
||||
(e.g. something like login credentials), should you consider
|
||||
making it mandatory.
|
||||
|
||||
constraint
|
||||
This is an additional constraint to be enforced on the parameter beyond
|
||||
its type or fixed allowed values set. This should be a predicate (a function
|
||||
that takes a single argument -- the user-supplied value -- and returns
|
||||
a ``bool`` indicating whether the constraint has been satisfied).
|
||||
|
||||
override
|
||||
A parameter name must be unique not only within an extension but also
|
||||
with that extension's class hierarchy. If you try to declare a parameter
|
||||
with the same name as already exists, you will get an error. If you do
|
||||
want to override a parameter from further up in the inheritance
|
||||
hierarchy, you can indicate that by setting ``override`` attribute to
|
||||
``True``.
|
||||
|
||||
When overriding, you do not need to specify every other attribute of the
|
||||
parameter, just the ones you what to override. Values for the rest will
|
||||
be taken from the parameter in the base class.
|
||||
|
||||
|
||||
Validation and cross-parameter constraints
|
||||
------------------------------------------
|
||||
|
||||
An extension will get validated at some point after constructions. When exactly
|
||||
this occurs depends on the extension type, but it *will* be validated before it
|
||||
is used.
|
||||
|
||||
You can implement ``validate`` method in your extension (that takes no arguments
|
||||
beyond the ``self``) to perform any additions *internal* validation in your
|
||||
extension. By "internal", I mean that you cannot make assumptions about the
|
||||
surrounding environment (e.g. that the device has been initialized).
|
||||
|
||||
The contract for ``validate`` method is that it should raise an exception
|
||||
(either ``wlauto.exceptions.ConfigError`` or extension-specific exception type -- see
|
||||
further on this page) if some validation condition has not, and cannot, been met.
|
||||
If the method returns without raising an exception, then the extension is in a
|
||||
valid internal state.
|
||||
|
||||
Note that ``validate`` can be used not only to verify, but also to impose a
|
||||
valid internal state. In particular, this where cross-parameter constraints can
|
||||
be resolved. If the ``default`` or ``allowed_values`` of one parameter depend on
|
||||
another parameter, there is no way to express that declaratively when specifying
|
||||
the parameters. In that case the dependent attribute should be left unspecified
|
||||
on creation and should instead be set inside ``validate``.
|
||||
|
||||
Logging
|
||||
-------
|
||||
|
||||
Every extension class has it's own logger that you can access through
|
||||
``self.logger`` inside the extension's methods. Generally, a :class:`Device` will log
|
||||
everything it is doing, so you shouldn't need to add much additional logging in
|
||||
your expansion's. But you might what to log additional information, e.g.
|
||||
what settings your extension is using, what it is doing on the host, etc.
|
||||
Operations on the host will not normally be logged, so your extension should
|
||||
definitely log what it is doing on the host. One situation in particular where
|
||||
you should add logging is before doing something that might take a significant amount
|
||||
of time, such as downloading a file.
|
||||
|
||||
|
||||
Documenting
|
||||
-----------
|
||||
|
||||
All extensions and their parameter should be documented. For extensions
|
||||
themselves, this is done through ``description`` class attribute. The convention
|
||||
for an extension description is that the first paragraph should be a short
|
||||
summary description of what the extension does and why one would want to use it
|
||||
(among other things, this will get extracted and used by ``wa list`` command).
|
||||
Subsequent paragraphs (separated by blank lines) can then provide a more
|
||||
detailed description, including any limitations and setup instructions.
|
||||
|
||||
For parameters, the description is passed as an argument on creation. Please
|
||||
note that if ``default``, ``allowed_values``, or ``constraint``, are set in the
|
||||
parameter, they do not need to be explicitly mentioned in the description (wa
|
||||
documentation utilities will automatically pull those). If the ``default`` is set
|
||||
in ``validate`` or additional cross-parameter constraints exist, this *should*
|
||||
be documented in the parameter description.
|
||||
|
||||
Both extensions and their parameters should be documented using reStructureText
|
||||
markup (standard markup for Python documentation). See:
|
||||
|
||||
http://docutils.sourceforge.net/rst.html
|
||||
|
||||
Aside from that, it is up to you how you document your extension. You should try
|
||||
to provide enough information so that someone unfamiliar with your extension is
|
||||
able to use it, e.g. you should document all settings and parameters your
|
||||
extension expects (including what the valid value are).
|
||||
|
||||
|
||||
Error Notification
|
||||
------------------
|
||||
|
||||
When you detect an error condition, you should raise an appropriate exception to
|
||||
notify the user. The exception would typically be :class:`ConfigError` or
|
||||
(depending the type of the extension)
|
||||
:class:`WorkloadError`/:class:`DeviceError`/:class:`InstrumentError`/:class:`ResultProcessorError`.
|
||||
All these errors are defined in :mod:`wlauto.exception` module.
|
||||
|
||||
:class:`ConfigError` should be raised where there is a problem in configuration
|
||||
specified by the user (either through the agenda or config files). These errors
|
||||
are meant to be resolvable by simple adjustments to the configuration (and the
|
||||
error message should suggest what adjustments need to be made. For all other
|
||||
errors, such as missing dependencies, mis-configured environment, problems
|
||||
performing operations, etc., the extension type-specific exceptions should be
|
||||
used.
|
||||
|
||||
If the extension itself is capable of recovering from the error and carrying
|
||||
on, it may make more sense to log an ERROR or WARNING level message using the
|
||||
extension's logger and to continue operation.
|
||||
|
||||
|
||||
Utils
|
||||
-----
|
||||
|
||||
Workload Automation defines a number of utilities collected under
|
||||
:mod:`wlauto.utils` subpackage. These utilities were created to help with the
|
||||
implementation of the framework itself, but may be also be useful when
|
||||
implementing extensions.
|
||||
|
||||
|
||||
Adding a Workload
|
||||
=================
|
||||
|
||||
.. note:: You can use ``wa create workload [name]`` script to generate a new workload
|
||||
structure for you. This script can also create the boilerplate for
|
||||
UI automation, if your workload needs it. See ``wa create -h`` for more
|
||||
details.
|
||||
|
||||
New workloads can be added by subclassing :class:`wlauto.core.workload.Workload`
|
||||
|
||||
|
||||
The Workload class defines the following interface::
|
||||
|
||||
class Workload(Extension):
|
||||
|
||||
name = None
|
||||
|
||||
def init_resources(self, context):
|
||||
pass
|
||||
|
||||
def setup(self, context):
|
||||
raise NotImplementedError()
|
||||
|
||||
def run(self, context):
|
||||
raise NotImplementedError()
|
||||
|
||||
def update_result(self, context):
|
||||
raise NotImplementedError()
|
||||
|
||||
def teardown(self, context):
|
||||
raise NotImplementedError()
|
||||
|
||||
def validate(self):
|
||||
pass
|
||||
|
||||
.. note:: Please see :doc:`conventions` section for notes on how to interpret
|
||||
this.
|
||||
|
||||
The interface should be implemented as follows
|
||||
|
||||
:name: This identifies the workload (e.g. it used to specify it in the
|
||||
agenda_.
|
||||
:init_resources: This method may be optionally override to implement dynamic
|
||||
resource discovery for the workload.
|
||||
**Added in version 2.1.3**
|
||||
:setup: Everything that needs to be in place for workload execution should
|
||||
be done in this method. This includes copying files to the device,
|
||||
starting up an application, configuring communications channels,
|
||||
etc.
|
||||
:run: This method should perform the actual task that is being measured.
|
||||
When this method exits, the task is assumed to be complete.
|
||||
|
||||
.. note:: Instrumentation is kicked off just before calling this
|
||||
method and is disabled right after, so everything in this
|
||||
method is being measured. Therefore this method should
|
||||
contain the least code possible to perform the operations
|
||||
you are interested in measuring. Specifically, things like
|
||||
installing or starting applications, processing results, or
|
||||
copying files to/from the device should be done elsewhere if
|
||||
possible.
|
||||
|
||||
:update_result: This method gets invoked after the task execution has
|
||||
finished and should be used to extract metrics and add them
|
||||
to the result (see below).
|
||||
:teardown: This could be used to perform any cleanup you may wish to do,
|
||||
e.g. Uninstalling applications, deleting file on the device, etc.
|
||||
|
||||
:validate: This method can be used to validate any assumptions your workload
|
||||
makes about the environment (e.g. that required files are
|
||||
present, environment variables are set, etc) and should raise
|
||||
a :class:`wlauto.exceptions.WorkloadError` if that is not the
|
||||
case. The base class implementation only makes sure sure that
|
||||
the name attribute has been set.
|
||||
|
||||
.. _agenda: agenda.html
|
||||
|
||||
Workload methods (except for ``validate``) take a single argument that is a
|
||||
:class:`wlauto.core.execution.ExecutionContext` instance. This object keeps
|
||||
track of the current execution state (such as the current workload, iteration
|
||||
number, etc), and contains, among other things, a
|
||||
:class:`wlauto.core.workload.WorkloadResult` instance that should be populated
|
||||
from the ``update_result`` method with the results of the execution. ::
|
||||
|
||||
# ...
|
||||
|
||||
def update_result(self, context):
|
||||
# ...
|
||||
context.result.add_metric('energy', 23.6, 'Joules', lower_is_better=True)
|
||||
|
||||
# ...
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
This example shows a simple workload that times how long it takes to compress a
|
||||
file of a particular size on the device.
|
||||
|
||||
.. note:: This is intended as an example of how to implement the Workload
|
||||
interface. The methodology used to perform the actual measurement is
|
||||
not necessarily sound, and this Workload should not be used to collect
|
||||
real measurements.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import os
|
||||
from wlauto import Workload, Parameter
|
||||
|
||||
class ZiptestWorkload(Workload):
|
||||
|
||||
name = 'ziptest'
|
||||
description = '''
|
||||
Times how long it takes to gzip a file of a particular size on a device.
|
||||
|
||||
This workload was created for illustration purposes only. It should not be
|
||||
used to collect actual measurements.
|
||||
|
||||
'''
|
||||
|
||||
parameters = [
|
||||
Parameter('file_size', kind=int, default=2000000,
|
||||
description='Size of the file (in bytes) to be gzipped.')
|
||||
]
|
||||
|
||||
def setup(self, context):
|
||||
# Generate a file of the specified size containing random garbage.
|
||||
host_infile = os.path.join(context.output_directory, 'infile')
|
||||
command = 'openssl rand -base64 {} > {}'.format(self.file_size, host_infile)
|
||||
os.system(command)
|
||||
# Set up on-device paths
|
||||
devpath = self.device.path # os.path equivalent for the device
|
||||
self.device_infile = devpath.join(self.device.working_directory, 'infile')
|
||||
self.device_outfile = devpath.join(self.device.working_directory, 'outfile')
|
||||
# Push the file to the device
|
||||
self.device.push_file(host_infile, self.device_infile)
|
||||
|
||||
def run(self, context):
|
||||
self.device.execute('cd {} && (time gzip {}) &>> {}'.format(self.device.working_directory,
|
||||
self.device_infile,
|
||||
self.device_outfile))
|
||||
|
||||
def update_result(self, context):
|
||||
# Pull the results file to the host
|
||||
host_outfile = os.path.join(context.output_directory, 'outfile')
|
||||
self.device.pull_file(self.device_outfile, host_outfile)
|
||||
# Extract metrics form the file's contents and update the result
|
||||
# with them.
|
||||
content = iter(open(host_outfile).read().strip().split())
|
||||
for value, metric in zip(content, content):
|
||||
mins, secs = map(float, value[:-1].split('m'))
|
||||
context.result.add_metric(metric, secs + 60 * mins)
|
||||
|
||||
def teardown(self, context):
|
||||
# Clean up on-device file.
|
||||
self.device.delete_file(self.device_infile)
|
||||
self.device.delete_file(self.device_outfile)
|
||||
|
||||
|
||||
|
||||
.. _GameWorkload:
|
||||
|
||||
Adding revent-dependent Workload:
|
||||
---------------------------------
|
||||
|
||||
:class:`wlauto.common.game.GameWorkload` is the base class for all the workloads
|
||||
that depend on :ref:`revent_files_creation` files. It implements all the methods
|
||||
needed to push the files to the device and run them. New GameWorkload can be
|
||||
added by subclassing :class:`wlauto.common.game.GameWorkload`:
|
||||
|
||||
The GameWorkload class defines the following interface::
|
||||
|
||||
class GameWorkload(Workload):
|
||||
|
||||
name = None
|
||||
package = None
|
||||
activity = None
|
||||
|
||||
The interface should be implemented as follows
|
||||
|
||||
:name: This identifies the workload (e.g. it used to specify it in the
|
||||
agenda_.
|
||||
:package: This is the name of the '.apk' package without its file extension.
|
||||
:activity: The name of the main activity that runs the package.
|
||||
|
||||
Example:
|
||||
--------
|
||||
|
||||
This example shows a simple GameWorkload that plays a game.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from wlauto.common.game import GameWorkload
|
||||
|
||||
class MyGame(GameWorkload):
|
||||
|
||||
name = 'mygame'
|
||||
package = 'com.mylogo.mygame'
|
||||
activity = 'myActivity.myGame'
|
||||
|
||||
Convention for Naming revent Files for :class:`wlauto.common.game.GameWorkload`
|
||||
-------------------------------------------------------------------------------
|
||||
|
||||
There is a convention for naming revent files which you should follow if you
|
||||
want to record your own revent files. Each revent file must start with the
|
||||
device name(case sensitive) then followed by a dot '.' then the stage name
|
||||
then '.revent'. All your custom revent files should reside at
|
||||
'~/.workload_automation/dependencies/WORKLOAD NAME/'. These are the current
|
||||
supported stages:
|
||||
|
||||
:setup: This stage is where the game is loaded. It is a good place to
|
||||
record revent here to modify the game settings and get it ready
|
||||
to start.
|
||||
:run: This stage is where the game actually starts. This will allow for
|
||||
more accurate results if the revent file for this stage only
|
||||
records the game being played.
|
||||
|
||||
For instance, to add a custom revent files for a device named mydevice and
|
||||
a workload name mygame, you create a new directory called mygame in
|
||||
'~/.workload_automation/dependencies/'. Then you add the revent files for
|
||||
the stages you want in ~/.workload_automation/dependencies/mygame/::
|
||||
|
||||
mydevice.setup.revent
|
||||
mydevice.run.revent
|
||||
|
||||
Any revent file in the dependencies will always overwrite the revent file in the
|
||||
workload directory. So it is possible for example to just provide one revent for
|
||||
setup in the dependencies and use the run.revent that is in the workload directory.
|
||||
|
||||
Adding an Instrument
|
||||
====================
|
||||
|
||||
Instruments can be used to collect additional measurements during workload
|
||||
execution (e.g. collect power readings). An instrument can hook into almost any
|
||||
stage of workload execution. A typical instrument would implement a subset of
|
||||
the following interface::
|
||||
|
||||
class Instrument(Extension):
|
||||
|
||||
name = None
|
||||
description = None
|
||||
|
||||
parameters = [
|
||||
]
|
||||
|
||||
def initialize(self, context):
|
||||
pass
|
||||
|
||||
def setup(self, context):
|
||||
pass
|
||||
|
||||
def start(self, context):
|
||||
pass
|
||||
|
||||
def stop(self, context):
|
||||
pass
|
||||
|
||||
def update_result(self, context):
|
||||
pass
|
||||
|
||||
def teardown(self, context):
|
||||
pass
|
||||
|
||||
def finalize(self, context):
|
||||
pass
|
||||
|
||||
This is similar to a Workload, except all methods are optional. In addition to
|
||||
the workload-like methods, instruments can define a number of other methods that
|
||||
will get invoked at various points during run execution. The most useful of
|
||||
which is perhaps ``initialize`` that gets invoked after the device has been
|
||||
initialised for the first time, and can be used to perform one-time setup (e.g.
|
||||
copying files to the device -- there is no point in doing that for each
|
||||
iteration). The full list of available methods can be found in
|
||||
:ref:`Signals Documentation <instrument_name_mapping>`.
|
||||
|
||||
|
||||
Prioritization
|
||||
--------------
|
||||
|
||||
Callbacks (e.g. ``setup()`` methods) for all instrumentation get executed at the
|
||||
same point during workload execution, one after another. The order in which the
|
||||
callbacks get invoked should be considered arbitrary and should not be relied
|
||||
on (e.g. you cannot expect that just because instrument A is listed before
|
||||
instrument B in the config, instrument A's callbacks will run first).
|
||||
|
||||
In some cases (e.g. in ``start()`` and ``stop()`` methods), it is important to
|
||||
ensure that a particular instrument's callbacks run a closely as possible to the
|
||||
workload's invocations in order to maintain accuracy of readings; or,
|
||||
conversely, that a callback is executed after the others, because it takes a
|
||||
long time and may throw off the accuracy of other instrumentation. You can do
|
||||
this by prepending ``fast_`` or ``slow_`` to your callbacks' names. For
|
||||
example::
|
||||
|
||||
class PreciseInstrument(Instument):
|
||||
|
||||
# ...
|
||||
|
||||
def fast_start(self, context):
|
||||
pass
|
||||
|
||||
def fast_stop(self, context):
|
||||
pass
|
||||
|
||||
# ...
|
||||
|
||||
``PreciseInstrument`` will be started after all other instrumentation (i.e.
|
||||
*just* before the workload runs), and it will stopped before all other
|
||||
instrumentation (i.e. *just* after the workload runs). It is also possible to
|
||||
use ``very_fast_`` and ``very_slow_`` prefixes when you want to be really
|
||||
sure that your callback will be the last/first to run.
|
||||
|
||||
If more than one active instrument have specified fast (or slow) callbacks, then
|
||||
their execution order with respect to each other is not guaranteed. In general,
|
||||
having a lot of instrumentation enabled is going to necessarily affect the
|
||||
readings. The best way to ensure accuracy of measurements is to minimize the
|
||||
number of active instruments (perhaps doing several identical runs with
|
||||
different instruments enabled).
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
Below is a simple instrument that measures the execution time of a workload::
|
||||
|
||||
class ExecutionTimeInstrument(Instrument):
|
||||
"""
|
||||
Measure how long it took to execute the run() methods of a Workload.
|
||||
|
||||
"""
|
||||
|
||||
name = 'execution_time'
|
||||
|
||||
def initialize(self, context):
|
||||
self.start_time = None
|
||||
self.end_time = None
|
||||
|
||||
def fast_start(self, context):
|
||||
self.start_time = time.time()
|
||||
|
||||
def fast_stop(self, context):
|
||||
self.end_time = time.time()
|
||||
|
||||
def update_result(self, context):
|
||||
execution_time = self.end_time - self.start_time
|
||||
context.result.add_metric('execution_time', execution_time, 'seconds')
|
||||
|
||||
|
||||
Adding a Result Processor
|
||||
=========================
|
||||
|
||||
A result processor is responsible for processing the results. This may
|
||||
involve formatting and writing them to a file, uploading them to a database,
|
||||
generating plots, etc. WA comes with a few result processors that output
|
||||
results in a few common formats (such as csv or JSON).
|
||||
|
||||
You can add your own result processors by creating a Python file in
|
||||
``~/.workload_automation/result_processors`` with a class that derives from
|
||||
:class:`wlauto.core.result.ResultProcessor`, which has the following interface::
|
||||
|
||||
class ResultProcessor(Extension):
|
||||
|
||||
name = None
|
||||
description = None
|
||||
|
||||
parameters = [
|
||||
]
|
||||
|
||||
def initialize(self, context):
|
||||
pass
|
||||
|
||||
def process_iteration_result(self, result, context):
|
||||
pass
|
||||
|
||||
def export_iteration_result(self, result, context):
|
||||
pass
|
||||
|
||||
def process_run_result(self, result, context):
|
||||
pass
|
||||
|
||||
def export_run_result(self, result, context):
|
||||
pass
|
||||
|
||||
def finalize(self, context):
|
||||
pass
|
||||
|
||||
|
||||
The method names should be fairly self-explanatory. The difference between
|
||||
"process" and "export" methods is that export methods will be invoke after
|
||||
process methods for all result processors have been generated. Process methods
|
||||
may generated additional artifacts (metrics, files, etc), while export methods
|
||||
should not -- the should only handle existing results (upload them to a
|
||||
database, archive on a filer, etc).
|
||||
|
||||
The result object passed to iteration methods is an instance of
|
||||
:class:`wlauto.core.result.IterationResult`, the result object passed to run
|
||||
methods is an instance of :class:`wlauto.core.result.RunResult`. Please refer to
|
||||
their API documentation for details.
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
Here is an example result processor that formats the results as a column-aligned
|
||||
table::
|
||||
|
||||
import os
|
||||
from wlauto import ResultProcessor
|
||||
from wlauto.utils.misc import write_table
|
||||
|
||||
|
||||
class Table(ResultProcessor):
|
||||
|
||||
name = 'table'
|
||||
description = 'Gerates a text file containing a column-aligned table with run results.'
|
||||
|
||||
def process_run_result(self, result, context):
|
||||
rows = []
|
||||
for iteration_result in result.iteration_results:
|
||||
for metric in iteration_result.metrics:
|
||||
rows.append([metric.name, str(metric.value), metric.units or '',
|
||||
metric.lower_is_better and '-' or '+'])
|
||||
|
||||
outfile = os.path.join(context.output_directory, 'table.txt')
|
||||
with open(outfile, 'w') as wfh:
|
||||
write_table(rows, wfh)
|
||||
|
||||
|
||||
Adding a Resource Getter
|
||||
========================
|
||||
|
||||
A resource getter is a new extension type added in version 2.1.3. A resource
|
||||
getter implement a method of acquiring resources of a particular type (such as
|
||||
APK files or additional workload assets). Resource getters are invoked in
|
||||
priority order until one returns the desired resource.
|
||||
|
||||
If you want WA to look for resources somewhere it doesn't by default (e.g. you
|
||||
have a repository of APK files), you can implement a getter for the resource and
|
||||
register it with a higher priority than the standard WA getters, so that it gets
|
||||
invoked first.
|
||||
|
||||
Instances of a resource getter should implement the following interface::
|
||||
|
||||
class ResourceGetter(Extension):
|
||||
|
||||
name = None
|
||||
resource_type = None
|
||||
priority = GetterPriority.environment
|
||||
|
||||
def get(self, resource, **kwargs):
|
||||
raise NotImplementedError()
|
||||
|
||||
The getter should define a name (as with all extensions), a resource
|
||||
type, which should be a string, e.g. ``'jar'``, and a priority (see `Getter
|
||||
Prioritization`_ below). In addition, ``get`` method should be implemented. The
|
||||
first argument is an instance of :class:`wlauto.core.resource.Resource`
|
||||
representing the resource that should be obtained. Additional keyword
|
||||
arguments may be used by the invoker to provide additional information about
|
||||
the resource. This method should return an instance of the resource that
|
||||
has been discovered (what "instance" means depends on the resource, e.g. it
|
||||
could be a file path), or ``None`` if this getter was unable to discover
|
||||
that resource.
|
||||
|
||||
Getter Prioritization
|
||||
---------------------
|
||||
|
||||
A priority is an integer with higher numeric values indicating a higher
|
||||
priority. The following standard priority aliases are defined for getters:
|
||||
|
||||
|
||||
:cached: The cached version of the resource. Look here first. This priority also implies
|
||||
that the resource at this location is a "cache" and is not the only version of the
|
||||
resource, so it may be cleared without losing access to the resource.
|
||||
:preferred: Take this resource in favour of the environment resource.
|
||||
:environment: Found somewhere under ~/.workload_automation/ or equivalent, or
|
||||
from environment variables, external configuration files, etc.
|
||||
These will override resource supplied with the package.
|
||||
:package: Resource provided with the package.
|
||||
:remote: Resource will be downloaded from a remote location (such as an HTTP server
|
||||
or a samba share). Try this only if no other getter was successful.
|
||||
|
||||
These priorities are defined as class members of
|
||||
:class:`wlauto.core.resource.GetterPriority`, e.g. ``GetterPriority.cached``.
|
||||
|
||||
Most getters in WA will be registered with either ``environment`` or
|
||||
``package`` priorities. So if you want your getter to override the default, it
|
||||
should typically be registered as ``preferred``.
|
||||
|
||||
You don't have to stick to standard priority levels (though you should, unless
|
||||
there is a good reason). Any integer is a valid priority. The standard priorities
|
||||
range from -20 to 20 in increments of 10.
|
||||
|
||||
Example
|
||||
-------
|
||||
|
||||
The following is an implementation of a getter for a workload APK file that
|
||||
looks for the file under
|
||||
``~/.workload_automation/dependencies/<workload_name>``::
|
||||
|
||||
import os
|
||||
import glob
|
||||
|
||||
from wlauto import ResourceGetter, GetterPriority, settings
|
||||
from wlauto.exceptions import ResourceError
|
||||
|
||||
|
||||
class EnvironmentApkGetter(ResourceGetter):
|
||||
|
||||
name = 'environment_apk'
|
||||
resource_type = 'apk'
|
||||
priority = GetterPriority.environment
|
||||
|
||||
def get(self, resource):
|
||||
resource_dir = _d(os.path.join(settings.dependency_directory, resource.owner.name))
|
||||
version = kwargs.get('version')
|
||||
found_files = glob.glob(os.path.join(resource_dir, '*.apk'))
|
||||
if version:
|
||||
found_files = [ff for ff in found_files if version.lower() in ff.lower()]
|
||||
if len(found_files) == 1:
|
||||
return found_files[0]
|
||||
elif not found_files:
|
||||
return None
|
||||
else:
|
||||
raise ResourceError('More than one .apk found in {} for {}.'.format(resource_dir,
|
||||
resource.owner.name))
|
||||
|
||||
.. _adding_a_device:
|
||||
|
||||
Adding a Device
|
||||
===============
|
||||
|
||||
At the moment, only Android devices are supported. Most of the functionality for
|
||||
interacting with a device is implemented in
|
||||
:class:`wlauto.common.AndroidDevice` and is exposed through ``generic_android``
|
||||
device interface, which should suffice for most purposes. The most common area
|
||||
where custom functionality may need to be implemented is during device
|
||||
initialization. Usually, once the device gets to the Android home screen, it's
|
||||
just like any other Android device (modulo things like differences between
|
||||
Android versions).
|
||||
|
||||
If your device doesn't not work with ``generic_device`` interface and you need
|
||||
to write a custom interface to handle it, you would do that by subclassing
|
||||
``AndroidDevice`` and then just overriding the methods you need. Typically you
|
||||
will want to override one or more of the following:
|
||||
|
||||
reset
|
||||
Trigger a device reboot. The default implementation just sends ``adb
|
||||
reboot`` to the device. If this command does not work, an alternative
|
||||
implementation may need to be provided.
|
||||
|
||||
hard_reset
|
||||
This is a harsher reset that involves cutting the power to a device
|
||||
(e.g. holding down power button or removing battery from a phone). The
|
||||
default implementation is a no-op that just sets some internal flags. If
|
||||
you're dealing with unreliable prototype hardware that can crash and
|
||||
become unresponsive, you may want to implement this in order for WA to
|
||||
be able to recover automatically.
|
||||
|
||||
connect
|
||||
When this method returns, adb connection to the device has been
|
||||
established. This gets invoked after a reset. The default implementation
|
||||
just waits for the device to appear in the adb list of connected
|
||||
devices. If this is not enough (e.g. your device is connected via
|
||||
Ethernet and requires an explicit ``adb connect`` call), you may wish to
|
||||
override this to perform the necessary actions before invoking the
|
||||
``AndroidDevice``\ s version.
|
||||
|
||||
init
|
||||
This gets called once at the beginning of the run once the connection to
|
||||
the device has been established. There is no default implementation.
|
||||
It's there to allow whatever custom initialisation may need to be
|
||||
performed for the device (setting properties, configuring services,
|
||||
etc).
|
||||
|
||||
Please refer to the API documentation for :class:`wlauto.common.AndroidDevice`
|
||||
for the full list of its methods and their functionality.
|
||||
|
||||
|
||||
Other Extension Types
|
||||
=====================
|
||||
|
||||
In addition to extension types covered above, there are few other, more
|
||||
specialized ones. They will not be covered in as much detail. Most of them
|
||||
expose relatively simple interfaces with only a couple of methods and it is
|
||||
expected that if the need arises to extend them, the API-level documentation
|
||||
that accompanies them, in addition to what has been outlined here, should
|
||||
provide enough guidance.
|
||||
|
||||
:commands: This allows extending WA with additional sub-commands (to supplement
|
||||
exiting ones outlined in the :ref:`invocation` section).
|
||||
:modules: Modules are "extensions for extensions". They can be loaded by other
|
||||
extensions to expand their functionality (for example, a flashing
|
||||
module maybe loaded by a device in order to support flashing).
|
||||
|
||||
|
||||
Packaging Your Extensions
|
||||
=========================
|
||||
|
||||
If your have written a bunch of extensions, and you want to make it easy to
|
||||
deploy them to new systems and/or to update them on existing systems, you can
|
||||
wrap them in a Python package. You can use ``wa create package`` command to
|
||||
generate appropriate boiler plate. This will create a ``setup.py`` and a
|
||||
directory for your package that you can place your extensions into.
|
||||
|
||||
For example, if you have a workload inside ``my_workload.py`` and a result
|
||||
processor in ``my_result_processor.py``, and you want to package them as
|
||||
``my_wa_exts`` package, first run the create command ::
|
||||
|
||||
wa create package my_wa_exts
|
||||
|
||||
This will create a ``my_wa_exts`` directory which contains a
|
||||
``my_wa_exts/setup.py`` and a subdirectory ``my_wa_exts/my_wa_exts`` which is
|
||||
the package directory for your extensions (you can rename the top-level
|
||||
``my_wa_exts`` directory to anything you like -- it's just a "container" for the
|
||||
setup.py and the package directory). Once you have that, you can then copy your
|
||||
extensions into the package directory, creating
|
||||
``my_wa_exts/my_wa_exts/my_workload.py`` and
|
||||
``my_wa_exts/my_wa_exts/my_result_processor.py``. If you have a lot of
|
||||
extensions, you might want to organize them into subpackages, but only the
|
||||
top-level package directory is created by default, and it is OK to have
|
||||
everything in there.
|
||||
|
||||
.. note:: When discovering extensions thorugh this mechanism, WA traveries the
|
||||
Python module/submodule tree, not the directory strucuter, therefore,
|
||||
if you are going to create subdirectories under the top level dictory
|
||||
created for you, it is important that your make sure they are valid
|
||||
Python packages; i.e. each subdirectory must contain a __init__.py
|
||||
(even if blank) in order for the code in that directory and its
|
||||
subdirectories to be discoverable.
|
||||
|
||||
At this stage, you may want to edit ``params`` structure near the bottom of
|
||||
the ``setup.py`` to add correct author, license and contact information (see
|
||||
"Writing the Setup Script" section in standard Python documentation for
|
||||
details). You may also want to add a README and/or a COPYING file at the same
|
||||
level as the setup.py. Once you have the contents of your package sorted,
|
||||
you can generate the package by running ::
|
||||
|
||||
cd my_wa_exts
|
||||
python setup.py sdist
|
||||
|
||||
This will generate ``my_wa_exts/dist/my_wa_exts-0.0.1.tar.gz`` package which
|
||||
can then be deployed on the target system with standard Python package
|
||||
management tools, e.g. ::
|
||||
|
||||
sudo pip install my_wa_exts-0.0.1.tar.gz
|
||||
|
||||
As part of the installation process, the setup.py in the package, will write the
|
||||
package's name into ``~/.workoad_automoation/packages``. This will tell WA that
|
||||
the package contains extension and it will load them next time it runs.
|
||||
|
||||
.. note:: There are no unistall hooks in ``setuputils``, so if you ever
|
||||
uninstall your WA extensions package, you will have to manually remove
|
||||
it from ``~/.workload_automation/packages`` otherwise WA will complain
|
||||
abou a missing package next time you try to run it.
|
||||
Reference in New Issue
Block a user