workload-automation/doc/source/developer_information/developer_reference/plugins.rst

.. plugins:


Plugins
=======

Workload Automation offers several plugin points (or plugin types). The most
interesting of these are

:workloads: These are the tasks that get executed and measured on the device. These
            can be benchmarks, high-level use cases, or pretty much anything else.
:targets: These are interfaces to the physical devices (development boards or end-user
          devices, such as smartphones) that use cases run on. Typically each model of a
          physical device would require its own interface class (though some functionality
          may be reused by subclassing from an existing base).
:instruments: Instruments allow collecting additional data from workload execution (e.g.
              system traces). Instruments are not specific to a particular workload. Instruments
              can hook into any stage of workload execution.
:output processors: These are used to format the results of workload execution once they have been
                    collected. Depending on the callback used, these will run either after each
                    iteration and/or at the end of the run, after all of the results have been
                    collected.

You can create a plugin by subclassing the appropriate base class, defining
appropriate methods and attributes, and putting the .py file containing the
class into the "plugins" subdirectory under ``~/.workload_automation`` (or
equivalent) where it will be automatically picked up by WA.


Plugin Basics
--------------

This section contains reference information common to plugins of all types.

.. _metrics:

Metrics
^^^^^^^
This is what WA uses to store a single metric collected from executing a workload.

    :name: the name of the metric. Uniquely identifies the metric
                 within the results.
    :value: The numerical value of the metric for this execution of a
                  workload. This can be either an int or a float.
    :units: Units for the collected value. Can be None if the value
                  has no units (e.g. it's a count or a standardised score).
    :lower_is_better: Boolean flag indicating where lower values are
                            better than higher ones. Defaults to False.
    :classifiers: A set of key-value pairs to further classify this
                        metric beyond current iteration (e.g. this can be used
                        to identify sub-tests).

Metrics can be added to WA output via the :ref:`context <context>`:


.. code-block:: python

  context.add_metric("score", 9001)
  context.add_metric("time", 2.35, "seconds", lower_is_better=True)

You only need to specify the name and the value for the metric. Units and
classifiers are optional, and, if not specified otherwise, it will be assumed
that higher values are better (``lower_is_better=False``).

The metric will be added to the result for the current job, if there is one;
otherwise, it will be added to the overall run result.

.. _artifact:

Artifacts
^^^^^^^^^
This is an artifact generated during execution/post-processing of a workload.
Unlike :ref:`metrics <metrics>`, this represents an actual artifact, such as a
file, generated.  This may be "output", such as trace, or it could be "meta
data" such as logs.  These are distinguished using the ``kind`` attribute, which
also helps WA decide how it should be handled. Currently supported kinds are:

        :log: A log file. Not part of the "output" as such but contains
              information about the run/workload execution that be useful for
              diagnostics/meta analysis.
        :meta: A file containing metadata. This is not part of the "output", but
               contains information that may be necessary to reproduce the
               results (contrast with ``log`` artifacts which are *not*
               necessary).
        :data: This file contains new data, not available otherwise and should
               be considered part of the "output" generated by WA. Most traces
               would fall into this category.
        :export: Exported version of results or some other artifact. This
                 signifies that this artifact does not contain any new data
                 that is not available elsewhere and that it may be safely
                 discarded without losing information.
        :raw: Signifies that this is a raw dump/log that is normally processed
              to extract useful information and is then discarded. In a sense,
              it is the opposite of ``export``, but in general may also be
              discarded.

              .. note:: whether a file is marked as ``log``/``data`` or ``raw``
                        depends on how important it is to preserve this file,
                        e.g. when archiving, vs how much space it takes up.
                        Unlike ``export`` artifacts which are (almost) always
                        ignored by other exporters as that would never result
                        in data loss, ``raw`` files *may* be processed by
                        exporters if they decided that the risk of losing
                        potentially (though unlikely) useful data is greater
                        than the time/space cost of handling the artifact (e.g.
                        a database uploader may choose to ignore ``raw``
                        artifacts, whereas a network filer archiver may choose
                        to archive them).

        .. note: The kind parameter is intended to represent the logical
                 function of a particular artifact, not it's intended means of
                 processing -- this is left entirely up to the output
                 processors.

As with :ref:`metrics`, artifacts are added via the :ref:`context <context>`:

.. code-block:: python

  context.add_artifact("benchmark-output", "bech-out.txt", kind="raw",
                       description="stdout from running the benchmark")

.. note:: The file *must* exist on the host by the point at which the artifact
          is added, otherwise an error will be raised.

The artifact will be added to the result of the current job, if there is one;
otherwise, it will be added to the overall run result. In some situations, you
may wish to add an artifact to the overall run while being inside a job context,
this can be done with ``add_run_artifact``:

.. code-block:: python

  context.add_run_artifact("score-summary", "scores.txt", kind="export",
         description="""
         Summary of the scores so far. Updated after
         every job.
         """)

In this case, you also need to make sure that the file represented by the
artifact is written to the output directory for the run and not the current job.

.. _metadata:

Metadata
^^^^^^^^

There may be additional data collected by your plugin that you want to record as
part of the result, but that does not fall under the definition of a "metric".
For example, you may want to record the version of the binary you're executing.
You can do this by adding a metadata entry:

.. code-block:: python

  context.add_metadata("exe-version", 1.3)


Metadata will be added either to the current job result, or to the run result,
depending on the current context. Metadata values can be scalars or nested
structures of dicts/sequences; the only constraint is that all constituent
objects of the value must be POD (Plain Old Data) types -- see :ref:`WA POD
types <wa-pods>`.

There is special support for handling metadata entries that are dicts of values.
The following call adds a metadata entry ``"versions"`` who's value is
``{"my_exe": 1.3}``:

.. code-block:: python

  context.add_metadata("versions", "my_exe", 1.3)

If you attempt to add a metadata entry that already exists, an error will be
raised, unless ``force=True`` is specified, in which case, it will be
overwritten.

Updating an existing entry whose value is a collection can be done with
``update_metadata``:

.. code-block:: python

  context.update_metadata("ran_apps", "my_exe")
  context.update_metadata("versions", "my_other_exe", "2.3.0")

The first call appends ``"my_exe"`` to the list at metadata entry
``"ran_apps"``. The second call updates the ``"versions"`` dict in the metadata
with an entry for ``"my_other_exe"``.

If an entry does not exit, ``update_metadata`` will create it, so it's
recommended to always use that for non-scalar entries, unless the intention is
specifically to ensure that the entry does not exist at the time of the call.

.. _classifiers:

Classifiers
^^^^^^^^^^^

Classifiers are key-value pairs of tags that can be attached to metrics,
artifacts, jobs, or the entire run. Run and job classifiers get propagated to
metrics and artifacts. Classifier keys should be strings, and their values
should be simple scalars (i.e. strings, numbers, or bools).

Classifiers can be thought of as "tags" that are used to annotate metrics and
artifacts, in order to make it easier to sort through them later. WA itself does
not do anything with them, however output processors will augment the output
they generate with them (for example, ``csv`` processor can add additional
columns for classifier keys).

Classifiers are typically added by the user to attach some domain-specific
information (e.g. experiment configuration identifier) to the results, see
:ref:`using classifiers <using-classifiers>`. However, plugins can also attach
additional classifiers, by specifying them in ``add_metric()`` and
``add_artifacts()`` calls.


Metadata vs Classifiers
^^^^^^^^^^^^^^^^^^^^^^^

Both metadata and classifiers are sets of essentially opaque key-value pairs
that get included in WA output. While they may seem somewhat similar and
interchangeable, they serve different purposes and are handled differently by
the framework.

Classifiers are used to annotate generated metrics and artifacts in order to
assist post-processing tools in sorting through them. Metadata is used to record
additional information that is not necessary for processing the results, but
that may be needed in order to reproduce them or to make sense of them in a
grander context.

These are specific differences in how they are handled:

- Classifiers are often provided by the user via the agenda (though can also be
  added by plugins). Metadata in only created by the framework and plugins.
- Classifier values must be simple scalars; metadata values can be nested
  collections, such as lists or dicts.
- Classifiers are used by output processors to augment the output the latter
  generated; metadata typically isn't.
- Classifiers are essentially associated with the individual metrics and
  artifacts (though in the agenda they're specified at workload, section, or
  global run levels); metadata is associated with a particular job or run, and
  not with metrics or artifacts.

--------------------

.. _execution-decorators:

Execution Decorators
---------------------

The following decorators are available for use in order to control how often a
method should be able to be executed.

For example, if we want to ensure that no matter how many iterations of a
particular workload are ran, we only execute the initialize method for that instance
once, we would use the decorator as follows:

.. code-block:: python

    from wa.utils.exec_control import once

    @once
    def initialize(self, context):
        # Perform one time initialization e.g. installing a binary to target
        # ..

@once_per_instance
^^^^^^^^^^^^^^^^^^
The specified method will be invoked only once for every bound instance within
the environment.

@once_per_class
^^^^^^^^^^^^^^^
The specified method will be invoked only once for all instances of a class
within the environment.

@once
^^^^^
The specified method will be invoked only once within the environment.

.. warning:: If a method containing a super call is decorated, this will also cause
             stop propagation up the hierarchy, unless this is the desired
             effect, additional functionality should be implemented in a
             separate decorated method which can then be called allowing for
             normal propagation to be retained.


--------------------

Utils
-----

Workload Automation defines a number of utilities collected under
:mod:`wa.utils` subpackage. These utilities were created to help with the
implementation of the framework itself, but may be also be useful when
implementing plugins.

--------------------

Workloads
---------

All of the type inherit from the same base :class:`Workload` and its API can be
seen in the :ref:`API <workload-api>` section.

Workload methods (except for ``validate``) take a single argument that is a
:class:`wa.framework.execution.ExecutionContext` instance. This object keeps
track of the current execution state (such as the current workload, iteration
number, etc), and contains, among other things, a
:class:`wa.framework.output.JobOutput` instance that should be populated from
the ``update_output`` method with the results of the execution. For more
information please see `the context`_ documentation. ::

        # ...

        def update_output(self, context):
           # ...
           context.add_metric('energy', 23.6, 'Joules', lower_is_better=True)

        # ...

.. _workload-types:

Workload Types
^^^^^^^^^^^^^^^^

There are multiple workload types that you can inherit from depending on the
purpose of your workload, the different types along with an output of their
intended use cases are outlined below.

.. _basic-workload:

Basic (:class:`wa.Workload <wa.framework.workload.Workload>`)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This type of the workload is the simplest type of workload and is left the to
developer to implement its full functionality.


.. _apk-workload:

Apk (:class:`wa.ApkWorkload <wa.framework.workload.ApkWorkload>`)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This workload will simply deploy and launch an android app in its basic form
with no UI interaction.

.. _uiautomator-workload:


UiAuto (:class:`wa.UiautoWorkload <wa.framework.workload.UiautoWorkload>`)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This workload is for android targets which will use UiAutomator to interact with
UI elements without a specific android app, for example performing manipulation
of android itself. This is the preferred type of automation as the results are
more portable and reproducible due to being able to wait for UI elements to
appear rather than having to rely on human recordings.

.. _apkuiautomator-workload:

ApkUiAuto (:class:`wa.ApkUiautoWorkload <wa.framework.workload.ApkUiautoWorkload>`)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The is the same as the UiAuto workload however it is also associated with an
android app e.g. AdobeReader and will automatically deploy and launch the
android app before running the automation.

.. _revent-workload:

Revent (:class:`wa.ReventWorkload <wa.framework.workload.ReventWorkload>`)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Revent workloads are designed primarily for games as these are unable to be
automated with UiAutomator due to the fact that they are rendered within a
single UI element. They require a recording to be performed manually and
currently will need re-recording for each different device. For more
information on revent workloads been please see :ref:`revent_files_creation`

.. _apkrevent-workload:

APKRevent (:class:`wa.ApkReventWorkload <wa.framework.workload.ApkReventWorkload>`)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The is the same as the Revent workload however it is also associated with an
android app e.g. AngryBirds and will automatically deploy and launch the android
app before running the automation.