doc: Restructure developer information

Re-organise and add additional information to developer guide.

doc/source/developer_information/developer_reference/plugins.rst

@@ -0,0 +1,376 @@
+.. plugins:
+
+
+Plugins
+=======
+
+Workload Automation offers several plugin points (or plugin types). The most
+interesting of these are
+
+:workloads: These are the tasks that get executed and measured on the device. These
+            can be benchmarks, high-level use cases, or pretty much anything else.
+:targets: These are interfaces to the physical devices (development boards or end-user
+          devices, such as smartphones) that use cases run on. Typically each model of a
+          physical device would require its own interface class (though some functionality
+          may be reused by subclassing from an existing base).
+:instruments: Instruments allow collecting additional data from workload execution (e.g.
+              system traces). Instruments are not specific to a particular workload. Instruments
+              can hook into any stage of workload execution.
+:output processors: These are used to format the results of workload execution once they have been
+                    collected. Depending on the callback used, these will run either after each
+                    iteration and/or at the end of the run, after all of the results have been
+                    collected.
+
+You can create a plugin by subclassing the appropriate base class, defining
+appropriate methods and attributes, and putting the .py file containing the
+class into the "plugins" subdirectory under ``~/.workload_automation`` (or
+equivalent) where it will be automatically picked up by WA.
+
+
+Plugin Basics
+--------------
+
+This section contains reference information common to plugins of all types.
+
+.. _metrics:
+
+Metrics
+^^^^^^^
+This is what WA uses to store a single metric collected from executing a workload.
+
+    :name: the name of the metric. Uniquely identifies the metric
+                 within the results.
+    :value: The numerical value of the metric for this execution of a
+                  workload. This can be either an int or a float.
+    :units: Units for the collected value. Can be None if the value
+                  has no units (e.g. it's a count or a standardised score).
+    :lower_is_better: Boolean flag indicating where lower values are
+                            better than higher ones. Defaults to False.
+    :classifiers: A set of key-value pairs to further classify this
+                        metric beyond current iteration (e.g. this can be used
+                        to identify sub-tests).
+
+Metrics can be added to WA output via the context:
+
+
+.. code-block:: python
+
+  context.add_metric("score", 9001)
+  context.add_metric("time", 2.35, "seconds", lower_is_better=True)
+
+You only need to specify the name and the value for the metric. Units and
+classifiers are optional, and, if not specified otherwise, it will be assumed
+that higher values are better (lower_is_better=False).
+
+The metric will be added to the result for the current job, if there is one;
+otherwise, it will be added to the overall run result.
+
+.. _artifact:
+
+Artifacts
+^^^^^^^^^
+This is an artifact generated during execution/post-processing of a workload.
+Unlike :ref:`metrics <metrics>`, this represents an actual artifact, such as a
+file, generated.  This may be "output", such as trace, or it could be "meta
+data" such as logs.  These are distinguished using the ``kind`` attribute, which
+also helps WA decide how it should be handled. Currently supported kinds are:
+
+        :log: A log file. Not part of the "output" as such but contains
+              information about the run/workload execution that be useful for
+              diagnostics/meta analysis.
+        :meta: A file containing metadata. This is not part of the "output", but
+               contains information that may be necessary to reproduce the
+               results (contrast with ``log`` artifacts which are *not*
+               necessary).
+        :data: This file contains new data, not available otherwise and should
+               be considered part of the "output" generated by WA. Most traces
+               would fall into this category.
+        :export: Exported version of results or some other artifact. This
+                 signifies that this artifact does not contain any new data
+                 that is not available elsewhere and that it may be safely
+                 discarded without losing information.
+        :raw: Signifies that this is a raw dump/log that is normally processed
+              to extract useful information and is then discarded. In a sense,
+              it is the opposite of ``export``, but in general may also be
+              discarded.
+
+              .. note:: whether a file is marked as ``log``/``data`` or ``raw``
+                        depends on how important it is to preserve this file,
+                        e.g. when archiving, vs how much space it takes up.
+                        Unlike ``export`` artifacts which are (almost) always
+                        ignored by other exporters as that would never result
+                        in data loss, ``raw`` files *may* be processed by
+                        exporters if they decided that the risk of losing
+                        potentially (though unlikely) useful data is greater
+                        than the time/space cost of handling the artifact (e.g.
+                        a database uploader may choose to ignore ``raw``
+                        artifacts, whereas a network filer archiver may choose
+                        to archive them).
+
+        .. note: The kind parameter is intended to represent the logical
+                 function of a particular artifact, not it's intended means of
+                 processing -- this is left entirely up to the output
+                 processors.
+
+As with :ref:`metrics`, artifacts are added via the context:
+
+.. code-block:: python
+
+  context.add_artifact("benchmark-output", "bech-out.txt", kind="raw",
+                       description="stdout from running the benchmark")
+
+.. note:: The file *must* exist on the host by the point at which the artifact
+          is added, otherwise an error will be raised.
+
+The artifact will be added to the result of the current job, if there is one;
+otherwise, it will be added to the overall run result. In some situations, you
+may wish to add an artifact to the overall run while being inside a job context,
+this can be done with ``add_run_artifact``:
+
+.. code-block:: python
+
+  context.add_run_artifact("score-summary", "scores.txt", kind="export",
+         description="""
+         Summary of the scores so far. Updated after
+         every job.
+         """)
+
+In this case, you also need to make sure that the file represented by the
+artifact is written to the output directory for the run and not the current job.
+
+.. _metadata:
+
+Metadata
+^^^^^^^^
+
+There may be additional data collected by your plugin that you want to record as
+part of the result, but that does not fall under the definition of a "metric".
+For example, you may want to record the version of the binary you're executing.
+You can do this by adding a metadata entry:
+
+.. code-block:: python
+
+  context.add_metadata("exe-version", 1.3)
+
+
+Metadata will be added either to the current job result, or to the run result,
+depending on the current context. Metadata values can be scalars or nested
+structures of dicts/sequences; the only constraint is that all constituent
+objects of the value must be POD (Plain Old Data) types -- see :ref:`WA POD
+types <wa-pods>`.
+
+There is special support for handling metadata entries that are dicts of values.
+The following call adds a metadata entry ``"versions"`` who's value is
+``{"my_exe": 1.3}``:
+
+.. code-block:: python
+
+  context.add_metadata("versions", "my_exe", 1.3)
+
+If you attempt to add a metadata entry that already exists, an error will be
+raised, unless ``force=True`` is specified, in which case, it will be
+overwritten.
+
+Updating an existing entry whose value is a collection can be done with
+``update_metadata``:
+
+.. code-block:: python
+
+  context.update_metadata("ran_apps", "my_exe")
+  context.update_metadata("versions", "my_other_exe", "2.3.0")
+
+The first call appends ``"my_exe"`` to the list at metadata entry
+``"ran_apps"``. The second call updates the ``"versions"`` dict in the metadata
+with an entry for ``"my_other_exe"``.
+
+If an entry does not exit, ``update_metadata`` will create it, so it's
+recommended to always use that for non-scalar entries, unless the intention is
+specifically to ensure that the entry does not exist at the time of the call.
+
+.. _classifiers:
+
+Classifiers
+^^^^^^^^^^^
+
+Classifiers are key-value pairs of tags that can be attached to metrics,
+artifacts, jobs, or the entire run. Run and job classifiers get propagated to
+metrics and artifacts. Classifier keys should be strings, and their values
+should be simple scalars (i.e. strings, numbers, or bools).
+
+Classifiers can be thought of as "tags" that are used to annotate metrics and
+artifacts, in order to make it easier to sort through them later. WA itself does
+not do anything with them, however output processors will augment the output
+they generate with them (for example, ``csv`` processor can add additional
+columns for classifier keys).
+
+Classifiers are typically added by the user to attach some domain-specific
+information (e.g. experiment configuration identifier) to the results, see
+:ref:`using classifiers <using-classifiers>`. However, plugins can also attach
+additional classifiers, by specifying them in ``add_metric()`` and
+``add_artifacts()`` calls.
+
+
+Metadata vs Classifiers
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Both metadata and classifiers are sets of essentially opaque key-value pairs
+that get included in WA output. While they may seem somewhat similar and
+interchangeable, they serve different purposes and are handled differently by
+the framework.
+
+Classifiers are used to annotate generated metrics and artifacts in order to
+assist post-processing tools in sorting through them. Metadata is used to record
+additional information that is not necessary for processing the results, but
+that may be needed in order to reproduce them or to make sense of them in a
+grander context.
+
+These are specific differences in how they are handled:
+
+- Classifiers are often provided by the user via the agenda (though can also be
+  added by plugins). Metadata in only created by the framework and plugins.
+- Classifier values must be simple scalars; metadata values can be nested
+  collections, such as lists or dicts.
+- Classifiers are used by output processors to augment the output the latter
+  generated; metadata typically isn't.
+- Classifiers are essentially associated with the individual metrics and
+  artifacts (though in the agenda they're specified at workload, section, or
+  global run levels); metadata is associated with a particular job or run, and
+  not with metrics or artifacts.
+
+
+.. _execution-decorators:
+
+Execution Decorators
+---------------------
+
+The following decorators are available for use in order to control how often a
+method should be able to be executed.
+
+For example, if we want to ensure that no matter how many iterations of a
+particular workload are ran, we only execute the initialize method for that instance
+once, we would use the decorator as follows:
+
+.. code-block:: python
+
+    from wa.utils.exec_control import once
+
+    @once
+    def initialize(self, context):
+        # Perform one time initialization e.g. installing a binary to target
+        # ..
+
+@once_per_instance
+^^^^^^^^^^^^^^^^^^
+The specified method will be invoked only once for every bound instance within
+the environment.
+
+@once_per_class
+^^^^^^^^^^^^^^^
+The specified method will be invoked only once for all instances of a class
+within the environment.
+
+@once
+^^^^^
+The specified method will be invoked only once within the environment.
+
+.. warning:: If a method containing a super call is decorated, this will also cause
+             stop propagation up the hierarchy, unless this is the desired
+             effect, additional functionality should be implemented in a
+             separate decorated method which can then be called allowing for
+             normal propagation to be retained.
+
+
+
+
+Utils
+^^^^^
+
+Workload Automation defines a number of utilities collected under
+:mod:`wa.utils` subpackage. These utilities were created to help with the
+implementation of the framework itself, but may be also be useful when
+implementing plugins.
+
+--------------------
+
+Workloads
+---------
+
+All of the type inherit from the same base :class:`Workload` and its API can be
+seen in the :ref:`API <workload-api>` section.
+
+Workload methods (except for ``validate``) take a single argument that is a
+:class:`wa.framework.execution.ExecutionContext` instance. This object keeps
+track of the current execution state (such as the current workload, iteration
+number, etc), and contains, among other things, a
+:class:`wa.framework.output.JobOutput` instance that should be populated from
+the ``update_output`` method with the results of the execution. For more
+information please see `the context`_ documentation. ::
+
+        # ...
+
+        def update_output(self, context):
+           # ...
+           context.add_metric('energy', 23.6, 'Joules', lower_is_better=True)
+
+        # ...
+
+.. _workload-types:
+
+Workload Types
+^^^^^^^^^^^^^^^^
+
+There are multiple workload types that you can inherit from depending on the
+purpose of your workload, the different types along with an output of their
+intended use cases are outlined below.
+
+.. _basic-workload:
+
+Basic (:class:`wa.Workload <wa.framework.workload.Workload>`)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This type of the workload is the simplest type of workload and is left the to
+developer to implement its full functionality.
+
+
+.. _apk-workload:
+
+Apk (:class:`wa.ApkWorkload <wa.framework.workload.ApkWorkload>`)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This workload will simply deploy and launch an android app in its basic form
+with no UI interaction.
+
+.. _uiautomator-workload:
+
+
+UiAuto (:class:`wa.UiautoWorkload <wa.framework.workload.UiautoWorkload>`)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+This workload is for android targets which will use UiAutomator to interact with
+UI elements without a specific android app, for example performing manipulation
+of android itself. This is the preferred type of automation as the results are
+more portable and reproducible due to being able to wait for UI elements to
+appear rather than having to rely on human recordings.
+
+.. _apkuiautomator-workload:
+
+ApkUiAuto (:class:`wa.ApkUiautoWorkload <wa.framework.workload.ApkUiautoWorkload>`)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The is the same as the UiAuto workload however it is also associated with an
+android app e.g. AdobeReader and will automatically deploy and launch the
+android app before running the automation.
+
+.. _revent-workload:
+
+Revent (:class:`wa.ReventWorkload <wa.framework.workload.ReventWorkload>`)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Revent workloads are designed primarily for games as these are unable to be
+automated with UiAutomator due to the fact that they are rendered within a
+single UI element. They require a recording to be performed manually and
+currently will need re-recording for each different device. For more
+information on revent workloads been please see :ref:`revent_files_creation`
+
+.. _apkrevent-workload:
+
+APKRevent (:class:`wa.ApkReventWorkload <wa.framework.workload.ApkReventWorkload>`)
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The is the same as the Revent workload however it is also associated with an
+android app e.g. AngryBirds and will automatically deploy and launch the android
+app before running the automation.