.. _writing-plugins: Writing Plugins ================ Workload Automation offers several plugin points (or plugin types).The most interesting of these are :workloads: These are the tasks that get executed and measured on the device. These can be benchmarks, high-level use cases, or pretty much anything else. :targets: These are interfaces to the physical devices (development boards or end-user devices, such as smartphones) that use cases run on. Typically each model of a physical device would require its own interface class (though some functionality may be reused by subclassing from an existing base). :instruments: Instruments allow collecting additional data from workload execution (e.g. system traces). Instruments are not specific to a particular Workload. Instruments can hook into any stage of workload execution. :output processors: These are used to format the results of workload execution once they have been collected. Depending on the callback used, these will run either after each iteration or at the end of the run, after all of the results have been collected. You can create an plugin by subclassing the appropriate base class, defining appropriate methods and attributes, and putting the .py file with the class into the "plugins" subdirectory under ``~/.workload_automation`` (or equivalent) where it will be automatically picked up by WA. Plugin Basics ---------------- This sub-section covers things common to implementing plugins of all types. It is recommended you familiarize yourself with the information here before proceeding onto guidance for specific plugin types. .. _context: The Context ^^^^^^^^^^^ The majority of methods in plugins accept a context argument. This is an instance of :class:`wa.framework.execution.ExecutionContext`. It contains information about the current state of execution of WA and keeps track of things like which workload is currently running. Notable methods of the context are: context.add_artifact(name, path, kind, description=None, classifier=None) This is how plugins can add :ref:`artifacts ` of various types to the run output folder for WA. context.add_metric(name, value, units=None, lower_is_better=False, classifiers=None) This method should be used to add :ref:`metrics ` that have been generated from a workload, this will allow WA to process the results accordingly depending on which output processors are enabled. Notable attributes of the context are: context.workload :class:`wa.framework.workload` object that is currently being executed. context.tm This is the target manager which can be used to access various information about the target including initialization parameters. context.current_job This is an instance of :class:`wa.framework.job.Job` and contains all the relevant information to the workload job being currently being executed. context.current_job.spec The current workload specification being executed. This is an instance of :class:`wa.framework.configuration.core.JobSpec` and defines the workload and the parameters under which it is being executed. context.current_job.current_iteration The current iteration of the spec that is being executed. Note that this is the iteration for that spec, i.e. the number of times that spec has been run, *not* the total number of all iterations have been executed so far. context.current_job_output This is the result object for the current iteration. This is an instance of :class:`wa.framework.output.JobOutput`. It contains the status of the iteration as well as the metrics and artefacts generated by the workload. In addition to these, context also defines a few useful paths (see below). Paths ^^^^^ You should avoid using hard-coded absolute paths in your plugins whenever possible, as they make your code too dependent on a particular environment and may mean having to make adjustments when moving to new (host and/or device) platforms. To help avoid hard-coded absolute paths, WA defines a number of standard locations. You should strive to define your paths relative to one of those. On the host ~~~~~~~~~~~ Host paths are available through the context object, which is passed to most plugin methods. context.run_output_directory This is the top-level output directory for all WA results (by default, this will be "wa_output" in the directory in which WA was invoked. context.output_directory This is the output directory for the current iteration. This will an iteration-specific subdirectory under the main results location. If there is no current iteration (e.g. when processing overall run results) this will point to the same location as ``root_output_directory``. Additionally, the global ``wa.settings`` object exposes on other location: settings.dependency_directory this is the root directory for all plugin dependencies (e.g. media files, assets etc) that are not included within the plugin itself. As per Python best practice, it is recommended that methods and values in ``os.path`` standard library module are used for host path manipulation. On the target ~~~~~~~~~~~~~ Workloads and instruments have a ``target`` attribute, which is an interface to the target used by WA. It defines the following location: target.working_directory This is the directory for all WA-related files on the target. All files deployed to the target should be pushed to somewhere under this location (the only exception being executables installed with ``target.install`` method). Since there could be a mismatch between path notation used by the host and the target, the ``os.path`` modules should *not* be used for on-target path manipulation. Instead target has an equipment module exposed through ``target.path`` attribute. This has all the same attributes and behaves the same way as ``os.path``, but is guaranteed to produce valid paths for the target, irrespective of the host's path notation. For example: .. code:: python result_file = self.target.path.join(self.target.working_directory, "result.txt") self.command = "{} -a -b -c {}".format(target_binary, result_file) .. note:: Output processors, unlike workloads and instruments, do not have their own target attribute as they are designed to be able to be ran offline. .. _metrics: Metrics ^^^^^^^ This is what WA uses to store a single metric collected from executing a workload. :name: the name of the metric. Uniquely identifies the metric within the results. :value: The numerical value of the metric for this execution of a workload. This can be either an int or a float. :units: Units for the collected value. Can be None if the value has no units (e.g. it's a count or a standardised score). :lower_is_better: Boolean flag indicating where lower values are better than higher ones. Defaults to False. :classifiers: A set of key-value pairs to further classify this metric beyond current iteration (e.g. this can be used to identify sub-tests). .. _artifact: Artifacts ^^^^^^^^^ This is an artifact generated during execution/post-processing of a workload. Unlike :ref:`metrics `, this represents an actual artifact, such as a file, generated. This may be "output", such as trace, or it could be "meta data" such as logs. These are distinguished using the ``kind`` attribute, which also helps WA decide how it should be handled. Currently supported kinds are: :log: A log file. Not part of the "output" as such but contains information about the run/workload execution that be useful for diagnostics/meta analysis. :meta: A file containing metadata. This is not part of the "output", but contains information that may be necessary to reproduce the results (contrast with ``log`` artifacts which are *not* necessary). :data: This file contains new data, not available otherwise and should be considered part of the "output" generated by WA. Most traces would fall into this category. :export: Exported version of results or some other artifact. This signifies that this artifact does not contain any new data that is not available elsewhere and that it may be safely discarded without losing information. :raw: Signifies that this is a raw dump/log that is normally processed to extract useful information and is then discarded. In a sense, it is the opposite of ``export``, but in general may also be discarded. .. note:: whether a file is marked as ``log``/``data`` or ``raw`` depends on how important it is to preserve this file, e.g. when archiving, vs how much space it takes up. Unlike ``export`` artifacts which are (almost) always ignored by other exporters as that would never result in data loss, ``raw`` files *may* be processed by exporters if they decided that the risk of losing potentially (though unlikely) useful data is greater than the time/space cost of handling the artifact (e.g. a database uploader may choose to ignore ``raw`` artifacts, where as a network filer archiver may choose to archive them). .. note: The kind parameter is intended to represent the logical function of a particular artifact, not it's intended means of processing -- this is left entirely up to the output processors. .. _resource-resolution: Dynamic Resource Resolution ^^^^^^^^^^^^^^^^^^^^^^^^^^^ The idea is to decouple resource identification from resource discovery. Workloads/instruments/devices/etc state *what* resources they need, and not *where* to look for them -- this instead is left to the resource resolver that is part of the execution context. The actual discovery of resources is performed by resource getters that are registered with the resolver. A resource type is defined by a subclass of :class:`wa.framework.resource.Resource`. An instance of this class describes a resource that is to be obtained. At minimum, a ``Resource`` instance has an owner (which is typically the object that is looking for the resource), but specific resource types may define other parameters that describe an instance of that resource (such as file names, URLs, etc). An object looking for a resource invokes a resource resolver with an instance of ``Resource`` describing the resource it is after. The resolver goes through the getters registered for that resource type in priority order attempting to obtain the resource; once the resource is obtained, it is returned to the calling object. If none of the registered getters could find the resource, ``None`` is returned instead. The most common kind of object looking for resources is a ``Workload``, and the ``Workload`` class defines :py:meth:`wa.framework.workload.Workload.init_resources` method that may be overridden by subclasses to perform resource resolution. For example, a workload looking for an executable file would do so like this:: from wa import Workload from wa.import Executable class MyBenchmark(Workload): # ... def init_resources(self, context): resource = Executable(self, self.target.abi, 'my_benchmark') host_exe = context.resolver.get(resource) # ... Currently available resource types are defined in :py:mod:`wa.framework.resources`. .. _deploying-executables: Deploying executables to a target ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Some targets may have certain restrictions on where executable binaries may be placed and how they should be invoked. To ensure your plugin works with as wide a range of targets as possible, you should use WA APIs for deploying and invoking executables on a target, as outlined below. As with other resources, host-side paths to the executable binary to be deployed should be obtained via the :ref:`resource resolver `. A special resource type, ``Executable`` is used to identify a binary to be deployed. This is similar to the regular ``File`` resource, however it takes an additional parameter that specifies the ABI for which executable was compiled. In order for the binary to be obtained in this way, it must be stored in one of the locations scanned by the resource resolver in a directory structure ``/bin//`` (where ``root`` is the base resource location to be searched, e.g. ``~/.workload_automation/dependencies/``, and ```` is the ABI for which the executable has been compiled, as returned by ``self.target.abi``). Once the path to the host-side binary has been obtained, it may be deployed using one of two methods from a `Target `_ instance -- ``install`` or ``install_if_needed``. The latter will check a version of that binary has been previously deployed by WA and will not try to re-install. .. code:: python from wa import Executable host_binary = context.resolver.get(Executable(self, self.target.abi, 'some_binary')) target_binary = self.target.install_if_needed(host_binary) .. note:: Please also note that the check is done based solely on the binary name. For more information please see: :func:`devlib.target.Target.install_if_needed` Both of the above methods will return the path to the installed binary on the target. The executable should be invoked *only* via that path; do **not** assume that it will be in ``PATH`` on the target (or that the executable with the same name in ``PATH`` is the version deployed by WA. For more information on how to implement this, please see the :ref:`how to guide `. Deploying assets ----------------- WA provides a generic mechanism for deploying assets during workload initialization. WA will automatically try to retrieve and deploy each asset to the target's working directory that is contained in a workloads ``deployable_assets`` attribute stored as a list. If the parameter ``cleanup_assets`` is set then any asset deployed will be removed again and the end of the run. If the workload requires a custom deployment mechanism the ``deploy_assets`` method can be overridden for that particular workload, in which case, either additional assets should have their on target paths added to the workload's ``deployed_assests`` attribute or the corresponding ``remove_assets`` method should also be implemented. Parameters ^^^^^^^^^^ All plugins can be parametrized. Parameters are specified using ``parameters`` class attribute. This should be a list of :class:`wa.framework.plugin.Parameter` instances. The following attributes can be specified on parameter creation: name This is the only mandatory argument. The name will be used to create a corresponding attribute in the plugin instance, so it must be a valid Python identifier. kind This is the type of the value of the parameter. This could be an callable. Normally this should be a standard Python type, e.g. ``int`` or ``float``, or one the types defined in :mod:`wa.utils.types`. If not explicitly specified, this will default to ``str``. .. note:: Irrespective of the ``kind`` specified, ``None`` is always a valid value for a parameter. If you don't want to allow ``None``, then set ``mandatory`` (see below) to ``True``. allowed_values A list of the only allowed values for this parameter. .. note:: For composite types, such as ``list_of_strings`` or ``list_of_ints`` in :mod:`wa.utils.types`, each element of the value will be checked against ``allowed_values`` rather than the composite value itself. default The default value to be used for this parameter if one has not been specified by the user. Defaults to ``None``. mandatory A ``bool`` indicating whether this parameter is mandatory. Setting this to ``True`` will make ``None`` an illegal value for the parameter. Defaults to ``False``. .. note:: Specifying a ``default`` will mean that this parameter will, effectively, be ignored (unless the user sets the param to ``None``). .. note:: Mandatory parameters are *bad*. If at all possible, you should strive to provide a sensible ``default`` or to make do without the parameter. Only when the param is absolutely necessary, and there really is no sensible default that could be given (e.g. something like login credentials), should you consider making it mandatory. constraint This is an additional constraint to be enforced on the parameter beyond its type or fixed allowed values set. This should be a predicate (a function that takes a single argument -- the user-supplied value -- and returns a ``bool`` indicating whether the constraint has been satisfied). override A parameter name must be unique not only within an plugin but also with that plugin's class hierarchy. If you try to declare a parameter with the same name as already exists, you will get an error. If you do want to override a parameter from further up in the inheritance hierarchy, you can indicate that by setting ``override`` attribute to ``True``. When overriding, you do not need to specify every other attribute of the parameter, just the ones you what to override. Values for the rest will be taken from the parameter in the base class. Validation and cross-parameter constraints ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ An plugin will get validated at some point after constructions. When exactly this occurs depends on the plugin type, but it *will* be validated before it is used. You can implement ``validate`` method in your plugin (that takes no arguments beyond the ``self``) to perform any additions *internal* validation in your plugin. By "internal", I mean that you cannot make assumptions about the surrounding environment (e.g. that the device has been initialized). The contract for ``validate`` method is that it should raise an exception (either ``wa.framework.exception.ConfigError`` or plugin-specific exception type -- see further on this page) if some validation condition has not, and cannot, been met. If the method returns without raising an exception, then the plugin is in a valid internal state. Note that ``validate`` can be used not only to verify, but also to impose a valid internal state. In particular, this where cross-parameter constraints can be resolved. If the ``default`` or ``allowed_values`` of one parameter depend on another parameter, there is no way to express that declaratively when specifying the parameters. In that case the dependent attribute should be left unspecified on creation and should instead be set inside ``validate``. Logging ^^^^^^^ Every plugin class has it's own logger that you can access through ``self.logger`` inside the plugin's methods. Generally, a :class:`Target` will log everything it is doing, so you shouldn't need to add much additional logging for device actions. However you might what to log additional information, e.g. what settings your plugin is using, what it is doing on the host, etc. (Operations on the host will not normally be logged, so your plugin should definitely log what it is doing on the host). One situation in particular where you should add logging is before doing something that might take a significant amount of time, such as downloading a file. Documenting ^^^^^^^^^^^ All plugins and their parameter should be documented. For plugins themselves, this is done through ``description`` class attribute. The convention for an plugin description is that the first paragraph should be a short summary description of what the plugin does and why one would want to use it (among other things, this will get extracted and used by ``wa list`` command). Subsequent paragraphs (separated by blank lines) can then provide a more detailed description, including any limitations and setup instructions. For parameters, the description is passed as an argument on creation. Please note that if ``default``, ``allowed_values``, or ``constraint``, are set in the parameter, they do not need to be explicitly mentioned in the description (wa documentation utilities will automatically pull those). If the ``default`` is set in ``validate`` or additional cross-parameter constraints exist, this *should* be documented in the parameter description. Both plugins and their parameters should be documented using reStructureText markup (standard markup for Python documentation). See: http://docutils.sourceforge.net/rst.html Aside from that, it is up to you how you document your plugin. You should try to provide enough information so that someone unfamiliar with your plugin is able to use it, e.g. you should document all settings and parameters your plugin expects (including what the valid values are). Error Notification ^^^^^^^^^^^^^^^^^^ When you detect an error condition, you should raise an appropriate exception to notify the user. The exception would typically be :class:`ConfigError` or (depending the type of the plugin) :class:`WorkloadError`/:class:`DeviceError`/:class:`InstrumentError`/:class:`OutputProcessorError`. All these errors are defined in :mod:`wa.framework.exception` module. :class:`ConfigError` should be raised where there is a problem in configuration specified by the user (either through the agenda or config files). These errors are meant to be resolvable by simple adjustments to the configuration (and the error message should suggest what adjustments need to be made. For all other errors, such as missing dependencies, mis-configured environment, problems performing operations, etc., the plugin type-specific exceptions should be used. If the plugin itself is capable of recovering from the error and carrying on, it may make more sense to log an ERROR or WARNING level message using the plugin's logger and to continue operation. .. _decorators: Execution Decorators --------------------- The following decorators are available for use in order to control how often a method should be able to be executed. For example, if we want to ensure that no matter how many iterations of a particular workload are ran, we only execute the initialize method for that instance once, we would use the decorator as follows: .. code-block:: python from wa.utils.exec_control import once @once def initialize(self, context): # Perform one time initialization e.g. installing a binary to target # .. @once_per_instance ^^^^^^^^^^^^^^^^^^ The specified method will be invoked only once for every bound instance within the environment. @once_per_class ^^^^^^^^^^^^^^^ The specified method will be invoked only once for all instances of a class within the environment. @once ^^^^^ The specified method will be invoked only once within the environment. .. warning:: If a method containing a super call is decorated, this will also cause stop propagation up the hierarchy, unless this is the desired effect, additional functionality should be implemented in a separate decorated method which can then be called allowing for normal propagation to be retained. Utils ^^^^^ Workload Automation defines a number of utilities collected under :mod:`wa.utils` subpackage. These utilities were created to help with the implementation of the framework itself, but may be also be useful when implementing plugins. Workloads --------- .. _workload-types: Workload Types ^^^^^^^^^^^^^^^^ .. _basic-workload: Basic (:class:`wa.Workload `) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This type of the workload is the simplest type of workload and is left the to developer to implement its full functionality. .. _apk-workload: Apk (:class:`wa.ApkWorkload `) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This workload will simply deploy and launch an android app in its basic form with no UI interaction. .. _uiautomator-workload: UiAuto (:class:`wa.UiautoWorkload `) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ This workload is for android targets which will use UiAutomator to interact with UI elements without a specific android app, for example performing manipulation of android itself. This is the preferred type of automation as the results are more portable and reproducible due to being able to wait for UI elements to appear rather than having to rely on human recordings. .. _apkuiautomator-workload: ApkUiAuto (:class:`wa.ApkUiautoWorkload `) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The is the same as the UiAuto workload however it is also associated with an android app e.g. AdobeReader and will automatically deploy and launch the android app before running the automation. .. _revent-workload: Revent (:class:`wa.ReventWorkload `) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Revent workloads are designed primarily for games as these are unable to be automated with UiAutomator due to the fact that they are rendered within a single UI element. They require a recording to be performed manually and currently will need re-recording for each different device. For more information on revent workloads been please see :ref:`revent_files_creation` .. _apkrevent-workload: APKRevent (:class:`wa.ApkReventWorkload `) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The is the same as the Revent workload however it is also associated with an android app e.g. AngryBirds and will automatically deploy and launch the android app before running the automation. .. _workload-interface: Workload Interface ^^^^^^^^^^^^^^^^^^^ The workload interface should be implemented as follows: :name: This identifies the workload (e.g. it used to specify it in the :ref:`agenda`. :init_resources: This method may be optionally override to implement dynamic resource discovery for the workload. This method executes early on, before the device has been initialized, so it should only be used to initialize resources that do not depend on the device to resolve. This method is executed once per run for each workload instance. :validate: This method can be used to validate any assumptions your workload makes about the environment (e.g. that required files are present, environment variables are set, etc) and should raise a :class:`wa.WorkloadError ` if that is not the case. The base class implementation only makes sure sure that the name attribute has been set. :initialize: This method is decorated with the ``@once_per_instance`` decorator, (for more information please see `Execution Decorators`_) therefore it will be executed exactly once per run (no matter how many instances of the workload there are). It will run after the device has been initialized, so it may be used to perform device-dependent initialization that does not need to be repeated on each iteration (e.g. as installing executables required by the workload on the device). :setup: Everything that needs to be in place for workload execution should be done in this method. This includes copying files to the device, starting up an application, configuring communications channels, etc. :run: This method should perform the actual task that is being measured. When this method exits, the task is assumed to be complete. .. note:: Instruments are kicked off just before calling this method and disabled right after, so everything in this method is being measured. Therefore this method should contain the least code possible to perform the operations you are interested in measuring. Specifically, things like installing or starting applications, processing results, or copying files to/from the device should be done elsewhere if possible. :extract_results: This method gets invoked after the task execution has finished and should be used to extract metrics from the target. :update_output: This method should be used to update the output within the specified execution context with the metrics and artifacts from this workload iteration. :teardown: This could be used to perform any cleanup you may wish to do, e.g. Uninstalling applications, deleting file on the device, etc. :finalize: This is the complement to ``initialize``. This will be executed exactly once at the end of the run. This should be used to perform any final clean up (e.g. uninstalling binaries installed in the ``initialize``). Workload methods (except for ``validate``) take a single argument that is a :class:`wa.framework.execution.ExecutionContext` instance. This object keeps track of the current execution state (such as the current workload, iteration number, etc), and contains, among other things, a :class:`wa.framework.output.JobOutput` instance that should be populated from the ``update_output`` method with the results of the execution. For more information please see `the context`_ documentation.:: # ... def update_output(self, context): # ... context.add_metric('energy', 23.6, 'Joules', lower_is_better=True) # ... .. _ReventWorkload: Adding Revent Workload ----------------------- There are two base classes that can be subclassed to create Revent based workloads depending on whether the workload is associated with an android Apk or not :class:`wa.ApkReventWorkload ` and :class:`wa.ReventWorkload ` respectively. They both implement all the methods needed to push the files to the device and run them. The revent workload classes define the following interfaces:: class ReventWorkload(Workload): name = None class ApkReventWorkload(Workload): name = None package_names = [] The interface should be implemented as follows :name: This identifies the workload (e.g. it used to specify it in the :ref:`agenda`. :package_names: This is a list of the android application apk packages names that are required to run the workload. .. instrument-reference: Adding an Instrument --------------------- Instruments can be used to collect additional measurements during workload execution (e.g. collect power readings). An instrument can hook into almost any stage of workload execution. Any new instrument should be a subclass of Instrument and it must have a name. When a new instrument is added to Workload Automation, the methods of the new instrument will be found automatically and hooked up to the supported signals. Once a signal is broadcasted, the corresponding registered method is invoked. Each method in Instrument must take two arguments, which are self and context. Supported method and their corresponding signals can be found in the :ref:`Signals Documentation ` To make implementations easier and common, the basic steps to add new instrument is similar to the steps to add new workload and an example can be found :ref:`here `. .. _instrument-api: The full interface of WA instruments is shown below:: class Instrument(Plugin): name = None description = None parameters = [ ] def initialize(self, context): """ This method will only be called once during the workload run therefore operations that only need to be performed initially should be performed her for example pushing the files to the target device, installing them. """ pass def setup(self, context): """ This method is invoked after the workload is setup. All the necessary setups should go inside this method. Setup, includes operations like clearing logs, additional configuration etc. """ pass def start(self, context): """ It is invoked just before the workload start execution. Here is where instrument measures start being registered/taken. """ pass def stop(self, context): """ It is invoked just after the workload execution stops. The measures should stop being taken/registered. """ pass def update_result(self, context): """ It is invoked after the workload updated its result. update_result is where the taken measures are added to the result so it can be processed by Workload Automation. """ pass def teardown(self, context): """ It is invoked after the workload is teared down. It is a good place to clean any logs generated by the instrument. """ pass def finalize(self, context): """ This method is the complement to the initialize method and will also only be called once so should be used to deleting/uninstalling files pushed to the device. """ pass This is similar to a Workload, except all methods are optional. In addition to the workload-like methods, instruments can define a number of other methods that will get invoked at various points during run execution. The most useful of which is perhaps ``initialize`` that gets invoked after the device has been initialised for the first time, and can be used to perform one-time setup (e.g. copying files to the device -- there is no point in doing that for each iteration). The full list of available methods can be found in :ref:`Signals Documentation `. .. _prioritization: Prioritization ^^^^^^^^^^^^^^ Callbacks (e.g. ``setup()`` methods) for all instruments get executed at the same point during workload execution, one after another. The order in which the callbacks get invoked should be considered arbitrary and should not be relied on (e.g. you cannot expect that just because instrument A is listed before instrument B in the config, instrument A's callbacks will run first). In some cases (e.g. in ``start()`` and ``stop()`` methods), it is important to ensure that a particular instrument's callbacks run a closely as possible to the workload's invocations in order to maintain accuracy of readings; or, conversely, that a callback is executed after the others, because it takes a long time and may throw off the accuracy of other instruments. You can do this by using decorators on the appropriate methods. The available decorators are: ``very_slow``, ``slow``, ``normal``, ``fast``, ``very_fast``, with ``very_fast`` running closest to the workload invocation and ``very_slow`` running furtherest away. For example:: from wa import very_fast # .. class PreciseInstrument(Instrument) # ... @very_fast def start(self, context): pass @very_fast def stop(self, context): pass # ... ``PreciseInstrument`` will be started after all other instruments (i.e. *just* before the workload runs), and it will stopped before all other instruments (i.e. *just* after the workload runs). If more than one active instrument have specified fast (or slow) callbacks, then their execution order with respect to each other is not guaranteed. In general, having a lot of instruments enabled is going to negatively affect the readings. The best way to ensure accuracy of measurements is to minimize the number of active instruments (perhaps doing several identical runs with different instruments enabled). Example ^^^^^^^ Below is a simple instrument that measures the execution time of a workload:: class ExecutionTimeInstrument(Instrument): """ Measure how long it took to execute the run() methods of a Workload. """ name = 'execution_time' def initialize(self, context): self.start_time = None self.end_time = None @fast def start(self, context): self.start_time = time.time() @fast def stop(self, context): self.end_time = time.time() def update_result(self, context): execution_time = self.end_time - self.start_time context.add_metric('execution_time', execution_time, 'seconds') .. include:: developer_reference/instrument_method_map.rst .. _adding-an-output-processor: Adding an Output processor ---------------------------- A output processor is responsible for processing the results. This may involve formatting and writing them to a file, uploading them to a database, generating plots, etc. WA comes with a few output processors that output results in a few common formats (such as csv or JSON). You can add your own output processors by creating a Python file in ``~/.workload_automation/result_processors`` with a class that derives from :class:`wa.OutputProcessor `, which has the following interface:: class OutputProcessor(Plugin): name = None description = None parameters = [ ] def initialize(self): pass def process_job_output(self, output, target_info, run_ouput): pass def export_job_output(self, output, target_info, run_ouput): pass def process_run_output(self, output, target_info): pass def export_run_output(self, output, target_info): pass def finalize(self): pass The method names should be fairly self-explanatory. The difference between "process" and "export" methods is that export methods will be invoke after process methods for all output processors have been generated. Process methods may generated additional artifacts (metrics, files, etc), while export methods should not -- the should only handle existing results (upload them to a database, archive on a filer, etc). The output object passed to job methods is an instance of :class:`wa.framework.output.JobOutput`, the output object passed to run methods is an instance of :class:`wa.RunOutput `. Please refer to their API documentation for details. Adding a Resource Getter ------------------------ A resource getter is a plugin that is designed to retrieve a resource (binaries, APK files or additional workload assets). Resource getters are invoked in priority order until one returns the desired resource. If you want WA to look for resources somewhere it doesn't by default (e.g. you have a repository of APK files), you can implement a getter for the resource and register it with a higher priority than the standard WA getters, so that it gets invoked first. Instances of a resource getter should implement the following interface:: class ResourceGetter(Plugin): name = None def register(self, resolver): raise NotImplementedError() The getter should define a name for itself (as with all plugins), in addition it should implement the ``register`` method. This involves registering a method with the resolver that should used to be called when trying to retrieve a resource (typically ``get``) along with it's priority (see `Getter Prioritization`_ below. That method should return an instance of the resource that has been discovered (what "instance" means depends on the resource, e.g. it could be a file path), or ``None`` if this getter was unable to discover that resource. Getter Prioritization ^^^^^^^^^^^^^^^^^^^^^ A priority is an integer with higher numeric values indicating a higher priority. The following standard priority aliases are defined for getters: :preferred: Take this resource in favour of the environment resource. :local: Found somewhere under ~/.workload_automation/ or equivalent, or from environment variables, external configuration files, etc. These will override resource supplied with the package. :lan: Resource will be retrieved from a locally mounted remote location (such as samba share) :remote: Resource will be downloaded from a remote location (such as an HTTP server) :package: Resource provided with the package. These priorities are defined as class members of :class:`wa.framework.resource.SourcePriority`, e.g. ``SourcePriority.preferred``. Most getters in WA will be registered with either ``local`` or ``package`` priorities. So if you want your getter to override the default, it should typically be registered as ``preferred``. You don't have to stick to standard priority levels (though you should, unless there is a good reason). Any integer is a valid priority. The standard priorities range from 0 to 50 in increments of 10. Example ^^^^^^^ The following is an implementation of a getter that searches for files in the users dependencies directory, typically ``~/.workload_automation/dependencies/`` It uses the ``get_from_location`` method to filter the available files in the provided directory appropriately:: import sys from wa import settings, from wa.framework.resource import ResourceGetter, SourcePriority from wa.framework.getters import get_from_location class UserDirectory(ResourceGetter): name = 'user' def register(self, resolver): resolver.register(self.get, SourcePriority.local) def get(self, resource): basepath = settings.dependencies_directory directory = _d(os.path.join(basepath, resource.owner.name)) return get_from_location(directory, resource) .. _adding_a_target: Adding a Target --------------- In WA3 a 'target' consists of a platform and a devlib target. The implementations of the targets are located in ``devlib``. WA3 will instantiate a devlib target passing relevant parameters parsed from the configuration. For more information about devlib targets please see `the documentation `_. The currently available platforms are: :generic: The 'standard' platform implementation of the target, this should work for the majority of use cases. :juno: A platform implementation specifically for the juno. :tc2: A platform implementation specifically for the tc2. :gem5: A platform implementation to interact with a gem5 simulation. The currently available targets from devlib are: :linux: A device running a Linux based OS. :android: A device running the Android OS. :local: Used to run locally on a linux based host. :chromeos: A device running chromeos, supporting an android container if available. For an example of adding you own customized version of an existing devlib target, please see the how to section :ref:`Adding a Custom Target `. Other Plugin Types --------------------- In addition to plugin types covered above, there are few other, more specialized ones. They will not be covered in as much detail. Most of them expose relatively simple interfaces with only a couple of methods and it is expected that if the need arises to extend them, the API-level documentation that accompanies them, in addition to what has been outlined here, should provide enough guidance. :commands: This allows extending WA with additional sub-commands (to supplement exiting ones outlined in the :ref:`invocation` section). :modules: Modules are "plugins for plugins". They can be loaded by other plugins to expand their functionality (for example, a flashing module maybe loaded by a device in order to support flashing). Packaging Your Plugins ---------------------- If your have written a bunch of plugins, and you want to make it easy to deploy them to new systems and/or to update them on existing systems, you can wrap them in a Python package. You can use ``wa create package`` command to generate appropriate boiler plate. This will create a ``setup.py`` and a directory for your package that you can place your plugins into. For example, if you have a workload inside ``my_workload.py`` and a result processor in ``my_result_processor.py``, and you want to package them as ``my_wa_exts`` package, first run the create command :: wa create package my_wa_exts This will create a ``my_wa_exts`` directory which contains a ``my_wa_exts/setup.py`` and a subdirectory ``my_wa_exts/my_wa_exts`` which is the package directory for your plugins (you can rename the top-level ``my_wa_exts`` directory to anything you like -- it's just a "container" for the setup.py and the package directory). Once you have that, you can then copy your plugins into the package directory, creating ``my_wa_exts/my_wa_exts/my_workload.py`` and ``my_wa_exts/my_wa_exts/my_result_processor.py``. If you have a lot of plugins, you might want to organize them into subpackages, but only the top-level package directory is created by default, and it is OK to have everything in there. .. note:: When discovering plugins through this mechanism, WA traverses the Python module/submodule tree, not the directory structure, therefore, if you are going to create subdirectories under the top level directory created for you, it is important that your make sure they are valid Python packages; i.e. each subdirectory must contain a __init__.py (even if blank) in order for the code in that directory and its subdirectories to be discoverable. At this stage, you may want to edit ``params`` structure near the bottom of the ``setup.py`` to add correct author, license and contact information (see "Writing the Setup Script" section in standard Python documentation for details). You may also want to add a README and/or a COPYING file at the same level as the setup.py. Once you have the contents of your package sorted, you can generate the package by running :: cd my_wa_exts python setup.py sdist This will generate ``my_wa_exts/dist/my_wa_exts-0.0.1.tar.gz`` package which can then be deployed on the target system with standard Python package management tools, e.g. :: sudo pip install my_wa_exts-0.0.1.tar.gz As part of the installation process, the setup.py in the package, will write the package's name into ``~/.workoad_automoation/packages``. This will tell WA that the package contains plugin and it will load them next time it runs. .. note:: There are no uninstall hooks in ``setuputils``, so if you ever uninstall your WA plugins package, you will have to manually remove it from ``~/.workload_automation/packages`` otherwise WA will complain about a missing package next time you try to run it.