sascha/workload-automation
1
0
Fork 0
mirror of https://github.com/ARM-software/workload-automation.git synced 2024-10-05 18:31:12 +01:00
Code Issues Releases Wiki Activity

doc: Reorganise Developer Guide/Reference

Browse Source
This commit is contained in:
Marc Bonnici 2018-06-26 14:38:53 +01:00 committed by setrofim
parent e66ae050a9
commit 116c260bae
5 changed files with 287 additions and 287 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
doc/Makefile
View File

@ -49,7 +49,7 @@ help:
clean: clean:
rm -rf $(BUILDDIR)/* rm -rf $(BUILDDIR)/*
rm -rf source/plugins/* rm -rf source/plugins/*
rm -rf source/developer_reference/instrument_method_map.rst rm -rf source/developer_guide/instrument_method_map.rst
rm -rf source/run_config/* rm -rf source/run_config/*
coverage: coverage:

2
doc/source/api/workload.rst
View File

@ -10,7 +10,7 @@ Workload
The base :class:`Workload` interface is as follows, and is the base class for The base :class:`Workload` interface is as follows, and is the base class for
all :ref:`workload types <workload-types>`. For more information about to all :ref:`workload types <workload-types>`. For more information about to
implement your own workload please see the implement your own workload please see the
:ref:`Developer How Tos <adding-a-workload>`. :ref:`Developer How Tos <adding-a-workload-example>`.
All instances of a workload will have the following attributes: All instances of a workload will have the following attributes:

2
doc/source/conf.py
View File

@ -314,7 +314,7 @@ def setup(app):
generate_target_documentation('plugins') generate_target_documentation('plugins')
generate_run_config_documentation('run_config') generate_run_config_documentation('run_config')
generate_meta_config_documentation('run_config') generate_meta_config_documentation('run_config')
generate_instrument_method_map(os.path.join('developer_information', 'developer_reference', generate_instrument_method_map(os.path.join('developer_information', 'developer_guide',
'instrument_method_map.rst')) 'instrument_method_map.rst'))
app.add_object_type('confval', 'confval', app.add_object_type('confval', 'confval',
objname='configuration value', objname='configuration value',

285
doc/source/developer_information/developer_guide/writing_plugins.rst
View File

@ -34,129 +34,6 @@ This sub-section covers things common to implementing plugins of all types. It
is recommended you familiarize yourself with the information here before is recommended you familiarize yourself with the information here before
proceeding onto guidance for specific plugin types. proceeding onto guidance for specific plugin types.
.. _context:
The Context
^^^^^^^^^^^
The majority of methods in plugins accept a context argument. This is an
instance of :class:`wa.framework.execution.ExecutionContext`. It contains
information about the current state of execution of WA and keeps track of things
like which workload is currently running.
Notable methods of the context are:
:context.get_resource(resource, strict=True):
This method should be used to retrieve a resource using the resource getters rather than using the ResourceResolver directly as this method additionally record any found resources hash in the output metadata.
:context.add_artifact(name, host_file_path, kind, description=None, classifier=None):
Plugins can add :ref:`artifacts <artifact>` of various kinds to the run
output directory for WA and associate them with a description and/or
:ref:`classifier <classifiers>`.
:context.add_metric(name, value, units=None, lower_is_better=False, classifiers=None):
This method should be used to add :ref:`metrics <metrics>` that have been
generated from a workload, this will allow WA to process the results
accordingly depending on which output processors are enabled.
Notable attributes of the context are:
:context.workload:
:class:`wa.framework.workload` object that is currently being executed.
:context.tm:
This is the target manager that can be used to access various information
about the target including initialization parameters.
:context.current_job:
This is an instance of :class:`wa.framework.job.Job` and contains all
the information relevant to the workload job currently being executed.
:context.current_job.spec:
The current workload specification being executed. This is an
instance of :class:`wa.framework.configuration.core.JobSpec`
and defines the workload and the parameters under which it is
being executed.
:context.current_job.current_iteration:
The current iteration of the spec that is being executed. Note that this
is the iteration for that spec, i.e. the number of times that spec has
been run, *not* the total number of all iterations have been executed so
far.
:context.job_output:
This is the output object for the current iteration which
is an instance of :class:`wa.framework.output.JobOutput`. It contains
the status of the iteration as well as the metrics and artifacts
generated by the workload.
In addition to these, context also defines a few useful paths (see below).
Paths
^^^^^
You should avoid using hard-coded absolute paths in your plugins whenever
possible, as they make your code too dependent on a particular environment and
may mean having to make adjustments when moving to new (host and/or device)
platforms. To help avoid hard-coded absolute paths, WA defines a number of
standard locations. You should strive to define your paths relative
to one of these.
On the host
~~~~~~~~~~~
Host paths are available through the context object, which is passed to most
plugin methods.
context.run_output_directory
This is the top-level output directory for all WA results (by default,
this will be "wa_output" in the directory in which WA was invoked.
context.output_directory
This is the output directory for the current iteration. This will an
iteration-specific subdirectory under the main results location. If
there is no current iteration (e.g. when processing overall run results)
this will point to the same location as ``root_output_directory``.
Additionally, the global ``wa.settings`` object exposes on other location:
settings.dependency_directory
this is the root directory for all plugin dependencies (e.g. media
files, assets etc) that are not included within the plugin itself.
As per Python best practice, it is recommended that methods and values in
``os.path`` standard library module are used for host path manipulation.
On the target
~~~~~~~~~~~~~
Workloads and instruments have a ``target`` attribute, which is an interface to
the target used by WA. It defines the following location:
target.working_directory
This is the directory for all WA-related files on the target. All files
deployed to the target should be pushed to somewhere under this location
(the only exception being executables installed with ``target.install``
method).
Since there could be a mismatch between path notation used by the host and the
target, the ``os.path`` modules should *not* be used for on-target path
manipulation. Instead target has an equipment module exposed through
``target.path`` attribute. This has all the same attributes and behaves the
same way as ``os.path``, but is guaranteed to produce valid paths for the target,
irrespective of the host's path notation. For example:
.. code:: python
result_file = self.target.path.join(self.target.working_directory, "result.txt")
self.command = "{} -a -b -c {}".format(target_binary, result_file)
.. note:: Output processors, unlike workloads and instruments, do not have their
own target attribute as they are designed to be able to be ran offline.
.. _resource-resolution: .. _resource-resolution:
Dynamic Resource Resolution Dynamic Resource Resolution
@ -271,166 +148,6 @@ additional assets should have their on target paths added to the workload's
``deployed_assests`` attribute or the corresponding ``remove_assets`` method ``deployed_assests`` attribute or the corresponding ``remove_assets`` method
should also be implemented. should also be implemented.
.. _plugin-parmeters:
Parameters
----------
All plugins can be parametrized. Parameters are specified using
``parameters`` class attribute. This should be a list of
:class:`wa.framework.plugin.Parameter` instances. The following attributes can be
specified on parameter creation:
:name:
This is the only mandatory argument. The name will be used to create a
corresponding attribute in the plugin instance, so it must be a valid
Python identifier.
:kind:
This is the type of the value of the parameter. This must be an
callable. Normally this should be a standard Python type, e.g. ``int``
or ``float``, or one the types defined in :mod:`wa.utils.types`.
If not explicitly specified, this will default to ``str``.
.. note:: Irrespective of the ``kind`` specified, ``None`` is always a
valid value for a parameter. If you don't want to allow
``None``, then set ``mandatory`` (see below) to ``True``.
:allowed_values:
A list of the only allowed values for this parameter.
.. note:: For composite types, such as ``list_of_strings`` or
``list_of_ints`` in :mod:`wa.utils.types`, each element of
the value will be checked against ``allowed_values`` rather
than the composite value itself.
:default:
The default value to be used for this parameter if one has not been
specified by the user. Defaults to ``None``.
:mandatory:
A ``bool`` indicating whether this parameter is mandatory. Setting this
to ``True`` will make ``None`` an illegal value for the parameter.
Defaults to ``False``.
.. note:: Specifying a ``default`` will mean that this parameter will,
effectively, be ignored (unless the user sets the param to ``None``).
.. note:: Mandatory parameters are *bad*. If at all possible, you should
strive to provide a sensible ``default`` or to make do without
the parameter. Only when the param is absolutely necessary,
and there really is no sensible default that could be given
(e.g. something like login credentials), should you consider
making it mandatory.
:constraint:
This is an additional constraint to be enforced on the parameter beyond
its type or fixed allowed values set. This should be a predicate (a function
that takes a single argument -- the user-supplied value -- and returns
a ``bool`` indicating whether the constraint has been satisfied).
:override:
A parameter name must be unique not only within an plugin but also
with that plugin's class hierarchy. If you try to declare a parameter
with the same name as already exists, you will get an error. If you do
want to override a parameter from further up in the inheritance
hierarchy, you can indicate that by setting ``override`` attribute to
``True``.
When overriding, you do not need to specify every other attribute of the
parameter, just the ones you what to override. Values for the rest will
be taken from the parameter in the base class.
Validation and cross-parameter constraints
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
A plugin will get validated at some point after construction. When exactly
this occurs depends on the plugin type, but it *will* be validated before it
is used.
You can implement ``validate`` method in your plugin (that takes no arguments
beyond the ``self``) to perform any additional *internal* validation in your
plugin. By "internal", I mean that you cannot make assumptions about the
surrounding environment (e.g. that the device has been initialized).
The contract for ``validate`` method is that it should raise an exception
(either ``wa.framework.exception.ConfigError`` or plugin-specific exception type -- see
further on this page) if some validation condition has not, and cannot, been met.
If the method returns without raising an exception, then the plugin is in a
valid internal state.
Note that ``validate`` can be used not only to verify, but also to impose a
valid internal state. In particular, this where cross-parameter constraints can
be resolved. If the ``default`` or ``allowed_values`` of one parameter depend on
another parameter, there is no way to express that declaratively when specifying
the parameters. In that case the dependent attribute should be left unspecified
on creation and should instead be set inside ``validate``.
Logging
-------
Every plugin class has it's own logger that you can access through
``self.logger`` inside the plugin's methods. Generally, a :class:`Target` will
log everything it is doing, so you shouldn't need to add much additional logging
for device actions. However you might what to log additional information, e.g.
what settings your plugin is using, what it is doing on the host, etc.
(Operations on the host will not normally be logged, so your plugin should
definitely log what it is doing on the host). One situation in particular where
you should add logging is before doing something that might take a significant
amount of time, such as downloading a file.
Documenting
-----------
All plugins and their parameter should be documented. For plugins
themselves, this is done through ``description`` class attribute. The convention
for an plugin description is that the first paragraph should be a short
summary description of what the plugin does and why one would want to use it
(among other things, this will get extracted and used by ``wa list`` command).
Subsequent paragraphs (separated by blank lines) can then provide a more
detailed description, including any limitations and setup instructions.
For parameters, the description is passed as an argument on creation. Please
note that if ``default``, ``allowed_values``, or ``constraint``, are set in the
parameter, they do not need to be explicitly mentioned in the description (wa
documentation utilities will automatically pull those). If the ``default`` is set
in ``validate`` or additional cross-parameter constraints exist, this *should*
be documented in the parameter description.
Both plugins and their parameters should be documented using reStructureText
markup (standard markup for Python documentation). See:
http://docutils.sourceforge.net/rst.html
Aside from that, it is up to you how you document your plugin. You should try
to provide enough information so that someone unfamiliar with your plugin is
able to use it, e.g. you should document all settings and parameters your
plugin expects (including what the valid values are).
Error Notification
------------------
When you detect an error condition, you should raise an appropriate exception to
notify the user. The exception would typically be :class:`ConfigError` or
(depending the type of the plugin)
:class:`WorkloadError`/:class:`DeviceError`/:class:`InstrumentError`/:class:`OutputProcessorError`.
All these errors are defined in :mod:`wa.framework.exception` module.
A :class:`ConfigError` should be raised where there is a problem in configuration
specified by the user (either through the agenda or config files). These errors
are meant to be resolvable by simple adjustments to the configuration (and the
error message should suggest what adjustments need to be made. For all other
errors, such as missing dependencies, mis-configured environment, problems
performing operations, etc., the plugin type-specific exceptions should be
used.
If the plugin itself is capable of recovering from the error and carrying
on, it may make more sense to log an ERROR or WARNING level message using the
plugin's logger and to continue operation.
.. _instrument-reference: .. _instrument-reference:
Adding an Instrument Adding an Instrument
@ -595,7 +312,7 @@ Below is a simple instrument that measures the execution time of a workload::
context.add_metric('execution_time', execution_time, 'seconds') context.add_metric('execution_time', execution_time, 'seconds')
.. include:: developer_information/developer_reference/instrument_method_map.rst .. include:: developer_information/developer_guide/instrument_method_map.rst
.. _adding-an-output-processor: .. _adding-an-output-processor:

283
doc/source/developer_information/developer_reference/plugins.rst
View File

@ -32,6 +32,289 @@ Plugin Basics
This section contains reference information common to plugins of all types. This section contains reference information common to plugins of all types.
.. _context:
The Context
^^^^^^^^^^^
The majority of methods in plugins accept a context argument. This is an
instance of :class:`wa.framework.execution.ExecutionContext`. It contains
information about the current state of execution of WA and keeps track of things
like which workload is currently running.
Notable methods of the context are:
:context.get_resource(resource, strict=True):
This method should be used to retrieve a resource using the resource getters rather than using the ResourceResolver directly as this method additionally record any found resources hash in the output metadata.
:context.add_artifact(name, host_file_path, kind, description=None, classifier=None):
Plugins can add :ref:`artifacts <artifact>` of various kinds to the run
output directory for WA and associate them with a description and/or
:ref:`classifier <classifiers>`.
:context.add_metric(name, value, units=None, lower_is_better=False, classifiers=None):
This method should be used to add :ref:`metrics <metrics>` that have been
generated from a workload, this will allow WA to process the results
accordingly depending on which output processors are enabled.
Notable attributes of the context are:
:context.workload:
:class:`wa.framework.workload` object that is currently being executed.
:context.tm:
This is the target manager that can be used to access various information
about the target including initialization parameters.
:context.current_job:
This is an instance of :class:`wa.framework.job.Job` and contains all
the information relevant to the workload job currently being executed.
:context.current_job.spec:
The current workload specification being executed. This is an
instance of :class:`wa.framework.configuration.core.JobSpec`
and defines the workload and the parameters under which it is
being executed.
:context.current_job.current_iteration:
The current iteration of the spec that is being executed. Note that this
is the iteration for that spec, i.e. the number of times that spec has
been run, *not* the total number of all iterations have been executed so
far.
:context.job_output:
This is the output object for the current iteration which
is an instance of :class:`wa.framework.output.JobOutput`. It contains
the status of the iteration as well as the metrics and artifacts
generated by the workload.
In addition to these, context also defines a few useful paths (see below).
Paths
^^^^^
You should avoid using hard-coded absolute paths in your plugins whenever
possible, as they make your code too dependent on a particular environment and
may mean having to make adjustments when moving to new (host and/or device)
platforms. To help avoid hard-coded absolute paths, WA defines a number of
standard locations. You should strive to define your paths relative
to one of these.
On the host
~~~~~~~~~~~
Host paths are available through the context object, which is passed to most
plugin methods.
context.run_output_directory
This is the top-level output directory for all WA results (by default,
this will be "wa_output" in the directory in which WA was invoked.
context.output_directory
This is the output directory for the current iteration. This will an
iteration-specific subdirectory under the main results location. If
there is no current iteration (e.g. when processing overall run results)
this will point to the same location as ``root_output_directory``.
Additionally, the global ``wa.settings`` object exposes on other location:
settings.dependency_directory
this is the root directory for all plugin dependencies (e.g. media
files, assets etc) that are not included within the plugin itself.
As per Python best practice, it is recommended that methods and values in
``os.path`` standard library module are used for host path manipulation.
On the target
~~~~~~~~~~~~~
Workloads and instruments have a ``target`` attribute, which is an interface to
the target used by WA. It defines the following location:
target.working_directory
This is the directory for all WA-related files on the target. All files
deployed to the target should be pushed to somewhere under this location
(the only exception being executables installed with ``target.install``
method).
Since there could be a mismatch between path notation used by the host and the
target, the ``os.path`` modules should *not* be used for on-target path
manipulation. Instead target has an equipment module exposed through
``target.path`` attribute. This has all the same attributes and behaves the
same way as ``os.path``, but is guaranteed to produce valid paths for the target,
irrespective of the host's path notation. For example:
.. code:: python
result_file = self.target.path.join(self.target.working_directory, "result.txt")
self.command = "{} -a -b -c {}".format(target_binary, result_file)
.. note:: Output processors, unlike workloads and instruments, do not have their
own target attribute as they are designed to be able to be ran offline.
.. _plugin-parmeters:
Parameters
^^^^^^^^^^^
All plugins can be parametrized. Parameters are specified using
``parameters`` class attribute. This should be a list of
:class:`wa.framework.plugin.Parameter` instances. The following attributes can be
specified on parameter creation:
:name:
This is the only mandatory argument. The name will be used to create a
corresponding attribute in the plugin instance, so it must be a valid
Python identifier.
:kind:
This is the type of the value of the parameter. This must be an
callable. Normally this should be a standard Python type, e.g. ``int``
or ``float``, or one the types defined in :mod:`wa.utils.types`.
If not explicitly specified, this will default to ``str``.
.. note:: Irrespective of the ``kind`` specified, ``None`` is always a
valid value for a parameter. If you don't want to allow
``None``, then set ``mandatory`` (see below) to ``True``.
:allowed_values:
A list of the only allowed values for this parameter.
.. note:: For composite types, such as ``list_of_strings`` or
``list_of_ints`` in :mod:`wa.utils.types`, each element of
the value will be checked against ``allowed_values`` rather
than the composite value itself.
:default:
The default value to be used for this parameter if one has not been
specified by the user. Defaults to ``None``.
:mandatory:
A ``bool`` indicating whether this parameter is mandatory. Setting this
to ``True`` will make ``None`` an illegal value for the parameter.
Defaults to ``False``.
.. note:: Specifying a ``default`` will mean that this parameter will,
effectively, be ignored (unless the user sets the param to ``None``).
.. note:: Mandatory parameters are *bad*. If at all possible, you should
strive to provide a sensible ``default`` or to make do without
the parameter. Only when the param is absolutely necessary,
and there really is no sensible default that could be given
(e.g. something like login credentials), should you consider
making it mandatory.
:constraint:
This is an additional constraint to be enforced on the parameter beyond
its type or fixed allowed values set. This should be a predicate (a function
that takes a single argument -- the user-supplied value -- and returns
a ``bool`` indicating whether the constraint has been satisfied).
:override:
A parameter name must be unique not only within an plugin but also
with that plugin's class hierarchy. If you try to declare a parameter
with the same name as already exists, you will get an error. If you do
want to override a parameter from further up in the inheritance
hierarchy, you can indicate that by setting ``override`` attribute to
``True``.
When overriding, you do not need to specify every other attribute of the
parameter, just the ones you what to override. Values for the rest will
be taken from the parameter in the base class.
Validation and cross-parameter constraints
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
A plugin will get validated at some point after construction. When exactly
this occurs depends on the plugin type, but it *will* be validated before it
is used.
You can implement ``validate`` method in your plugin (that takes no arguments
beyond the ``self``) to perform any additional *internal* validation in your
plugin. By "internal", I mean that you cannot make assumptions about the
surrounding environment (e.g. that the device has been initialized).
The contract for ``validate`` method is that it should raise an exception
(either ``wa.framework.exception.ConfigError`` or plugin-specific exception type -- see
further on this page) if some validation condition has not, and cannot, been met.
If the method returns without raising an exception, then the plugin is in a
valid internal state.
Note that ``validate`` can be used not only to verify, but also to impose a
valid internal state. In particular, this where cross-parameter constraints can
be resolved. If the ``default`` or ``allowed_values`` of one parameter depend on
another parameter, there is no way to express that declaratively when specifying
the parameters. In that case the dependent attribute should be left unspecified
on creation and should instead be set inside ``validate``.
Logging
^^^^^^^
Every plugin class has it's own logger that you can access through
``self.logger`` inside the plugin's methods. Generally, a :class:`Target` will
log everything it is doing, so you shouldn't need to add much additional logging
for device actions. However you might what to log additional information, e.g.
what settings your plugin is using, what it is doing on the host, etc.
(Operations on the host will not normally be logged, so your plugin should
definitely log what it is doing on the host). One situation in particular where
you should add logging is before doing something that might take a significant
amount of time, such as downloading a file.
Documenting
^^^^^^^^^^^
All plugins and their parameter should be documented. For plugins
themselves, this is done through ``description`` class attribute. The convention
for an plugin description is that the first paragraph should be a short
summary description of what the plugin does and why one would want to use it
(among other things, this will get extracted and used by ``wa list`` command).
Subsequent paragraphs (separated by blank lines) can then provide a more
detailed description, including any limitations and setup instructions.
For parameters, the description is passed as an argument on creation. Please
note that if ``default``, ``allowed_values``, or ``constraint``, are set in the
parameter, they do not need to be explicitly mentioned in the description (wa
documentation utilities will automatically pull those). If the ``default`` is set
in ``validate`` or additional cross-parameter constraints exist, this *should*
be documented in the parameter description.
Both plugins and their parameters should be documented using reStructureText
markup (standard markup for Python documentation). See:
http://docutils.sourceforge.net/rst.html
Aside from that, it is up to you how you document your plugin. You should try
to provide enough information so that someone unfamiliar with your plugin is
able to use it, e.g. you should document all settings and parameters your
plugin expects (including what the valid values are).
Error Notification
^^^^^^^^^^^^^^^^^^
When you detect an error condition, you should raise an appropriate exception to
notify the user. The exception would typically be :class:`ConfigError` or
(depending the type of the plugin)
:class:`WorkloadError`/:class:`DeviceError`/:class:`InstrumentError`/:class:`OutputProcessorError`.
All these errors are defined in :mod:`wa.framework.exception` module.
A :class:`ConfigError` should be raised where there is a problem in configuration
specified by the user (either through the agenda or config files). These errors
are meant to be resolvable by simple adjustments to the configuration (and the
error message should suggest what adjustments need to be made. For all other
errors, such as missing dependencies, mis-configured environment, problems
performing operations, etc., the plugin type-specific exceptions should be
used.
If the plugin itself is capable of recovering from the error and carrying
on, it may make more sense to log an ERROR or WARNING level message using the
plugin's logger and to continue operation.
.. _metrics: .. _metrics:
Metrics Metrics