sascha/workload-automation
1
0
Fork 0
mirror of https://github.com/ARM-software/workload-automation.git synced 2025-02-20 11:58:55 +00:00
Code Issues Releases Wiki Activity

doc/user_reference: Restructure Agenda reference

Browse Source
This commit is contained in:
Marc Bonnici 2018-06-21 14:54:22 +01:00 committed by setrofim
parent 6c93590062
commit b6e91630b8
1 changed files with 161 additions and 86 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

247
doc/source/user_information/user_reference/agenda.rst
View File

@ -4,16 +4,86 @@ Agenda
------
An agenda can be thought of as defining an experiment as it specifies what is to
be done during a Workload Automation run. This includes which workloads will be
run, with what configuration and which augmentations will be enabled, etc.
An agenda can be thought of as a way to define an experiment as it specifies
what is to be done during a Workload Automation run. This includes which
workloads will be run, with what configuration and which augmentations will be
enabled, etc. Agenda syntax is designed to be both succinct and expressive and
is written using YAML notation.
Agenda syntax is designed to be both succinct and expressive and are written
using YAML notation.
There are three valid top level entries which are:
:ref:`config <config-agenda-entry>`, :ref:`workloads <workloads-agenda-entry>`,
:ref:`sections <sections-agenda-entry>`.
There are three valid top level entries which are: ``"config"``, ``"workloads"``,
``"sections"``.
An example agenda can be seen here:
.. code-block:: yaml
config: # General configuration for the run
user_directory: ~/.workload_automation/
default_output_directory: 'wa_output'
augmentations: # A list of all augmentations to be enabled and disabled.
- trace-cmd
- csv
- ~dmesg # Disable the dmseg augmentation
iterations: 1 # How many iterations to run each workload by default
device: generic_android
device_config:
device: R32C801B8XY # Th adb name of our device we want to run on
disable_selinux: true
load_default_modules: true
package_data_directory: /data/data
trace-cmd: # Provide config for the trace-cmd augmentation.
buffer_size_step: 1000
events:
- sched*
- irq*
- power*
- thermal*
no_install: false
report: true
report_on_target: false
csv: # Provide config for the csv augmentation
use_all_classifiers: true
sections: # Configure what sections we want and their settings
- id: LITTLES # Run workloads just on the LITTLE cores
runtime_parameters: # Supply RT parameters to be used for this section
num_little_cores: 4
num_big_cores: 0
- id: BIGS # Run workloads just on the big cores
runtime_parameters: # Supply RT parameters to be used for this section
num_big_cores: 4
num_little_cores: 0
workloads: # List which workloads should be ran
- name: benchmarkpi
augmentations:
- ~trace-cmd # Disable the trace-cmd instrument for this workload
iterations: 2 # Override the global number of iteration for this workload
params: # Specify workload parameters for this workload
cleanup_assets: true
exact_abi: false
force_install: false
install_timeout: 300
markers_enabled: false
prefer_host_package: true
strict: false
uninstall: false
- dhrystone # Run the dhrystone workload with all default config
This agenda will result in a total of 6 jobs being executed on our Android
device, 4 of which running the BenchmarkPi workload with its customized workload
parameters and 2 running dhrystone with its default configuration. The first 3
will be running on only the little cores and the latter running on the big
cores. For all of the jobs executed the output will be processed by the ``csv``
processor,(plus any additional processors enabled in the default config file),
however trace data will only be collected for the dhrystone jobs.
.. _config-agenda-entry:
config
^^^^^^^
@ -21,77 +91,71 @@ config
This section is used to provide overall configuration for WA and its run. The
``config`` section of an agenda will be merged with any other configuration
files provided (including the default config file) and merged with the most
specific configuration taking precedence (see :ref:`here <config-merging>` for
more information.
specific configuration taking precedence (see
:ref:`Config Merging <config-merging>` for more information. The only
restriction is that ``run_name`` can only be specified in the config section
of an agenda as this would not make sense to set as a default.
Within this section there are multiple distinct types of configuration that can be
provided.
Within this section there are multiple distinct types of configuration that can
be provided. However in addition to the options listed here all configuration
that is available for :ref:`sections <sections-agenda-entry>` can also be entered
here and will be globally applied.
Run Config
~~~~~~~~~~
Configuration
"""""""""""""
The first is to configure the behaviour of WA and how a run as a
whole will behave. The most common that you may want to specify are:
whole will behave. The most common options that that you may want to specify are:
- :confval:`device` - The name of the 'device' that you wish to perform the run
on. This name is a combination of a devlib
`Platform <http://devlib.readthedocs.io/en/latest/platform.html>`_ and
`Target <http://devlib.readthedocs.io/en/latest/target.html>`_.
To see the available options please use ``wa list targets``.
- :confval:`device_config` - The is a dict mapping allowing you to configure
which target to connect to (e.g. ``host`` for an SSH connection or ``device``
to specific an ADB name) as well as configure other options for the device for
example the ``working_directory`` or the list of ``modules`` to be loaded onto
the device.
:device: The name of the 'device' that you wish to perform the run
on. This name is a combination of a devlib
`Platform <http://devlib.readthedocs.io/en/latest/platform.html>`_ and
`Target <http://devlib.readthedocs.io/en/latest/target.html>`_. To
see the available options please use ``wa list targets``.
:device_config: The is a dict mapping allowing you to configure which target
to connect to (e.g. ``host`` for an SSH connection or
``device`` to specific an ADB name) as well as configure other
options for the device for example the ``working_directory``
or the list of ``modules`` to be loaded onto the device.
:execution_order: Defines the order in which the agenda spec will be executed.
:reboot_policy: Defines when during execution of a run a Device will be rebooted.
:max_retries: The maximum number of times failed jobs will be retried before giving up.
:allow_phone_home: Prevent running any workloads that are marked with ‘phones_home’.
For more information and a full list of these configuration options please see
:ref:`Run Configuration <run-configuration>`.
Meta Configuration
~~~~~~~~~~~~~~~~~~
The next type of configuration options are the `"Meta Configuration"` options
(for a full list please see :ref:`here <meta-configuration>`) and these are used
to configure the behaviour of WA framework itself, for example directory
locations to be used or logging configuration.
:ref:`Run Configuration <run-configuration>` and
:ref:`"Meta Configuration" <meta-configuration>`.
Plugins
~~~~~~~
You can also use this section to supply configuration for specific plugins, such
as augmentations, workloads, resource getters etc. To do this the plugin name
you wish to configure should be provided as an entry in this section and should
contain a mapping of configuration options to their desired settings. If
configuration is supplied for a plugin that is not currently enabled then it will
simply be ignored. This allows for plugins to be temporarily removed
without also having to remove their configuration, or to provide a set of
defaults for a plugin which can then be overridden.
"""""""
:augmentations: Specify a list of which augmentations should be enabled (or if
prefixed with a ``~``, disabled).
Some plugins provide global aliases which can set one or more configuration
options at once, and these can also bee specified here. For example specifying
the entry ``remote_assets_url`` with a corresponding value will set the URL the
http resource getter will attempt to search for any missing assets at.
.. note:: While augmentations can be enabled and disabled on a per workload
basis, they cannot yet be re-configured part way through a run and the
configuration provided as part of an agenda config section or separate
config file will be used for all jobs in a WA run.
:<plugin_name>: You can also use this section to supply configuration for
specific plugins, such as augmentations, workloads, resource getters etc.
To do this the plugin name you wish to configure should be provided as an
entry in this section and should contain a mapping of configuration
options to their desired settings. If configuration is supplied for a
plugin that is not currently enabled then it will simply be ignored. This
allows for plugins to be temporarily removed without also having to remove
their configuration, or to provide a set of defaults for a plugin which
can then be overridden.
augmentations
"""""""""""""
As mentioned above this section should be used to configure augmentations, both
to specify which should be enabled and disabled and also to provide any relevant
configuration. The ``"augmentation"`` entry, if present, should be a list of
augmentations that should be enabled (or if prefixed with a ``~``, disabled).
:<global_alias>: Some plugins provide global aliases which can set one or more
configuration options at once, and these can also bee specified here. For
example specifying the entry ``remote_assets_url`` with a corresponding
value will set the URL the http resource getter will attempt to search for
any missing assets at.
.. note:: While augmentations can be enabled and disabled on a per workload
basis, they cannot yet be re-configured part way through a run and the
configuration provided as part of an agenda config section or separate
config file will be used for all jobs in a WA run.
workloads
"""""""""
In addition to configuring individual workloads both in the ``workloads`` and
``sections`` entries you can also provide configuration at this level which will
apply globally in the same style mentioned below. Any configuration provided
here will be overridden if specified again in subsequent sections.
---------------------------
.. _workloads-agenda-entry:
workloads
^^^^^^^^^
@ -103,27 +167,32 @@ here will be the most specific and therefore override any other more generalised
configuration for that particular workload spec. The valid entries are as
follows:
- :confval:`workload_name` (Mandatory) - The name of the workload to be ran
- :confval:`iterations` - Specify how many iterations the workload should be ran
- :confval:`label` - Similar to IDs but do not have the uniqueness restriction.
If specified, labels will be used by some output processors instead of (or in
addition to) the workload name. For example, the csv output processor will put
the label in the "workload" column of the CSV file.
- :confval:`augmentations` - The instruments and output processors to enable (or
disabled using a ~) during this workload.
- :confval:`classifiers` Classifiers allow you to tag metrics from this workload
spec which are often used to help identify what runtime parameters were used
when post processing results.
- :confval:`workload_parameters` [*workload_params*] - Any parameters to
configure that particular workload in a dict form.
:workload_name: **(Mandatory)** The name of the workload to be ran
:iterations: Specify how many iterations the workload should be ran
:label: Similar to IDs but do not have the uniqueness restriction.
If specified, labels will be used by some output processors instead of (or in
addition to) the workload name. For example, the csv output processor will put
the label in the "workload" column of the CSV file.
:augmentations: The instruments and output processors to enable (or
disabled using a ~) during this workload.
:classifiers: Classifiers allow you to tag metrics from this workload
spec which are often used to help identify what runtime parameters were used
when post processing results.
:workload_parameters: Any parameters to
configure that particular workload in a dict form.
Alias: ``workload_params``
.. note:: You can see available parameters for a given workload with the
:ref:`show command <show-command>`.
:ref:`show command <show-command>` or look it up in the
:ref:`Plugin Reference <plugin-reference>`.
- :confval:`runtime_parameters` [*runtime_parms*] - A dict mapping of any
runtime parameters that should be set for the device for that particular
workload. For available parameters please see :ref:`runtime parameters
<runtime-parameters>`.
:runtime_parameters: A dict mapping of any runtime parameters that should be set
for the device for that particular workload. For available
parameters please see
:ref:`runtime parameters <runtime-parameters>`.
Alias: ``runtime_parms``
.. note:: Unless specified elsewhere these configurations will not be
undone once the workload has finished. I.e. if the frequency of a
@ -137,13 +206,19 @@ follows:
be interpreted as ``runtime_params``.
---------------------------
.. _sections-agenda-entry:
sections
^^^^^^^^
Sections are used for for grouping sets of configuration together in order to
reduce the need for duplicated configuration (for more information please see
:ref:`here <sections>`). Each section specified will be applied for each entry
in the ``workloads`` section. The valid configuration entries are the same
as the ``"workloads"`` section as mentioned above, except you can
additionally specify a "workloads" entry which can be provided with the same
configuration entries as the ``"workloads"`` top level entry.
:ref:`Sections <sections>`). Each section specified will be applied for each
entry in the ``workloads`` section. The valid configuration entries are the
same as the ``"workloads"`` section as mentioned above, except you can
additionally specify:
:workloads: An entry which can be provided with the same configuration entries
as the :ref:`workloads <workloads-agenda-entry>` top level entry.