mirror of
https://github.com/ARM-software/workload-automation.git
synced 2025-10-31 07:04:17 +00:00
doc/user_reference: Restructure Agenda reference
This commit is contained in:
Marc Bonnici
setrofim
@@ -4,16 +4,86 @@ Agenda
|
|||||||
------
|
------
|
||||||
|
|
||||||
|
|
||||||
An agenda can be thought of as defining an experiment as it specifies what is to
|
An agenda can be thought of as a way to define an experiment as it specifies
|
||||||
be done during a Workload Automation run. This includes which workloads will be
|
what is to be done during a Workload Automation run. This includes which
|
||||||
run, with what configuration and which augmentations will be enabled, etc.
|
workloads will be run, with what configuration and which augmentations will be
|
||||||
|
enabled, etc. Agenda syntax is designed to be both succinct and expressive and
|
||||||
|
is written using YAML notation.
|
||||||
|
|
||||||
Agenda syntax is designed to be both succinct and expressive and are written
|
There are three valid top level entries which are:
|
||||||
using YAML notation.
|
:ref:`config <config-agenda-entry>`, :ref:`workloads <workloads-agenda-entry>`,
|
||||||
|
:ref:`sections <sections-agenda-entry>`.
|
||||||
|
|
||||||
There are three valid top level entries which are: ``"config"``, ``"workloads"``,
|
An example agenda can be seen here:
|
||||||
``"sections"``.
|
|
||||||
|
|
||||||
|
.. code-block:: yaml
|
||||||
|
|
||||||
|
config: # General configuration for the run
|
||||||
|
user_directory: ~/.workload_automation/
|
||||||
|
default_output_directory: 'wa_output'
|
||||||
|
augmentations: # A list of all augmentations to be enabled and disabled.
|
||||||
|
- trace-cmd
|
||||||
|
- csv
|
||||||
|
- ~dmesg # Disable the dmseg augmentation
|
||||||
|
|
||||||
|
iterations: 1 # How many iterations to run each workload by default
|
||||||
|
|
||||||
|
device: generic_android
|
||||||
|
device_config:
|
||||||
|
device: R32C801B8XY # Th adb name of our device we want to run on
|
||||||
|
disable_selinux: true
|
||||||
|
load_default_modules: true
|
||||||
|
package_data_directory: /data/data
|
||||||
|
|
||||||
|
trace-cmd: # Provide config for the trace-cmd augmentation.
|
||||||
|
buffer_size_step: 1000
|
||||||
|
events:
|
||||||
|
- sched*
|
||||||
|
- irq*
|
||||||
|
- power*
|
||||||
|
- thermal*
|
||||||
|
no_install: false
|
||||||
|
report: true
|
||||||
|
report_on_target: false
|
||||||
|
csv: # Provide config for the csv augmentation
|
||||||
|
use_all_classifiers: true
|
||||||
|
|
||||||
|
sections: # Configure what sections we want and their settings
|
||||||
|
- id: LITTLES # Run workloads just on the LITTLE cores
|
||||||
|
runtime_parameters: # Supply RT parameters to be used for this section
|
||||||
|
num_little_cores: 4
|
||||||
|
num_big_cores: 0
|
||||||
|
|
||||||
|
- id: BIGS # Run workloads just on the big cores
|
||||||
|
runtime_parameters: # Supply RT parameters to be used for this section
|
||||||
|
num_big_cores: 4
|
||||||
|
num_little_cores: 0
|
||||||
|
|
||||||
|
workloads: # List which workloads should be ran
|
||||||
|
- name: benchmarkpi
|
||||||
|
augmentations:
|
||||||
|
- ~trace-cmd # Disable the trace-cmd instrument for this workload
|
||||||
|
iterations: 2 # Override the global number of iteration for this workload
|
||||||
|
params: # Specify workload parameters for this workload
|
||||||
|
cleanup_assets: true
|
||||||
|
exact_abi: false
|
||||||
|
force_install: false
|
||||||
|
install_timeout: 300
|
||||||
|
markers_enabled: false
|
||||||
|
prefer_host_package: true
|
||||||
|
strict: false
|
||||||
|
uninstall: false
|
||||||
|
- dhrystone # Run the dhrystone workload with all default config
|
||||||
|
|
||||||
|
This agenda will result in a total of 6 jobs being executed on our Android
|
||||||
|
device, 4 of which running the BenchmarkPi workload with its customized workload
|
||||||
|
parameters and 2 running dhrystone with its default configuration. The first 3
|
||||||
|
will be running on only the little cores and the latter running on the big
|
||||||
|
cores. For all of the jobs executed the output will be processed by the ``csv``
|
||||||
|
processor,(plus any additional processors enabled in the default config file),
|
||||||
|
however trace data will only be collected for the dhrystone jobs.
|
||||||
|
|
||||||
|
.. _config-agenda-entry:
|
||||||
|
|
||||||
config
|
config
|
||||||
^^^^^^^
|
^^^^^^^
|
||||||
@@ -21,77 +91,71 @@ config
|
|||||||
This section is used to provide overall configuration for WA and its run. The
|
This section is used to provide overall configuration for WA and its run. The
|
||||||
``config`` section of an agenda will be merged with any other configuration
|
``config`` section of an agenda will be merged with any other configuration
|
||||||
files provided (including the default config file) and merged with the most
|
files provided (including the default config file) and merged with the most
|
||||||
specific configuration taking precedence (see :ref:`here <config-merging>` for
|
specific configuration taking precedence (see
|
||||||
more information.
|
:ref:`Config Merging <config-merging>` for more information. The only
|
||||||
|
restriction is that ``run_name`` can only be specified in the config section
|
||||||
|
of an agenda as this would not make sense to set as a default.
|
||||||
|
|
||||||
Within this section there are multiple distinct types of configuration that can be
|
Within this section there are multiple distinct types of configuration that can
|
||||||
provided.
|
be provided. However in addition to the options listed here all configuration
|
||||||
|
that is available for :ref:`sections <sections-agenda-entry>` can also be entered
|
||||||
|
here and will be globally applied.
|
||||||
|
|
||||||
Run Config
|
Configuration
|
||||||
~~~~~~~~~~
|
"""""""""""""
|
||||||
|
|
||||||
The first is to configure the behaviour of WA and how a run as a
|
The first is to configure the behaviour of WA and how a run as a
|
||||||
whole will behave. The most common that you may want to specify are:
|
whole will behave. The most common options that that you may want to specify are:
|
||||||
|
|
||||||
- :confval:`device` - The name of the 'device' that you wish to perform the run
|
:device: The name of the 'device' that you wish to perform the run
|
||||||
on. This name is a combination of a devlib
|
on. This name is a combination of a devlib
|
||||||
`Platform <http://devlib.readthedocs.io/en/latest/platform.html>`_ and
|
`Platform <http://devlib.readthedocs.io/en/latest/platform.html>`_ and
|
||||||
`Target <http://devlib.readthedocs.io/en/latest/target.html>`_.
|
`Target <http://devlib.readthedocs.io/en/latest/target.html>`_. To
|
||||||
To see the available options please use ``wa list targets``.
|
see the available options please use ``wa list targets``.
|
||||||
- :confval:`device_config` - The is a dict mapping allowing you to configure
|
:device_config: The is a dict mapping allowing you to configure which target
|
||||||
which target to connect to (e.g. ``host`` for an SSH connection or ``device``
|
to connect to (e.g. ``host`` for an SSH connection or
|
||||||
to specific an ADB name) as well as configure other options for the device for
|
``device`` to specific an ADB name) as well as configure other
|
||||||
example the ``working_directory`` or the list of ``modules`` to be loaded onto
|
options for the device for example the ``working_directory``
|
||||||
the device.
|
or the list of ``modules`` to be loaded onto the device.
|
||||||
|
:execution_order: Defines the order in which the agenda spec will be executed.
|
||||||
|
:reboot_policy: Defines when during execution of a run a Device will be rebooted.
|
||||||
|
:max_retries: The maximum number of times failed jobs will be retried before giving up.
|
||||||
|
:allow_phone_home: Prevent running any workloads that are marked with ‘phones_home’.
|
||||||
|
|
||||||
For more information and a full list of these configuration options please see
|
For more information and a full list of these configuration options please see
|
||||||
:ref:`Run Configuration <run-configuration>`.
|
:ref:`Run Configuration <run-configuration>` and
|
||||||
|
:ref:`"Meta Configuration" <meta-configuration>`.
|
||||||
Meta Configuration
|
|
||||||
~~~~~~~~~~~~~~~~~~
|
|
||||||
|
|
||||||
The next type of configuration options are the `"Meta Configuration"` options
|
|
||||||
(for a full list please see :ref:`here <meta-configuration>`) and these are used
|
|
||||||
to configure the behaviour of WA framework itself, for example directory
|
|
||||||
locations to be used or logging configuration.
|
|
||||||
|
|
||||||
|
|
||||||
Plugins
|
Plugins
|
||||||
~~~~~~~
|
"""""""
|
||||||
You can also use this section to supply configuration for specific plugins, such
|
:augmentations: Specify a list of which augmentations should be enabled (or if
|
||||||
as augmentations, workloads, resource getters etc. To do this the plugin name
|
prefixed with a ``~``, disabled).
|
||||||
you wish to configure should be provided as an entry in this section and should
|
|
||||||
contain a mapping of configuration options to their desired settings. If
|
|
||||||
configuration is supplied for a plugin that is not currently enabled then it will
|
|
||||||
simply be ignored. This allows for plugins to be temporarily removed
|
|
||||||
without also having to remove their configuration, or to provide a set of
|
|
||||||
defaults for a plugin which can then be overridden.
|
|
||||||
|
|
||||||
Some plugins provide global aliases which can set one or more configuration
|
.. note:: While augmentations can be enabled and disabled on a per workload
|
||||||
options at once, and these can also bee specified here. For example specifying
|
basis, they cannot yet be re-configured part way through a run and the
|
||||||
the entry ``remote_assets_url`` with a corresponding value will set the URL the
|
configuration provided as part of an agenda config section or separate
|
||||||
http resource getter will attempt to search for any missing assets at.
|
config file will be used for all jobs in a WA run.
|
||||||
|
|
||||||
|
:<plugin_name>: You can also use this section to supply configuration for
|
||||||
|
specific plugins, such as augmentations, workloads, resource getters etc.
|
||||||
|
To do this the plugin name you wish to configure should be provided as an
|
||||||
|
entry in this section and should contain a mapping of configuration
|
||||||
|
options to their desired settings. If configuration is supplied for a
|
||||||
|
plugin that is not currently enabled then it will simply be ignored. This
|
||||||
|
allows for plugins to be temporarily removed without also having to remove
|
||||||
|
their configuration, or to provide a set of defaults for a plugin which
|
||||||
|
can then be overridden.
|
||||||
|
|
||||||
augmentations
|
:<global_alias>: Some plugins provide global aliases which can set one or more
|
||||||
"""""""""""""
|
configuration options at once, and these can also bee specified here. For
|
||||||
As mentioned above this section should be used to configure augmentations, both
|
example specifying the entry ``remote_assets_url`` with a corresponding
|
||||||
to specify which should be enabled and disabled and also to provide any relevant
|
value will set the URL the http resource getter will attempt to search for
|
||||||
configuration. The ``"augmentation"`` entry, if present, should be a list of
|
any missing assets at.
|
||||||
augmentations that should be enabled (or if prefixed with a ``~``, disabled).
|
|
||||||
|
|
||||||
.. note:: While augmentations can be enabled and disabled on a per workload
|
---------------------------
|
||||||
basis, they cannot yet be re-configured part way through a run and the
|
|
||||||
configuration provided as part of an agenda config section or separate
|
|
||||||
config file will be used for all jobs in a WA run.
|
|
||||||
|
|
||||||
workloads
|
|
||||||
"""""""""
|
|
||||||
In addition to configuring individual workloads both in the ``workloads`` and
|
|
||||||
``sections`` entries you can also provide configuration at this level which will
|
|
||||||
apply globally in the same style mentioned below. Any configuration provided
|
|
||||||
here will be overridden if specified again in subsequent sections.
|
|
||||||
|
|
||||||
|
.. _workloads-agenda-entry:
|
||||||
|
|
||||||
workloads
|
workloads
|
||||||
^^^^^^^^^
|
^^^^^^^^^
|
||||||
@@ -103,27 +167,32 @@ here will be the most specific and therefore override any other more generalised
|
|||||||
configuration for that particular workload spec. The valid entries are as
|
configuration for that particular workload spec. The valid entries are as
|
||||||
follows:
|
follows:
|
||||||
|
|
||||||
- :confval:`workload_name` (Mandatory) - The name of the workload to be ran
|
:workload_name: **(Mandatory)** The name of the workload to be ran
|
||||||
- :confval:`iterations` - Specify how many iterations the workload should be ran
|
:iterations: Specify how many iterations the workload should be ran
|
||||||
- :confval:`label` - Similar to IDs but do not have the uniqueness restriction.
|
:label: Similar to IDs but do not have the uniqueness restriction.
|
||||||
If specified, labels will be used by some output processors instead of (or in
|
If specified, labels will be used by some output processors instead of (or in
|
||||||
addition to) the workload name. For example, the csv output processor will put
|
addition to) the workload name. For example, the csv output processor will put
|
||||||
the label in the "workload" column of the CSV file.
|
the label in the "workload" column of the CSV file.
|
||||||
- :confval:`augmentations` - The instruments and output processors to enable (or
|
:augmentations: The instruments and output processors to enable (or
|
||||||
disabled using a ~) during this workload.
|
disabled using a ~) during this workload.
|
||||||
- :confval:`classifiers` Classifiers allow you to tag metrics from this workload
|
:classifiers: Classifiers allow you to tag metrics from this workload
|
||||||
spec which are often used to help identify what runtime parameters were used
|
spec which are often used to help identify what runtime parameters were used
|
||||||
when post processing results.
|
when post processing results.
|
||||||
- :confval:`workload_parameters` [*workload_params*] - Any parameters to
|
:workload_parameters: Any parameters to
|
||||||
configure that particular workload in a dict form.
|
configure that particular workload in a dict form.
|
||||||
|
|
||||||
|
Alias: ``workload_params``
|
||||||
|
|
||||||
.. note:: You can see available parameters for a given workload with the
|
.. note:: You can see available parameters for a given workload with the
|
||||||
:ref:`show command <show-command>`.
|
:ref:`show command <show-command>` or look it up in the
|
||||||
|
:ref:`Plugin Reference <plugin-reference>`.
|
||||||
|
|
||||||
- :confval:`runtime_parameters` [*runtime_parms*] - A dict mapping of any
|
:runtime_parameters: A dict mapping of any runtime parameters that should be set
|
||||||
runtime parameters that should be set for the device for that particular
|
for the device for that particular workload. For available
|
||||||
workload. For available parameters please see :ref:`runtime parameters
|
parameters please see
|
||||||
<runtime-parameters>`.
|
:ref:`runtime parameters <runtime-parameters>`.
|
||||||
|
|
||||||
|
Alias: ``runtime_parms``
|
||||||
|
|
||||||
.. note:: Unless specified elsewhere these configurations will not be
|
.. note:: Unless specified elsewhere these configurations will not be
|
||||||
undone once the workload has finished. I.e. if the frequency of a
|
undone once the workload has finished. I.e. if the frequency of a
|
||||||
@@ -137,13 +206,19 @@ follows:
|
|||||||
be interpreted as ``runtime_params``.
|
be interpreted as ``runtime_params``.
|
||||||
|
|
||||||
|
|
||||||
|
---------------------------
|
||||||
|
|
||||||
|
.. _sections-agenda-entry:
|
||||||
|
|
||||||
sections
|
sections
|
||||||
^^^^^^^^
|
^^^^^^^^
|
||||||
|
|
||||||
Sections are used for for grouping sets of configuration together in order to
|
Sections are used for for grouping sets of configuration together in order to
|
||||||
reduce the need for duplicated configuration (for more information please see
|
reduce the need for duplicated configuration (for more information please see
|
||||||
:ref:`here <sections>`). Each section specified will be applied for each entry
|
:ref:`Sections <sections>`). Each section specified will be applied for each
|
||||||
in the ``workloads`` section. The valid configuration entries are the same
|
entry in the ``workloads`` section. The valid configuration entries are the
|
||||||
as the ``"workloads"`` section as mentioned above, except you can
|
same as the ``"workloads"`` section as mentioned above, except you can
|
||||||
additionally specify a "workloads" entry which can be provided with the same
|
additionally specify:
|
||||||
configuration entries as the ``"workloads"`` top level entry.
|
|
||||||
|
:workloads: An entry which can be provided with the same configuration entries
|
||||||
|
as the :ref:`workloads <workloads-agenda-entry>` top level entry.
|
||||||
|
|||||||
Reference in New Issue
Block a user