mirror of
https://github.com/ARM-software/workload-automation.git
synced 2025-10-29 22:24:51 +00:00
doc: Restructure
Restructure the documentation to be split into `User Information` and `Developer Information`, and split the how to guides into their corresponding section.
This commit is contained in:
149
doc/source/user_information/user_reference/agenda.rst
Normal file
149
doc/source/user_information/user_reference/agenda.rst
Normal file
@@ -0,0 +1,149 @@
|
||||
.. _agenda-reference:
|
||||
|
||||
Agenda
|
||||
------
|
||||
|
||||
|
||||
An agenda can be thought of as defining an experiment as it specifies what is to
|
||||
be done during a Workload Automation run. This includes which workloads will be
|
||||
run, with what configuration and which augmentations will be enabled, etc.
|
||||
|
||||
Agenda syntax is designed to be both succinct and expressive and are written
|
||||
using YAML notation.
|
||||
|
||||
There are three valid top level entries which are: ``"config"``, ``"workloads"``,
|
||||
``"sections"``.
|
||||
|
||||
|
||||
config
|
||||
^^^^^^^
|
||||
|
||||
This section is used to provide overall configuration for WA and its run. The
|
||||
``config`` section of an agenda will be merged with any other configuration
|
||||
files provided (including the default config file) and merged with the most
|
||||
specific configuration taking precedence (see :ref:`here <config-merging>` for
|
||||
more information.
|
||||
|
||||
Within this section there are multiple distinct types of configuration that can be
|
||||
provided.
|
||||
|
||||
Run Config
|
||||
~~~~~~~~~~
|
||||
|
||||
The first is to configure the behaviour of WA and how a run as a
|
||||
whole will behave. The most common that you may want to specify are:
|
||||
|
||||
- :confval:`device` - The name of the 'device' that you wish to perform the run
|
||||
on. This name is a combination of a devlib
|
||||
`Platform <http://devlib.readthedocs.io/en/latest/platform.html>`_ and
|
||||
`Target <http://devlib.readthedocs.io/en/latest/target.html>`_.
|
||||
To see the available options please use ``wa list targets``.
|
||||
- :confval:`device_config` - The is a dict mapping allowing you to configure
|
||||
which target to connect to (e.g. ``host`` for an SSH connection or ``device``
|
||||
to specific an ADB name) as well as configure other options for the device for
|
||||
example the ``working_directory`` or the list of ``modules`` to be loaded onto
|
||||
the device.
|
||||
|
||||
For more information and a full list of these configuration options please see
|
||||
:ref:`Run Configuration <run-configuration>`.
|
||||
|
||||
Meta Configuration
|
||||
~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The next type of configuration options are the `"Meta Configuration"` options
|
||||
(for a full list please see :ref:`here <meta-configuration>`) and these are used
|
||||
to configure the behaviour of WA framework itself, for example directory
|
||||
locations to be used or logging configuration.
|
||||
|
||||
|
||||
Plugins
|
||||
~~~~~~~
|
||||
You can also use this section to supply configuration for specific plugins, such
|
||||
as augmentations, workloads, resource getters etc. To do this the plugin name
|
||||
you wish to configure should be provided as an entry in this section and should
|
||||
contain a mapping of configuration options to their desired settings. If
|
||||
configuration is supplied for a plugin that is not currently enabled then it will
|
||||
simply be ignored. This allows for plugins to be temporarily removed
|
||||
without also having to remove their configuration, or to provide a set of
|
||||
defaults for a plugin which can then be overridden.
|
||||
|
||||
Some plugins provide global aliases which can set one or more configuration
|
||||
options at once, and these can also bee specified here. For example specifying
|
||||
the entry ``remote_assets_url`` with a corresponding value will set the URL the
|
||||
http resource getter will attempt to search for any missing assets at.
|
||||
|
||||
|
||||
augmentations
|
||||
"""""""""""""
|
||||
As mentioned above this section should be used to configure augmentations, both
|
||||
to specify which should be enabled and disabled and also to provide any relevant
|
||||
configuration. The ``"augmentation"`` entry, if present, should be a list of
|
||||
augmentations that should be enabled (or if prefixed with a ``~``, disabled).
|
||||
|
||||
.. note:: While augmentations can be enabled and disabled on a per workload
|
||||
basis, they cannot yet be re-configured part way through a run and the
|
||||
configuration provided as part of an agenda config section or separate
|
||||
config file will be used for all jobs in a WA run.
|
||||
|
||||
workloads
|
||||
"""""""""
|
||||
In addition to configuring individual workloads both in the ``workloads`` and
|
||||
``sections`` entries you can also provide configuration at this level which will
|
||||
apply globally in the same style mentioned below. Any configuration provided
|
||||
here will be overridden if specified again in subsequent sections.
|
||||
|
||||
|
||||
workloads
|
||||
^^^^^^^^^
|
||||
|
||||
Here you can specify a list of workloads to be ran. If you wish to run a
|
||||
workload with all default values then you can specify the workload name directly
|
||||
as an entry, otherwise a dict mapping should be provided. Any settings provided
|
||||
here will be the most specific and therefore override any other more generalised
|
||||
configuration for that particular workload spec. The valid entries are as
|
||||
follows:
|
||||
|
||||
- :confval:`workload_name` (Mandatory) - The name of the workload to be ran
|
||||
- :confval:`iterations` - Specify how many iterations the workload should be ran
|
||||
- :confval:`label` - Similar to IDs but do not have the uniqueness restriction.
|
||||
If specified, labels will be used by some output processors instead of (or in
|
||||
addition to) the workload name. For example, the csv output processor will put
|
||||
the label in the "workload" column of the CSV file.
|
||||
- :confval:`augmentations` - The instruments and output processors to enable (or
|
||||
disabled using a ~) during this workload.
|
||||
- :confval:`classifiers` Classifiers allow you to tag metrics from this workload
|
||||
spec which are often used to help identify what runtime parameters were used
|
||||
when post processing results.
|
||||
- :confval:`workload_parameters` [*workload_params*] - Any parameters to
|
||||
configure that particular workload in a dict form.
|
||||
|
||||
.. note:: You can see available parameters for a given workload with the
|
||||
:ref:`show command <show-command>`.
|
||||
|
||||
- :confval:`runtime_parameters` [*runtime_parms*] - A dict mapping of any
|
||||
runtime parameters that should be set for the device for that particular
|
||||
workload. For available parameters please see :ref:`runtime parameters
|
||||
<runtime-parameters>`.
|
||||
|
||||
.. note:: Unless specified elsewhere these configurations will not be
|
||||
undone once the workload has finished. I.e. if the frequency of a
|
||||
core is changed it will remain at that frequency until otherwise
|
||||
changed.
|
||||
|
||||
.. note:: There is also a shorter ``params`` alias available, however this alias will be
|
||||
interpreted differently depending on whether it is used in workload
|
||||
entry, in which case it will be interpreted as ``workload_params``, or
|
||||
at global config or section (see below) level, in which case it will
|
||||
be interpreted as ``runtime_params``.
|
||||
|
||||
|
||||
sections
|
||||
^^^^^^^^
|
||||
|
||||
Sections are used for for grouping sets of configuration together in order to
|
||||
reduce the need for duplicated configuration (for more information please see
|
||||
:ref:`here <sections>`). Each section specified will be applied for each entry
|
||||
in the ``workloads`` section. The valid configuration entries are the same
|
||||
as the ``"workloads"`` section as mentioned above, except you can
|
||||
additionally specify a "workloads" entry which can be provided with the same
|
||||
configuration entries as the ``"workloads"`` top level entry.
|
||||
102
doc/source/user_information/user_reference/configuration.rst
Normal file
102
doc/source/user_information/user_reference/configuration.rst
Normal file
@@ -0,0 +1,102 @@
|
||||
.. _configuration-specification:
|
||||
|
||||
|
||||
Configuration
|
||||
=============
|
||||
|
||||
.. include:: user_information/user_reference/agenda.rst
|
||||
|
||||
---------------------
|
||||
|
||||
.. _run-configuration:
|
||||
|
||||
Run Configuration
|
||||
------------------
|
||||
In addition to specifying run execution parameters through an agenda, the
|
||||
behaviour of WA can be modified through configuration file(s). The default
|
||||
configuration file is ``~/.workload_automation/config.yaml`` (the location can
|
||||
be changed by setting ``WA_USER_DIRECTORY`` environment variable, see
|
||||
:ref:`envvars` section below). This file will be created when you first run WA
|
||||
if it does not already exist. This file must always exist and will always be
|
||||
loaded. You can add to or override the contents of that file on invocation of
|
||||
Workload Automation by specifying an additional configuration file using
|
||||
``--config`` option. Variables with specific names will be picked up by the
|
||||
framework and used to modify the behaviour of Workload automation.
|
||||
|
||||
---------------------
|
||||
|
||||
.. _available_settings:
|
||||
|
||||
.. include:: run_config/Run_Configuration.rst
|
||||
|
||||
---------------------
|
||||
|
||||
.. _meta-configuration:
|
||||
|
||||
Meta Configuration
|
||||
------------------
|
||||
|
||||
There are also a couple of settings are used to provide additional metadata
|
||||
for a run. These may get picked up by instruments or output processors to
|
||||
attach context to results.
|
||||
|
||||
.. include:: run_config/Meta_Configuration.rst
|
||||
|
||||
---------------------
|
||||
|
||||
.. _envvars:
|
||||
|
||||
Environment Variables
|
||||
---------------------
|
||||
|
||||
In addition to standard configuration described above, WA behaviour can be
|
||||
altered through environment variables. These can determine where WA looks for
|
||||
various assets when it starts.
|
||||
|
||||
.. confval:: WA_USER_DIRECTORY
|
||||
|
||||
This is the location WA will look for config.yaml, plugins, dependencies,
|
||||
and it will also be used for local caches, etc. If this variable is not set,
|
||||
the default location is ``~/.workload_automation`` (this is created when WA
|
||||
is installed).
|
||||
|
||||
.. note:: This location **must** be writable by the user who runs WA.
|
||||
|
||||
|
||||
.. confval:: WA_LOG_BUFFER_CAPACITY
|
||||
|
||||
Specifies the capacity (in log records) for the early log handler which is
|
||||
used to buffer log records until a log file becomes available. If the is not
|
||||
set, the default value of ``1000`` will be used. This should sufficient for
|
||||
most scenarios, however this may need to be increased, e.g. if plugin loader
|
||||
scans a very large number of locations; this may also be set to a lower
|
||||
value to reduce WA's memory footprint on memory-constrained hosts.
|
||||
|
||||
---------------------
|
||||
|
||||
.. include:: user_information/user_reference/runtime_parameters.rst
|
||||
|
||||
---------------------
|
||||
|
||||
.. _config-merging:
|
||||
|
||||
Configuration Merging
|
||||
---------------------
|
||||
WA configuration can come from various sources of increasing priority, as well
|
||||
as being specified in a generic and specific manner. For example WA's global
|
||||
config file would be considered the least specific vs the parameters of a
|
||||
workload in an agenda which would be the most specific. WA has two rules for the
|
||||
priority of configuration:
|
||||
|
||||
- Configuration from higher priority sources overrides configuration from
|
||||
lower priority sources.
|
||||
- More specific configuration overrides less specific configuration.
|
||||
|
||||
There is a situation where these two rules come into conflict. When a generic
|
||||
configuration is given in config source of high priority and a specific
|
||||
configuration is given in a config source of lower priority. In this situation
|
||||
it is not possible to know the end users intention and WA will error.
|
||||
|
||||
This functionality allows for defaults for plugins, targets etc. to be
|
||||
configured at a global level and then seamless overridden without the need to
|
||||
remove the high level configuration.
|
||||
349
doc/source/user_information/user_reference/invocation.rst
Normal file
349
doc/source/user_information/user_reference/invocation.rst
Normal file
@@ -0,0 +1,349 @@
|
||||
.. _invocation:
|
||||
|
||||
Commands
|
||||
========
|
||||
|
||||
Installing the wa package will add ``wa`` command to your system,
|
||||
which you can run from anywhere. This has a number of sub-commands, which can
|
||||
be viewed by executing ::
|
||||
|
||||
wa -h
|
||||
|
||||
Individual sub-commands are discussed in detail below.
|
||||
|
||||
.. _run-command:
|
||||
|
||||
Run
|
||||
---
|
||||
|
||||
The most common sub-command you will use is ``run``. This will run the specified
|
||||
workload(s) and process its resulting output. This takes a single mandatory
|
||||
argument which specifies what you want WA to run. This could be either a workload
|
||||
name, or a path to an agenda" file that allows to specify multiple workloads as
|
||||
well as a lot additional configuration (see :ref:`agenda` section for details).
|
||||
Executing ::
|
||||
|
||||
wa run -h
|
||||
|
||||
Will display help for this subcommand that will look something like this:
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
usage: wa run [-h] [-c CONFIG] [-v] [--version] [-d DIR] [-f] [-i ID]
|
||||
[--disable INSTRUMENT]
|
||||
AGENDA
|
||||
|
||||
Execute automated workloads on a remote device and process the resulting
|
||||
output.
|
||||
|
||||
positional arguments:
|
||||
AGENDA Agenda for this workload automation run. This defines
|
||||
which workloads will be executed, how many times, with
|
||||
which tunables, etc. See example agendas in
|
||||
/usr/local/lib/python2.7/dist-packages/wa for an
|
||||
example of how this file should be structured.
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
-c CONFIG, --config CONFIG
|
||||
specify an additional config.yaml
|
||||
-v, --verbose The scripts will produce verbose output.
|
||||
--version show program's version number and exit
|
||||
-d DIR, --output-directory DIR
|
||||
Specify a directory where the output will be
|
||||
generated. If the directory already exists, the script
|
||||
will abort unless -f option (see below) is used, in
|
||||
which case the contents of the directory will be
|
||||
overwritten. If this option is not specified, then
|
||||
wa_output will be used instead.
|
||||
-f, --force Overwrite output directory if it exists. By default,
|
||||
the script will abort in this situation to prevent
|
||||
accidental data loss.
|
||||
-i ID, --id ID Specify a workload spec ID from an agenda to run. If
|
||||
this is specified, only that particular spec will be
|
||||
run, and other workloads in the agenda will be
|
||||
ignored. This option may be used to specify multiple
|
||||
IDs.
|
||||
--disable INSTRUMENT Specify an instrument or output processor to disable
|
||||
from the command line. This equivalent to adding
|
||||
"~{metavar}" to the instruments list in the
|
||||
agenda. This can be used to temporarily disable a
|
||||
troublesome instrument for a particular run without
|
||||
introducing permanent change to the config (which one
|
||||
might then forget to revert). This option may be
|
||||
specified multiple times.
|
||||
|
||||
.. _list-command:
|
||||
|
||||
List
|
||||
----
|
||||
|
||||
This lists all plugins of a particular type. For example ::
|
||||
|
||||
wa list instruments
|
||||
|
||||
will list all instruments currently included in WA. The list will consist of
|
||||
plugin names and short descriptions of the functionality they offer e.g.
|
||||
|
||||
.. code-block:: none
|
||||
|
||||
#..
|
||||
cpufreq: Collects dynamic frequency (DVFS) settings before and after
|
||||
workload execution.
|
||||
dmesg: Collected dmesg output before and during the run.
|
||||
energy_measurement: This instrument is designed to be used as an interface to
|
||||
the various energy measurement instruments located
|
||||
in devlib.
|
||||
execution_time: Measure how long it took to execute the run() methods of
|
||||
a Workload.
|
||||
file_poller: Polls the given files at a set sample interval. The values
|
||||
are output in CSV format.
|
||||
fps: Measures Frames Per Second (FPS) and associated metrics for
|
||||
a workload.
|
||||
#..
|
||||
|
||||
|
||||
You can use the same syntax to quickly display information about ``commands``,
|
||||
``energy_instrument_backends``, ``instruments``, ``output_processors``, ``resource_getters``,
|
||||
``targets`` and ``workloads``
|
||||
|
||||
.. _show-command:
|
||||
|
||||
Show
|
||||
----
|
||||
|
||||
This will show detailed information about an plugin (workloads, targets,
|
||||
instruments etc.), including a full description and any relevant
|
||||
parameters/configuration that are available. For example executing ::
|
||||
|
||||
wa show benchmarkpi
|
||||
|
||||
will produce something like: ::
|
||||
|
||||
|
||||
benchmarkpi
|
||||
-----------
|
||||
|
||||
Measures the time the target device takes to run and complete the Pi
|
||||
calculation algorithm.
|
||||
|
||||
http://androidbenchmark.com/howitworks.php
|
||||
|
||||
from the website:
|
||||
|
||||
The whole idea behind this application is to use the same Pi calculation
|
||||
algorithm on every Android Device and check how fast that process is.
|
||||
Better calculation times, conclude to faster Android devices. This way you
|
||||
can also check how lightweight your custom made Android build is. Or not.
|
||||
|
||||
As Pi is an irrational number, Benchmark Pi does not calculate the actual Pi
|
||||
number, but an approximation near the first digits of Pi over the same
|
||||
calculation circles the algorithms needs.
|
||||
|
||||
So, the number you are getting in milliseconds is the time your mobile device
|
||||
takes to run and complete the Pi calculation algorithm resulting in a
|
||||
approximation of the first Pi digits.
|
||||
|
||||
parameters
|
||||
~~~~~~~~~~
|
||||
|
||||
cleanup_assets : boolean
|
||||
If ``True``, if assets are deployed as part of the workload they
|
||||
will be removed again from the device as part of finalize.
|
||||
|
||||
default: ``True``
|
||||
|
||||
package_name : str
|
||||
The package name that can be used to specify
|
||||
the workload apk to use.
|
||||
|
||||
install_timeout : integer
|
||||
Timeout for the installation of the apk.
|
||||
|
||||
constraint: ``value > 0``
|
||||
|
||||
default: ``300``
|
||||
|
||||
version : str
|
||||
The version of the package to be used.
|
||||
|
||||
variant : str
|
||||
The variant of the package to be used.
|
||||
|
||||
strict : boolean
|
||||
Whether to throw an error if the specified package cannot be found
|
||||
on host.
|
||||
|
||||
force_install : boolean
|
||||
Always re-install the APK, even if matching version is found already installed
|
||||
on the device.
|
||||
|
||||
uninstall : boolean
|
||||
If ``True``, will uninstall workload's APK as part of teardown.'
|
||||
|
||||
exact_abi : boolean
|
||||
If ``True``, workload will check that the APK matches the target
|
||||
device ABI, otherwise any suitable APK found will be used.
|
||||
|
||||
markers_enabled : boolean
|
||||
If set to ``True``, workloads will insert markers into logs
|
||||
at various points during execution. These markers may be used
|
||||
by other plugins or post-processing scripts to provide
|
||||
measurements or statistics for specific parts of the workload
|
||||
execution.
|
||||
|
||||
.. note:: You can also use this command to view global settings by using ``wa show settings``
|
||||
|
||||
|
||||
.. _create-command:
|
||||
|
||||
Create
|
||||
------
|
||||
|
||||
This aids in the creation of new WA-related objects for example agendas and workloads.
|
||||
For more detailed information on creating workloads please see the
|
||||
:ref:`adding a workload <adding-a-workload>` section for more details.
|
||||
|
||||
As an example to create an agenda that will run the dhrystone and memcpy workloads
|
||||
that will use the status and hwmon augmentations, run each test 3 times and save
|
||||
into the file ``my_agenda.yaml`` the following command can be used::
|
||||
|
||||
wa create agenda dhrystone memcpy status hwmon -i 3 -o my_agenda.yaml
|
||||
|
||||
Which will produce something like::
|
||||
|
||||
config:
|
||||
augmentations:
|
||||
- status
|
||||
- hwmon
|
||||
status: {}
|
||||
hwmon: {}
|
||||
iterations: 3
|
||||
workloads:
|
||||
- name: dhrystone
|
||||
params:
|
||||
cleanup_assets: true
|
||||
delay: 0
|
||||
duration: 0
|
||||
mloops: 0
|
||||
taskset_mask: 0
|
||||
threads: 4
|
||||
- name: memcpy
|
||||
params:
|
||||
buffer_size: 5242880
|
||||
cleanup_assets: true
|
||||
cpus: null
|
||||
iterations: 1000
|
||||
|
||||
This will be populated with default values which can then be customised for the
|
||||
particular use case.
|
||||
|
||||
.. _process-command:
|
||||
|
||||
Process
|
||||
--------
|
||||
|
||||
This command allows for output processors to be ran on data that was produced by
|
||||
a previous run.
|
||||
|
||||
There are 2 ways of specifying which processors you wish to use, either passing
|
||||
them directly as arguments to the process command with the ``--processor``
|
||||
argument or by providing an additional config file with the ``--config``
|
||||
argument. Please note that by default the process command will not rerun
|
||||
processors that have already been ran during the run, in order to force a rerun
|
||||
of the processors you can specific the ``--force`` argument.
|
||||
|
||||
Additionally if you have a directory containing multiple run directories you can
|
||||
specify the ``--recursive`` argument which will cause WA to walk the specified
|
||||
directory processing all the WA output sub-directories individually.
|
||||
|
||||
|
||||
As an example if we had performed multiple experiments and have the various WA
|
||||
output directories in our ``my_experiments`` directory, and we now want to process
|
||||
the outputs with a tool that only supports CSV files. We can easily generate CSV
|
||||
files for all the runs contained in our directory using the CSV processor by
|
||||
using the following command::
|
||||
|
||||
wa process -r -p csv my_experiments
|
||||
|
||||
|
||||
.. _record_command:
|
||||
|
||||
Record
|
||||
------
|
||||
|
||||
This command simplifies the process of recording revent files. It will
|
||||
automatically deploy revent and has options to automatically open apps and
|
||||
record specified stages of a workload. Revent allows you to record raw inputs
|
||||
such as screen swipes or button presses. This can be useful for recording inputs
|
||||
for workloads such as games that don't have XML UI layouts that can be used with
|
||||
UIAutomator. As a drawback from this, revent recordings are specific to the
|
||||
device type they were recorded on. WA uses two parts to the names of revent
|
||||
recordings in the format, ``{device_name}.{suffix}.revent``. - device_name can
|
||||
either be specified manually with the ``-d`` argument or it can be automatically
|
||||
determined. On Android device it will be obtained from ``build.prop``, on Linux
|
||||
devices it is obtained from ``/proc/device-tree/model``. - suffix is used by WA
|
||||
to determine which part of the app execution the recording is for, currently
|
||||
these are either ``setup``, ``run``, ``extract_results`` or ``teardown``. All
|
||||
stages except ``run`` are optional for playback and to specify which stages
|
||||
should be recorded the ``-s``, ``-r``, ``-e`` or ``-t`` arguments respectively,
|
||||
or optionally ``-a`` to indicate all stages should be recorded.
|
||||
|
||||
|
||||
The full set of options for this command are::
|
||||
|
||||
usage: wa record [-h] [-c CONFIG] [-v] [--version] [-d DEVICE] [-o FILE] [-s]
|
||||
[-r] [-e] [-t] [-a] [-C] [-p PACKAGE | -w WORKLOAD]
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
-c CONFIG, --config CONFIG
|
||||
specify an additional config.yaml
|
||||
-v, --verbose The scripts will produce verbose output.
|
||||
--version show program's version number and exit
|
||||
-d DEVICE, --device DEVICE
|
||||
Specify the device on which to run. This will take
|
||||
precedence over the device (if any) specified in
|
||||
configuration.
|
||||
-o FILE, --output FILE
|
||||
Specify the output file
|
||||
-s, --setup Record a recording for setup stage
|
||||
-r, --run Record a recording for run stage
|
||||
-e, --extract_results Record a recording for extract_results stage
|
||||
-t, --teardown Record a recording for teardown stage
|
||||
-a, --all Record recordings for available stages
|
||||
-C, --clear Clear app cache before launching it
|
||||
-p PACKAGE, --package PACKAGE
|
||||
Android package to launch before recording
|
||||
-w WORKLOAD, --workload WORKLOAD
|
||||
Name of a revent workload (mostly games)
|
||||
|
||||
For more information please see :ref:`Revent Recording <revent-recording>`.
|
||||
|
||||
.. _replay-command:
|
||||
|
||||
Replay
|
||||
------
|
||||
|
||||
Alongside ``record`` wa also has a command to playback a single recorded revent
|
||||
file. It behaves similar to the ``record`` command taking a subset of the same
|
||||
options allowing you to automatically launch a package on the device ::
|
||||
|
||||
usage: wa replay [-h] [-c CONFIG] [-v] [--debug] [--version] [-p PACKAGE] [-C]
|
||||
revent
|
||||
|
||||
positional arguments:
|
||||
revent The name of the file to replay
|
||||
|
||||
optional arguments:
|
||||
-h, --help show this help message and exit
|
||||
-c CONFIG, --config CONFIG
|
||||
specify an additional config.py
|
||||
-v, --verbose The scripts will produce verbose output.
|
||||
--debug Enable debug mode. Note: this implies --verbose.
|
||||
--version show program's version number and exit
|
||||
-p PACKAGE, --package PACKAGE
|
||||
Package to launch before recording
|
||||
-C, --clear Clear app cache before launching it
|
||||
|
||||
For more information please see :ref:`Revent Replaying <revent_replaying>`.
|
||||
132
doc/source/user_information/user_reference/output_directory.rst
Normal file
132
doc/source/user_information/user_reference/output_directory.rst
Normal file
@@ -0,0 +1,132 @@
|
||||
.. _output_directory_structure:
|
||||
|
||||
Output Directory Structure
|
||||
==========================
|
||||
|
||||
This is an overview of WA output directory structure.
|
||||
|
||||
.. note:: In addition to files and subdirectories described here,
|
||||
other content may present in the output directory for
|
||||
a run, depending on the enabled augmentations.
|
||||
|
||||
Overview
|
||||
--------
|
||||
|
||||
The output directory will contain a subdirectory for every job iteration that
|
||||
was run, as well as some additional entries. The following diagram illustrates
|
||||
the typical structure of WA output directory::
|
||||
|
||||
wa_output/
|
||||
├── __meta/
|
||||
│ ├── config.json
|
||||
│ ├── jobs.json
|
||||
│ ├── raw_config
|
||||
│ │ ├── cfg0-config.yaml
|
||||
│ │ └── agenda.yaml
|
||||
│ ├── run_info.json
|
||||
│ └── target_info.json
|
||||
├── __failed/
|
||||
│ └── wk1-dhrystone-1-attempt1
|
||||
├── wk1-dhrystone-1/
|
||||
│ └── result.json
|
||||
├── wk1-dhrystone-2/
|
||||
│ └── result.json
|
||||
├── wk2-memcpy-1/
|
||||
│ └── result.json
|
||||
├── wk2-memcpy-2/
|
||||
│ └── result.json
|
||||
├── result.json
|
||||
└── run.log
|
||||
|
||||
This is the directory structure that would be generated after running two
|
||||
iterations each of ``dhrystone`` and ``memcpy`` workloads with no augmentations
|
||||
enabled, and with the first attempt at the first iteration of dhrystone having
|
||||
failed.
|
||||
|
||||
Output Directory Entries
|
||||
------------------------
|
||||
|
||||
result.json
|
||||
Contains a JSON structure describing the result of the execution,
|
||||
including collected metrics and artifacts. There will be one for each
|
||||
job execution, and one for the overall run. The run ``result.json`` will
|
||||
only contain metrics/artifacts for the run as a whole, and will not
|
||||
contain results for individual jobs.
|
||||
|
||||
You typically would not access ``result.json`` files directly. Instead
|
||||
you would either enable augmentations to format the results in easier to
|
||||
manage form (such as CSV table), or use :ref:`output_processing_api` to
|
||||
access the results from scripts.
|
||||
|
||||
|
||||
run.log
|
||||
This is a log of everything that happened during the run, including all
|
||||
interactions with the target, and all the decisions made by the
|
||||
framework. The output is equivalent to what you would see on the console
|
||||
when running with ``--verbose`` option.
|
||||
|
||||
.. note:: WA source contains a syntax file for Vim that will color the
|
||||
initial part of each log line, in a similar way to what you
|
||||
see on the console. This may be useful for quickly spotting
|
||||
error and warning messages when scrolling through the log.
|
||||
|
||||
https://github.com/ARM-software/workload-automation/blob/next/extras/walog.vim
|
||||
|
||||
__meta
|
||||
This directory contains configuration and run metadata. See
|
||||
:ref:`config_and_meta` below for details.
|
||||
|
||||
__failed
|
||||
This directory will only be present if one or more job executions has
|
||||
failed and were re-run. This directory contains output directories for
|
||||
the failed attempts.
|
||||
|
||||
job execution output subdirectory
|
||||
Each subdirectory will be named ``<job id>_<workload label>_<iteration
|
||||
number>``, and will, at minimum, contain a ``result.json`` (see above).
|
||||
Additionally, it may contain raw output from the workload, and any
|
||||
additional artifacts (e.g. traces) generated by augmentations. Finally,
|
||||
if workload execution has failed, WA may gather some additional logging
|
||||
(such as the UI state at the time of failure) and place it here.
|
||||
|
||||
|
||||
.. _config_and_meta:
|
||||
|
||||
Configuration and Metadata
|
||||
--------------------------
|
||||
|
||||
As stated above, the ``__meta`` directory contains run configuration and
|
||||
metadata. Typically, you would not access these files directly, but would use
|
||||
the :ref:`output_processing_api` to query the metadata.
|
||||
|
||||
For more details about WA configuration see :ref:`configuration-specification`.
|
||||
|
||||
config.json
|
||||
Contains the overall run configuration, such as target interface
|
||||
configuration, and job execution order, and various "meta-configuration"
|
||||
settings, such as default output path, verbosity level, and logging
|
||||
formatting.
|
||||
|
||||
jobs.json
|
||||
Final configuration for all jobs, including enabled augmentations,
|
||||
workload and runtime parameters, etc.
|
||||
|
||||
raw_config
|
||||
This directory contains copies of config file(s) and the agenda that
|
||||
were parsed in order to generate configuration for this run. Each config
|
||||
file is prefixed with ``cfg<N>-``, where ``<N>`` is the number
|
||||
indicating the order (with respect to the other other config files) in
|
||||
which it was parsed, e.g. ``cfg0-config.yaml`` is always a copy of
|
||||
``$WA_USER_DIRECTORY/config.yaml``. The one file without a prefix is the
|
||||
agenda.
|
||||
|
||||
run_info.json
|
||||
Run metadata, e.g. duration, start/end timestamps and duration.
|
||||
|
||||
target_info.json
|
||||
Extensive information about the target. This includes information about
|
||||
the target's CPUS configuration, kernel and userspace versions, etc. The
|
||||
exact content will vary depending on the target type (Android vs Linux)
|
||||
and what could accessed on a particular device (e.g. if
|
||||
``/proc/config.gz`` exists on the target, the kernel config will be
|
||||
included).
|
||||
@@ -0,0 +1,237 @@
|
||||
.. _runtime-parameters:
|
||||
|
||||
Runtime Parameters
|
||||
------------------
|
||||
|
||||
.. contents:: Contents
|
||||
:local:
|
||||
|
||||
Runtime parameters are options that can be specified to automatically configure
|
||||
device at runtime. They can be specified at the global level in the agenda or
|
||||
for individual workloads.
|
||||
|
||||
Example
|
||||
^^^^^^^
|
||||
Say we want to perform an experiment on an Android big.LITTLE devices to compare
|
||||
the power consumption between the big and LITTLE clusters running the dhrystone
|
||||
and benchmarkpi workloads. Assuming we have additional instrumentation active
|
||||
for this device that can measure the power the device is consuming, to reduce
|
||||
external factors we want to ensure that the device is in airplane mode turned on
|
||||
for all our tests and the screen is off only for our dhrystone run. We will then
|
||||
run 2 :ref:`sections <sections>` will each enable a single cluster on the
|
||||
device, set the cores to their maximum frequency and disable all available idle
|
||||
states.
|
||||
|
||||
.. code-block:: yaml
|
||||
|
||||
config:
|
||||
runtime_parameters:
|
||||
airplane_mode: true
|
||||
#..
|
||||
workloads:
|
||||
- name: dhrystone
|
||||
iterations: 1
|
||||
runtime_parameters:
|
||||
screen_on: false
|
||||
- name: benchmarkpi
|
||||
iterations: 1
|
||||
sections:
|
||||
- id: LITTLES
|
||||
runtime_parameters:
|
||||
num_little_cores: 4
|
||||
little_governor: userspace
|
||||
little_frequency: max
|
||||
little_idle_states: none
|
||||
num_big_cores: 0
|
||||
|
||||
- id: BIGS
|
||||
runtime_parameters:
|
||||
num_big_cores: 4
|
||||
big_governor: userspace
|
||||
big_frequency: max
|
||||
big_idle_states: none
|
||||
num_little_cores: 0
|
||||
|
||||
|
||||
HotPlug
|
||||
^^^^^^^
|
||||
|
||||
Parameters:
|
||||
|
||||
:num_cores: An ``int`` that specifies the total number of cpu cores to be online.
|
||||
|
||||
:num_<core_name>_cores: An ``int`` that specifies the total number of that particular core
|
||||
to be online, the target will be queried and if the core_names can
|
||||
be determine a parameter for each of the unique core names will be
|
||||
available.
|
||||
|
||||
:cpu<core_no>_online: A ``boolean`` that specifies whether that particular cpu, e.g. cpu0 will
|
||||
be online.
|
||||
|
||||
If big.LITTLE is detected for the device and additional 2 parameters are available:
|
||||
|
||||
:num_big_cores: An ``int`` that specifies the total number of `big` cpu cores to be online.
|
||||
|
||||
:num_little_cores: An ``int`` that specifies the total number of `little` cpu cores to be online.
|
||||
|
||||
|
||||
|
||||
.. Note:: Please note that if the device in question is operating its own dynamic
|
||||
hotplugging then WA may be unable to set the CPU state or will be overridden.
|
||||
Unfortunately the method of disabling dynamic hot plugging will vary from
|
||||
device to device.
|
||||
|
||||
|
||||
CPUFreq
|
||||
^^^^^^^
|
||||
|
||||
:frequency: An ``int`` that can be used to specify a frequency for all cores if there are common frequencies available.
|
||||
|
||||
.. Note:: When settings the frequency, if the governor is not set to userspace then WA will attempt to set the maximum
|
||||
and minimum frequencies to mimic the desired behaviour.
|
||||
|
||||
:max_frequency: An ``int`` that can be used to specify a maximum frequency for all cores if there are common frequencies available.
|
||||
|
||||
:min_frequency: An ``int`` that can be used to specify a minimum frequency for all cores if there are common frequencies available.
|
||||
|
||||
:governor: A ``string`` that can be used to specify the governor for all cores if there are common governors available.
|
||||
|
||||
:governor: A ``string`` that can be used to specify the governor for all cores if there are common governors available.
|
||||
|
||||
:governor_tunable: A ``dict`` that can be used to specify governor
|
||||
tunables for all cores, unlike the other common parameters these are not
|
||||
validated at the beginning of the run therefore incorrect values will cause
|
||||
an error during runtime.
|
||||
|
||||
:<core_name>_frequency: An ``int`` that can be used to specify a frequency for cores of a particular type e.g. 'A72'.
|
||||
|
||||
:<core_name>_max_frequency: An ``int`` that can be used to specify a maximum frequency for cores of a particular type e.g. 'A72'.
|
||||
|
||||
:<core_name>_min_frequency: An ``int`` that can be used to specify a minimum frequency for cores of a particular type e.g. 'A72'.
|
||||
|
||||
:<core_name>_governor: A ``string`` that can be used to specify the governor for cores of a particular type e.g. 'A72'.
|
||||
|
||||
:<core_name>_governor: A ``string`` that can be used to specify the governor for cores of a particular type e.g. 'A72'.
|
||||
|
||||
:<core_name>_governor_tunable: A ``dict`` that can be used to specify governor
|
||||
tunables for cores of a particular type e.g. 'A72', these are not
|
||||
validated at the beginning of the run therefore incorrect values will cause
|
||||
an error during runtime.
|
||||
|
||||
|
||||
:cpu<no>_frequency: An ``int`` that can be used to specify a frequency for a particular core e.g. 'cpu0'.
|
||||
|
||||
:cpu<no>_max_frequency: An ``int`` that can be used to specify a maximum frequency for a particular core e.g. 'cpu0'.
|
||||
|
||||
:cpu<no>_min_frequency: An ``int`` that can be used to specify a minimum frequency for a particular core e.g. 'cpu0'.
|
||||
|
||||
:cpu<no>_governor: A ``string`` that can be used to specify the governor for a particular core e.g. 'cpu0'.
|
||||
|
||||
:cpu<no>_governor: A ``string`` that can be used to specify the governor for a particular core e.g. 'cpu0'.
|
||||
|
||||
:cpu<no>_governor_tunable: A ``dict`` that can be used to specify governor
|
||||
tunables for a particular core e.g. 'cpu0', these are not
|
||||
validated at the beginning of the run therefore incorrect values will cause
|
||||
an error during runtime.
|
||||
|
||||
|
||||
If big.LITTLE is detected for the device an additional set of parameters are available:
|
||||
|
||||
:big_frequency: An ``int`` that can be used to specify a frequency for the big cores.
|
||||
|
||||
:big_max_frequency: An ``int`` that can be used to specify a maximum frequency for the big cores.
|
||||
|
||||
:big_min_frequency: An ``int`` that can be used to specify a minimum frequency for the big cores.
|
||||
|
||||
:big_governor: A ``string`` that can be used to specify the governor for the big cores.
|
||||
|
||||
:big_governor: A ``string`` that can be used to specify the governor for the big cores.
|
||||
|
||||
:big_governor_tunable: A ``dict`` that can be used to specify governor
|
||||
tunables for the big cores, these are not
|
||||
validated at the beginning of the run therefore incorrect values will cause
|
||||
an error during runtime.
|
||||
|
||||
:little_frequency: An ``int`` that can be used to specify a frequency for the little cores.
|
||||
|
||||
:little_max_frequency: An ``int`` that can be used to specify a maximum frequency for the little cores.
|
||||
|
||||
:little_min_frequency: An ``int`` that can be used to specify a minimum frequency for the little cores.
|
||||
|
||||
:little_governor: A ``string`` that can be used to specify the governor for the little cores.
|
||||
|
||||
:little_governor: A ``string`` that can be used to specify the governor for the little cores.
|
||||
|
||||
:little_governor_tunable: A ``dict`` that can be used to specify governor
|
||||
tunables for the little cores, these are not
|
||||
validated at the beginning of the run therefore incorrect values will cause
|
||||
an error during runtime.
|
||||
|
||||
|
||||
CPUIdle
|
||||
^^^^^^^
|
||||
|
||||
:idle_states: A ``string`` or list of strings which can be used to specify what
|
||||
idles states should be enabled for all cores if there are common
|
||||
idle states available. 'all' and 'none' are also valid entries as a
|
||||
shorthand
|
||||
|
||||
:<core_name>_idle_states: A ``string`` or list of strings which can be used to
|
||||
specify what idles states should be enabled for cores of a particular type
|
||||
e.g. 'A72'. 'all' and 'none' are also valid entries as a shorthand
|
||||
:cpu<no>_idle_states: A ``string`` or list of strings which can be used to
|
||||
specify what idles states should be enabled for a particular core e.g.
|
||||
'cpu0'. 'all' and 'none' are also valid entries as a shorthand
|
||||
|
||||
If big.LITTLE is detected for the device and additional set of parameters are available:
|
||||
|
||||
:big_idle_states: A ``string`` or list of strings which can be used to specify
|
||||
what idles states should be enabled for the big cores. 'all' and 'none' are
|
||||
also valid entries as a shorthand
|
||||
:little_idle_states: A ``string`` or list of strings which can be used to
|
||||
specify what idles states should be enabled for the little cores. 'all' and
|
||||
'none' are also valid entries as a shorthand.
|
||||
|
||||
|
||||
Android Specific Runtime Parameters
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
:brightness: An ``int`` between 0 and 255 (inclusive) to specify the brightness
|
||||
the screen should be set to. Defaults to ``127``.
|
||||
|
||||
:airplane_mode: A ``boolean`` to specify whether airplane mode should be
|
||||
enabled for the device.
|
||||
|
||||
:rotation: A ``String`` to specify the screen orientation for the device. Valid
|
||||
entries are ``NATURAL``, ``LEFT``, ``INVERTED``, ``RIGHT``.
|
||||
|
||||
:screen_on: A ``boolean`` to specify whether the devices screen should be
|
||||
turned on. Defaults to ``True``.
|
||||
|
||||
.. _setting-sysfiles:
|
||||
|
||||
Setting Sysfiles
|
||||
^^^^^^^^^^^^^^^^
|
||||
In order to perform additional configuration of a target the ``sysfile_values``
|
||||
runtime parameter can be used. The value for this parameter is a mapping (an
|
||||
associative array, in YAML) of file paths onto values that should be written
|
||||
into those files. ``sysfile_values`` is the only runtime parameter that is
|
||||
available for any (Linux) device. Other runtime parameters will depend on the
|
||||
specifics of the device used (e.g. its CPU cores configuration) as detailed
|
||||
above.
|
||||
|
||||
.. note:: By default WA will attempt to verify that the any sysfile values were
|
||||
written correctly by reading the node back and comparing the two values. If
|
||||
you do not wish this check to happen, for example the node you are writing to
|
||||
is write only, you can append an ``!`` to the file path to disable this
|
||||
verification.
|
||||
|
||||
For example the following configuration could be used to enable and verify that cpu0
|
||||
is online, however will not attempt to check that its governor have been set to
|
||||
userspace::
|
||||
|
||||
- name: dhrystone
|
||||
runtime_params:
|
||||
sysfile_values:
|
||||
/sys/devices/system/cpu/cpu0/online: 1
|
||||
/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor!: userspace
|
||||
Reference in New Issue
Block a user