sascha/workload-automation
1
0
Fork 0
mirror of https://github.com/ARM-software/workload-automation.git synced 2025-10-15 00:04:09 +01:00
Code Issues Releases Wiki Activity

Documentation: Update WA3 Documentation

Browse Source 
Update the documentation and build system for producing documentation for
WA3 with support for automatic building on readthedocs.

Note: This is currently a WIP.
This commit is contained in:
Marc Bonnici
2018-01-19 10:49:56 +00:00
committed by setrofim
parent 98bed3822a
commit bc87eacde2
47 changed files with 4803 additions and 3961 deletions
Download Patch File Download Diff File Expand all files Collapse all files

54
doc/source/user_reference/configuration.rst Normal file
View File

@@ -0,0 +1,54 @@
.. _configuration-specification:
Configuration
=============
Run Configuration
------------------
In addition to specifying run execution parameters through an agenda, the
behaviour of WA can be modified through configuration file(s). The default
configuration file is ``~/.workload_automation/config.yaml`` (the location can
be changed by setting ``WA_USER_DIRECTORY`` environment variable, see
:ref:`envvars` section below). This file will be created when you first run WA
if it does not already exist. This file must always exist and will always be
loaded. You can add to or override the contents of that file on invocation of
Workload Automation by specifying an additional configuration file using
``--config`` option. Variables with specific names will be picked up by the
framework and used to modify the behaviour of Workload automation.
.. _available_settings:
.. include:: run_config/Run_Configuration.rst
Meta Configuration
------------------
There are also a couple of settings are used to provide additional metadata
for a run. These may get picked up by instruments or output processors to
attach context to results.
.. include:: run_config/Meta_Configuration.rst
.. _envvars:
Environment Variables
---------------------
In addition to standard configuration described above, WA behaviour can be
altered through environment variables. These can determine where WA looks for
various assets when it starts.
.. confval:: WA_USER_DIRECTORY
This is the location WA will look for config.yaml, plugins, dependencies,
and it will also be used for local caches, etc. If this variable is not set,
the default location is ``~/.workload_automation`` (this is created when WA
is installed).
.. note:: This location **must** be writable by the user who runs WA.
.. include:: user_reference/runtime_parameters.rst

317
doc/source/user_reference/invocation.rst Normal file
View File

@@ -0,0 +1,317 @@
.. _invocation:
Commands
========
Installing the wa package will add ``wa`` command to your system,
which you can run from anywhere. This has a number of sub-commands, which can
be viewed by executing ::
wa -h
Individual sub-commands are discussed in detail below.
.. _run-command:
Run
---
The most common sub-command you will use is ``run``. This will run specified
workload(s) and process resulting output. This takes a single mandatory
argument that specifies what you want WA to run. This could be either a
workload name, or a path to an "agenda" file that allows to specify multiple
workloads as well as a lot additional configuration (see :ref:`agenda`
section for details). Executing ::
wa run -h
Will display help for this subcommand that will look something like this::
usage: wa run [-h] [-c CONFIG] [-v] [--version] [-d DIR] [-f] [-i ID]
[--disable INSTRUMENT]
AGENDA
Execute automated workloads on a remote device and process the resulting
output.
positional arguments:
AGENDA Agenda for this workload automation run. This defines
which workloads will be executed, how many times, with
which tunables, etc. See example agendas in
/usr/local/lib/python2.7/dist-packages/wa for an
example of how this file should be structured.
optional arguments:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
specify an additional config.yaml
-v, --verbose The scripts will produce verbose output.
--version show program's version number and exit
-d DIR, --output-directory DIR
Specify a directory where the output will be
generated. If the directory already exists, the script
will abort unless -f option (see below) is used, in
which case the contents of the directory will be
overwritten. If this option is not specified, then
wa_output will be used instead.
-f, --force Overwrite output directory if it exists. By default,
the script will abort in this situation to prevent
accidental data loss.
-i ID, --id ID Specify a workload spec ID from an agenda to run. If
this is specified, only that particular spec will be
run, and other workloads in the agenda will be
ignored. This option may be used to specify multiple
IDs.
--disable INSTRUMENT Specify an instrument or output processor to disable
from the command line. This equivalent to adding
"~{metavar}" to the instruments list in the
agenda. This can be used to temporarily disable a
troublesome instrument for a particular run without
introducing permanent change to the config (which one
might then forget to revert). This option may be
specified multiple times.
.. _list-command:
List
----
This lists all plugins of a particular type. For example ::
wa list instruments
will list all instruments currently included in WA. The list will consist of
plugin names and short descriptions of the functionality they offer e.g.
.. code-block:: none
#..
cpufreq: Collects dynamic frequency (DVFS) settings before and after
workload execution.
dmesg: Collected dmesg output before and during the run.
energy_measurement: This instrument is designed to be used as an interface to
the various energy measurement instruments located
in devlib.
execution_time: Measure how long it took to execute the run() methods of
a Workload.
file_poller: Polls the given files at a set sample interval. The values
are output in CSV format.
fps: Measures Frames Per Second (FPS) and associated metrics for
a workload.
#..
You can use the same syntax to quickly display information about ``commands``,
``energy_instrument_backends``, ``instruments``, ``output_processors``, ``resource_getters``,
``targets`` and ``workloads``
.. _show-command:
Show
----
This will show detailed information about an plugin, including more in-depth
description and any parameters/configuration that are available. For example
executing ::
wa show benchmarkpi
will produce something like: ::
benchmarkpi
-----------
Measures the time the target device takes to run and complete the Pi
calculation algorithm.
http://androidbenchmark.com/howitworks.php
from the website:
The whole idea behind this application is to use the same Pi calculation
algorithm on every Android Device and check how fast that proccess is.
Better calculation times, conclude to faster Android devices. This way you
can also check how lightweight your custom made Android build is. Or not.
As Pi is an irrational number, Benchmark Pi does not calculate the actual Pi
number, but an approximation near the first digits of Pi over the same
calculation circles the algorithms needs.
So, the number you are getting in miliseconds is the time your mobile device
takes to run and complete the Pi calculation algorithm resulting in a
approximation of the first Pi digits.
parameters
~~~~~~~~~~
cleanup_assets : boolean
If ``True``, if assets are deployed as part of the workload they
will be removed again from the device as part of finalize.
default: ``True``
package_name : str
The package name that can be used to specify
the workload apk to use.
install_timeout : integer
Timeout for the installation of the apk.
constraint: ``value > 0``
default: ``300``
version : str
The version of the package to be used.
variant : str
The variant of the package to be used.
strict : boolean
Whether to throw an error if the specified package cannot be found
on host.
force_install : boolean
Always re-install the APK, even if matching version is found already installed
on the device.
uninstall : boolean
If ``True``, will uninstall workload's APK as part of teardown.'
exact_abi : boolean
If ``True``, workload will check that the APK matches the target
device ABI, otherwise any suitable APK found will be used.
markers_enabled : boolean
If set to ``True``, workloads will insert markers into logs
at various points during execution. These markes may be used
by other plugins or post-processing scripts to provide
measurments or statistics for specific parts of the workload
execution.
.. _create-command:
Create
------
This aids in the creation of new WA-related objects for example agendas and workloads.
For more detailed information on creating workloads please see the
:ref:`adding a workload <adding-a-workload>` section for more details.
agendas:
As an example to create an agenda that will run the dhrystone and memcpy workloads
that will use the status and hwmon augumentations, run each test 3 times and save
into the file ``my_agenda.yaml`` the following command can be used::
wa create agenda dhrystone memcpy status hwmon -i 3 -o my_agenda.yaml
Which will produce something like::
config:
augmentations:
- status
- hwmon
status: {}
hwmon: {}
iterations: 3
workloads:
- name: dhrystone
params:
cleanup_assets: true
delay: 0
duration: 0
mloops: 0
taskset_mask: 0
threads: 4
- name: memcpy
params:
buffer_size: 5242880
cleanup_assets: true
cpus: null
iterations: 1000
This will be populated with default values which can then be customised for the
particular use case.
.. _record_command:
Record
------
This command simiplifies the process of recording revent files. It will
automatically deploy revent and has options to automatically open apps and
record specified stages of a workload. Revent allows you to record raw inputs
such as screen swipes or button presses. This can be useful for recording inputs
for workloads such as games that don't have XML UI layouts that can be used with
UIAutomator. As a drawback from this, revent recordings are specific to the
device type they were recorded on. WA uses two parts to the names of revent
recordings in the format, ``{device_name}.{suffix}.revent``. - device_name can
either be specified manually with the ``-d`` argument or it can be automatically
determined. On Android device it will be obtained from ``build.prop``, on Linux
devices it is obtained from ``/proc/device-tree/model``. - suffix is used by WA
to determine which part of the app execution the recording is for, currently
these are either ``setup``, ``run``, ``extract_results`` or ``teardown``. All
stages except ``run`` are optional for playback and to specify which stages
shoule be recorded the ``-s``, ``-r``, ``-e`` or ``-t`` arguments respectively,
or optionally ``-a`` to indicate all stages should be recorded.
The full set of options for this command are::
usage: wa record [-h] [-c CONFIG] [-v] [--version] [-d DEVICE] [-o FILE] [-s]
[-e] [-t] [-a] [-C] [-p PACKAGE | -w WORKLOAD]
optional arguments:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
specify an additional config.yaml
-v, --verbose The scripts will produce verbose output.
--version show program's version number and exit
-d DEVICE, --device DEVICE
Specify the device on which to run. This will take
precedence over the device (if any) specified in
configuration.
-o FILE, --output FILE
Specify the output file
-s, --setup Record a recording for setup stage
-e, --extract_results
Record a recording for extract_results stage
-t, --teardown Record a recording for teardown stage
-a, --all Record recordings for available stages
-C, --clear Clear app cache before launching it
-p PACKAGE, --package PACKAGE
Android package to launch before recording
-w WORKLOAD, --workload WORKLOAD
Name of a revent workload (mostly games)
For more information please see :ref:`Revent Recording <revent-recording>`.
.. _replay-command:
Replay
------
Along side ``record`` wa also has a command to playback a single recorded revent file.
It behaves similar to the ``record`` command taking a subset of the same options allowing you to automatically launch a package on the device ::
usage: wa replay [-h] [-c CONFIG] [-v] [--debug] [--version] [-p PACKAGE] [-C]
revent
positional arguments:
revent The name of the file to replay
optional arguments:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
specify an additional config.py
-v, --verbose The scripts will produce verbose output.
--debug Enable debug mode. Note: this implies --verbose.
--version show program's version number and exit
-p PACKAGE, --package PACKAGE
Package to launch before recording
-C, --clear Clear app cache before launching it
For more information please see :ref:`Revent Replaying <revent_replaying>`.

82
doc/source/user_reference/output_api.rst Normal file
View File

@@ -0,0 +1,82 @@
.. _output-api:
Output API
==========
WA3 now has an output API that can be used to post process a run's
:ref:`Output Directory Structure<output_directory>` which can be performed by using WA's
``RunOutput`` object.
Example:
If we have an existing WA output called ``wa_output`` in the current working
directory we can initialize a ``RunOutput`` as follows:
.. code-block:: python
In [1]: from wa import RunOutput
...:
...: output_folder = 'wa_output'
...: run_output = RunOutput(output_folder)
From here we can retrieve different information about the run. For example if we
want to see what the status of the run was and retrieve the metrics recorded from
the first run we can do the following:
.. code-block:: python
In [2]: run_output.status
Out[2]: OK(7)
In [3]: run_output.jobs
Out[3]:
[<wa.framework.output.JobOutput at 0x7f70358a1f10>,
<wa.framework.output.JobOutput at 0x7f70358a1150>,
<wa.framework.output.JobOutput at 0x7f7035862810>,
<wa.framework.output.JobOutput at 0x7f7035875090>]
In [4]: job_1 = run_output.jobs[0]
In [5]: job_1.label
Out[5]: u'dhrystone'
In [6]: job_1.metrics
Out[6]:
[<thread 0 score: 14423105 (+)>,
<thread 0 DMIPS: 8209 (+)>,
<thread 1 score: 14423105 (+)>,
<thread 1 DMIPS: 8209 (+)>,
<thread 2 score: 14423105 (+)>,
<thread 2 DMIPS: 8209 (+)>,
<thread 3 score: 18292638 (+)>,
<thread 3 DMIPS: 10411 (+)>,
<thread 4 score: 17045532 (+)>,
<thread 4 DMIPS: 9701 (+)>,
<thread 5 score: 14150917 (+)>,
<thread 5 DMIPS: 8054 (+)>,
<time: 0.184497 seconds (-)>,
<total DMIPS: 52793 (+)>,
<total score: 92758402 (+)>]
We can also retrieve information about the device that the run was performed on:
.. code-block:: python
In [7]: run_output.target_info.os
Out[7]: u'android'
In [8]: run_output.target_info.os_version
Out[8]:
OrderedDict([(u'all_codenames', u'REL'),
(u'incremental', u'3687331'),
(u'preview_sdk', u'0'),
(u'base_os', u''),
(u'release', u'7.1.1'),
(u'codename', u'REL'),
(u'security_patch', u'2017-03-05'),
(u'sdk', u'25')])

229
doc/source/user_reference/runtime_parameters.rst Normal file
View File

@@ -0,0 +1,229 @@
.. _runtime-parmeters:
Runtime Parameters
------------------
.. contents:: Contents
:local:
Runtime parameters are options that can be specified to automatically configure
device at runtime. They can be specified at the global level in the agenda or
for individual workloads.
Example
^^^^^^^
Say we want to perform an experiment on an Android big.LITTLE devices to compare
the power consumption between the big and LITTLE clusters running the dhrystone
workload. Assuming we have additional instrumentation active for this device
that we can measure the power the device is consuming, to reduce external
factors we want to ensure that the device has its screen turned off and that it
is in airplane mode turned on for both tests. We will then run 2 :ref:`sections
<sections>` will each enable a single cluster on the device, set the cores to their
maximum frequency and disable all available idle states.
.. code-block:: yaml
#..
workloads:
- name: dhrystone
iterations: 1
runtime_parameters:
airplane_mode: true
screen_on: false
sections:
- id: LITTLES
runtime_parameters:
num_little_cores: 4
little_governor: userspace
little_frequency: max
little_idle_states: none
num_big_cores: 0
- id: BIGS
runtime_parameters:
num_big_cores: 4
big_governor: userspace
big_frequency: max
big_idle_states: none
num_little_cores: 0
HotPlug
^^^^^^^
Parameters:
:num_cores: An ``int`` that specifies the total number of cpu cores to be online.
:num_<core_name>_cores: An ``int`` that specifies the total number of that particular core
to be online, the target will be queried and if the core_names can
be determine a parameter for each of the unique core names will be
available.
:cpu<core_no>_online: A ``boolean`` that specifies whether that particular cpu, e.g. cpu0 will
be online.
If big.LITTLE is detected for the device and additional 2 parameters are available:
:num_big_cores: An ``int`` that specifies the total number of `big` cpu cores to be online.
:num_little_cores: An ``int`` that specifies the total number of `little` cpu cores to be online.
.. Note:: Please note that if the device in question is operating its own dynamic
hotplugging then WA may be unable to set the CPU state or will be overridden.
Unfortunately the method of disabling dynamic hot plugging will vary from
device to device.
CPUFreq
^^^^^^^
:frequency: An ``int`` that can be used to specify a frequency for all cores if there are common frequencies available.
.. Note:: When settings the frequency, if the governor is not set to userspace then WA will attempt to set the maximum
and minimum frequencies to mimic the desired behaviour.
:max_frequency: An ``int`` that can be used to specify a maximum frequency for all cores if there are common frequencies available.
:min_frequency: An ``int`` that can be used to specify a minimum frequency for all cores if there are common frequencies available.
:governor: A ``string`` that can be used to specify the governor for all cores if there are common governors available.
:governor: A ``string`` that can be used to specify the governor for all cores if there are common governors available.
:governor_tunable: A ``dict`` that can be used to specify governor
tunables for all cores, unlike the other common parameters these are not
validated at the beginning of the run therefore incorrect values will cause
an error during runtime.
:<core_name>_frequency: An ``int`` that can be used to specify a frequency for cores of a particular type e.g. 'A72'.
:<core_name>_max_frequency: An ``int`` that can be used to specify a maximum frequency for cores of a particular type e.g. 'A72'.
:<core_name>_min_frequency: An ``int`` that can be used to specify a minimum frequency for cores of a particular type e.g. 'A72'.
:<core_name>_governor: A ``string`` that can be used to specify the governor for cores of a particular type e.g. 'A72'.
:<core_name>_governor: A ``string`` that can be used to specify the governor for cores of a particular type e.g. 'A72'.
:<core_name>_governor_tunable: A ``dict`` that can be used to specify governor
tunables for cores of a particular type e.g. 'A72', these are not
validated at the beginning of the run therefore incorrect values will cause
an error during runtime.
:cpu<no>_frequency: An ``int`` that can be used to specify a frequency for a particular core e.g. 'cpu0'.
:cpu<no>_max_frequency: An ``int`` that can be used to specify a maximum frequency for a particular core e.g. 'cpu0'.
:cpu<no>_min_frequency: An ``int`` that can be used to specify a minimum frequency for a particular core e.g. 'cpu0'.
:cpu<no>_governor: A ``string`` that can be used to specify the governor for a particular core e.g. 'cpu0'.
:cpu<no>_governor: A ``string`` that can be used to specify the governor for a particular core e.g. 'cpu0'.
:cpu<no>_governor_tunable: A ``dict`` that can be used to specify governor
tunables for a particular core e.g. 'cpu0', these are not
validated at the beginning of the run therefore incorrect values will cause
an error during runtime.
If big.LITTLE is detected for the device an additional set of parameters are available:
:big_frequency: An ``int`` that can be used to specify a frequency for the big cores.
:big_max_frequency: An ``int`` that can be used to specify a maximum frequency for the big cores.
:big_min_frequency: An ``int`` that can be used to specify a minimum frequency for the big cores.
:big_governor: A ``string`` that can be used to specify the governor for the big cores.
:big_governor: A ``string`` that can be used to specify the governor for the big cores.
:big_governor_tunable: A ``dict`` that can be used to specify governor
tunables for the big cores, these are not
validated at the beginning of the run therefore incorrect values will cause
an error during runtime.
:little_frequency: An ``int`` that can be used to specify a frequency for the little cores.
:little_max_frequency: An ``int`` that can be used to specify a maximum frequency for the little cores.
:little_min_frequency: An ``int`` that can be used to specify a minimum frequency for the little cores.
:little_governor: A ``string`` that can be used to specify the governor for the little cores.
:little_governor: A ``string`` that can be used to specify the governor for the little cores.
:little_governor_tunable: A ``dict`` that can be used to specify governor
tunables for the little cores, these are not
validated at the beginning of the run therefore incorrect values will cause
an error during runtime.
CPUIdle
^^^^^^^
:idle_states: A ``string`` or list of strings which can be used to specify what
idles states should be enabled for all cores if there are common frequencies
available. 'all' and 'none' are also valid entries as a shorthand
:<core_name>_idle_states: A ``string`` or list of strings which can be used to
specify what idles states should be enabled for cores of a particular type
e.g. 'A72'. 'all' and 'none' are also valid entries as a shorthand
:cpu<no>_idle_states: A ``string`` or list of strings which can be used to
specify what idles states should be enabled for a particular core e.g.
'cpu0'. 'all' and 'none' are also valid entries as a shorthand
If big.LITTLE is detected for the device and additional set of parameters are available:
:big_idle_states: A ``string`` or list of strings which can be used to specify
what idles states should be enabled for the big cores. 'all' and 'none' are
also valid entries as a shorthand
:little_idle_states: A ``string`` or list of strings which can be used to
specify what idles states should be enabled for the little cores. 'all' and
'none' are also valid entries as a shorthand.
Android Specific Runtime Parameters
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
:brightness: An ``int`` between 0 and 255 (inclusive) to specify the brightness
the screen should be set to. Defaults to ``127``.
:airplane_mode: A ``boolean`` to specify whether airplane mode should be
enabled for the device.
:rotation: A ``String`` to specify the screen orientation for the device. Valid
entries are ``NATURAL``, ``LEFT``, ``INVERTED``, ``RIGHT``.
:screen_on: A ``boolean`` to specify whether the devices screen should be
turned on. Defaults to ``true``.
Setting Sysfiles
^^^^^^^^^^^^^^^^
In order to perform additional configuration of a target the ``sysfile_values``
runtime parameter can be used. The value for this parameter is a mapping (an
associative array, in YAML) of file paths onto values that should be written
into those files. sysfile_values is the only runtime parameter that is available
for any (Linux) device. Other runtime parameters will depend on the specifics of
the device used (e.g. its CPU cores configuration) as detailed above.
.. note:: By default WA will attempt to verify that the sysfile value was
written correctly by reading the node back and comparing the two values. If
you do not wish this check to happen, for example the node you are writing to
is write only, you can append an ``!`` to the file path to disable this
verification.
For example the following configuration could be used to enable and verify that cpu0
is online, however will not attempt to check that it's governor have been set to
userspace::
- name: dhrystone
runtime_params:
sysfile_values:
/sys/devices/system/cpu/cpu0/online: 1
/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor!: userspace