sascha/workload-automation
1
0
Fork 0
mirror of https://github.com/ARM-software/workload-automation.git synced 2025-10-30 14:44:09 +00:00
Code Issues Releases Wiki Activity

doc: Restructure

Browse Source 
Restructure the documentation to be split into `User Information` and
`Developer Information`, and split the how to guides into their
corresponding section.
This commit is contained in:
Marc Bonnici
2018-06-20 17:43:52 +01:00
committed by setrofim
parent 3c0f1968c5
commit 6c93590062
28 changed files with 122 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files

528
doc/source/user_information/user_guide.rst Normal file
View File

@@ -0,0 +1,528 @@
.. _user-guide:
**********
User Guide
**********
This guide will show you how to quickly start running workloads using
Workload Automation 3.
.. contents:: Contents
:depth: 2
:local:
---------------------------------------------------------------
Install
=======
.. note:: This is a quick summary. For more detailed instructions, please see
the :ref:`installation` section.
Make sure you have Python 2.7 and a recent Android SDK with API level 18 or above
installed on your system. A complete install of the Android SDK is required, as
WA uses a number of its utilities, not just adb. For the SDK, make sure that either
``ANDROID_HOME`` environment variable is set, or that ``adb`` is in your ``PATH``.
.. Note:: If you plan to run Workload Automation on Linux devices only, SSH is required,
and Android SDK is optional if you wish to run WA on Android devices at a
later time.
However, you would be starting off with a limited number of workloads that
will run on Linux devices.
In addition to the base Python 2.7 install, you will also need to have ``pip``
(Python's package manager) installed as well. This is usually a separate package.
Once you have those, you can install WA with::
sudo -H pip install wa
This will install Workload Automation on your system, along with its mandatory
dependencies.
Alternatively we provide a Dockerfile that which can be used to create a Docker
image for running WA along with its dependencies. More information can be found
:ref:`here <dockerfile>`.
(Optional) Verify installation
-------------------------------
Once the tarball has been installed, try executing ::
wa -h
You should see a help message outlining available subcommands.
(Optional) APK files
--------------------
A large number of WA workloads are installed as APK files. These cannot be
distributed with WA and so you will need to obtain those separately.
For more details, please see the :ref:`installation <apk_files>` section.
List Command
============
In order to get started with using WA we first we need to find
out what is available to use. In order to do this we can use the :ref:`list <list-command>`
command followed by the type of plugin that you wish to see.
For example to see what workloads are available along with a short description
of each you run::
wa list workloads
Which will give an output in the format of:
.. code-block:: none
adobereader: The Adobe Reader workflow carries out the following typical
productivity tasks.
androbench: Executes storage performance benchmarks
angrybirds_rio: Angry Birds Rio game.
antutu: Executes Antutu 3D, UX, CPU and Memory tests
applaunch: This workload launches and measures the launch time of applications
for supporting workloads.
benchmarkpi: Measures the time the target device takes to run and complete the
Pi calculation algorithm.
dhrystone: Runs the Dhrystone benchmark.
exoplayer: Android ExoPlayer
geekbench: Geekbench provides a comprehensive set of benchmarks engineered to
quickly and accurately measure
processor and memory performance.
#..
The same syntax can be used to display ``commands``,
``energy_instrument_backends``, ``instruments``, ``output_processors``,
``resource_getters``, ``targets``. Once you have found the plugin you are
looking for you can use the :ref:`show <show-command>` command to display more
detailed information. Alternatively please see the
:ref:`Plugin Reference <plugin-reference>` for an online version.
Show Command
============
If you want to learn more information about a particular plugin, such as the
parameters it supports, you can use the "show" command::
wa show dhrystone
If you have ``pandoc`` installed on your system, this will display man
page-like description of the plugin, and the parameters it supports. If you do
not have ``pandoc``, you will instead see the same information as raw
restructured text.
Configure Your Device
=====================
There are multiple options for configuring your device depending on your
particular use case.
You can either add your configuration to the default configuration file
``config.yaml``, under the ``$WA_USER_HOME/`` directory or you can specify it in
the ``config`` section of your agenda directly.
Alternatively if you are using multiple devices, you may want to create separate
config files for each of your devices you will be using. This allows you to
specify which device you would like to use for a particular run and pass it as
an argument when invoking with the ``-c`` flag.
::
wa run dhrystone -c my_device.yaml
By default WA will use the “most specific” configuration available for example
any configuration specified inside an agenda will override a passed
configuration file which will in turn overwrite the default configuration file.
.. note:: For a more information about configuring your
device please see :ref:`Setting Up A Device <setting-up-a-device>`.
Android
-------
By default, the device WA will use is set to 'generic_android'. WA is configured
to work with a generic Android device through ``adb``. If you only have one
device listed when you execute ``adb devices``, and your device has a standard
Android configuration, then no extra configuration is required.
However, if your device is connected via network, you will have to manually
execute ``adb connect <device ip>`` (or specify this in your
:ref:`agenda <agenda>`) so that it appears in the device listing.
If you have multiple devices connected, you will need to tell WA which one you
want it to use. You can do that by setting ``device`` in the device_config section.
.. code-block:: yaml
# ...
device_config:
device: 'abcdef0123456789'
# ...
# ...
Linux
-----
First, set the device to 'generic_linux'
.. code-block:: yaml
# ...
device: 'generic_linux'
# ...
Find the device_config section and add these parameters
.. code-block:: yaml
# ...
device_config:
host: '192.168.0.100'
username: 'root'
password: 'password'
# ...
# ...
Parameters:
- Host is the IP of your target Linux device
- Username is the user for the device
- Password is the password for the device
Enabling and Disabling Augmentations
---------------------------------------
Augmentations are the collective name for "instruments" and "output
processors" in WA3.
Some augmentations are enabled by default after your initial install of WA,
which are specified in the ``config.yaml`` file located in your
``WA_USER_DIRECTORY``, typically ``~/.workload_autoamation``.
.. note:: Some Linux devices may not be able to run certain augmentations
provided by WA (e.g. cpufreq is disabled or unsupported by the
device).
.. code-block:: yaml
# ...
augmentations:
# Records the time it took to run the workload
- execution_time
# Collects /proc/interrupts before and after execution and does a diff.
- interrupts
# Collects the contents of/sys/devices/system/cpu before and after
# execution and does a diff.
- cpufreq
# Generate a txt file containing general status information about
# which runs failed and which were successful.
- status
# ...
If you only wanted to keep the 'execution_time' instrument enabled, you can comment out
the rest of the list augmentations to disable them.
This should give you basic functionality. If you are working with a development
board or you need some advanced functionality additional configuration may be required.
Please see the :ref:`device setup <setting-up-a-device>` section for more details.
.. note:: In WA2 'Instrumentation' and 'Result Processors' were divided up into their
own sections in the agenda. In WA3 they now fall under the same category of
'augmentations'. For compatibility the old naming structure is still valid
however using the new entry names is recommended.
Running Your First Workload
===========================
The simplest way to run a workload is to specify it as a parameter to WA ``run``
:ref:`run <run-command>` sub-command::
wa run dhrystone
You will see INFO output from WA as it executes each stage of the run. A
completed run output should look something like this::
INFO Creating output directory.
INFO Initializing run
INFO Connecting to target
INFO Setting up target
INFO Initializing execution context
INFO Generating jobs
INFO Loading job wk1 (dhrystone) [1]
INFO Installing instruments
INFO Installing output processors
INFO Starting run
INFO Initializing run
INFO Initializing job wk1 (dhrystone) [1]
INFO Running job wk1
INFO Configuring augmentations
INFO Configuring target for job wk1 (dhrystone) [1]
INFO Setting up job wk1 (dhrystone) [1]
INFO Running job wk1 (dhrystone) [1]
INFO Tearing down job wk1 (dhrystone) [1]
INFO Completing job wk1
INFO Job completed with status OK
INFO Finalizing run
INFO Finalizing job wk1 (dhrystone) [1]
INFO Done.
INFO Run duration: 9 seconds
INFO Ran a total of 1 iterations: 1 OK
INFO Results can be found in wa_output
Once the run has completed, you will find a directory called ``wa_output``
in the location where you have invoked ``wa run``. Within this directory,
you will find a "results.csv" file which will contain results obtained for
dhrystone, as well as a "run.log" file containing detailed log output for
the run. You will also find a sub-directory called 'wk1-dhrystone-1' that
contains the results for that iteration. Finally, you will find various additional
information in the ``wa_output/__meta`` subdirectory for example information
extracted from the target and a copy of the agenda file. The contents of
iteration-specific subdirectories will vary from workload to workload, and,
along with the contents of the main output directory, will depend on the
augmentations that were enabled for that run.
The ``run`` sub-command takes a number of options that control its behaviour,
you can view those by executing ``wa run -h``. Please see the :ref:`invocation`
section for details.
Create an Agenda
================
Simply running a single workload is normally of little use. Typically, you would
want to specify several workloads, setup the device state and, possibly, enable
additional augmentations. To do this, you would need to create an "agenda" for
the run that outlines everything you want WA to do.
Agendas are written using YAML_ markup language. A simple agenda might look
like this:
.. code-block:: yaml
config:
augmentations:
- ~execution_time
- json
iterations: 2
workloads:
- memcpy
- name: dhrystone
params:
mloops: 5
threads: 1
This agenda:
- Specifies two workloads: memcpy and dhrystone.
- Specifies that dhrystone should run in one thread and execute five million loops.
- Specifies that each of the two workloads should be run twice.
- Enables json output processor, in addition to the output processors enabled in
the config.yaml.
- Disables execution_time instrument, if it is enabled in the config.yaml
An agenda can be created using WA's ``create`` :ref:`command <using-the-create-command>`
or in a text editor and saved as a YAML file.
For more options please see the :ref:`agenda` documentation.
.. _YAML: http://en.wikipedia.org/wiki/YAML
.. _using-the-create-command:
Using the Create Command
-------------------------
The easiest way to create an agenda is to use the 'create' command. For more
in-depth information please see the :ref:`Create Command <create-command>` documentation.
In order to populate the agenda with relevant information you can supply all of
the plugins you wish to use as arguments to the command, for example if we want
to create an agenda file for running ``dhystrone`` on a 'generic android' device and we
want to enable the ``execution_time`` and ``trace-cmd`` instruments and display the
metrics using the ``csv`` output processor. We would use the following command::
wa create agenda generic_android dhrystone execution_time trace-cmd csv -o my_agenda.yaml
This will produce a `my_agenda.yaml` file containing all the relevant
configuration for the specified plugins along with their default values as shown
below:
.. code-block:: yaml
config:
augmentations:
- execution_time
- trace-cmd
- csv
iterations: 1
device: generic_android
device_config:
adb_server: null
big_core: null
core_clusters: null
core_names: null
device: null
disable_selinux: true
executables_directory: null
load_default_modules: true
logcat_poll_period: null
model: null
modules: null
package_data_directory: /data/data
shell_prompt: !<tag:wa:regex> '8:^.*(shell|root)@.*:/\S* [#$] '
working_directory: null
execution_time: {}
trace-cmd:
buffer_size: null
buffer_size_step: 1000
events:
- sched*
- irq*
- power*
- thermal*
functions: null
no_install: false
report: true
report_on_target: false
csv:
extra_columns: null
use_all_classifiers: false
workloads:
- name: dhrystone
params:
cleanup_assets: true
delay: 0
duration: 0
mloops: 0
taskset_mask: 0
threads: 4
Run Command
============
These examples show some useful options that can be used with WA's ``run`` command.
Once we have created an agenda to use it with WA we can pass it as a argument to
the run command e.g.::
wa run <path/to/agenda> (e.g. wa run ~/myagenda.yaml)
By default WA will use the "wa_output" directory to stores its output however to
redirect the output to a different directory we can use::
wa run dhrystone -d my_output_directory
We can also tell WA to use additional config files by supplying it with
the ``-c`` argument. One use case for passing additional config files is if you
have multiple devices you wish test with WA, you can store the relevant device
configuration in individual config files and then pass the file corresponding to
the device you wish to use for that particular test.
.. note:: As previously mentioned, any more specific configuration present in
the agenda file will overwrite the corresponding config parameters
specified in the config file(s).
::
wa run -c myconfig.yaml ~/myagenda.yaml
To use the same output directory but override the existing contents to
store new dhrystone results we can specify the ``-f`` argument::
wa run -f dhrystone
To display verbose output while running memcpy::
wa run --verbose memcpy
.. _output_directory:
Output
======
The output directory will contain subdirectories for each job that was run,
which will in turn contain the generated metrics and artifacts for each job.
The directory will also contain a ``run.log`` file containing the complete log
output for the run, and a ``__meta`` directory with the configuration and
metadata for the run. Metrics are serialized inside ``result.json`` files inside
each job's subdirectory. There may also be a ``__failed`` directory containing
failed attempts for jobs that have been re-run.
Augmentations may add additional files at the run or job directory level. The
default configuration has ``status`` and ``csv`` augmentations enabled which
generate a ``status.txt`` containing status summary for the run and individual
jobs, and a ``results.csv`` containing metrics from all jobs in a CSV table,
respectively.
See :ref:`output_directory_structure` for more information.
In order to make it easier to access WA results from scripts, WA provides an API
that parses the contents of the output directory:
.. code-block:: pycon
>>> from wa import RunOutput
>>> ro = RunOutput('./wa_output')
>>> for job in ro.jobs:
... if job.status != 'OK':
... print 'Job "{}" did not complete successfully: {}'.format(job, job.status)
... continue
... print 'Job "{}":'.format(job)
... for metric in job.metrics:
... if metric.units:
... print '\t{}: {} {}'.format(metric.name, metric.value, metric.units)
... else:
... print '\t{}: {}'.format(metric.name, metric.value)
...
Job "wk1-dhrystone-1":
thread 0 score: 20833333
thread 0 DMIPS: 11857
thread 1 score: 24509804
thread 1 DMIPS: 13950
thread 2 score: 18011527
thread 2 DMIPS: 10251
thread 3 score: 26371308
thread 3 DMIPS: 15009
time: 1.001251 seconds
total DMIPS: 51067
total score: 89725972
execution_time: 1.4834280014 seconds
See :ref:`output_processing_api` for details.
Uninstall
=========
If you have installed Workload Automation via ``pip``, then run this command to
uninstall it::
sudo pip uninstall wa
.. Note:: It will *not* remove any user configuration (e.g. the ~/.workload_automation
directory).
Upgrade
=======
To upgrade Workload Automation to the latest version via ``pip``, run::
sudo pip install --upgrade --no-deps wa