sascha/workload-automation
1
0
Fork 0
mirror of https://github.com/ARM-software/workload-automation.git synced 2025-09-01 19:02:31 +01:00
Code Issues Releases Wiki Activity

doc: update and extend Output docs

Browse Source 
- Move output processing API docs into developer reference.
- Add a section on output directory structure to user reference.
- Rewrite the output section of the user guide to provide a very
  brief overview of the output directory and to link to the two
  sections above.
This commit is contained in:
Sergei Trofimov
2018-05-18 15:41:05 +01:00
committed by Marc Bonnici
parent 5291e389dc
commit 192bebbc23
5 changed files with 185 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files

102
doc/source/user_reference/output_api.rst
View File

@@ -1,102 +0,0 @@
.. _output-api:
Output API
==========
WA3 now has an output API that can be used to post process a run's
:ref:`Output Directory Structure<output_directory>` which can be performed by using WA's
``RunOutput`` object.
Example:
If we have an existing WA output called ``wa_output`` in the current working
directory we can initialize a ``RunOutput`` as follows:
.. code-block:: python
In [1]: from wa import RunOutput
...:
...: output_folder = 'wa_output'
...: run_output = RunOutput(output_folder)
From here we can retrieve different information about the run. For example if we
want to see what the overall status of the run was, along with the runtime
parameters and the metrics recorded from the first job was we can do the following:
.. code-block:: python
In [2]: run_output.status
Out[2]: OK(7)
# List all of the jobs for the run
In [3]: run_output.jobs
Out[3]:
[<wa.framework.output.JobOutput at 0x7f70358a1f10>,
<wa.framework.output.JobOutput at 0x7f70358a1150>,
<wa.framework.output.JobOutput at 0x7f7035862810>,
<wa.framework.output.JobOutput at 0x7f7035875090>]
# Examine the first job that was ran
In [4]: job_1 = run_output.jobs[0]
In [5]: job_1.label
Out[5]: u'dhrystone'
# Print out all the runtime parameters and their values for this job
In [6]: for k, v in job_1.spec.runtime_parameters.iteritems():
...: print (k, v)
(u'airplane_mode': False)
(u'brightness': 100)
(u'governor': 'userspace')
(u'big_frequency': 1700000)
(u'little_frequency': 1400000)
# Print out all the metrics avalible for this job
In [7]: job_1.metrics
Out[7]:
[<thread 0 score: 14423105 (+)>,
<thread 0 DMIPS: 8209 (+)>,
<thread 1 score: 14423105 (+)>,
<thread 1 DMIPS: 8209 (+)>,
<thread 2 score: 14423105 (+)>,
<thread 2 DMIPS: 8209 (+)>,
<thread 3 score: 18292638 (+)>,
<thread 3 DMIPS: 10411 (+)>,
<thread 4 score: 17045532 (+)>,
<thread 4 DMIPS: 9701 (+)>,
<thread 5 score: 14150917 (+)>,
<thread 5 DMIPS: 8054 (+)>,
<time: 0.184497 seconds (-)>,
<total DMIPS: 52793 (+)>,
<total score: 92758402 (+)>]
We can also retrieve information about the target that the run was performed on
for example:
.. code-block:: python
# Print out the target's abi:
In [9]: run_output.target_info.abi
Out[9]: u'arm64'
# The os the target was running
In [9]: run_output.target_info.os
Out[9]: u'android'
# And other information about the os version
In [10]: run_output.target_info.os_version
Out[10]:
OrderedDict([(u'all_codenames', u'REL'),
(u'incremental', u'3687331'),
(u'preview_sdk', u'0'),
(u'base_os', u''),
(u'release', u'7.1.1'),
(u'codename', u'REL'),
(u'security_patch', u'2017-03-05'),
(u'sdk', u'25')])

132
doc/source/user_reference/output_directory.rst Normal file
View File

@@ -0,0 +1,132 @@
.. _output_directory_structure:
Output Directory Structure
==========================
This is an overview of WA output directory structure.
.. note:: In addition to files and subdirectories described here,
other content may present in the output directory for
a run, depending on the enabled augmentations.
Overview
--------
The output directory will contain a subdirectory for every job iteration that
was run, as well as some additional entries. The following diagram illustrates
the typical structure of WA output directory::
wa_output/
├── __meta/
│ ├── config.json
│ ├── jobs.json
│ ├── raw_config
│ │   ├── cfg0-config.yaml
│ │   └── agenda.yaml
│ ├── run_info.json
│ └── target_info.json
├── __failed/
│ └── wk1-dhrystone-1-attempt1
├── wk1-dhrystone-1/
│ └── result.json
├── wk1-dhrystone-2/
│ └── result.json
├── wk2-memcpy-1/
│ └── result.json
├── wk2-memcpy-2/
│ └── result.json
├── result.json
└── run.log
This is the directory structure that would be generated after running two
iterations each of ``dhrystone`` and ``memcpy`` workloads with no augmentations
enabled, and with the first attempt at the first iteration of dhrystone having
failed.
Output Directory Entries
------------------------
result.json
Contains a JSON structure describing the result of the execution,
including collected metrics and artifacts. There will be one for each
job execution, and one for the overall run. The run ``result.json`` will
only contain metrics/artifacts for the run as a whole, and will not
contain results for individual jobs.
You typically would not access ``result.json`` files directly. Instead
you would either enable augmentations to format the results in easier to
manage form (such as CSV table), or use :ref:`output_processing_api` to
access the results from scripts.
run.log
This is a log of everything that happened during the run, including all
interactions with the target, and all the decisions made by the
framework. The output is equivalent to what you would see on the console
when running with ``--verbose`` option.
.. note:: WA source contains a syntax file for Vim that will color the
initial part of each log line, in a similar way to what you
see on the console. This may be useful for quickly spotting
error and warning messages when scrolling through the log.
https://github.com/ARM-software/workload-automation/blob/next/extras/walog.vim
__meta
This directory contains configuration and run metadata. See
:ref:`config_and_meta` below for details.
__failed
This directory will only be present if one or more job executions has
failed and were re-run. This directory contains output directories for
the failed attempts.
job execution output subdirectory
Each subdirectory will be named ``<job id>_<workload label>_<iteration
number>``, and will, at minimum, contain a ``result.json`` (see above).
Additionally, it may contain raw output from the workload, and any
additional artifacts (e.g. traces) generated by augmentations. Finally,
if workload execution has failed, WA may gather some additional logging
(such as the UI state at the time of failure) and place it here.
.. _config_and_meta:
Configuration and Metadata
--------------------------
As stated above, the ``__meta`` directory contains run configuration and
metadata. Typically, you would not access these files directly, but would use
the :ref:`output_processing_api` to query the metadata.
For more details about WA configuration see :ref:`configuration-specification`.
config.json
Contains the overall run configuration, such as target interface
configuration, and job execution order, and various "meta-configuration"
settings, such as default output path, verbosity level, and logging
formatting.
jobs.json
Final configuration for all jobs, including enabled augmentations,
workload and runtime parameters, etc.
raw_config
This directory contains copies of config file(s) and the agenda that
were parsed in order to generate configuration for this run. Each config
file is prefixed with ``cfg<N>-``, where ``<N>`` is the number
indicating the order (with respect to the other other config files) in
which it was parsed, e.g. ``cfg0-config.yaml`` is always a copy of
``$WA_USER_DIRECTORY/config.yaml``. The one file without a prefix is the
agenda.
run_info.json
Run metadata, e.g. duration, start/end timestamps and duration.
target_info.json
Extensive information about the target. This includes information about
the target's CPUS configuration, kernel and userspace versions, etc. The
exact content will vary depending on the target type (Android vs Linux)
and what could accessed on a particular device (e.g. if
``/proc/config.gz`` exists on the target, the kernel config will be
included).