sascha/workload-automation
1
0
Fork 0
mirror of https://github.com/ARM-software/workload-automation.git synced 2025-01-18 20:11:20 +00:00
Code Issues Releases Wiki Activity

doc: update and extend Output docs

Browse Source 
- Move output processing API docs into developer reference.
- Add a section on output directory structure to user reference.
- Rewrite the output section of the user guide to provide a very
  brief overview of the output directory and to link to the two
  sections above.
This commit is contained in:
Sergei Trofimov 2018-05-18 15:41:05 +01:00 committed by Marc Bonnici
parent 5291e389dc
commit 192bebbc23
5 changed files with 185 additions and 25 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
doc/source/developer_reference.rst
View File

@ -11,6 +11,7 @@ Developer Reference
--------------------------------------------------------------------------------
.. include:: developer_reference/output_processing_api.rst
.. include:: developer_reference/writing_plugins.rst
.. include:: developer_reference/contributing.rst
.. include:: developer_reference/revent.rst

6
doc/source/user_reference/output_api.rst → doc/source/developer_reference/output_processing_api.rst
View File

@ -1,7 +1,7 @@
.. _output-api:
.. _output_processing_api:
Output API
==========
Output Processing API
=====================
WA3 now has an output API that can be used to post process a run's
:ref:`Output Directory Structure<output_directory>` which can be performed by using WA's

69
doc/source/user_guide.rst
View File

@ -448,32 +448,59 @@ To display verbose output while running memcpy::
.. _output_directory:
Output Directory
================
Output
======
The exact contents on the output directory will depend on configuration options
used, augmentations enabled, etc.
The output directory will contain subdirectories for each job that was run,
which will in turn contain the generated metrics and artifacts for each job.
The directory will also contain a ``run.log`` file containing the complete log
output for the run, and a ``__meta`` directory with the configuration and
metadata for the run. Metrics are serialized inside ``result.json`` files inside
each job's subdirectory. There may also be a ``__failed`` directory containing
failed attempts for jobs that have been re-run.
At the top level, there will be a ``run.log`` file containing the complete log
output for the execution. The contents of this file is equivalent to what you
would get in the console when using --verbose option.
Augmentations may add additional files at the run or job directory level. The
default configuration has ``status`` and ``csv`` augmentations enabled which
generate a ``status.txt`` containing status summary for the run and individual
jobs, and a ``results.csv`` containing metrics from all jobs in a CSV table,
respectively.
The output directory will also contain a ``results.json`` file that lists any
overall run metrics and artifacts that have been collected during the run.
Depending on the augmentations that were enabled there may be other results
files in different formats, for example the ``csv``
:ref:`output processor <output-processors>`. In addition you will find a subdirectory
for each successful iteration executed with output and its own ``results.json``
file for that specific iteration.
See :ref:`output_directory_structure` for more information.
If a job fails to complete for some reason, then the output directory for that
job will be moved into a new directory called ``__failed``. If the job was
running on a platform that supports android then WA will take a screen capture
and UI dump from the device.
In order to make it easier to access WA results from scripts, WA provides an API
that parses the contents of the output directory:
Finally, there will be a ``__meta`` subdirectory. This will contain a copy of
the agenda file used to run the workloads along with other configuration
files that were used for the execution of that run.
.. code-block:: pycon
>>> from wa import RunOutput
>>> ro = RunOutput('./wa_output')
>>> for job in ro.jobs:
... if job.status != 'OK':
... print 'Job "{}" did not complete successfully: {}'.format(job, job.status)
... continue
... print 'Job "{}":'.format(job)
... for metric in job.metrics:
... if metric.units:
... print '\t{}: {} {}'.format(metric.name, metric.value, metric.units)
... else:
... print '\t{}: {}'.format(metric.name, metric.value)
...
Job "wk1-dhrystone-1":
thread 0 score: 20833333
thread 0 DMIPS: 11857
thread 1 score: 24509804
thread 1 DMIPS: 13950
thread 2 score: 18011527
thread 2 DMIPS: 10251
thread 3 score: 26371308
thread 3 DMIPS: 15009
time: 1.001251 seconds
total DMIPS: 51067
total score: 89725972
execution_time: 1.4834280014 seconds
See :ref:`output_processing_api` for details.
Uninstall
=========

2
doc/source/user_reference.rst
View File

@ -13,4 +13,4 @@ User Reference
.. include:: user_reference/configuration.rst
.. include:: user_reference/invocation.rst
.. include:: user_reference/output_api.rst
.. include:: user_reference/output_directory.rst

132
doc/source/user_reference/output_directory.rst Normal file
View File

@ -0,0 +1,132 @@
.. _output_directory_structure:
Output Directory Structure
==========================
This is an overview of WA output directory structure.
.. note:: In addition to files and subdirectories described here,
other content may present in the output directory for
a run, depending on the enabled augmentations.
Overview
--------
The output directory will contain a subdirectory for every job iteration that
was run, as well as some additional entries. The following diagram illustrates
the typical structure of WA output directory::
wa_output/
├── __meta/
│ ├── config.json
│ ├── jobs.json
│ ├── raw_config
│ │   ├── cfg0-config.yaml
│ │   └── agenda.yaml
│ ├── run_info.json
│ └── target_info.json
├── __failed/
│ └── wk1-dhrystone-1-attempt1
├── wk1-dhrystone-1/
│ └── result.json
├── wk1-dhrystone-2/
│ └── result.json
├── wk2-memcpy-1/
│ └── result.json
├── wk2-memcpy-2/
│ └── result.json
├── result.json
└── run.log
This is the directory structure that would be generated after running two
iterations each of ``dhrystone`` and ``memcpy`` workloads with no augmentations
enabled, and with the first attempt at the first iteration of dhrystone having
failed.
Output Directory Entries
------------------------
result.json
Contains a JSON structure describing the result of the execution,
including collected metrics and artifacts. There will be one for each
job execution, and one for the overall run. The run ``result.json`` will
only contain metrics/artifacts for the run as a whole, and will not
contain results for individual jobs.
You typically would not access ``result.json`` files directly. Instead
you would either enable augmentations to format the results in easier to
manage form (such as CSV table), or use :ref:`output_processing_api` to
access the results from scripts.
run.log
This is a log of everything that happened during the run, including all
interactions with the target, and all the decisions made by the
framework. The output is equivalent to what you would see on the console
when running with ``--verbose`` option.
.. note:: WA source contains a syntax file for Vim that will color the
initial part of each log line, in a similar way to what you
see on the console. This may be useful for quickly spotting
error and warning messages when scrolling through the log.
https://github.com/ARM-software/workload-automation/blob/next/extras/walog.vim
__meta
This directory contains configuration and run metadata. See
:ref:`config_and_meta` below for details.
__failed
This directory will only be present if one or more job executions has
failed and were re-run. This directory contains output directories for
the failed attempts.
job execution output subdirectory
Each subdirectory will be named ``<job id>_<workload label>_<iteration
number>``, and will, at minimum, contain a ``result.json`` (see above).
Additionally, it may contain raw output from the workload, and any
additional artifacts (e.g. traces) generated by augmentations. Finally,
if workload execution has failed, WA may gather some additional logging
(such as the UI state at the time of failure) and place it here.
.. _config_and_meta:
Configuration and Metadata
--------------------------
As stated above, the ``__meta`` directory contains run configuration and
metadata. Typically, you would not access these files directly, but would use
the :ref:`output_processing_api` to query the metadata.
For more details about WA configuration see :ref:`configuration-specification`.
config.json
Contains the overall run configuration, such as target interface
configuration, and job execution order, and various "meta-configuration"
settings, such as default output path, verbosity level, and logging
formatting.
jobs.json
Final configuration for all jobs, including enabled augmentations,
workload and runtime parameters, etc.
raw_config
This directory contains copies of config file(s) and the agenda that
were parsed in order to generate configuration for this run. Each config
file is prefixed with ``cfg<N>-``, where ``<N>`` is the number
indicating the order (with respect to the other other config files) in
which it was parsed, e.g. ``cfg0-config.yaml`` is always a copy of
``$WA_USER_DIRECTORY/config.yaml``. The one file without a prefix is the
agenda.
run_info.json
Run metadata, e.g. duration, start/end timestamps and duration.
target_info.json
Extensive information about the target. This includes information about
the target's CPUS configuration, kernel and userspace versions, etc. The
exact content will vary depending on the target type (Android vs Linux)
and what could accessed on a particular device (e.g. if
``/proc/config.gz`` exists on the target, the kernel config will be
included).