From 192bebbc232251914c61a412a2628efb7843997c Mon Sep 17 00:00:00 2001 From: Sergei Trofimov Date: Fri, 18 May 2018 15:41:05 +0100 Subject: [PATCH] doc: update and extend Output docs - Move output processing API docs into developer reference. - Add a section on output directory structure to user reference. - Rewrite the output section of the user guide to provide a very brief overview of the output directory and to link to the two sections above. --- doc/source/developer_reference.rst | 1 + .../output_processing_api.rst} | 6 +- doc/source/user_guide.rst | 69 ++++++--- doc/source/user_reference.rst | 2 +- .../user_reference/output_directory.rst | 132 ++++++++++++++++++ 5 files changed, 185 insertions(+), 25 deletions(-) rename doc/source/{user_reference/output_api.rst => developer_reference/output_processing_api.rst} (97%) create mode 100644 doc/source/user_reference/output_directory.rst diff --git a/doc/source/developer_reference.rst b/doc/source/developer_reference.rst index 175ab26b..1ca32b60 100644 --- a/doc/source/developer_reference.rst +++ b/doc/source/developer_reference.rst @@ -11,6 +11,7 @@ Developer Reference -------------------------------------------------------------------------------- +.. include:: developer_reference/output_processing_api.rst .. include:: developer_reference/writing_plugins.rst .. include:: developer_reference/contributing.rst .. include:: developer_reference/revent.rst diff --git a/doc/source/user_reference/output_api.rst b/doc/source/developer_reference/output_processing_api.rst similarity index 97% rename from doc/source/user_reference/output_api.rst rename to doc/source/developer_reference/output_processing_api.rst index a2f51528..992d0e6b 100644 --- a/doc/source/user_reference/output_api.rst +++ b/doc/source/developer_reference/output_processing_api.rst @@ -1,7 +1,7 @@ -.. _output-api: +.. _output_processing_api: -Output API -========== +Output Processing API +===================== WA3 now has an output API that can be used to post process a run's :ref:`Output Directory Structure` which can be performed by using WA's diff --git a/doc/source/user_guide.rst b/doc/source/user_guide.rst index 4c495b7f..61ed763c 100644 --- a/doc/source/user_guide.rst +++ b/doc/source/user_guide.rst @@ -448,32 +448,59 @@ To display verbose output while running memcpy:: .. _output_directory: -Output Directory -================ +Output +====== -The exact contents on the output directory will depend on configuration options -used, augmentations enabled, etc. +The output directory will contain subdirectories for each job that was run, +which will in turn contain the generated metrics and artifacts for each job. +The directory will also contain a ``run.log`` file containing the complete log +output for the run, and a ``__meta`` directory with the configuration and +metadata for the run. Metrics are serialized inside ``result.json`` files inside +each job's subdirectory. There may also be a ``__failed`` directory containing +failed attempts for jobs that have been re-run. -At the top level, there will be a ``run.log`` file containing the complete log -output for the execution. The contents of this file is equivalent to what you -would get in the console when using --verbose option. +Augmentations may add additional files at the run or job directory level. The +default configuration has ``status`` and ``csv`` augmentations enabled which +generate a ``status.txt`` containing status summary for the run and individual +jobs, and a ``results.csv`` containing metrics from all jobs in a CSV table, +respectively. -The output directory will also contain a ``results.json`` file that lists any -overall run metrics and artifacts that have been collected during the run. -Depending on the augmentations that were enabled there may be other results -files in different formats, for example the ``csv`` -:ref:`output processor `. In addition you will find a subdirectory -for each successful iteration executed with output and its own ``results.json`` -file for that specific iteration. +See :ref:`output_directory_structure` for more information. -If a job fails to complete for some reason, then the output directory for that -job will be moved into a new directory called ``__failed``. If the job was -running on a platform that supports android then WA will take a screen capture -and UI dump from the device. +In order to make it easier to access WA results from scripts, WA provides an API +that parses the contents of the output directory: -Finally, there will be a ``__meta`` subdirectory. This will contain a copy of -the agenda file used to run the workloads along with other configuration -files that were used for the execution of that run. + +.. code-block:: pycon + + >>> from wa import RunOutput + >>> ro = RunOutput('./wa_output') + >>> for job in ro.jobs: + ... if job.status != 'OK': + ... print 'Job "{}" did not complete successfully: {}'.format(job, job.status) + ... continue + ... print 'Job "{}":'.format(job) + ... for metric in job.metrics: + ... if metric.units: + ... print '\t{}: {} {}'.format(metric.name, metric.value, metric.units) + ... else: + ... print '\t{}: {}'.format(metric.name, metric.value) + ... + Job "wk1-dhrystone-1": + thread 0 score: 20833333 + thread 0 DMIPS: 11857 + thread 1 score: 24509804 + thread 1 DMIPS: 13950 + thread 2 score: 18011527 + thread 2 DMIPS: 10251 + thread 3 score: 26371308 + thread 3 DMIPS: 15009 + time: 1.001251 seconds + total DMIPS: 51067 + total score: 89725972 + execution_time: 1.4834280014 seconds + +See :ref:`output_processing_api` for details. Uninstall ========= diff --git a/doc/source/user_reference.rst b/doc/source/user_reference.rst index 1b794a6b..e2365aa3 100644 --- a/doc/source/user_reference.rst +++ b/doc/source/user_reference.rst @@ -13,4 +13,4 @@ User Reference .. include:: user_reference/configuration.rst .. include:: user_reference/invocation.rst -.. include:: user_reference/output_api.rst +.. include:: user_reference/output_directory.rst diff --git a/doc/source/user_reference/output_directory.rst b/doc/source/user_reference/output_directory.rst new file mode 100644 index 00000000..e68b94a1 --- /dev/null +++ b/doc/source/user_reference/output_directory.rst @@ -0,0 +1,132 @@ +.. _output_directory_structure: + +Output Directory Structure +========================== + +This is an overview of WA output directory structure. + +.. note:: In addition to files and subdirectories described here, + other content may present in the output directory for + a run, depending on the enabled augmentations. + +Overview +-------- + +The output directory will contain a subdirectory for every job iteration that +was run, as well as some additional entries. The following diagram illustrates +the typical structure of WA output directory:: + + wa_output/ + ├── __meta/ + │ ├── config.json + │ ├── jobs.json + │ ├── raw_config + │ │   ├── cfg0-config.yaml + │ │   └── agenda.yaml + │ ├── run_info.json + │ └── target_info.json + ├── __failed/ + │ └── wk1-dhrystone-1-attempt1 + ├── wk1-dhrystone-1/ + │ └── result.json + ├── wk1-dhrystone-2/ + │ └── result.json + ├── wk2-memcpy-1/ + │ └── result.json + ├── wk2-memcpy-2/ + │ └── result.json + ├── result.json + └── run.log + +This is the directory structure that would be generated after running two +iterations each of ``dhrystone`` and ``memcpy`` workloads with no augmentations +enabled, and with the first attempt at the first iteration of dhrystone having +failed. + +Output Directory Entries +------------------------ + +result.json + Contains a JSON structure describing the result of the execution, + including collected metrics and artifacts. There will be one for each + job execution, and one for the overall run. The run ``result.json`` will + only contain metrics/artifacts for the run as a whole, and will not + contain results for individual jobs. + + You typically would not access ``result.json`` files directly. Instead + you would either enable augmentations to format the results in easier to + manage form (such as CSV table), or use :ref:`output_processing_api` to + access the results from scripts. + + +run.log + This is a log of everything that happened during the run, including all + interactions with the target, and all the decisions made by the + framework. The output is equivalent to what you would see on the console + when running with ``--verbose`` option. + + .. note:: WA source contains a syntax file for Vim that will color the + initial part of each log line, in a similar way to what you + see on the console. This may be useful for quickly spotting + error and warning messages when scrolling through the log. + + https://github.com/ARM-software/workload-automation/blob/next/extras/walog.vim + +__meta + This directory contains configuration and run metadata. See + :ref:`config_and_meta` below for details. + +__failed + This directory will only be present if one or more job executions has + failed and were re-run. This directory contains output directories for + the failed attempts. + +job execution output subdirectory + Each subdirectory will be named ``__``, and will, at minimum, contain a ``result.json`` (see above). + Additionally, it may contain raw output from the workload, and any + additional artifacts (e.g. traces) generated by augmentations. Finally, + if workload execution has failed, WA may gather some additional logging + (such as the UI state at the time of failure) and place it here. + + +.. _config_and_meta: + +Configuration and Metadata +-------------------------- + +As stated above, the ``__meta`` directory contains run configuration and +metadata. Typically, you would not access these files directly, but would use +the :ref:`output_processing_api` to query the metadata. + +For more details about WA configuration see :ref:`configuration-specification`. + +config.json + Contains the overall run configuration, such as target interface + configuration, and job execution order, and various "meta-configuration" + settings, such as default output path, verbosity level, and logging + formatting. + +jobs.json + Final configuration for all jobs, including enabled augmentations, + workload and runtime parameters, etc. + +raw_config + This directory contains copies of config file(s) and the agenda that + were parsed in order to generate configuration for this run. Each config + file is prefixed with ``cfg-``, where ```` is the number + indicating the order (with respect to the other other config files) in + which it was parsed, e.g. ``cfg0-config.yaml`` is always a copy of + ``$WA_USER_DIRECTORY/config.yaml``. The one file without a prefix is the + agenda. + +run_info.json + Run metadata, e.g. duration, start/end timestamps and duration. + +target_info.json + Extensive information about the target. This includes information about + the target's CPUS configuration, kernel and userspace versions, etc. The + exact content will vary depending on the target type (Android vs Linux) + and what could accessed on a particular device (e.g. if + ``/proc/config.gz`` exists on the target, the kernel config will be + included).