mirror of
https://github.com/ARM-software/workload-automation.git
synced 2024-10-06 10:51:13 +01:00
03235d0b59
- move TargetInfo and RunInfo docs into the new output API doc - remove the old developer_reference/APIs.rst as obsolete - Update output API tag reference to point to the new doc
538 lines
18 KiB
ReStructuredText
538 lines
18 KiB
ReStructuredText
.. _output_processing_api:
|
|
|
|
Output
|
|
======
|
|
|
|
A WA output directory can be accessed via a :class:`RunOutput` object. There are
|
|
two ways of getting one -- either instantiate it with a path to a WA output
|
|
directory, or use :func:`discover_wa_outputs` to traverse a directory tree
|
|
iterating over all WA output directories found.
|
|
|
|
.. function:: discover_wa_outputs(path)
|
|
|
|
Recursively traverse ``path`` looking for WA output directories. Return
|
|
an iterator over :class:`RunOutput` objects for each discovered output.
|
|
|
|
:param path: The directory to scan for WA output
|
|
|
|
|
|
.. class:: RunOutput(path)
|
|
|
|
The main interface into a WA output directory.
|
|
|
|
:param path: must be the path to the top-level output directory (the one
|
|
containing ``__meta`` subdirectory and ``run.log``).
|
|
|
|
Example
|
|
-------
|
|
|
|
To demonstrate how we can use the output API if we have an existing WA output
|
|
called ``wa_output`` in the current working directory we can initialize a
|
|
``RunOutput`` as follows:
|
|
|
|
.. code-block:: python
|
|
|
|
In [1]: from wa import RunOutput
|
|
...:
|
|
...: output_directory = 'wa_output'
|
|
...: run_output = RunOutput(output_directory)
|
|
|
|
|
|
|
|
From here we can retrieve various information about the run. For example if we
|
|
want to see what the overall status of the run was, along with the runtime
|
|
parameters and the metrics recorded from the first job was we can do the following:
|
|
|
|
.. code-block:: python
|
|
|
|
In [2]: run_output.status
|
|
Out[2]: OK(7)
|
|
|
|
# List all of the jobs for the run
|
|
In [3]: run_output.jobs
|
|
Out[3]:
|
|
[<wa.framework.output.JobOutput at 0x7f70358a1f10>,
|
|
<wa.framework.output.JobOutput at 0x7f70358a1150>,
|
|
<wa.framework.output.JobOutput at 0x7f7035862810>,
|
|
<wa.framework.output.JobOutput at 0x7f7035875090>]
|
|
|
|
# Examine the first job that was ran
|
|
In [4]: job_1 = run_output.jobs[0]
|
|
|
|
In [5]: job_1.label
|
|
Out[5]: u'dhrystone'
|
|
|
|
# Print out all the runtime parameters and their values for this job
|
|
In [6]: for k, v in job_1.spec.runtime_parameters.iteritems():
|
|
...: print (k, v)
|
|
(u'airplane_mode': False)
|
|
(u'brightness': 100)
|
|
(u'governor': 'userspace')
|
|
(u'big_frequency': 1700000)
|
|
(u'little_frequency': 1400000)
|
|
|
|
# Print out all the metrics avalible for this job
|
|
In [7]: job_1.metrics
|
|
Out[7]:
|
|
[<thread 0 score: 14423105 (+)>,
|
|
<thread 0 DMIPS: 8209 (+)>,
|
|
<thread 1 score: 14423105 (+)>,
|
|
<thread 1 DMIPS: 8209 (+)>,
|
|
<thread 2 score: 14423105 (+)>,
|
|
<thread 2 DMIPS: 8209 (+)>,
|
|
<thread 3 score: 18292638 (+)>,
|
|
<thread 3 DMIPS: 10411 (+)>,
|
|
<thread 4 score: 17045532 (+)>,
|
|
<thread 4 DMIPS: 9701 (+)>,
|
|
<thread 5 score: 14150917 (+)>,
|
|
<thread 5 DMIPS: 8054 (+)>,
|
|
<time: 0.184497 seconds (-)>,
|
|
<total DMIPS: 52793 (+)>,
|
|
<total score: 92758402 (+)>]
|
|
|
|
|
|
|
|
We can also retrieve information about the target that the run was performed on
|
|
for example:
|
|
|
|
.. code-block:: python
|
|
|
|
# Print out the target's abi:
|
|
In [9]: run_output.target_info.abi
|
|
Out[9]: u'arm64'
|
|
|
|
# The os the target was running
|
|
In [9]: run_output.target_info.os
|
|
Out[9]: u'android'
|
|
|
|
# And other information about the os version
|
|
In [10]: run_output.target_info.os_version
|
|
Out[10]:
|
|
OrderedDict([(u'all_codenames', u'REL'),
|
|
(u'incremental', u'3687331'),
|
|
(u'preview_sdk', u'0'),
|
|
(u'base_os', u''),
|
|
(u'release', u'7.1.1'),
|
|
(u'codename', u'REL'),
|
|
(u'security_patch', u'2017-03-05'),
|
|
(u'sdk', u'25')])
|
|
|
|
|
|
|
|
:class:`RunOutput`
|
|
------------------
|
|
|
|
:class:`RunOutput` provides access to the output of a WA :term:`run`, including metrics,
|
|
artifacts, metadata, and configuration. It has the following attributes:
|
|
|
|
|
|
``jobs``
|
|
A list of :class:`JobOutput` objects for each job that was executed during
|
|
the run.
|
|
|
|
``status``
|
|
Run status. This indicates whether the run has completed without problems
|
|
(``Status.OK``) or if there were issues.
|
|
|
|
``metrics``
|
|
A list of :class:`Metric`\ s for the run.
|
|
|
|
.. note:: these are *overall run* metrics only. Metrics for individual
|
|
jobs are contained within the corresponding :class:`JobOutput`\ s.
|
|
|
|
``artifacts``
|
|
A list of :class:`Artifact`\ s for the run. These are usually backed by a
|
|
file and can contain traces, raw data, logs, etc.
|
|
|
|
.. note:: these are *overall run* artifacts only. Artifacts for individual
|
|
jobs are contained within the corresponding :class:`JobOutput`\ s.
|
|
|
|
``info``
|
|
A :ref:`RunInfo <run-info-api>` object that contains information about the run
|
|
itself for example it's duration, name, uuid etc.
|
|
|
|
``target_info``
|
|
A :ref:`TargetInfo <target-info-api>` object which can be used to access
|
|
various information about the target that was used during the run for example
|
|
it's ``abi``, ``hostname``, ``os`` etc.
|
|
|
|
``run_config``
|
|
A :ref:`RunConfiguration <run-configuration>` object that can be used to
|
|
access all the configuration of the run itself, for example the
|
|
``reboot_policy``, ``execution_order``, ``device_config`` etc.
|
|
|
|
``classifiers``
|
|
:ref:`classifiers <classifiers>` defined for the entire run.
|
|
|
|
``metadata``
|
|
:ref:`metadata <metadata>` associated with the run.
|
|
|
|
``events``
|
|
A list of any events logged during the run, that are not associated with a
|
|
particular job.
|
|
|
|
``event_summary``
|
|
A condensed summary of any events that occurred during the run.
|
|
|
|
``augmentations``
|
|
A list of the :term:`augmentation`\ s that were enabled during the run (these
|
|
augmentations may or may not have been active for a particular job).
|
|
|
|
``basepath``
|
|
A (relative) path to the WA output directory backing this object.
|
|
|
|
|
|
methods
|
|
~~~~~~~
|
|
|
|
.. method:: RunOutput.get_artifact(name)
|
|
|
|
Return the :class:`Artifact` specified by ``name``. This will only look
|
|
at the run artifacts; this will not search the artifacts of the individual
|
|
jobs.
|
|
|
|
:param name: The name of the artifact who's path to retrieve.
|
|
:return: The :class:`Artifact` with that name
|
|
:raises HostError: If the artifact with the specified name does not exist.
|
|
|
|
|
|
.. method:: RunOutput.get_artifact_path(name)
|
|
|
|
Return the path to the file backing the artifact specified by ``name``. This
|
|
will only look at the run artifacts; this will not search the artifacts of
|
|
the individual jobs.
|
|
|
|
:param name: The name of the artifact who's path to retrieve.
|
|
:return: The path to the artifact
|
|
:raises HostError: If the artifact with the specified name does not exist.
|
|
|
|
|
|
.. method:: RunOutput.get_metric(name)
|
|
|
|
Return the :class:`Metric` associated with the run (not the individual jobs)
|
|
with the specified `name`.
|
|
|
|
:return: The :class`Metric` object for the metric with the specified name.
|
|
|
|
|
|
.. method:: RunOutput.get_job_spec(spec_id)
|
|
|
|
Return the :class:`JobSpec` with the specified `spec_id`. A :term:`spec`
|
|
describes the job to be executed. Each :class:`Job` has an associated
|
|
:class:`JobSpec`, though a single :term:`spec` can be associated with
|
|
multiple :term:`job`\ s (If the :term:`spec` specifies multiple iterations).
|
|
|
|
.. method:: RunOutput.list_workloads()
|
|
|
|
List unique workload labels that featured in this run. The labels will be
|
|
in the order in which they first ran.
|
|
|
|
:return: A list of `str` labels of workloads that were part of this run.
|
|
|
|
|
|
:class:`JobOutput`
|
|
------------------
|
|
|
|
:class:`JobOutput` provides access to the output of a single :term:`job`
|
|
executed during a WA :term:`run`, including metrics,
|
|
artifacts, metadata, and configuration. It has the following attributes:
|
|
|
|
``status``
|
|
Job status. This indicates whether the job has completed without problems
|
|
(``Status.OK``) or if there were issues.
|
|
|
|
.. note:: Under typical configuration, WA will make a number of attempts to
|
|
re-run a job in case of issue. This status (and the rest of the
|
|
output) will represent the the latest attempt. I.e. a
|
|
``Status.OK`` indicates that the latest attempt was successful,
|
|
but it does mean that there weren't prior failures. You can check
|
|
the ``retry`` attribute (see below) to whether this was the first
|
|
attempt or not.
|
|
|
|
``retry``
|
|
Retry number for this job. If a problem is detected during job execution, the
|
|
job will be re-run up to :confval:`max_retries` times. This indicates the
|
|
final retry number for the output. A value of ``0`` indicates that the job
|
|
succeeded on the first attempt, and no retries were necessary.
|
|
|
|
.. note:: Outputs for previous attempts are moved into ``__failed``
|
|
subdirectory of WA output. These are currently not exposed via the
|
|
API.
|
|
|
|
``id``
|
|
The ID of the :term:`spec` associated with with job. This ID is unique to
|
|
the spec, but not necessary to the job -- jobs representing multiple
|
|
iterations of the same spec will share the ID.
|
|
|
|
``iteration``
|
|
The iteration number of this job. Together with the ``id`` (above), this
|
|
uniquely identifies a job with a run.
|
|
|
|
``label``
|
|
The workload label associated with this job. Usually, this will be the name
|
|
or :term:`alias` of the workload, however maybe overwritten by the user in
|
|
the :term:`agenda`.
|
|
|
|
``metrics``
|
|
A list of :class:`Metric`\ s for the job.
|
|
|
|
``artifacts``
|
|
A list of :class:`Artifact`\ s for the job These are usually backed by a
|
|
file and can contain traces, raw data, logs, etc.
|
|
|
|
``classifiers``
|
|
:ref:`classifiers <classifiers>` defined for the job.
|
|
|
|
``metadata``
|
|
:ref:`metadata <metadata>` associated with the job.
|
|
|
|
``events``
|
|
A list of any events logged during the execution of the job.
|
|
|
|
``event_summary``
|
|
A condensed summary of any events that occurred during the execution of the
|
|
job.
|
|
|
|
``augmentations``
|
|
A list of the :term:`augmentation`\ s that were enabled for this job. This may
|
|
be different from overall augmentations specified for the run, as they may be
|
|
enabled/disabled on per-job basis.
|
|
|
|
``basepath``
|
|
A (relative) path to the WA output directory backing this object.
|
|
|
|
|
|
methods
|
|
~~~~~~~
|
|
|
|
.. method:: RunOutput.get_artifact(name)
|
|
|
|
Return the :class:`Artifact` specified by ``name`` associated with this job.
|
|
|
|
:param name: The name of the artifact who's path to retrieve.
|
|
:return: The :class:`Artifact` with that name
|
|
:raises HostError: If the artifact with the specified name does not exist.
|
|
|
|
|
|
.. method:: RunOutput.get_artifact_path(name)
|
|
|
|
Return the path to the file backing the artifact specified by ``name``,
|
|
associated with this job.
|
|
|
|
:param name: The name of the artifact who's path to retrieve.
|
|
:return: The path to the artifact
|
|
:raises HostError: If the artifact with the specified name does not exist.
|
|
|
|
|
|
.. method:: RunOutput.get_metric(name)
|
|
|
|
Return the :class:`Metric` associated with this job with the specified
|
|
`name`.
|
|
|
|
:return: The :class`Metric` object for the metric with the specified name.
|
|
|
|
|
|
:class:`Metric`
|
|
---------------
|
|
|
|
A metric represent a single numerical measurement/score collected as a result of
|
|
running the workload. It would be generated either by the workload or by one of
|
|
the augmentations active during the execution of the workload.
|
|
|
|
A :class:`Metric` has the following attributes:
|
|
|
|
name
|
|
The name of the metric.
|
|
|
|
.. note:: A name of the metric is not necessarily unique, even for the same
|
|
job. Some workloads internally run multiple sub-tests, each
|
|
generating a metric with the same name. In such cases,
|
|
:term:`classifier`\ s are used to distinguish between them.
|
|
|
|
value
|
|
The value of the metrics collected.
|
|
|
|
|
|
units
|
|
The units of the metrics. This maybe ``None`` if the metric has no units.
|
|
|
|
|
|
lower_is_better
|
|
The default assumption is that higher metric values are better. This may be
|
|
overridden by setting this to ``True``, e.g. if metrics such as "run time"
|
|
or "latency". WA does not use this internally (at the moment) but this may
|
|
be used by external parties to sensibly process WA results in a generic way.
|
|
|
|
|
|
classifiers
|
|
These can be user-defined :term:`classifier`\ s propagated from the job/run,
|
|
or they may have been added by the workload to help distinguish between
|
|
otherwise identical metrics.
|
|
|
|
|
|
:class:`Artifact`
|
|
-----------------
|
|
|
|
An artifact is a file that is created on the host as part of executing a
|
|
workload. This could be trace, logging, raw output, or pretty much anything
|
|
else. Pretty much every file under WA output directory that is not already
|
|
represented by some other framework object will have an :class:`Artifact`
|
|
associated with it.
|
|
|
|
An :class:`Artifact` has the following attributes:
|
|
|
|
|
|
name
|
|
The name of this artifact. This will be unique for the job/run (unlike
|
|
metric names). This is intended as a consistent "handle" for this artifact.
|
|
The actual file name for the artifact may vary from job to job (e.g. some
|
|
benchmarks that create files with results include timestamps in the file
|
|
names), however the name will always be the same.
|
|
|
|
path
|
|
Partial path to the file associated with this artifact. Often, this is just
|
|
the file name. To get the complete path that maybe used to access the file,
|
|
use :func:`get_artifact_path` of the corresponding output object.
|
|
|
|
|
|
kind
|
|
Describes the nature of this artifact to facilitate generic processing.
|
|
Possible kinds are:
|
|
|
|
:log: A log file. Not part of the "output" as such but contains
|
|
information about the run/workload execution that be useful for
|
|
diagnostics/meta analysis.
|
|
:meta: A file containing metadata. This is not part of the "output", but
|
|
contains information that may be necessary to reproduce the
|
|
results (contrast with ``log`` artifacts which are *not*
|
|
necessary).
|
|
:data: This file contains new data, not available otherwise and should
|
|
be considered part of the "output" generated by WA. Most traces
|
|
would fall into this category.
|
|
:export: Exported version of results or some other artifact. This
|
|
signifies that this artifact does not contain any new data
|
|
that is not available elsewhere and that it may be safely
|
|
discarded without losing information.
|
|
:raw: Signifies that this is a raw dump/log that is normally processed
|
|
to extract useful information and is then discarded. In a sense,
|
|
it is the opposite of ``export``, but in general may also be
|
|
discarded.
|
|
|
|
.. note:: whether a file is marked as ``log``/``data`` or ``raw``
|
|
depends on how important it is to preserve this file,
|
|
e.g. when archiving, vs how much space it takes up.
|
|
Unlike ``export`` artifacts which are (almost) always
|
|
ignored by other exporters as that would never result
|
|
in data loss, ``raw`` files *may* be processed by
|
|
exporters if they decided that the risk of losing
|
|
potentially (though unlikely) useful data is greater
|
|
than the time/space cost of handling the artifact (e.g.
|
|
a database uploader may choose to ignore ``raw``
|
|
artifacts, where as a network filer archiver may choose
|
|
to archive them).
|
|
|
|
.. note:: The kind parameter is intended to represent the logical
|
|
function of a particular artifact, not it's intended means of
|
|
processing -- this is left entirely up to the output
|
|
processors.
|
|
|
|
description
|
|
This may be used by the artifact's creator to provide additional free-form
|
|
information about the artifact. In practice, this is often ``None``
|
|
|
|
|
|
classifiers
|
|
Job- and run-level :term:`classifier`\ s will be propagated to the artifact.
|
|
|
|
|
|
Additional run info
|
|
-------------------
|
|
|
|
:class:`RunOutput` object has ``target_info`` and ``run_info`` attributes that
|
|
contain structures that provide additional information about the run and device.
|
|
|
|
.. _target-info-api:
|
|
|
|
:class:`TargetInfo`
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
The :class:`TargetInfo` class presents various pieces of information about the
|
|
target device. An instance of this class will be instantiated and populated
|
|
automatically from the devlib `target
|
|
<http://devlib.readthedocs.io/en/latest/target.html>`_ created during a WA run
|
|
and serialized to a json file as part of the metadata exported
|
|
by WA at the end of a run.
|
|
|
|
The available attributes of the class are as follows:
|
|
|
|
target
|
|
The name of the target class that was uised ot interact with the device
|
|
during the run E.g. ``"AndroidTarget"``, ``"LinuxTarget"`` etc.
|
|
|
|
cpus
|
|
A list of :class:`CpuInfo` objects describing the capabilities of each CPU.
|
|
|
|
os
|
|
A generic name of the OS the target was running (e.g. ``"android"``).
|
|
|
|
os_version
|
|
A dict that contains a mapping of OS version elements to their values. This
|
|
mapping is OS-specific.
|
|
|
|
abi
|
|
The ABI of the target device.
|
|
|
|
hostname
|
|
The hostname of the the device the run was executed on.
|
|
|
|
is_rooted
|
|
A boolean value specifying whether root was detected on the device.
|
|
|
|
kernel_version
|
|
The version of the kernel on the target device. This returns a
|
|
:class:`KernelVersion` instance that has separate version and release
|
|
fields.
|
|
|
|
kernel_config
|
|
A :class:`KernelConfig` instance that contains parsed kernel config from the
|
|
target device. This may be ``None`` if the kernel config could not be
|
|
extracted.
|
|
|
|
sched_features
|
|
A list of the available tweaks to the scheduler, if available from the
|
|
device.
|
|
|
|
hostid
|
|
The unique identifier of the particular device the WA run was executed on.
|
|
|
|
|
|
.. _run-info-api:
|
|
|
|
:class:`RunInfo`
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
The :class:`RunInfo` provides general run information. It has the following
|
|
attributes:
|
|
|
|
|
|
uuid
|
|
A unique identifier for that particular run.
|
|
|
|
run_name
|
|
The name of the run (if provided)
|
|
|
|
project
|
|
The name of the project the run belongs to (if provided)
|
|
|
|
project_stage
|
|
The project stage the run is associated with (if provided)
|
|
|
|
duration
|
|
The length of time the run took to complete.
|
|
|
|
start_time
|
|
The time the run was stared.
|
|
|
|
end_time
|
|
The time at which the run finished.
|