sascha/workload-automation
1
0
Fork 0
mirror of https://github.com/ARM-software/workload-automation.git synced 2025-10-01 01:22:35 +01:00
Code Issues Releases Wiki Activity

doc: Restructure

Browse Source 
Restructure the documentation to be split into `User Information` and
`Developer Information`, and split the how to guides into their
corresponding section.
This commit is contained in:
Marc Bonnici
2018-06-20 17:43:52 +01:00
committed by setrofim
parent 3c0f1968c5
commit 6c93590062
28 changed files with 122 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
doc/source/developer_information/developer_reference/WA_Execution.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 42 KiB

2
doc/source/developer_information/developer_reference/WA_Signal_Dispatch.svg Normal file
View File

File diff suppressed because one or more lines are too long
Side by Side

After

Width:  |  Height:  |  Size: 63 KiB

55
doc/source/developer_information/developer_reference/contributing.rst Normal file
View File

@@ -0,0 +1,55 @@
Contributing Code
=================
We welcome code contributions via GitHub pull requests. To help with
maintainability of the code line we ask that the code uses a coding style
consistent with the rest of WA code. Briefly, it is
- `PEP8 <https://www.python.org/dev/peps/pep-0008/>`_ with line length and block
comment rules relaxed (the wrapper for PEP8 checker inside ``dev_scripts``
will run it with appropriate configuration).
- Four-space indentation (*no tabs!*).
- Title-case for class names, underscore-delimited lower case for functions,
methods, and variables.
- Use descriptive variable names. Delimit words with ``'_'`` for readability.
Avoid shortening words, skipping vowels, etc (common abbreviations such as
"stats" for "statistics", "config" for "configuration", etc are OK). Do
*not* use Hungarian notation (so prefer ``birth_date`` over ``dtBirth``).
New extensions should also follow implementation guidelines specified in the
:ref:`writing-plugins` section of the documentation.
We ask that the following checks are performed on the modified code prior to
submitting a pull request:
.. note:: You will need pylint and pep8 static checkers installed::
pip install pep8
pip install pylint
It is recommended that you install via pip rather than through your
distribution's package manager because the latter is likely to
contain out-of-date version of these tools.
- ``./dev_scripts/pylint`` should be run without arguments and should produce no
output (any output should be addressed by making appropriate changes in the
code or adding a pylint ignore directive, if there is a good reason for
keeping the code as is).
- ``./dev_scripts/pep8`` should be run without arguments and should produce no
output (any output should be addressed by making appropriate changes in the
code).
- If the modifications touch core framework (anything under ``wa/framework``), unit
tests should be run using ``nosetests``, and they should all pass.
- If significant additions have been made to the framework, unit
tests should be added to cover the new functionality.
- If modifications have been made to documentation (this includes description
attributes for Parameters and Extensions), documentation should be built to
make sure no errors or warning during build process, and a visual inspection
of new/updated sections in resulting HTML should be performed to ensure
everything renders as expected.
Once you have your contribution is ready, please follow instructions in `GitHub
documentation <https://help.github.com/articles/creating-a-pull-request/>`_ to
create a pull request.

158
doc/source/developer_information/developer_reference/execution_model.rst Normal file
View File

@@ -0,0 +1,158 @@
Framework Overview
==================
Execution Model
---------------
At the high level, the execution model looks as follows:
.. image:: developer_information/developer_reference/WA_Execution.svg
:scale: 100 %
After some initial setup, the framework initializes the device, loads and
initialized instruments and output processors and begins executing jobs defined
by the workload specs in the agenda. Each job executes in basic stages:
initialize
Perform any once-per-run initialization of a workload instance, i.e.
binary resource resolution.
setup
Initial setup for the workload is performed. E.g. required assets are
deployed to the devices, required services or applications are launched,
etc. Run time configuration of the device for the workload is also
performed at this time.
setup_rerun (apk based workloads only)
For some apk based workloads the application is required to be started
twice. If the ``requires_rerun`` attribute of the workload is set to
``True`` then after the first setup method is called the application
will be killed and then restarted. This method can then be used to
perform any additional setup required.
run
This is when the workload actually runs. This is defined as the part of
the workload that is to be measured. Exactly what happens at this stage
depends entirely on the workload.
extract results
Extract any results that have been generated during the execution of the
workload from the device and back to that target. Any files pulled from
the devices should be added as artifacts to the run context.
update output
Perform any required parsing and processing of any collected results and
add any generated metrics to the run context.
teardown
Final clean up is performed, e.g. applications may closed, files
generated during execution deleted, etc.
Signals are dispatched (see :ref:`below <signal_dispatch>`) at each stage of
workload execution, which installed instruments can hook into in order to
collect measurements, alter workload execution, etc. Instruments implementation
usually mirrors that of workloads, defining initialization, setup, teardown and
output processing stages for a particular instrument. Instead of a ``run``
method instruments usually implement ``start`` and ``stop`` methods instead
which triggered just before and just after a workload run. However, the signal
dispatch mechanism gives a high degree of flexibility to instruments allowing
them to hook into almost any stage of a WA run (apart from the very early
initialization).
Metrics and artifacts generated by workloads and instruments are accumulated by
the framework and are then passed to active output processors. This happens
after each individual workload execution and at the end of the run. A output
processor may chose to act at either or both of these points.
Control Flow
------------
This section goes into more detail explaining the relationship between the major
components of the framework and how control passes between them during a run. It
will only go through the major transitions and interactions and will not attempt
to describe every single thing that happens.
.. note:: This is the control flow for the ``wa run`` command which is the main
functionality of WA. Other commands are much simpler and most of what
is described below does not apply to them.
#. :class:`wa.framework.entrypoint` parses the command from the arguments, creates a
:class:`wa.framework.configuration.execution.ConfigManager` and executes the run
command (:class:`wa.commands.run.RunCommand`) passing it the ConfigManger.
#. Run command initializes the output directory and creates a
:class:`wa.framework.configuration.parsers.AgendaParser` and will parser an
agenda and populate the ConfigManger based on the command line arguments.
Finally it instantiates a :class:`wa.framework.execution.Executor` and
passes it the completed ConfigManager.
#. The Executor uses the ConfigManager to create a
:class:`wa.framework.configuration.core.RunConfiguration` and fully defines the
configuration for the run (which will be serialised into ``__meta`` subdirectory
under the output directory).
#. The Executor proceeds to instantiate a TargetManager, used to handle the
device connection and configuration, and a
:class:`wa.framework.execution.ExecutionContext` which is used to track the
current state of the run execution and also serves as a means of
communication between the core framework and plugins. After this any required
instruments and output processors are initialized and installed.
#. Finally, the Executor instantiates a :class:`wa.framework.execution.Runner`,
initializes its job queue with workload specs from the RunConfiguraiton, and
kicks it off.
#. The Runner performs the run time configuration of the device and goes
through the workload specs (in the order defined by ``execution_order``
setting), running each spec according to the execution model described in the
previous section and sending signals (see below) at appropriate points during
execution.
#. At the end of the run, the control is briefly passed back to the Executor,
which outputs a summary for the run.
.. _signal_dispatch:
Signal Dispatch
---------------
WA uses the `louie <https://github.com/11craft/louie/>`_ (formerly,
pydispatcher) library for signal dispatch. Callbacks can be registered for
signals emitted during the run. WA uses a version of louie that has been
modified to introduce priority to registered callbacks (so that callbacks that
are know to be slow can be registered with a lower priority and therefore do not
interfere with other callbacks).
This mechanism is abstracted for instruments. Methods of an
:class:`wa.framework.Instrument` subclass automatically get hooked to
appropriate signals based on their names when the instrument is "installed"
for the run. Priority can then be specified by adding ``extremely_fast``,
``very_fast``, ``fast`` , ``slow``, ``very_slow`` or ``extremely_slow``
:ref:`decorators <instruments_method_map>` to the method definitions.
The full list of method names and the signals they map to may be viewed
:ref:`here <instruments_method_map>`.
Signal dispatching mechanism may also be used directly, for example to
dynamically register callbacks at runtime or allow plugins other than
``Instruments`` to access stages of the run they are normally not aware of.
Signals can be either paired or non paired signals. Non paired signals are one
off signals that are sent to indicate special events or transitions in execution
stages have occurred for example ``TARGET_CONNECTED``. Paired signals are used to
signify the start and end of a particular event. If the start signal has been
sent the end signal is guaranteed to also be sent, whether the operation was a
successes or not, however in the case of correct operation an additional success
signal will also be sent. For example in the event of a successful reboot of the
the device, the following signals will be sent ``BEFORE_REBOOT``,
``SUCCESSFUL_REBOOT`` and ``AFTER_REBOOT``.
An overview of what signals are sent at which point during execution can be seen
below. Most of the paired signals have been removed from the diagram for clarity
and shown as being dispatched from a particular stage of execution, however in
reality these signals will be sent just before and just after these stages are
executed. As mentioned above for each of these signals there will be at least 2
and up to 3 signals sent. If the "BEFORE_X" signal (sent just before the stage
is ran) is sent then the "AFTER_X" (sent just after the stage is ran) signal is
guaranteed to also be sent, and under normal operation a "SUCCESSFUL_X" signal
is also sent just after stage has been completed. The diagram also lists the
conditional signals that can be sent at any time during execution if something
unexpected happens, for example an error occurs or the user aborts the run.
.. image:: developer_information/developer_reference/WA_Signal_Dispatch.svg
:scale: 100 %
See Also
--------
- :ref:`Instrumentation Signal-Method Mapping <instruments_method_map>`.

337
doc/source/developer_information/developer_reference/revent.rst Normal file
View File

@@ -0,0 +1,337 @@
Revent Recordings
=================
Convention for Naming revent Files for Revent Workloads
-------------------------------------------------------------------------------
There is a convention for naming revent files which you should follow if you
want to record your own revent files. Each revent file must start with the
device name(case sensitive) then followed by a dot '.' then the stage name
then '.revent'. All your custom revent files should reside at
``'~/.workload_automation/dependencies/WORKLOAD NAME/'``. These are the current
supported stages:
:setup: This stage is where the application is loaded (if present). It is
a good place to record an revent here to perform any tasks to get
ready for the main part of the workload to start.
:run: This stage is where the main work of the workload should be performed.
This will allow for more accurate results if the revent file for this
stage only records the main actions under test.
:extract_results: This stage is used after the workload has been completed
to retrieve any metrics from the workload e.g. a score.
:teardown: This stage is where any final actions should be performed to
clean up the workload.
Only the run stage is mandatory, the remaining stages will be replayed if a
recording is present otherwise no actions will be performed for that particular
stage.
For instance, to add a custom revent files for a device named "mydevice" and
a workload name "myworkload", you need to add the revent files to the directory
``/home/$WA_USER_HOME/dependencies/myworkload/revent_files`` creating it if
necessary. ::
mydevice.setup.revent
mydevice.run.revent
mydevice.extract_results.revent
mydevice.teardown.revent
Any revent file in the dependencies will always overwrite the revent file in the
workload directory. So for example it is possible to just provide one revent for
setup in the dependencies and use the run.revent that is in the workload directory.
File format of revent recordings
--------------------------------
You do not need to understand recording format in order to use revent. This
section is intended for those looking to extend revent in some way, or to
utilize revent recordings for other purposes.
Format Overview
^^^^^^^^^^^^^^^
Recordings are stored in a binary format. A recording consists of three
sections::
+-+-+-+-+-+-+-+-+-+-+-+
| Header |
+-+-+-+-+-+-+-+-+-+-+-+
| |
| Device Description |
| |
+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| Event Stream |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+
The header contains metadata describing the recording. The device description
contains information about input devices involved in this recording. Finally,
the event stream contains the recorded input events.
All fields are either fixed size or prefixed with their length or the number of
(fixed-sized) elements.
.. note:: All values below are little endian
Recording Header
^^^^^^^^^^^^^^^^
An revent recoding header has the following structure
* It starts with the "magic" string ``REVENT`` to indicate that this is an
revent recording.
* The magic is followed by a 16 bit version number. This indicates the format
version of the recording that follows. Current version is ``2``.
* The next 16 bits indicate the type of the recording. This dictates the
structure of the Device Description section. Valid values are:
``0``
This is a general input event recording. The device description
contains a list of paths from which the events where recorded.
``1``
This a gamepad recording. The device description contains the
description of the gamepad used to create the recording.
* The header is zero-padded to 128 bits.
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 'R' | 'E' | 'V' | 'E' |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| 'N' | 'T' | Version |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Mode | PADDING |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| PADDING |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Device Description
^^^^^^^^^^^^^^^^^^
This section describes the input devices used in the recording. Its structure is
determined by the value of ``Mode`` field in the header.
general recording
^^^^^^^^^^^^^^^^^
.. note:: This is the only format supported prior to version ``2``.
The recording has been made from all available input devices. This section
contains the list of ``/dev/input`` paths for the devices, prefixed with total
number of the devices recorded.
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of devices |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Device paths +-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Similarly, each device path is a length-prefixed string. Unlike C strings, the
path is *not* NULL-terminated.
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Length of device path |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| Device path |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
gamepad recording
^^^^^^^^^^^^^^^^^
The recording has been made from a specific gamepad. All events in the stream
will be for that device only. The section describes the device properties that
will be used to create a virtual input device using ``/dev/uinput``. Please
see ``linux/input.h`` header in the Linux kernel source for more information
about the fields in this section.
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| bustype | vendor |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| product | version |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| name_length |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| name |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ev_bits |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| key_bits (96 bytes) |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| rel_bits (96 bytes) |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| abs_bits (96 bytes) |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| num_absinfo |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| |
| |
| absinfo entries |
| |
| |
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Each ``absinfo`` entry consists of six 32 bit values. The number of entries is
determined by the ``abs_bits`` field.
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| value |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| minimum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| maximum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| fuzz |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| flat |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| resolution |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Event stream
^^^^^^^^^^^^
The majority of an revent recording will be made up of the input events that were
recorded. The event stream is prefixed with the number of events in the stream,
and start and end times for the recording.
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of events |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Number of events (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Start Time Seconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Start Time Seconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Start Time Microseconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Start Time Microseconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| End Time Seconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| End Time Seconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| End Time Microseconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| End Time Microseconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
| Events |
| |
| |
| +-+-+-+-+-+-+-+-+-+-+-+-+
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Event structure
^^^^^^^^^^^^^^^
Each event entry structured as follows:
* An unsigned short integer representing which device from the list of device paths
this event is for (zero indexed). E.g. Device ID = 3 would be the 4th
device in the list of device paths.
* A unsigned long integer representing the number of seconds since "epoch" when
the event was recorded.
* A unsigned long integer representing the microseconds part of the timestamp.
* An unsigned integer representing the event type
* An unsigned integer representing the event code
* An unsigned integer representing the event value
For more information about the event type, code and value please read:
https://www.kernel.org/doc/Documentation/input/event-codes.txt
::
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Device ID | Timestamp Seconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp Seconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp Seconds (cont.) | stamp Micoseconds |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp Micoseconds (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Timestamp Micoseconds (cont.) | Event Type |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Event Code | Event Value |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Event Value (cont.) |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Parser
^^^^^^
WA has a parser for revent recordings. This can be used to work with revent
recordings in scripts. Here is an example:
.. code:: python
from wa.utils.revent import ReventRecording
with ReventRecording('/path/to/recording.revent') as recording:
print "Recording: {}".format(recording.filepath)
print "There are {} input events".format(recording.num_events)
print "Over a total of {} seconds".format(recording.duration)

123
doc/source/developer_information/developer_reference/serialization.rst Normal file
View File

@@ -0,0 +1,123 @@
.. _serialization:
Serialization
=============
Overview of Serialization
-------------------------
WA employs a serialization mechanism in order to store some of its internal
structures inside the output directory. Serialization is performed in two
stages:
1. A serializable object is converted into a POD (Plain Old Data) structure
consisting of primitive Python types, and a few additional types (see
:ref:`wa-pods` below).
2. The POD structure is serialized into a particular format by a generic
parser for that format. Currently, `yaml` and `json` are supported.
Deserialization works in reverse order -- first the serialized text is parsed
into a POD, which is then converted to the appropriate object.
Implementing Serializable Objects
---------------------------------
In order to be considered serializable, an object must either be a POD, or it
must implement the ``to_pod()`` method and ``from_pod`` static/class method,
which will perform the conversion to/form pod.
As an example, below as a (somewhat trimmed) implementation of the ``Event``
class:
.. code-block:: python
class Event(object):
@staticmethod
def from_pod(pod):
instance = Event(pod['message'])
instance.timestamp = pod['timestamp']
return instance
def __init__(self, message):
self.timestamp = datetime.utcnow()
self.message = message
def to_pod(self):
return dict(
timestamp=self.timestamp,
message=self.message,
)
Serialization API
-----------------
.. function:: read_pod(source, fmt=None)
.. function:: write_pod(pod, dest, fmt=None)
These read and write PODs from a file. The format will be inferred, if
possible, from the extension of the file, or it may be specified explicitly
with ``fmt``. ``source`` and ``dest`` can be either strings, in which case
they will be interpreted as paths, or they can be file-like objects.
.. function:: is_pod(obj)
Returns ``True`` if ``obj`` is a POD, and ``False`` otherwise.
.. function:: dump(o, wfh, fmt='json', \*args, \*\*kwargs)
.. function:: load(s, fmt='json', \*args, \*\*kwargs)
These implment an altenative serialization interface, which matches the
interface exposed by the parsers for the supported formats.
.. _wa-pods:
WA POD Types
------------
POD types are types that can be handled by a serializer directly, without a need
for any additional information. These consist of the build-in python types ::
list
tuple
dict
set
str
unicode
int
float
bool
...the standard library types ::
OrderedDict
datetime
...and the WA-defined types ::
regex_type
none_type
level
cpu_mask
Any structure consisting entirely of these types is a POD and can be serialized
and then deserialized without losing information. It is important to note that
only these specific types are considered POD, their subclasses are *not*.
.. note:: ``dict``\ s get deserialized as ``OrderedDict``\ s.
Serialization Formats
---------------------
WA utilizes two serialization formats: YAML and JSON. YAML is used for files
intended to be primarily written and/or read by humans; JSON is used for files
intended to be primarily written and/or read by WA and other programs.
The parsers and serializers for these formats used by WA have been modified to
handle additional types (e.g. regular expressions) that are typically not
supported by the formats. This was done in such a way that the resulting files
are still valid and can be parsed by any parser for that format.

1262
doc/source/developer_information/developer_reference/writing_plugins.rst Normal file
View File

File diff suppressed because it is too large Load Diff