Framework Overview ================== Execution Model --------------- At the high level, the execution model looks as follows: .. image:: developer_reference/WA_Execution.svg :scale: 100 % After some initial setup, the framework initializes the device, loads and initialized instruments and output processors and begins executing jobs defined by the workload specs in the agenda. Each job executes in basic stages: initialize Perform any once-per-run initialization of a workload instance, i.e. binary resource resolution. setup Initial setup for the workload is performed. E.g. required assets are deployed to the devices, required services or applications are launched, etc. Run time configuration of the device for the workload is also performed at this time. run This is when the workload actually runs. This is defined as the part of the workload that is to be measured. Exactly what happens at this stage depends entirely on the workload. extract results Extract any results that have been generated during the execution of the workload from the device and back to that target. Any files pulled from the devices should be added as artifacts to the run context. update output Perform any required parsing and processing of any collected results and add any generated metrics to the run context. teardown Final clean up is performed, e.g. applications may closed, files generated during execution deleted, etc. Signals are dispatched (see :ref:`below `) at each stage of workload execution, which installed instruments can hook into in order to collect measurements, alter workload execution, etc. Instruments implementation usually mirrors that of workloads, defining initialization, setup, teardown and output processing stages for a particular instrument. Instead of a ``run`` method instruments usually implement ``start`` and ``stop`` methods instead which triggered just before and just after a workload run. However, the signal dispatch mechanism gives a high degree of flexibility to instruments allowing them to hook into almost any stage of a WA run (apart from the very early initialization). Metrics and artifacts generated by workloads and instruments are accumulated by the framework and are then passed to active output processors. This happens after each individual workload execution and at the end of the run. A output processor may chose to act at either or both of these points. Control Flow ------------ This section goes into more detail explaining the relationship between the major components of the framework and how control passes between them during a run. It will only go through the major transitions and interactions and will not attempt to describe every single thing that happens. .. note:: This is the control flow for the ``wa run`` command which is the main functionality of WA. Other commands are much simpler and most of what is described below does not apply to them. #. :class:`wa.framework.entrypoint` parses the command from the arguments, creates a :class:`wa.framework.configuration.execution.ConfigManager` and executes the run command (:class:`wa.commands.run.RunCommand`) passing it the ConfigManger. #. Run command initializes the output directory and creates a :class:`wa.framework.configuration.parsers.AgendaParser` and will parser an agenda and populate the ConfigManger based on the command line arguments. Finally it instantiates a :class:`wa.framework.execution.Executor` and passes it the completed ConfigManager. #. The Executor uses the ConfigManager to create a :class:`wa.framework.configuration.core.RunConfiguration` and fully defines the configuration for the run (which will be serialised into ``__meta`` subdirectory under the output directory). #. The Executor proceeds to instantiate a TargetManager, used to handle the device connection and configuration, and a :class:`wa.framework.execution.ExecutionContext` which is used to track the current state of the run execution and also serves as a means of communication between the core framework and plugins. After this any required instruments and output processors are initialized and installed. #. Finally, the Executor instantiates a :class:`wa.framework.execution.Runner`, initializes its job queue with workload specs from the RunConfiguraiton, and kicks it off. #. The Runner performs the run time configuration of the device and goes through the workload specs (in the order defined by ``execution_order`` setting), running each spec according to the execution model described in the previous section and sending signals (see below) at appropriate points during execution. #. At the end of the run, the control is briefly passed back to the Executor, which outputs a summary for the run. .. _signal_dispatch: Signal Dispatch --------------- WA uses the `louie `_ (formerly, pydispatcher) library for signal dispatch. Callbacks can be registered for signals emitted during the run. WA uses a version of louie that has been modified to introduce priority to registered callbacks (so that callbacks that are know to be slow can be registered with a lower priority and therefore do not interfere with other callbacks). This mechanism is abstracted for instruments. Methods of an :class:`wa.framework.Instrument` subclass automatically get hooked to appropriate signals based on their names when the instrument is "installed" for the run. Priority can then be specified by adding ``extremely_fast``, ``very_fast``, ``fast`` , ``slow``, ``very_slow`` or ``extremely_slow`` :ref:`decorators ` to the method definitions. The full list of method names and the signals they map to may be viewed :ref:`here `. Signal dispatching mechanism may also be used directly, for example to dynamically register callbacks at runtime or allow plugins other than ``Instruments`` to access stages of the run they are normally not aware of. Signals can be either paired or non paired signals. Non paired signals are one off signals that are sent to indicate special events or transitions in execution stages have occurred for example ``TARGET_CONNECTED``. Paired signals are used to signify the start and end of a particular event. If the start signal has been sent the end signal is guaranteed to also be sent, whether the operation was a successes or not, however in the case of correct operation an additional success signal will also be sent. For example in the event of a successful reboot of the the device, the following signals will be sent ``BEFORE_REBOOT``, ``SUCCESSFUL_REBOOT`` and ``AFTER_REBOOT``. An overview of what signals are sent at which point during execution can be seen below. Most of the paired signals have been removed from the diagram for clarity and shown as being dispatched from a particular stage of execution, however in reality these signals will be sent just before and just after these stages are executed. As mentioned above for each of these signals there will be at least 2 and up to 3 signals sent. If the "BEFORE_X" signal (sent just before the stage is ran) is sent then the "AFTER_X" (sent just after the stage is ran) signal is guaranteed to also be sent, and under normal operation a "SUCCESSFUL_X" signal is also sent just after stage has been completed. The diagram also lists the conditional signals that can be sent at any time during execution if something unexpected happens, for example an error occurs or the user aborts the run. .. image:: developer_reference/WA_Signal_Dispatch.svg :scale: 100 % See Also -------- - :ref:`Instrumentation Signal-Method Mapping `.