- Have the job manage the indent of each of its stages
- Switch output_processor to using indentcontext
- Move the "Configuring augmentations" log line into Job -- the
rest of the stages are logged there, but this was done by the
Executor for some reason.
Only run finalize() for a job if initialize has succeed. finalize()
should be able to assume that initialize() has succeed, without needing
to check that that file have been created, variables set, etc.
If a target error occurs, check whether the target is unresponsive. If
it is, attempt to hard reset it if possible, or gracefully terminate
execution if not.
Save classifiers at Result as well as Metric level. Reason: when
processing output, one might want to filter complete results, as well as
individual metrics. While it is in theory possible to get the
classifiers for a job by simply extracting the common classifiers
between all metrics, this fails when there are no metrics generated for
a job (note that one might still want to process the output in this
case, e.g. for the artifacts).
Now that job output processing happens even on failure, this will result
in workload result processing also being triggered. We should not try to
extract failed workload results, so check the job status before
attempting to do so.
Set context for the loggers of the Runner, the workloads and the
installed instruments and processors. Errors/warnings logged by these
entities will be automatically added as events.
- Re-order Status entries so that higher severity entries have higher
enum values.
- Add set_status() to Job that ensures that a status is only set if it
is of higher severity (e.g. a Job that has been marked as PARTIAL by
an instrument will not be overwritten as OK by the runner).
- Retry no generates a new job, rather than re-enqueuing the existing
object; this ensures that the output status is tracked properly.
- Adjust ManagedCallback to set set job status to FAILED if it sees a
WorkloadError, and to PARTIAL other wise. The idea being that
instruments raise WorkloadError if they have a reason to believe
workload did not execute properly and indicated failure even if the
workload itself has failed to detect it (e.g. FPS instrument detecting
crashed content, where the workload might lack any feedback regarding
the crash). Other errors would indicate an issue with the instrument
itself, and so the job is marked as PARTIAL, as there is no reason to
suspect that the workload is at fault and the other results generated
for this execution may be valid.
Previously initialize and finalize were being called for each iteration
of each workload at the start/end of the run which is incorrect
behaviour. To prevent this, each iteration of a workload now shares a
single instance of the workload combined with the 'once_per_instance' decorator
to ensure that the methods are only invoked once per set of workload
runs.
- Workload's update_result stage has now been broken up into two parts:
extract_results and update_output. This is to allow the assistant to
pull output from the target in between the two stages.
- Updated assistant implementations for Linux and Android targets from
the exisiting code.
- Extended target descriptor code to handle assistants and their
parameters as well.
- Updated the target manager to actually make use of the assistants.
- Implemented result processor infrastructured
- Corrected some status tracking issues (differed between states
and output).
- Added "csv" and "status" result processors (these will be the default
enabled).
- Fixed up some of the signal map for instrumentation
- Changed how priorites are specified -- no longer method name prefixes
but dedicated decorators, including an easy way of specifying a custom
priority level (no longer need to manually connect signals)
- Updated ExecutionTimeInstrument to work with the new system
- Also removed some dead code
