--diable can now be used to disable both intruments and
result_processors from the command line (overriding configuration
files). Rename variable names to reflect this.
Replace update_enabled_instruments with update_augmentations, which
correctly splits instruments and results processors from augmentations
configuration values.
While instruments and processors are now handled uniformly in
configuration, they still need to be handled differently during actual
execution. This is the point where they get split again.
Remllace merge_instruments_result_processors with merge_agumentations
which is updated to properly merge the new unified "augmentations"
configurations.
Rename ConfigurationPoint "instrumenation" to "augmentations". Unlike
WA2, in W3, instruments and result_porcessors are handled identically in
configuration (though there're still distinct pulgin types).
"instrumentation" and "result_porcessors" entries in configuration are
now aliases for the same configuration point, which is renamed to
"augmentations" to reflect the fact that it contains both types of
plugins.
Expose PluginLoader's get_plugin_class via PluginCache. PluginCache is
basically intented to act like a PluginLoader, except when instantiating
plugins, the "default" parameter values are taken form the cached
configruation, rather than the actual defaults defined in the Plugin's
parmaters.
This maintains the default behaviour of bailing out immediately if any workload
fails in initialize(), but adds a setting, bail_on_init_failure, to change this
behaviour optionally. This can be useful where WA is being used more as a batch
processor.
Currently if you get confused and write a config with something like:
energy_measurement:
acme_cape
Then you get an error when we try to 'iteritems' on the value
'acme_cape'. Instead, explicitly check for this case.
Some workloads, such as Geekbench, may phone home and report data about the
device they are running on. This poses a risk for users that are testing on
unreleased or otherwise confidential devices - perhaps they use a standard
agenda to run a large battery of tests, in which case they may forget to disable
these dangerous workloads.
This provides a mechanism to prevent running those workloads from running by
setting allow_phone_home=False in the user configuration.
Currently if your config contains
typo_for_a_global_alias: 1
You will get an error where we try to call 'iteritems' on the value 1. This
commit re-orders the checks so that you instead get an error for the
unrecognised 'typo_for_a_global_alias'.
When an agenda contains multiple sections, the same workload entry might
be used in construction of multiple job specs. Job spec construction may
mangle the workload entry. To prevent this from impacting other
jobs, use a deep copy of the workload entry when constructing a job
spec.
PluginCache.get_plugin_config assumes that no more configuration is to
be processed, and therefore config is final. As such, it is validating
that mandatory parameters are set. This assumption is invalid for
workload_parameters, however, as those need to be resolved on per-spec
basis, and cannot be globally cached.
This commit adds a prameter for get_plugin_config that indicates whether
or not it should consider the config to be final.
Connection settings were not being properly extracted from device_config
and not used. Meaning it was impossible to e.g. specify the adb device
to use for a run.
This ensures that connection settings are extracted, validated, and used
properly.
- Re-order Status entries so that higher severity entries have higher
enum values.
- Add set_status() to Job that ensures that a status is only set if it
is of higher severity (e.g. a Job that has been marked as PARTIAL by
an instrument will not be overwritten as OK by the runner).
- Retry no generates a new job, rather than re-enqueuing the existing
object; this ensures that the output status is tracked properly.
- Adjust ManagedCallback to set set job status to FAILED if it sees a
WorkloadError, and to PARTIAL other wise. The idea being that
instruments raise WorkloadError if they have a reason to believe
workload did not execute properly and indicated failure even if the
workload itself has failed to detect it (e.g. FPS instrument detecting
crashed content, where the workload might lack any feedback regarding
the crash). Other errors would indicate an issue with the instrument
itself, and so the job is marked as PARTIAL, as there is no reason to
suspect that the workload is at fault and the other results generated
for this execution may be valid.
- What used to be enum.values is now enum.levels.
- Add enum.names and enum.values that are lists of enum's levels' names
and values respectively.
- Add a check on creation to make sure that provided level names do not
conflict with the atomatically created members.
When both global and config sections are present in an agenda, they both
will be processed by the ConfigManager. In each case, the path to the
agenda file was specified as the source, which resulted ConfigManager
complaining that the same source is being added twice. To avoid that,
the source is now suffixed with the section name.
Previously a config point would try to validate the name attribute of the object
it was meant to be setting, rather than what it was meant to be setting it to.
- Implemented result processor infrastructured
- Corrected some status tracking issues (differed between states
and output).
- Added "csv" and "status" result processors (these will be the default
enabled).
"params" is interpreted differently in section vs workload entires in
the agenda; previously, this was handled in the generic entry
construciton function by examining the ID prefix of the entry to
distinguish between the two.
This is unreliable as the user may specify their own IDs that won't have
the expected prefixes. To handle this, the "params" alias resolution for
sections is now handled in section specific part of entry processing
(workloads are handled autmatically because that is the default for
the corresponding JobConfig config point).
