Update the File and Executable resource string representations to
actually include the words "file" and "executable" respectively to make
messages containing these representations (e.g. when a resource is not
found) clearer.
--diable can now be used to disable both intruments and
result_processors from the command line (overriding configuration
files). Rename variable names to reflect this.
Replace update_enabled_instruments with update_augmentations, which
correctly splits instruments and results processors from augmentations
configuration values.
While instruments and processors are now handled uniformly in
configuration, they still need to be handled differently during actual
execution. This is the point where they get split again.
Remllace merge_instruments_result_processors with merge_agumentations
which is updated to properly merge the new unified "augmentations"
configurations.
Rename ConfigurationPoint "instrumenation" to "augmentations". Unlike
WA2, in W3, instruments and result_porcessors are handled identically in
configuration (though there're still distinct pulgin types).
"instrumentation" and "result_porcessors" entries in configuration are
now aliases for the same configuration point, which is renamed to
"augmentations" to reflect the fact that it contains both types of
plugins.
Expose PluginLoader's get_plugin_class via PluginCache. PluginCache is
basically intented to act like a PluginLoader, except when instantiating
plugins, the "default" parameter values are taken form the cached
configruation, rather than the actual defaults defined in the Plugin's
parmaters.
This maintains the default behaviour of bailing out immediately if any workload
fails in initialize(), but adds a setting, bail_on_init_failure, to change this
behaviour optionally. This can be useful where WA is being used more as a batch
processor.
Changes to the Status enum introduced by 31a535b5 and a9959550 broke
ran Jobs summary status at the end of the run. This fixes it so that the
total number of jobs and individual status counts are reported
correctly.
The generic "frequency" runtime parameter was only being set when there
are common frequences between avialable cores. It should always be set,
even if there are no frequencies in common, as it still valid to use it
with special values "min" and "max", in which case it should resolve
correctly to the appropriate frequencies.
Set context for the loggers of the Runner, the workloads and the
installed instruments and processors. Errors/warnings logged by these
entities will be automatically added as events.
This enables you to do things like:
global:
runtime_parameters:
airplane_mode: true # This improves repeatability of energy measurements
workloads:
- name: foo
- name: bar
- name: geekbench
runtime_parameters:
airplane_mode: false # Geekbench requires network, and we don't care
# about energy for that workload anyway
Fully matching an APK resource requires the file to be present locally,
so that its metadata can be queries. HTTP getter was matching against a
remote path so the match was failing.
The matching now happens in two stages == first partial path-only
matches are established. Secondly, all partial matches are downloaded
and final match occurs against downloaded files.
This method is used to partially match a resource; its implementation
cannot rely on the resource file actually being present and must match
against the specified path alone.
match() implementation now defaults to match_path(), as for most
resource types, the path is sufficient to uniquely match a resource.
Currently if you get confused and write a config with something like:
energy_measurement:
acme_cape
Then you get an error when we try to 'iteritems' on the value
'acme_cape'. Instead, explicitly check for this case.
- Make sure TargetManager.finalize() actually gets called at the end
of the run.
- Overrule the "diconnect" parameter behavior for gem5 and make sure it
always disconnects. This necessary for stats to be generated properly.
Gem5Platform requires a host output directory as one if it's
instantiation parameters. This is not something we want to expose a
configuration parameter to the user, as for WA, the standard output
directory ought to be used.
Up to this point, WA's target instatiation process assumed that all
parameters came from the user, and there was no way for WA itself to set
them. This commit adds extra_platform_parms argument to
instantiate_target, to remedi this.
extra_platform_parms is then used to set the host output directory for
gem5 appropriately.
(linux|android)_gem5 devices were being described as having gem5
platforms but standard linux/android connections. This commit fixes this
by making sure that the gem5 connection is used.
Runtime param names from config were matched against names available on
for the target using an overly-loose regex that resulted in the wrong
config point being returned.
Use caseless_string matching instead.
Some workloads, such as Geekbench, may phone home and report data about the
device they are running on. This poses a risk for users that are testing on
unreleased or otherwise confidential devices - perhaps they use a standard
agenda to run a large battery of tests, in which case they may forget to disable
these dangerous workloads.
This provides a mechanism to prevent running those workloads from running by
setting allow_phone_home=False in the user configuration.
Currently if your config contains
typo_for_a_global_alias: 1
You will get an error where we try to call 'iteritems' on the value 1. This
commit re-orders the checks so that you instead get an error for the
unrecognised 'typo_for_a_global_alias'.
- Add "activity" property to PackageManager exposing the underlying
apk_info attribute.
- Add "view" class attribute to ApkWorkload, initialized to None
- If "view" is not set for a particular workload, set a default based on
the package and activity of the underlying workload.
If an error occurs in a ManagedCallback that is invoked outside of a
job, re-raise rather than attempting to update the status of the
non-existent job.
- do not attempt to replay setup if a setup recording has not been
provided (perviously, setup was mandatory).
- update the apk initialization to the correct method inside the record
command.
When an agenda contains multiple sections, the same workload entry might
be used in construction of multiple job specs. Job spec construction may
mangle the workload entry. To prevent this from impacting other
jobs, use a deep copy of the workload entry when constructing a job
spec.
PluginCache.get_plugin_config assumes that no more configuration is to
be processed, and therefore config is final. As such, it is validating
that mandatory parameters are set. This assumption is invalid for
workload_parameters, however, as those need to be resolved on per-spec
basis, and cannot be globally cached.
This commit adds a prameter for get_plugin_config that indicates whether
or not it should consider the config to be final.
Connection settings were not being properly extracted from device_config
and not used. Meaning it was impossible to e.g. specify the adb device
to use for a run.
This ensures that connection settings are extracted, validated, and used
properly.
- Re-order Status entries so that higher severity entries have higher
enum values.
- Add set_status() to Job that ensures that a status is only set if it
is of higher severity (e.g. a Job that has been marked as PARTIAL by
an instrument will not be overwritten as OK by the runner).
- Retry no generates a new job, rather than re-enqueuing the existing
object; this ensures that the output status is tracked properly.
- Adjust ManagedCallback to set set job status to FAILED if it sees a
WorkloadError, and to PARTIAL other wise. The idea being that
instruments raise WorkloadError if they have a reason to believe
workload did not execute properly and indicated failure even if the
workload itself has failed to detect it (e.g. FPS instrument detecting
crashed content, where the workload might lack any feedback regarding
the crash). Other errors would indicate an issue with the instrument
itself, and so the job is marked as PARTIAL, as there is no reason to
suspect that the workload is at fault and the other results generated
for this execution may be valid.
Rename "package" parameter to "package_name" (to be cosnsitent with
package_names class attribute). "package" is now a property containting
the name of the actual package used after the APK is resolved.
- Add workload property as the shortcut for accessing the workoad for
the current job.
- Add set_status method that setts the status of the current job.
- Add get_metric method that searches for a metric in the output for the
current job, or failing that, in the run output.
- What used to be enum.values is now enum.levels.
- Add enum.names and enum.values that are lists of enum's levels' names
and values respectively.
- Add a check on creation to make sure that provided level names do not
conflict with the atomatically created members.
Previously any apk found would be selected even if the abi did not match
the device, now an apk is only selected if it is compatible with the
device. Additionally the 'exact_abi' parameter has been added to allow only
selecting an apk if a devices primary abi's native code is present in
the apk, or there is no native code.
Due to the fact that uiauto apk files built on different machines will
be signed with different keys, adb will fail to overwrite a previous
version even when set to replace. This commit now will uninstall the
previous uiauto apk file if present before attempting to install the new
version.
Previously initialize and finalize were being called for each iteration
of each workload at the start/end of the run which is incorrect
behaviour. To prevent this, each iteration of a workload now shares a
single instance of the workload combined with the 'once_per_instance' decorator
to ensure that the methods are only invoked once per set of workload
runs.
To support runs that require different versions of an application
the discovery/installation process should be performed at the beginning
of each iteration and therefore has been moved into setup.
In order to be able to pass the package name of the app under test to
the instrumented test, the generation of the uiautomator commands needed
to be postponed until after the apk resolution had been performed.
The starting activity is not always listed in an apk or required to start
an application to its default state, for example youtube. Therefore now the
starting activity will only be used if one is available.
In order to pass arguments to the instrumented test via the command line
they need to be encoded, this commit utilises a ParamDict to ensure
parameters are encoded correctly before passing them as arguments.
When both global and config sections are present in an agenda, they both
will be processed by the ConfigManager. In each case, the path to the
agenda file was specified as the source, which resulted ConfigManager
complaining that the same source is being added twice. To avoid that,
the source is now suffixed with the section name.
Before performing feature discovery on a device, all cores are attempted to be
onlined, however due to power management on some phones, cores can be immediately
hotplugged out again therefore causing the command to file. This commit now just
logs a message to warn that the online process failed and the information obtained
from a device may be incomplete.
Previously if one of the runtime parameters failed to initialize it would cause
the the entire setup process to fail. Now if an error is encountered e.g. a core
went offline when trying to read its information, that particular parameter will
be disabled along with it's supported parameters. This means that only if a user
attempts to use the RT parameter will execution be stopped however it will raise
a potentially misleading unsupported parameter error.
This commit removes the UI elements from ApkUIWorkload to provide a base
ApkWorkload class in order to support workloads that do not require a GUI.
It also remove the `@once` decorator as apk resolution may need to be
run more than once per run.
Previously apk resolution would fail if the application apk was not present on
device as this was used to retrieve information e.g. the package name. To work
around this, the resolution process now supports searching the device for either
a list of package names retrieved from the workload or as a parameter passed by
the user at runtime and if found on the device will be pulled to the host to
perform its analysis.
As uiautomator2 uses apk for instrumented tests this allows for distinguishing
between a normal application apk and a uiauto test apk based on the apks package
name.
The new method of using uiautomation 2 is by using instrumented apk files rather
than JAR files. This commit updates the base workload has been updated to
install/uninstall the new uiautomation APK files and to invoke the relevant
instrumentation.
WA2 Extensions had two features that have proven more trouble then they
are worth, and so they were removed from WA3 Plugins:
- Virtual methods (methods that automatically invoked super's version
without having to explicitly do that).
- It used to be possible to use strings or tuples to specify parameters
for extensions, and those were automatically "promoted" to Parameter
instances by the metaclass. This feature was never actually used.
This commit cleans up residual code for these features from Plugin
implementations and removes tests that used to exercise them.
Removing old and unsused files:
- wa/framework/old_output.py: superseded by output.py in the same dir
- the entire wlauto tree: replaced by wa/ tree; it's stale by now
anyway.
- log.py and actor.py from framework/ as neither is used.
assistant_params are now handled consistently with the other kinds of
params in TargetDescription, and they are now part of the default config
generated.
