The report_results parameter has been removed as the parsing functionality
is now performed by the `uxperf` processor therefore to report the
results of the application launch time ensure the processor has been
enabled.
Changes from checking if there are deployable assets to checking for
discovered assets. This allows for automated pushing of assets without
invoking the resource getters, for example pre discovered files.
Logcat entries are split on a ':' followed by a space so ensure this
is present in the line instead of just a ':'. Some devices have entires
without the trailing space causing an error.
Rename loop variable inside _write_outputs so that it doesn't clash with
the argument that gets passed (this resulted in writing to the wrong
location when writing a run artifact from a job).
Add a simplified version of the uxperf processor. This version only
extracts and calculates durations of UX_PERF sections from logcat.
Currently, this does not correlate them agains frames/fps data.
Now that job output processing happens even on failure, this will result
in workload result processing also being triggered. We should not try to
extract failed workload results, so check the job status before
attempting to do so.
Ensure that job output is processed even if a workload fails. This is
because output processing includes things like extracting logs, which
we still want to happen on failure.
Job status is now also set correctly when an error occurs during output
processing rather than actual running of the workload. Previously, the
status would be correctly set to PARTIAL in the inner except clause,
but the exception is then re-raised, and the status was "upgraded" to
FAILED in the outer except clause.
Do not concatenate start/stop markers to the action string with a '_',
delimit them with a space instead. This will make easier to parse the
message afterwards.
Mandate that either a package name is specified or the owner defines
supported package names. This prevents any apk file in the correct
location incorrectly being resolved.
Some workloads can insert timestamped markers into the logs if this
parameter is enabled in their automation. These markers can then be
used to look at measurements for specific parts of the execution.
These two lists are updated by workloads as assets are discovered and
deployed. As these are defined as class attributes in the base Workload
class, the same two list instances are shared between all classes. This
results in subsequent workloads re-deploying assets from the previous
workloads, and may lead to mismatches with deployable_assets, which is
overwritten on per-workload basis.
Make these into instance attriutes to prevent these issues.
Some targets need more than 5 seconds to re-connect to a wifi network
after disabling airplane mode. To fix that, poll for network up to 20
seconds and show a warning if unreachable.
Upon clearing data of a previously installed application the runtime
permissions can also be cleared, to avoid having to reinstall the
application all available permissions are explicitly granted to the
required application.
Allows for automatically pushing a set of arbitrary assets to the device
during initialization and adds a parameter to specify whether the assets should
be removed again at the end of the run.
Adds various methods that are used in UxPerf workloads and additional
helper class for UxPerf specific functions.
Adds a class to contain UxPerf specific functions.
In order to properly enable verbose logging for plugin enumeration,
verbosity option must be extracted before commands are processed, and
therefore before all valid arguments are known.
ArgumentParser.parse_known_args() is used for this, which will collect
all unexpected arguments into an "extras" list, rather than raising an
error. This, however, has an outstanding bug, where if say option '-v'
is known and '-f is unknown, args '-v -f' will be parsed correctly (with
'-f' being collected into extras), but '-vf' will not, resulting in an
error.
To get around this, pre-process sys.argv to detect joined options and
manually split them into separate args.
We currently populate results_so_far with a JobOutput for each Job and then a
Result for the RunOutput. This results in a bug when trying to access the
id/label/iteration.
This is fixed by always ensuring the we store Output objects and not
Results (results_so_far is renamed to outputs_so_far to reflect this), and
treating the RunOutput specially in _write_outputs.
Update the File and Executable resource string representations to
actually include the words "file" and "executable" respectively to make
messages containing these representations (e.g. when a resource is not
found) clearer.
--diable can now be used to disable both intruments and
result_processors from the command line (overriding configuration
files). Rename variable names to reflect this.
Replace update_enabled_instruments with update_augmentations, which
correctly splits instruments and results processors from augmentations
configuration values.
While instruments and processors are now handled uniformly in
configuration, they still need to be handled differently during actual
execution. This is the point where they get split again.
Remllace merge_instruments_result_processors with merge_agumentations
which is updated to properly merge the new unified "augmentations"
configurations.
Rename ConfigurationPoint "instrumenation" to "augmentations". Unlike
WA2, in W3, instruments and result_porcessors are handled identically in
configuration (though there're still distinct pulgin types).
"instrumentation" and "result_porcessors" entries in configuration are
now aliases for the same configuration point, which is renamed to
"augmentations" to reflect the fact that it contains both types of
plugins.
