The `install_timeout` variable is used to allow extra time when
installing an apk, if the apk is not present on the host system it
will be pulled back to the host, therefore we should allow extra time
when transferring the file in the other direction as well.
Add process command to run output processors over data produced by
a previous run.
Default processing is to skip the processors that were used during
the run and to process the specified directory only. Options to
use additional processors, force reprocessing and allow a directory
walk to discover additional output directories.
If the http getter is used but a connection is not avalible, when
attempting to fetch the resource index an error will be raised. Now we
ignore the error allowing for the remaining getters to be used.
Port manual workload from WA2. This workload allows using WA to
configure and instrument the target, while letting the user perform
arbitrary actions in place of running a fixed workload.
since Linaro's modification have been merged in upstream version,
we can remove this comment
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
resistor_values is an array of float and not int as the shunt resistors
are most of the time lower than 1 Ohm.
With kind=list_of_ints, all resistor values are rounded to 0. Replace it
by list_of_numbers
sample_rate_hz is an int used to get sample period and to compute energy.
Changes its type from str to int
With these 2 fixes, the daq instrument has been successfully tested with
NI DAQ6211
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Make sure runtime_params and boot_params are to_pod()'d when they get
written, as they are obj_dict's that json serializer does not know how
to handle otherwise.
Ensure that boot and runtime parameters always get initialized to an
obj_dict even when not explicitly specified. This will ensure they get
serialized and deserialized properly.
Arm Energy Probe with arm_probe has been recently added in devlib's
instrument.
Add the arm_energy_probe in the WA list of Energy measurement instruments
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
energy instruments might need to save platform specific configurations in
order to help to reproduce the use case. Such configuration should be saved in
__meta directory.
Add the directory path as a parameter of get_instruments() to save such
informations
Suggested-by: setrofim@gmail.com
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Catch exceptions raised when attempting to set value of a config point,
and re-raise as ConfigError with name associated with the value in the
config file.
Strings are iterable, so can be used to instantiate sets (resulting in a
set of chars). This is never what we want for toggle_set's though, and
may result in difficult-to-interpret errors when parsing configuration,
so raise a TypeError if attempting to create a toggle_set with a string.
Now allows a `prefer_host_package` parameter to be set to control
whether a package located on the host system (`True`) or on the target
(`False`) should be favoured during resolution.
`pull_apk` now returns the path to the apk it pulled from the device so
WA can keep track of it and avoid conflicts with other apks present
on the host.
Upon initilisation WA3 will now also check if a WA3 config file
exisits, rather than just the exsistance of the config directory.
If a `config.yaml` file is not found WA will look for a WA2 config.py
file and convert it to the new format, otherwise a default config file
will be generated.
Record UI state if an error occurs during setup, run, and output
processing stages (for other stages, the UI state is unlikely to be
relevant as they typically would not include UI manipulation).
Only run finalize() for a job if initialize has succeed. finalize()
should be able to assume that initialize() has succeed, without needing
to check that that file have been created, variables set, etc.
Ensure the workload uses the passed working directory for attaching
images instead of a hard coded name and allow for finding of the attach
button by description as well as name.
Fix an issue where AndroidWorkload.pull_apk would sometimes get the
wrong package if the desired package name is a substring of another
package name. Rather than using a regex to match the package name, use
the new get_package_info method to match the name exactly.
Not all benchmarks are available on all phones and will report "Not
supported" To allow for this update the regexes and if not a valid score
set any unsupported results to 'NaN'
This is WA2 sqlite result processor ported to WA3 output processor API.
It is schema-compatible with the WA2 version (in fact, there are *no*
schema changes).
Note: There is a medium-term goal of adding a more generic SQL database
exporter with a more complete schema sitting on top of something like
sqlalchemy to make it agnostic of DBMS implementations. At point, this
will likely be deprecated.
Update the 'package' argument description to specify it is for android
packages and add error checking to ensure that if specified, the target
supports android.
Change the recording of the run stage during workload recording mode to be
optional as the user may want to use the command to re-record a specific
stage of a workload without affecting the workload recording.
Hitting CTRL-C will abort execution of the current job, but will still
trigger run finalization, and possibly, post-processing and teardown of
the current job. If an exception is raised during this
post-process/teardown, the previous exception state (for the
KeybardInterrupt) will be clobbered. That means that, after the new
exception has been handled, WA would attempt to execute the next job,
rather than go to finalization of the run.
To avoid this, set a flag in the context upon catching KeybardInterrupt,
and check this flag before attempting to execute the next job in the
queue.
Rather than relying on a custom Logger with a context to add events when
they are logged, have the runner register hooks for corresponding
signals that do that.
The issue with the previous approach is that the event ended up
being added too late -- after the current job was already cleared, so
all events were added at run level.
With the new approach, the event will be added when it is first logged,
while there is still a current job being set. There will be no
duplication for Exceptions being re-raised and handled at different
levels because log_error() will ensure that each Exception is logged
only once.
- Add SimpleTargetDescriptor that simply returns a global list of target
descriptions.
- Add create_target_description function to populate that list with a
description crated from the specified components.
- Add add_description_for_target helper function to create a description
for the specified target class, and specified components. For
components that are not explicitly specified, attempt to guess
sensible defaults based on the target class' bases.
- fix TargetDescription._set to set lists rather than dicts for params
to be consistent with how params are set after creation (should
probably get rid of this entirely...)
- fix the comment describing the structure of TARGETS
- Keep track of logged exceptions inside log_error itself.
- signal: log the exception, if there is one in the finally clause of
the signal wrapper; this will ensure that the error will be logged
closer to the command that originated.
- entrypoint: use log.log_error for top-level error logging, rather than
the entrypoint logger directly; this will ensure that errors are not
repeated unnecessarily.
- Log CTRL-C message at zeroth indent level to make it easier to see in
the non-verbose output where it occurred.
If a target error occurs, check whether the target is unresponsive. If
it is, attempt to hard reset it if possible, or gracefully terminate
execution if not.
Add a method to check that the target is still responsive, and cache
the value so that the target does not need to be repeatedly probed
during various stages of wrapping up execution.
We remove the requirement for rt-app to run on a rooted device as this
does not appear to be required. We make sure that we run the workload
from a directory where we have write permissions by first changing the
directory before executing the workload (in_directory argument of
invoke).
In the case that the device is rooted, we run rt-app as
root. Otherwise, we simply run the workload.
By default rt-app writes to stderr, not to stdout. Therefore, if we
only sample the stdout, we get no output from the workload itself.
With this change, we redirect stderr to stdout to allow us to get the
workload output.
ExecutionContext.end_run() does final updates to the run info in the run
output (final status, run duration, etc). This was previously accessed
via self.output in the context. Typically, this would correctly resolve
to the run output, as there would be no current job. However, in the
event of a crash, current_job would be set, and this would resolve to
the job output itself, resulting in run info not being updated. Use
run_output to avoid this.
- RebootPolicy: correctly handle being passed another instance of
RebootPolicy on creation; this a requirement for it to be usable as a
ConfigurationPoint kind.
- RebootPolicy: move static method to above __init__ (consistency).
- RunConfiguration: handle deserialization of 'device_config' entry.
- RunConfiguration: move static method to above __init__ (consistency).
Save classifiers at Result as well as Metric level. Reason: when
processing output, one might want to filter complete results, as well as
individual metrics. While it is in theory possible to get the
classifiers for a job by simply extracting the common classifiers
between all metrics, this fails when there are no metrics generated for
a job (note that one might still want to process the output in this
case, e.g. for the artifacts).
Set target_binary as a class, rather than instance, attribute. This
happens only only once per run, and setting it as instance attribute the
first time, makes it unavailable for subsequent instances of the same
workload.
Do not explode if a result file for a job is missing when loading a
RunOutput. Specify job status as "UNKNOWN" and add the exception from
attempting to load the file to the events.
Fix an issue introduced by commit 42fb3eb
framework/workload: make "no matching package" message more useful
PackageHandler was incorrectly passing self.name into the error message
instead of self.owner.name.
Make the error message reported when no matching Android app package
could be found more useful by including the workload name, and, if
applicable, sought version.
Always prepend the format for the file into the downloaded filepath to
ensure matching is correctly performed. Previously the ogg file did not
specify this causing it to be re-downloaded for each run.
Rename the `start` method to `run` as this is what is what is called
by the threading module's `start` method, otherwise this causes the
polling to be done in the main thread blocking execution.
Add support for loading plugins from packages that register themselves
with WA by writing their names into $WA_USER_DIRECTORY/packages.
This is functionality that existed in WA2 and was omitted in WA3 until
now.
Output processor names are converted to identifiers when they are
loaded. Ensure the same thing happens when looking up installed output
processors with get_output_processor().
In devlib the default working directory has changed causing the workload
to look in the wrong directory, now the workload should use the passed
working directory and extract the relevant name from there.
Both the instrument and utility functions were called `trace-cmd.py`.
`"-"`'s are not valid python identifiers therefore when trying to import
the `trace-cmd` module this causes an error.
- Add an option to the poller binary insert a marker into ftrace that
aligns with the first output entry. The option is exposed as an
instrument parameter.
- If this parameter is set, the resulting .csv will be post-processed to
update the timestamps to align with trace-cmd output.
- Change poller artifact names to use - instead of _ to be consistent
with trace-cmd artifact naming.
Allows setting of select android runtime configuration for devices running
chromeos and which support android. Currently only 'brightness' is functioning
correctly therefore the other parameters are only enable for standard android devices.
Previously when validating cpufreq runtime parameters it would check too
early whether the cpu was online, not allowing for the fact that the cpu
might be being onlined before the parameter was committed to the device.
To improve reproducibility of tests, it could help to ensure that the
same screen orientation is used on media playback.
This adds a new workload parameter which allows to ensure a LANDSCAPE
mode, which can be useful to better represent video playback scenarios.
When not enforced by the agenda, we enforce a PORTRAIT orientation.
Signed-off-by: Patrick Bellasi <patrick.bellasi@arm.com>
Fix the setting of runtime parameters when the casing of the parameter in
the agenda does not match the "canonical" casing.
To make the writing of agendas easier, the casing of the parameters is
supposed to be ignored. To achieve this, parameter names are converted to
caseless_string type before they are looked up.
caseless_string's do not work with dicts. Both __contains__ (used for
the "in" operator) and __getitem__ (used for the [] operator) implement
hash-based look up, and it is not possible to have a caseless_string
match against multiple hashes to cover the different casing
possibilities.
So instead, iterate over the items in parameters dict, comparing the
caseless_string name to the key, and returning the value if it matches.
On some devices an installed apk filepath can contain an '=' character
which was previously used to end the regex match. Now match with the
package name as well to ensure the file path is extracted correctly.
Uses os.eviron as the default environment for creation of a
MetaConfiguration object, this allows an instance to be created without
providing any parameters and fixes re-creating an instance from pod.
Changed location of revent binary on target to be a class attribute so that
it is shared across all instances. Also changed discovery method to a
static method which can be memoized to prevent the target being queried
unnecessarily.
use the new get_target_description() when initializing the target inside
TargetManager, instead of manually searching through results of
list_target_descriptions().
The show command will try to use pandoc and man to display the relevant
information and will fall back to printing the markdown if pandoc is not
available.
Instead of taking an extension as a parameter and assuming there it has
an attribute 'parameters' instead take a list of parameters directly
which allows the method to be used with parameter lists with different
names.
Make TargetInfo an attribute of run output, replacing the read/write
methods for the targetfile. Instead, always load it on creation, if
targetfile exists (useful for external scripts), and have a method to
set it after creation (uselful during WA run, where the output is
created before connecting to the target).
Fix TargetInfo.from_pod(). As part of creating the POD, some of devlib's
classes such as Cpuinfo, get serialized. Since they are from devlib,
they do not implement WA's serialization interface, and are instead
serialized by extracting their internal structures. They are then not
deserialized properly, since their __init__'s expect text rather than
parsed structures.
This implements deserialization functions for these classes which
initialize them properly from TargetInfo POD.
Fix output when running for a specified number of loops (rather than
duration) and adding delay to threads. Previously, the output for the
last thread was not printed due to a missing check for a negative
duration.
Don't construct an ArtifactType in Output.Add_artifact, the Artifact
class does that for us.
Next, fix the use of a nonexistent attribute Artifact.valid_kinds
Previously only the top level help message would ever be displayed,
this was caused by 'parse_known_commands' automatically displaying the
default help message and exiting before any of the custom plugins are
loaded. Now ensure this flag is never passed into the method.
Remove wa.framework.plugin.Artifact and associated references. The name
of the class clashes with the class from output and can potentially
cause confusion.
The original intention for this was to be an "expected artifact
descriptor" of sorts that plugins can specify for validation purposes,
but that functionality was never implemented. Given that the framework
has undergone significant changes since this was implemented, it's not
clear that this is the best way to go about the original goal.
Therefore remove this for now.
To reflect that the current implementation of ReventWorkload uses an APK
file on the device the name has been changed appropriately and the
relevant workloads have been updated.
- Use get_config_point_map() to ensure aliases are parsed from
configuraiton.
- Only set the default for a name only if it matches the corresponding
param's name (i.e. don't set for alises).
The report_results parameter has been removed as the parsing functionality
is now performed by the `uxperf` processor therefore to report the
results of the application launch time ensure the processor has been
enabled.
Changes from checking if there are deployable assets to checking for
discovered assets. This allows for automated pushing of assets without
invoking the resource getters, for example pre discovered files.
Logcat entries are split on a ':' followed by a space so ensure this
is present in the line instead of just a ':'. Some devices have entires
without the trailing space causing an error.
Rename loop variable inside _write_outputs so that it doesn't clash with
the argument that gets passed (this resulted in writing to the wrong
location when writing a run artifact from a job).
Add a simplified version of the uxperf processor. This version only
extracts and calculates durations of UX_PERF sections from logcat.
Currently, this does not correlate them agains frames/fps data.
Now that job output processing happens even on failure, this will result
in workload result processing also being triggered. We should not try to
extract failed workload results, so check the job status before
attempting to do so.
Ensure that job output is processed even if a workload fails. This is
because output processing includes things like extracting logs, which
we still want to happen on failure.
Job status is now also set correctly when an error occurs during output
processing rather than actual running of the workload. Previously, the
status would be correctly set to PARTIAL in the inner except clause,
but the exception is then re-raised, and the status was "upgraded" to
FAILED in the outer except clause.
Do not concatenate start/stop markers to the action string with a '_',
delimit them with a space instead. This will make easier to parse the
message afterwards.
Mandate that either a package name is specified or the owner defines
supported package names. This prevents any apk file in the correct
location incorrectly being resolved.
Some workloads can insert timestamped markers into the logs if this
parameter is enabled in their automation. These markers can then be
used to look at measurements for specific parts of the execution.
These two lists are updated by workloads as assets are discovered and
deployed. As these are defined as class attributes in the base Workload
class, the same two list instances are shared between all classes. This
results in subsequent workloads re-deploying assets from the previous
workloads, and may lead to mismatches with deployable_assets, which is
overwritten on per-workload basis.
Make these into instance attriutes to prevent these issues.
Some targets need more than 5 seconds to re-connect to a wifi network
after disabling airplane mode. To fix that, poll for network up to 20
seconds and show a warning if unreachable.
Upon clearing data of a previously installed application the runtime
permissions can also be cleared, to avoid having to reinstall the
application all available permissions are explicitly granted to the
required application.
Allows for automatically pushing a set of arbitrary assets to the device
during initialization and adds a parameter to specify whether the assets should
be removed again at the end of the run.
Adds various methods that are used in UxPerf workloads and additional
helper class for UxPerf specific functions.
Adds a class to contain UxPerf specific functions.
In order to properly enable verbose logging for plugin enumeration,
verbosity option must be extracted before commands are processed, and
therefore before all valid arguments are known.
ArgumentParser.parse_known_args() is used for this, which will collect
all unexpected arguments into an "extras" list, rather than raising an
error. This, however, has an outstanding bug, where if say option '-v'
is known and '-f is unknown, args '-v -f' will be parsed correctly (with
'-f' being collected into extras), but '-vf' will not, resulting in an
error.
To get around this, pre-process sys.argv to detect joined options and
manually split them into separate args.
We currently populate results_so_far with a JobOutput for each Job and then a
Result for the RunOutput. This results in a bug when trying to access the
id/label/iteration.
This is fixed by always ensuring the we store Output objects and not
Results (results_so_far is renamed to outputs_so_far to reflect this), and
treating the RunOutput specially in _write_outputs.
Update the File and Executable resource string representations to
actually include the words "file" and "executable" respectively to make
messages containing these representations (e.g. when a resource is not
found) clearer.
--diable can now be used to disable both intruments and
result_processors from the command line (overriding configuration
files). Rename variable names to reflect this.
Replace update_enabled_instruments with update_augmentations, which
correctly splits instruments and results processors from augmentations
configuration values.
While instruments and processors are now handled uniformly in
configuration, they still need to be handled differently during actual
execution. This is the point where they get split again.
Remllace merge_instruments_result_processors with merge_agumentations
which is updated to properly merge the new unified "augmentations"
configurations.
Rename ConfigurationPoint "instrumenation" to "augmentations". Unlike
WA2, in W3, instruments and result_porcessors are handled identically in
configuration (though there're still distinct pulgin types).
"instrumentation" and "result_porcessors" entries in configuration are
now aliases for the same configuration point, which is renamed to
"augmentations" to reflect the fact that it contains both types of
plugins.
Expose PluginLoader's get_plugin_class via PluginCache. PluginCache is
basically intented to act like a PluginLoader, except when instantiating
plugins, the "default" parameter values are taken form the cached
configruation, rather than the actual defaults defined in the Plugin's
parmaters.
Now that we add an event when logging errors, we need to avoid passing an
exception object directly to the logger. Otherwise we end up adding the
unserializable Exception object to the events list, which causes an error when
trying to write the output.
This maintains the default behaviour of bailing out immediately if any workload
fails in initialize(), but adds a setting, bail_on_init_failure, to change this
behaviour optionally. This can be useful where WA is being used more as a batch
processor.
Changes to the Status enum introduced by 31a535b5 and a9959550 broke
ran Jobs summary status at the end of the run. This fixes it so that the
total number of jobs and individual status counts are reported
correctly.
The generic "frequency" runtime parameter was only being set when there
are common frequences between avialable cores. It should always be set,
even if there are no frequencies in common, as it still valid to use it
with special values "min" and "max", in which case it should resolve
correctly to the appropriate frequencies.
Set context for the loggers of the Runner, the workloads and the
installed instruments and processors. Errors/warnings logged by these
entities will be automatically added as events.
Add ContextLogger wich extends logging.Logger's warning and error
methods to automatically add the message to the context's events (if a
context has been provided).
This enables you to do things like:
global:
runtime_parameters:
airplane_mode: true # This improves repeatability of energy measurements
workloads:
- name: foo
- name: bar
- name: geekbench
runtime_parameters:
airplane_mode: false # Geekbench requires network, and we don't care
# about energy for that workload anyway
For some reason I didn't look at the WA2 instrument until afer I
wrote this, so it's different in how it handles 'before/after'
quantities (which I called 'momentary' in the code) - instead of
adding a 'mean' it just adds two separate metrics: one for before and
one for after. Personally I think this is more useful (esp. for
temperature) but if necessary I can revert to the old behaviour.
This also doesn't add the helpful classifier telling you the hwmon
device, because we don't have immediate access to that info from the
devlib Instrument. If necessary I'm sure it could be added later.
Furthermore it doesn't have a parameter to filter the sensors, just
collects from them all.
Where backends have the capability to collect from multiple devlib
Instruments, EnergyMeasurement currently appends 'device' keys to the
names of metrics reported by those Instruments. Where multiple
Instruments are indeed used, it then sums up equivalent metrics to
produce the 'normal' metric without the 'device' keys
appended.
E.g. If you have two ACME 'devices' enabled you will get something
like
'device_total_energy_iio:device0' and
'device_total_energy_iio:device1', which will be summed to produce
'device_total_energy'.
However when only one Instrument is actually instantiated, this is
not needed and not done. Therefore just directly produce the metric
with the name originally given by devlib (i.e. in the example,
'device_total_energy').
Fully matching an APK resource requires the file to be present locally,
so that its metadata can be queries. HTTP getter was matching against a
remote path so the match was failing.
The matching now happens in two stages == first partial path-only
matches are established. Secondly, all partial matches are downloaded
and final match occurs against downloaded files.
This method is used to partially match a resource; its implementation
cannot rely on the resource file actually being present and must match
against the specified path alone.
match() implementation now defaults to match_path(), as for most
resource types, the path is sufficient to uniquely match a resource.
Devlib's get_data API can return None when something goes wrong -
this currently results in us attempting to add a non-existent
artifact.
To avoid this confusion, check and use the return value from
get_data, instead of assuming that it will write data to its
'outfile' parameter.
Currently if you get confused and write a config with something like:
energy_measurement:
acme_cape
Then you get an error when we try to 'iteritems' on the value
'acme_cape'. Instead, explicitly check for this case.
- Fix start/stop methods of sysfile_getter, which where still using the
WA2 mechanic of prefixing "slow_" to the name to indicate priority,
and consequently weren't being invoked.
- Call super inside cpufreq's valide(() to make sure tempfs path is set.
- Make sure TargetManager.finalize() actually gets called at the end
of the run.
- Overrule the "diconnect" parameter behavior for gem5 and make sure it
always disconnects. This necessary for stats to be generated properly.
devlib's CGgroups module logs things at INFO level. The information it
logs is too detailed for what WA considers "INFO", and should be logged
at DEBUG level instead.
Gem5Platform requires a host output directory as one if it's
instantiation parameters. This is not something we want to expose a
configuration parameter to the user, as for WA, the standard output
directory ought to be used.
Up to this point, WA's target instatiation process assumed that all
parameters came from the user, and there was no way for WA itself to set
them. This commit adds extra_platform_parms argument to
instantiate_target, to remedi this.
extra_platform_parms is then used to set the host output directory for
gem5 appropriately.
(linux|android)_gem5 devices were being described as having gem5
platforms but standard linux/android connections. This commit fixes this
by making sure that the gem5 connection is used.
Seem I cherry-picked the geekbench workload from my WIP branch
without testing properly, and missed a fixup commit. Here's the
content of that commit, which fixes some missing updates from WA2 ->
WA3 APIs.
Devlib's ACME instrument uses iio-capture under the hood, which can
only capture data from one IIO device at a time. Devlib's instrument
API expects to produce a single CSV file for the Instrument, with a
single axis of sample timestamps. These two things cannot be
correctly reconciled without changing the devlib Instrument API -
get_data would need to be able to return two distinct sets of data.
Instead, where required WA will instantiate the ACME instrument
multiple times (once for each IIO device), producing two separate CSV
files. Aggregated energy info (as opposed to timestamped samples of
energy data) _can_ be meaningfully combined from multiple IIO
devices, so the derived stats are summed.
This requires altering the internal API of the energy_measurement
module - rather than a single instrument the super-instrument now has
a dictionary mapping 'device' keys (this is really an arbitrary key,
but I think that 'device' is a justifiable metaphor for whatever
might separate multiple instruments, plus it is the terminology used
by IIO/ACME) to Instrument objects. Existing instruments simply
return a dict with a single entry, with None as the key.
It may in future be possible to solve this problem in devlib's ACME
instrument by avoiding the use of the iio-capture tool. In that case,
it should be possible to basically revert this patch and get the
required funcitonality from devlib.
Runtime param names from config were matched against names available on
for the target using an overly-loose regex that resulted in the wrong
config point being returned.
Use caseless_string matching instead.
As paramters defined by energy_measurement instrument backends were not
used in instantiating the backends, but the underlying devlib
instrument, the values did not undergo the usual Plugin parameter
setting perocedure. In particuar, type conversion would not take place.
Make sure backend parameters are validated properly by setting them on
an obj_dict before passing them to the devlib instrument.
Some workloads, such as Geekbench, may phone home and report data about the
device they are running on. This poses a risk for users that are testing on
unreleased or otherwise confidential devices - perhaps they use a standard
agenda to run a large battery of tests, in which case they may forget to disable
these dangerous workloads.
This provides a mechanism to prevent running those workloads from running by
setting allow_phone_home=False in the user configuration.
Currently if your config contains
typo_for_a_global_alias: 1
You will get an error where we try to call 'iteritems' on the value 1. This
commit re-orders the checks so that you instead get an error for the
unrecognised 'typo_for_a_global_alias'.
Differences from original:
- Standard renaming and rearrangement of imports
- UiAutomation.java now uses the setup() and runWorkload() phases which weren't
present before.
Due to the nature of modern Android devices, in order to usefully
interpret the power* events, you more and more often need to know the
kernel's view of temperature too. Therefore I think enabling the
thermal event group by default makes sense.
- Add "activity" property to PackageManager exposing the underlying
apk_info attribute.
- Add "view" class attribute to ApkWorkload, initialized to None
- If "view" is not set for a particular workload, set a default based on
the package and activity of the underlying workload.
- Ensure that CLOCK_MONOTONIC is used for all timestamps, so that the
timestamps collected by revent and those reported in the struct
input_event wouild match.
- Do not attempt to free events array inside fini_revent_recroding() if
no events have been recorded.
- Add missing \n in some debug messages.
- Add end of recording debug message.
- Make sure start/end timestamps are properly adjusted when there are
zero events.
- Rename adjust_event_times --> adjust_timestamps, as the latter better
fits its current function.
- Add a couple of debug messages.
DerivedMeasurements in devlib was changed to return DerivedMetrics
instead of Measurements. The former do not have "channel" attribute, so
just pass its name to WA's metrics.
If an error occurs in a ManagedCallback that is invoked outside of a
job, re-raise rather than attempting to update the status of the
non-existent job.
The begining and end of recording timestamps are now stored as part of
the recording. This allows to correctly replay recordings where the
first event occurs some time after the recording started, or the last
event some time before recording ended.
Add pre-built revent binaries for the armeabi and arm64 architectures.
- do not attempt to replay setup if a setup recording has not been
provided (perviously, setup was mandatory).
- update the apk initialization to the correct method inside the record
command.
When an agenda contains multiple sections, the same workload entry might
be used in construction of multiple job specs. Job spec construction may
mangle the workload entry. To prevent this from impacting other
jobs, use a deep copy of the workload entry when constructing a job
spec.
PluginCache.get_plugin_config assumes that no more configuration is to
be processed, and therefore config is final. As such, it is validating
that mandatory parameters are set. This assumption is invalid for
workload_parameters, however, as those need to be resolved on per-spec
basis, and cannot be globally cached.
This commit adds a prameter for get_plugin_config that indicates whether
or not it should consider the config to be final.
Connection settings were not being properly extracted from device_config
and not used. Meaning it was impossible to e.g. specify the adb device
to use for a run.
This ensures that connection settings are extracted, validated, and used
properly.
- Re-order Status entries so that higher severity entries have higher
enum values.
- Add set_status() to Job that ensures that a status is only set if it
is of higher severity (e.g. a Job that has been marked as PARTIAL by
an instrument will not be overwritten as OK by the runner).
- Retry no generates a new job, rather than re-enqueuing the existing
object; this ensures that the output status is tracked properly.
- Adjust ManagedCallback to set set job status to FAILED if it sees a
WorkloadError, and to PARTIAL other wise. The idea being that
instruments raise WorkloadError if they have a reason to believe
workload did not execute properly and indicated failure even if the
workload itself has failed to detect it (e.g. FPS instrument detecting
crashed content, where the workload might lack any feedback regarding
the crash). Other errors would indicate an issue with the instrument
itself, and so the job is marked as PARTIAL, as there is no reason to
suspect that the workload is at fault and the other results generated
for this execution may be valid.
Rename "package" parameter to "package_name" (to be cosnsitent with
package_names class attribute). "package" is now a property containting
the name of the actual package used after the APK is resolved.
- Add workload property as the shortcut for accessing the workoad for
the current job.
- Add set_status method that setts the status of the current job.
- Add get_metric method that searches for a metric in the output for the
current job, or failing that, in the run output.
It seems that occasionally an inclomplete recording file is pulled. Add
a one-second (target time) delay to allow SIGINT to be properly handled
and recording to be updated, followed by a sync to ensure the file has
been written.
The interactive governor isn't standard any more (and was
Android-only anyway). Remove this default so you don't get errors for
kernels that don't support it.
Delvib now is capable of performing postprocessing of MeasurementCSV
files, instead of calculating additional metrics in WA this will be
performed externally. Currently support has been added for calculating
average power and cumulative energy.
- What used to be enum.values is now enum.levels.
- Add enum.names and enum.values that are lists of enum's levels' names
and values respectively.
- Add a check on creation to make sure that provided level names do not
conflict with the atomatically created members.
Previously any apk found would be selected even if the abi did not match
the device, now an apk is only selected if it is compatible with the
device. Additionally the 'exact_abi' parameter has been added to allow only
selecting an apk if a devices primary abi's native code is present in
the apk, or there is no native code.
- Remove reference to default events from the overall workload
documentation. It was, as of recently, outdated, and was also
redundant, as the actual defaults will be in the parameter-specific
documentation.
- Remove reference to Android-specific trace-cmd binary -- this was not
true for a long time.
- Clarify that the on-host trace-cmd binary is now optional due to the
report_on_target config.
Due to the fact that uiauto apk files built on different machines will
be signed with different keys, adb will fail to overwrite a previous
version even when set to replace. This commit now will uninstall the
previous uiauto apk file if present before attempting to install the new
version.
Previously initialize and finalize were being called for each iteration
of each workload at the start/end of the run which is incorrect
behaviour. To prevent this, each iteration of a workload now shares a
single instance of the workload combined with the 'once_per_instance' decorator
to ensure that the methods are only invoked once per set of workload
runs.
Previously there was no differentiation between the same method at different
inheritance levels, therefore using the once_per_instance_decorator with super call would
prevent the super call from executing as they were considered the same
instance.
To support runs that require different versions of an application
the discovery/installation process should be performed at the beginning
of each iteration and therefore has been moved into setup.
In order to be able to pass the package name of the app under test to
the instrumented test, the generation of the uiautomator commands needed
to be postponed until after the apk resolution had been performed.
The starting activity is not always listed in an apk or required to start
an application to its default state, for example youtube. Therefore now the
starting activity will only be used if one is available.
In order to pass arguments to the instrumented test via the command line
they need to be encoded, this commit utilises a ParamDict to ensure
parameters are encoded correctly before passing them as arguments.
When both global and config sections are present in an agenda, they both
will be processed by the ConfigManager. In each case, the path to the
agenda file was specified as the source, which resulted ConfigManager
complaining that the same source is being added twice. To avoid that,
the source is now suffixed with the section name.
Add to/from_pod implementations to level, so that it may be serialized
properly. In order to make sure that serialization is a reversible
operation, ensure that the level value is numeric.
Before performing feature discovery on a device, all cores are attempted to be
onlined, however due to power management on some phones, cores can be immediately
hotplugged out again therefore causing the command to file. This commit now just
logs a message to warn that the online process failed and the information obtained
from a device may be incomplete.
Previously if one of the runtime parameters failed to initialize it would cause
the the entire setup process to fail. Now if an error is encountered e.g. a core
went offline when trying to read its information, that particular parameter will
be disabled along with it's supported parameters. This means that only if a user
attempts to use the RT parameter will execution be stopped however it will raise
a potentially misleading unsupported parameter error.
This commit removes the UI elements from ApkUIWorkload to provide a base
ApkWorkload class in order to support workloads that do not require a GUI.
It also remove the `@once` decorator as apk resolution may need to be
run more than once per run.
Previously apk resolution would fail if the application apk was not present on
device as this was used to retrieve information e.g. the package name. To work
around this, the resolution process now supports searching the device for either
a list of package names retrieved from the workload or as a parameter passed by
the user at runtime and if found on the device will be pulled to the host to
perform its analysis.
As uiautomator2 uses apk for instrumented tests this allows for distinguishing
between a normal application apk and a uiauto test apk based on the apks package
name.
The new method of using uiautomation 2 is by using instrumented apk files rather
than JAR files. This commit updates the base workload has been updated to
install/uninstall the new uiautomation APK files and to invoke the relevant
instrumentation.
This re-adds the parser for the trace-cmd text trace, after it got
dropped along with wlauto sub-tree. There are minor differences from the
old WA2 parser.
- Changed to work with devlib start/stop markers
- TraceCmdTrace renamed to TraceCmdParser
- The file path is now passed to the parse() method rather than
on creation.
- detect whether markers are present in the trace and do not filter
if they aren't
In the case where the device screen was off prior to the execution of
the workload, but the user did not set screen_off, the screen will be
turned on during the setup. Previously, it would remain on for the
subsequent execution, however the correct behavior is for the workload
to always restore the screen to its previous state.
Tests now reside in the root of the repo, rather than in wa package.
This means they will no longer packaged and installed in user
deployments (they're only useful for developers).
WA2 Extensions had two features that have proven more trouble then they
are worth, and so they were removed from WA3 Plugins:
- Virtual methods (methods that automatically invoked super's version
without having to explicitly do that).
- It used to be possible to use strings or tuples to specify parameters
for extensions, and those were automatically "promoted" to Parameter
instances by the metaclass. This feature was never actually used.
This commit cleans up residual code for these features from Plugin
implementations and removes tests that used to exercise them.
- correct the import for merge_config_values fuction
- fixed expected result for set-with-list case; sequence merge will
produce unique values.
- check that the merge result type matches the other's type
Removing old and unsused files:
- wa/framework/old_output.py: superseded by output.py in the same dir
- the entire wlauto tree: replaced by wa/ tree; it's stale by now
anyway.
- log.py and actor.py from framework/ as neither is used.
assistant_params are now handled consistently with the other kinds of
params in TargetDescription, and they are now part of the default config
generated.
Updated the record command to allow revent recordings to be made.
They can be performed in 3 ways; the default is a standard recording
as in WA2, it can start a specified package before starting the
recording or a workload can be specified which launches the workload
and prompts the user to record each stage, naming and storing the recordings
appropriately.
Added a new ReventWorkload as a base for revent based workloads.
Updated ReventGui to use `revent_recorder` and now uses `target.model`
instead of `taget.name`.
Previously if a parameter was not specified via config it would not
be used during initialisation even if the parameter had a default
value. Now any parameters with default values are populated as necessary.
Revent files are automatically placed in the sub folder `revent_files` in the
workload directory when recording, therefore when trying to retrieve recordings
the getter now looks inside of the sub directory.
Previously trying to use the `__base_filepath` import from inside the `Package`
class, resulted in the error "`_Package__base_filepath` is not defined."
Added a workload type to handle workloads that have both an APK with an
application and associated automation JAR. Added benchmarkpi
implementation using using the new workload.
- get_by_extension was comparing the expected extension to the entire
tuple returned by os.path.splitext(), rather than just the extension
part.
- UserDirectory getter was looking in the root dependencies directory,
rather than the subdirectory for the owner.
- Filer getter was not handling non-existing paths properly.