Now that we add an event when logging errors, we need to avoid passing an
exception object directly to the logger. Otherwise we end up adding the
unserializable Exception object to the events list, which causes an error when
trying to write the output.
This maintains the default behaviour of bailing out immediately if any workload
fails in initialize(), but adds a setting, bail_on_init_failure, to change this
behaviour optionally. This can be useful where WA is being used more as a batch
processor.
Changes to the Status enum introduced by 31a535b5 and a9959550 broke
ran Jobs summary status at the end of the run. This fixes it so that the
total number of jobs and individual status counts are reported
correctly.
The generic "frequency" runtime parameter was only being set when there
are common frequences between avialable cores. It should always be set,
even if there are no frequencies in common, as it still valid to use it
with special values "min" and "max", in which case it should resolve
correctly to the appropriate frequencies.
Set context for the loggers of the Runner, the workloads and the
installed instruments and processors. Errors/warnings logged by these
entities will be automatically added as events.
Add ContextLogger wich extends logging.Logger's warning and error
methods to automatically add the message to the context's events (if a
context has been provided).
This enables you to do things like:
global:
runtime_parameters:
airplane_mode: true # This improves repeatability of energy measurements
workloads:
- name: foo
- name: bar
- name: geekbench
runtime_parameters:
airplane_mode: false # Geekbench requires network, and we don't care
# about energy for that workload anyway
For some reason I didn't look at the WA2 instrument until afer I
wrote this, so it's different in how it handles 'before/after'
quantities (which I called 'momentary' in the code) - instead of
adding a 'mean' it just adds two separate metrics: one for before and
one for after. Personally I think this is more useful (esp. for
temperature) but if necessary I can revert to the old behaviour.
This also doesn't add the helpful classifier telling you the hwmon
device, because we don't have immediate access to that info from the
devlib Instrument. If necessary I'm sure it could be added later.
Furthermore it doesn't have a parameter to filter the sensors, just
collects from them all.
Where backends have the capability to collect from multiple devlib
Instruments, EnergyMeasurement currently appends 'device' keys to the
names of metrics reported by those Instruments. Where multiple
Instruments are indeed used, it then sums up equivalent metrics to
produce the 'normal' metric without the 'device' keys
appended.
E.g. If you have two ACME 'devices' enabled you will get something
like
'device_total_energy_iio:device0' and
'device_total_energy_iio:device1', which will be summed to produce
'device_total_energy'.
However when only one Instrument is actually instantiated, this is
not needed and not done. Therefore just directly produce the metric
with the name originally given by devlib (i.e. in the example,
'device_total_energy').
Fully matching an APK resource requires the file to be present locally,
so that its metadata can be queries. HTTP getter was matching against a
remote path so the match was failing.
The matching now happens in two stages == first partial path-only
matches are established. Secondly, all partial matches are downloaded
and final match occurs against downloaded files.
This method is used to partially match a resource; its implementation
cannot rely on the resource file actually being present and must match
against the specified path alone.
match() implementation now defaults to match_path(), as for most
resource types, the path is sufficient to uniquely match a resource.
Devlib's get_data API can return None when something goes wrong -
this currently results in us attempting to add a non-existent
artifact.
To avoid this confusion, check and use the return value from
get_data, instead of assuming that it will write data to its
'outfile' parameter.
Currently if you get confused and write a config with something like:
energy_measurement:
acme_cape
Then you get an error when we try to 'iteritems' on the value
'acme_cape'. Instead, explicitly check for this case.
- Fix start/stop methods of sysfile_getter, which where still using the
WA2 mechanic of prefixing "slow_" to the name to indicate priority,
and consequently weren't being invoked.
- Call super inside cpufreq's valide(() to make sure tempfs path is set.
- Make sure TargetManager.finalize() actually gets called at the end
of the run.
- Overrule the "diconnect" parameter behavior for gem5 and make sure it
always disconnects. This necessary for stats to be generated properly.
devlib's CGgroups module logs things at INFO level. The information it
logs is too detailed for what WA considers "INFO", and should be logged
at DEBUG level instead.
Gem5Platform requires a host output directory as one if it's
instantiation parameters. This is not something we want to expose a
configuration parameter to the user, as for WA, the standard output
directory ought to be used.
Up to this point, WA's target instatiation process assumed that all
parameters came from the user, and there was no way for WA itself to set
them. This commit adds extra_platform_parms argument to
instantiate_target, to remedi this.
extra_platform_parms is then used to set the host output directory for
gem5 appropriately.
(linux|android)_gem5 devices were being described as having gem5
platforms but standard linux/android connections. This commit fixes this
by making sure that the gem5 connection is used.
Seem I cherry-picked the geekbench workload from my WIP branch
without testing properly, and missed a fixup commit. Here's the
content of that commit, which fixes some missing updates from WA2 ->
WA3 APIs.
Devlib's ACME instrument uses iio-capture under the hood, which can
only capture data from one IIO device at a time. Devlib's instrument
API expects to produce a single CSV file for the Instrument, with a
single axis of sample timestamps. These two things cannot be
correctly reconciled without changing the devlib Instrument API -
get_data would need to be able to return two distinct sets of data.
Instead, where required WA will instantiate the ACME instrument
multiple times (once for each IIO device), producing two separate CSV
files. Aggregated energy info (as opposed to timestamped samples of
energy data) _can_ be meaningfully combined from multiple IIO
devices, so the derived stats are summed.
This requires altering the internal API of the energy_measurement
module - rather than a single instrument the super-instrument now has
a dictionary mapping 'device' keys (this is really an arbitrary key,
but I think that 'device' is a justifiable metaphor for whatever
might separate multiple instruments, plus it is the terminology used
by IIO/ACME) to Instrument objects. Existing instruments simply
return a dict with a single entry, with None as the key.
It may in future be possible to solve this problem in devlib's ACME
instrument by avoiding the use of the iio-capture tool. In that case,
it should be possible to basically revert this patch and get the
required funcitonality from devlib.
Runtime param names from config were matched against names available on
for the target using an overly-loose regex that resulted in the wrong
config point being returned.
Use caseless_string matching instead.
As paramters defined by energy_measurement instrument backends were not
used in instantiating the backends, but the underlying devlib
instrument, the values did not undergo the usual Plugin parameter
setting perocedure. In particuar, type conversion would not take place.
Make sure backend parameters are validated properly by setting them on
an obj_dict before passing them to the devlib instrument.
Some workloads, such as Geekbench, may phone home and report data about the
device they are running on. This poses a risk for users that are testing on
unreleased or otherwise confidential devices - perhaps they use a standard
agenda to run a large battery of tests, in which case they may forget to disable
these dangerous workloads.
This provides a mechanism to prevent running those workloads from running by
setting allow_phone_home=False in the user configuration.
Currently if your config contains
typo_for_a_global_alias: 1
You will get an error where we try to call 'iteritems' on the value 1. This
commit re-orders the checks so that you instead get an error for the
unrecognised 'typo_for_a_global_alias'.
Differences from original:
- Standard renaming and rearrangement of imports
- UiAutomation.java now uses the setup() and runWorkload() phases which weren't
present before.
Due to the nature of modern Android devices, in order to usefully
interpret the power* events, you more and more often need to know the
kernel's view of temperature too. Therefore I think enabling the
thermal event group by default makes sense.
- Add "activity" property to PackageManager exposing the underlying
apk_info attribute.
- Add "view" class attribute to ApkWorkload, initialized to None
- If "view" is not set for a particular workload, set a default based on
the package and activity of the underlying workload.
- Ensure that CLOCK_MONOTONIC is used for all timestamps, so that the
timestamps collected by revent and those reported in the struct
input_event wouild match.
- Do not attempt to free events array inside fini_revent_recroding() if
no events have been recorded.
- Add missing \n in some debug messages.
- Add end of recording debug message.
- Make sure start/end timestamps are properly adjusted when there are
zero events.
- Rename adjust_event_times --> adjust_timestamps, as the latter better
fits its current function.
- Add a couple of debug messages.
DerivedMeasurements in devlib was changed to return DerivedMetrics
instead of Measurements. The former do not have "channel" attribute, so
just pass its name to WA's metrics.
If an error occurs in a ManagedCallback that is invoked outside of a
job, re-raise rather than attempting to update the status of the
non-existent job.
The begining and end of recording timestamps are now stored as part of
the recording. This allows to correctly replay recordings where the
first event occurs some time after the recording started, or the last
event some time before recording ended.
Add pre-built revent binaries for the armeabi and arm64 architectures.
- do not attempt to replay setup if a setup recording has not been
provided (perviously, setup was mandatory).
- update the apk initialization to the correct method inside the record
command.
When an agenda contains multiple sections, the same workload entry might
be used in construction of multiple job specs. Job spec construction may
mangle the workload entry. To prevent this from impacting other
jobs, use a deep copy of the workload entry when constructing a job
spec.
PluginCache.get_plugin_config assumes that no more configuration is to
be processed, and therefore config is final. As such, it is validating
that mandatory parameters are set. This assumption is invalid for
workload_parameters, however, as those need to be resolved on per-spec
basis, and cannot be globally cached.
This commit adds a prameter for get_plugin_config that indicates whether
or not it should consider the config to be final.
Connection settings were not being properly extracted from device_config
and not used. Meaning it was impossible to e.g. specify the adb device
to use for a run.
This ensures that connection settings are extracted, validated, and used
properly.
- Re-order Status entries so that higher severity entries have higher
enum values.
- Add set_status() to Job that ensures that a status is only set if it
is of higher severity (e.g. a Job that has been marked as PARTIAL by
an instrument will not be overwritten as OK by the runner).
- Retry no generates a new job, rather than re-enqueuing the existing
object; this ensures that the output status is tracked properly.
- Adjust ManagedCallback to set set job status to FAILED if it sees a
WorkloadError, and to PARTIAL other wise. The idea being that
instruments raise WorkloadError if they have a reason to believe
workload did not execute properly and indicated failure even if the
workload itself has failed to detect it (e.g. FPS instrument detecting
crashed content, where the workload might lack any feedback regarding
the crash). Other errors would indicate an issue with the instrument
itself, and so the job is marked as PARTIAL, as there is no reason to
suspect that the workload is at fault and the other results generated
for this execution may be valid.
Rename "package" parameter to "package_name" (to be cosnsitent with
package_names class attribute). "package" is now a property containting
the name of the actual package used after the APK is resolved.
