Add a workload for lat_mem_rd and bw_mem tests of LMbench benchmark, a suite of
portable ANSI/C microbenchmarks for UNIX/POSIX.
This is a port of corresponding workload from WA2.
Ensure the serialized state gets written out in case of early abort
(before the run has been fully initialized and job execution has
started), and that the appropriate final output is generated.
This requires that ExecutionContext is created much earlier (in order
for postamble to be possible), and that KeyboardInteruptException is
caught and handled properly by the Executor.
Add a convenience method for invoking write_info(), write_state(), and
write_result() of run_output. This will make it easier to call it from
multiple places.
TargetManger.get_target_info() gets invoked twice: first on context
creation, and then a second lime shortly afterwards to populate
RunOutput. Change target_info attribute inside the context to be a
property that redirects to run_output. This will allow the context to be
created earlier, before a target connection is available.
Add run.log as an Artifact to the output, once it has been created.
Every file outsize of __meta (excepting result.json) should have an
associated artifact to describe it.
Add wa and devlib versions to RunOutput's metadata on creation. They
already get logged, but this will make them readily accessible to
scripts and output processors.
When iterating over PLATFORMS tuples, account for the
platform_target_defaults introduced by 7470fb5ee.
Also, correct the tuple description in the comments.
The ValueError does not given enough information in the case of an empty file
extension. Add a special case to handle files with no extension and
prompt the user.
Call resolver's get() method, rather than get_resource(), incorrectly
introduced in the previous commit. Also add get() alias to the context
to allow it to be used as a resolver.
Since 2ff06af63, resources are obtained via context.get_resource, rather
than context.resolve.get_resource. LightContext used by the record
command was not updated to implement that method. This fixes the issue
Allow specifying "augmentations" and "all" as the plugin kind to lost.
In the case of of the former, instruments and output processors get
listed. In the case of the latter, every plugin kind gets listed.
This should have been handled by the @total_ordering decorator, but
isn't due to
https://bugs.python.org/issue25732
(briefly, total_ordering is back-ported from Python 3, where the base
object provides the default implementation of __ne__ based on __eq__, so
total_ordering did not override it; this, however does not happen in
Python 2).
Also update unit tests to catch this edge case.
During Python 3 migration, the izip call inside diff_interrupt_files has
been replaced with a zip call (zip returning an iterator in Python 3).
Import zip from builtins to ensure that it also produces an iterator in
Python 2.
Also fix the associated unit test by importing the correct function.
This instrument will log serial traffic to a file. The idea is to
assist with debugging development boards, as otherwise there is a
chance that crash reports, for example, are lost when the device is
reset by WA.
It logs the pre-run and post-run traffic to preamble_serial.log and
postamble_serial.log , respectively, in the top-level output
directory. Serial logs for each workload are stored in the appropriate
workload directory as serial.log.
As we are tying up the serial port, we stop the logging when the
device is reset to allow board such as the Arm Juno to be reset via
the serial connection.
Python 3 has changed the behavior of subparsers so that they are no
longer mandatory by default. As a result, executing just "wa" under
Python 3 results in a random error, rather than a help message.
Fix this by making the subparsers mandatory.
Certain devices present a night time reading popup alert when opening the books workload.
This can throw off the workload so I've added a function to check for this and dismiss if neccessary.
Add a decorator to indicate that a callback runs entirely host-side and
does not rely on a connection to the target. This means it will be
invoked even if the target was detected to be unresponsive.
- Add "hard" parameter to TargetManager's reboot().
- Pass in context to TargetManager's reboot() and
verify_target_responsive() -- this is needed for signal dispatch.
- Call self.reboot() instead of self.target.reboot() when attempting a
reset for unresponsive targets inside TargetManager, in order to
ensure that appropriate signals are dispatched.
Fix a regression introduced with Python 3 port -- JSON deserializer
should check for basestring rather than str when deciding whether to try
to decode a custom type.
In a previous commit we filtered parameters so that '-h' is never passed
to 'parse_known_commands' to ensure our own custom help message is displayed,
now also filter for '--help' for the same reason.
Previously incorrect method calls were used to try and convert 'max' and
'min' into their corresponding frequencies, now call the correct methods
and refactor the conversion into a separate method.
Modify execution so that output processors' process_job_output() (but
not export_job_output()) is wrapped by the JOB_OUTPUT_PROCESSED signals.
This makes sense conceptually, and is more useful in practice, as there
are already WORKLOAD_RESULT_EXTRACTION and WORKLOAD_OUTPUT_UPDATE
signals sent by the job, if it's desirable to run before output
processors, but after the job results have been collected.
Since daq-server has been split out into a separate package, daq setup
is no longer part of WA documentation. This resulted in a warning when
building the docs.
As part of resolving a resource, record its MD5 hash in the output
metadata. In order to enable this, resource resolution is now done via
the context, rather than directly via the ResourceResolver (in order not
to introduce a context dependency in the Gui object, context now
emulates the resolver interface).
Metadata is a key-value mapping for arbitrary data, similar to
classifiers. Unlike classifiers, metadata does not directly relate to
the results of the execution, but to the execution itself, and typically
would not be processed by Output Processors in the same way as
classifiers. Metadata can also be a lot more free-form in it's value
structure; while classifier values are simple scalars, metadata values
can be arbitrary POD structures.
If the device is to perform an initial reboot, allow up to `max_retries`
from the run configuration for the reboot to succeed and add logging to
inform the user.
Treat the optional reboot of the device as part of the job itself so that
if the reboot fails it will be retried in accordance to the job
configuration.
Instead of starting and stopping the target from inside the Context,
move the calls to the Runner. This allows for the any initilizaton
errors to be dealt with as part of the job and retried as
appropriate.
The implementation of the `counter` was duplicated in both
`wa.utils.counters` and `wa.utils.types.counter` the latter of which is
currently unsed, therefore port missing functionality to the former and
remove.
Replace the ad-hoc get_plugin() function in the show command (which was
broken for aliases) with a call to pluginloader.get_plugin_class(),
which correctly handles alias resolution.
- expose pluginloader's resolve_alias in plugin_cache (it's supposed
to be a transparent proxy for pluginloader).
- resolve aliases inside _set_plugin_defaults() to make sure the
correct defaults are used when creating job specs.
Expose the RebootPolicy as an attribute of the ExectionContext and ensure
that that the target is not automatically rebooted if the reboot
policy is set to "Never".
Rename the parameter to be clearer on the effect of the policy, as this
will cause WA to reset the device for each new job it runs regardless of
the iteration number.
We only need to deploy the gui once, therefore move the `deploy` method
call into `initialize` the corresponding `remove` method is already in
`finalize`.
Previously the `once_per_instance` used the output of `__repr__` to
identify the class of where the decorated method was called from. For
classes that override this method to produce the same output for
different instances of the same class this caused different instances to
be mistakenly treated as one. Now use the hash of the containing class
instead of the string representation and update the tests to catch this
error.
Update workloads that allow for specifying which cpus to be ran on to
all use the same interface while maintaining backwards compatibility with
the existing configuration options.
Add a cpu_mask type to allow for users to use a consistent way of representing a
cpus mask with properties to provide conversions between the various required
forms. The mask can be specified directly as a mask, as a list of cpus indexes
or a sysfs-style string.
Also add support for the type to be serialized/deserialzed for the json
and yaml serializers.
Add cpustates output processor. This is more-or-less a port of the
cpustates processor from WA, however there are some differences:
- Idle states are now tracked individually per-CPU. This will fix
processing traces from targets that have different number of idle
states on different clusters.
- Simplify the parameter list for report_power_stats:
- Replace paths to individual report files with a path to a single
directory. A subdirectory will be created under it which will
contain all the reports.
- Replace the individual bits bits of information about CPUs (core
names, idle states, etc) with a list of CpuInfo objects.
- Clean up and simplify the code a bit:
- Make all reports mandatory -- the marginal cost of generating
an additional report is minimal compared to tracking power states
in the first place.
- Standardize the interface for Reporters and Reports.
- Rename some of the reports to something a bit more meaningful.
- The stand-alone command line interface is not ported for now, as it
is now possible to run this offline on existing results using
"wa process".
Move the check_for_markers and events parameters from parser() to
__init__(). These parameters control the behavior of the parser, and do
not relate to a particular trace file, so it makes more sense to have
them there.
Running additional processors will generate new metrics and artifacts.
Update the serialized metadata with the new artifacts/metrics after
running all processors.
Extracting trace and pulling the trace file from the target can take a
relatively long time, as the trace file can be tens, and even hundreds,
of MB. Add an info-level log message to inform the user why WA execution
appears to have paused for a few seconds.
- Have the job manage the indent of each of its stages
- Switch output_processor to using indentcontext
- Move the "Configuring augmentations" log line into Job -- the
rest of the stages are logged there, but this was done by the
Executor for some reason.
Add hostid entry, as reported by busybox's hostid applet, to TargetInfo.
This is intended to be a "unique 32-bit identifier for the current
machine". In practice, it may not be set, or may be generated from
something like an IP address, so may not be unique. However, if set, it
is still likely to be more unique than kernel/os version so may be worth
recording.
Replace Target.cpuinfo, which contained parsed contents of
/proc/cpuinfo, with a list of more comprehensive CpuInfo objects which
include cpufreq and cpuidle information as well.
- Use indentcontext inside initialize_run to make sure log level
is dedented on error.
- Ensure Executor postamble always runs, event if runner errors.
- Fix format_duration() to handle sub-second timedeltas.
Add a function to return WA version suffixed with the commit hash for
current HEAD. This only works if WA was deployed via
python setup.py develop
otherwise, the output is identical to get_wa_version().
When scanning modules for plugins, only consider classes defined in that
module, but not those imported into it. This will prevent log output
being polluted with repeated "Skipping..." messages for imported plugin
bases.
Ensure buffered records are indented properly by saving the indent level
at the time the record was generated as part of the record, and
preferring that over the current indent level inside LineFormatter.
Move the logging of the WA version and command line used to invoke it
from run command into main() inside entrypoint. That way, these will be
the first entries in the log file, now that early log records are no
longer lost.
Buffer early log records until a log file becomes available, flush
records to said file afterwards. This ensures that early log output,
which, among other things, contains plugin discovery logging, is not
lost. Buffer capacity is configurable via an environment variable.
If more then one of augmentation entries ("augmentations",
"instrumenatation", "output_processors") is specified in the same
config they were being merged into a single entry.
This can cause issues with '~~' (drop everything glyph) being "resolved"
during this merge and therefore not actually used to drop all previous
augmentations.
Union'ing, instead of merging, configs at the same level avoids that.
Note that this is semantically correct, as we're not enforcing
precedence at the same level of config, so merge semantics do not apply.
Add support for "~~" special value that indicates that, when merging
into another set, all values from that set should in fact be dropped.
Apart from the unique merge semantics, "~~" just like any other "~"
value.
- The merge_with and merge_into implementation was reversed. Fix this, so that
it is implemented correctly, i.e. that
a.merge_with(b) === merge(a, b)
a.merge_into(b) === merge(b, a)
- Ensure that in case of merge_into(), a toggle_set is always returned.
Delay the split of augmentations into instruments and output processors
until they are actually being accessed; keep them as a single set until
then. This makes tracking of merges from various configurations sources
easier
- Add missing descriptions to backends.
- Describe how to get instrument-specific parameter descriptions in the
help for instrument_parameters param of EnergyMeasurementInstrument
stress-ng will stress test a computer system in various selectable
ways. It was designed to exercise various physical subsystems of a
computer as well as the various operating system kernel interfaces.
The rt-app workload parses the rt-app JSON file in order to override
some options (specifically duration). Previously, if the JSON was
syntactically incorrect, an uninformative ValueError was raised. Now
we raise a ConfigError with appropriate message prompting the user to
fix the file.
Add an instrument which wraps the devlib screencapture poller, which
allows us to capture a screenshot from the device every `period`. The
`period` is configurable based on the user requirements.
All captured screenshots are named according to the timestamp on the
device, and are placed into an output directory (screen-capture)
within the job directory.
Port sysbench workload from WA2.
sysbench is a modular, cross-platform and multi-threaded benchmark tool
for evaluating OS parameters that are important for a system running a
database under intensive load.
If WA is ran without any workloads being specified, previously an index
error was thrown, now check that at least one global workload has been
specified and if not, provide a more helpful error message.
Add extra_plugin_paths setting which gets populated from WA_PLUGIN_PATHS
environment variable. This allows specifying additional locations to
scan for WA plugins.
The `install_timeout` variable is used to allow extra time when
installing an apk, if the apk is not present on the host system it
will be pulled back to the host, therefore we should allow extra time
when transferring the file in the other direction as well.
Add process command to run output processors over data produced by
a previous run.
Default processing is to skip the processors that were used during
the run and to process the specified directory only. Options to
use additional processors, force reprocessing and allow a directory
walk to discover additional output directories.
If the http getter is used but a connection is not avalible, when
attempting to fetch the resource index an error will be raised. Now we
ignore the error allowing for the remaining getters to be used.
Port manual workload from WA2. This workload allows using WA to
configure and instrument the target, while letting the user perform
arbitrary actions in place of running a fixed workload.
since Linaro's modification have been merged in upstream version,
we can remove this comment
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
resistor_values is an array of float and not int as the shunt resistors
are most of the time lower than 1 Ohm.
With kind=list_of_ints, all resistor values are rounded to 0. Replace it
by list_of_numbers
sample_rate_hz is an int used to get sample period and to compute energy.
Changes its type from str to int
With these 2 fixes, the daq instrument has been successfully tested with
NI DAQ6211
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Make sure runtime_params and boot_params are to_pod()'d when they get
written, as they are obj_dict's that json serializer does not know how
to handle otherwise.
Ensure that boot and runtime parameters always get initialized to an
obj_dict even when not explicitly specified. This will ensure they get
serialized and deserialized properly.
Arm Energy Probe with arm_probe has been recently added in devlib's
instrument.
Add the arm_energy_probe in the WA list of Energy measurement instruments
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
energy instruments might need to save platform specific configurations in
order to help to reproduce the use case. Such configuration should be saved in
__meta directory.
Add the directory path as a parameter of get_instruments() to save such
informations
Suggested-by: setrofim@gmail.com
Signed-off-by: Vincent Guittot <vincent.guittot@linaro.org>
Catch exceptions raised when attempting to set value of a config point,
and re-raise as ConfigError with name associated with the value in the
config file.
Strings are iterable, so can be used to instantiate sets (resulting in a
set of chars). This is never what we want for toggle_set's though, and
may result in difficult-to-interpret errors when parsing configuration,
so raise a TypeError if attempting to create a toggle_set with a string.
Now allows a `prefer_host_package` parameter to be set to control
whether a package located on the host system (`True`) or on the target
(`False`) should be favoured during resolution.
`pull_apk` now returns the path to the apk it pulled from the device so
WA can keep track of it and avoid conflicts with other apks present
on the host.
Upon initilisation WA3 will now also check if a WA3 config file
exisits, rather than just the exsistance of the config directory.
If a `config.yaml` file is not found WA will look for a WA2 config.py
file and convert it to the new format, otherwise a default config file
will be generated.
Record UI state if an error occurs during setup, run, and output
processing stages (for other stages, the UI state is unlikely to be
relevant as they typically would not include UI manipulation).
Only run finalize() for a job if initialize has succeed. finalize()
should be able to assume that initialize() has succeed, without needing
to check that that file have been created, variables set, etc.
Ensure the workload uses the passed working directory for attaching
images instead of a hard coded name and allow for finding of the attach
button by description as well as name.
Fix an issue where AndroidWorkload.pull_apk would sometimes get the
wrong package if the desired package name is a substring of another
package name. Rather than using a regex to match the package name, use
the new get_package_info method to match the name exactly.
Not all benchmarks are available on all phones and will report "Not
supported" To allow for this update the regexes and if not a valid score
set any unsupported results to 'NaN'
This is WA2 sqlite result processor ported to WA3 output processor API.
It is schema-compatible with the WA2 version (in fact, there are *no*
schema changes).
Note: There is a medium-term goal of adding a more generic SQL database
exporter with a more complete schema sitting on top of something like
sqlalchemy to make it agnostic of DBMS implementations. At point, this
will likely be deprecated.
Update the 'package' argument description to specify it is for android
packages and add error checking to ensure that if specified, the target
supports android.
Change the recording of the run stage during workload recording mode to be
optional as the user may want to use the command to re-record a specific
stage of a workload without affecting the workload recording.
Hitting CTRL-C will abort execution of the current job, but will still
trigger run finalization, and possibly, post-processing and teardown of
the current job. If an exception is raised during this
post-process/teardown, the previous exception state (for the
KeybardInterrupt) will be clobbered. That means that, after the new
exception has been handled, WA would attempt to execute the next job,
rather than go to finalization of the run.
To avoid this, set a flag in the context upon catching KeybardInterrupt,
and check this flag before attempting to execute the next job in the
queue.
Rather than relying on a custom Logger with a context to add events when
they are logged, have the runner register hooks for corresponding
signals that do that.
The issue with the previous approach is that the event ended up
being added too late -- after the current job was already cleared, so
all events were added at run level.
With the new approach, the event will be added when it is first logged,
while there is still a current job being set. There will be no
duplication for Exceptions being re-raised and handled at different
levels because log_error() will ensure that each Exception is logged
only once.
- Add SimpleTargetDescriptor that simply returns a global list of target
descriptions.
- Add create_target_description function to populate that list with a
description crated from the specified components.
- Add add_description_for_target helper function to create a description
for the specified target class, and specified components. For
components that are not explicitly specified, attempt to guess
sensible defaults based on the target class' bases.
- fix TargetDescription._set to set lists rather than dicts for params
to be consistent with how params are set after creation (should
probably get rid of this entirely...)
- fix the comment describing the structure of TARGETS
- Keep track of logged exceptions inside log_error itself.
- signal: log the exception, if there is one in the finally clause of
the signal wrapper; this will ensure that the error will be logged
closer to the command that originated.
- entrypoint: use log.log_error for top-level error logging, rather than
the entrypoint logger directly; this will ensure that errors are not
repeated unnecessarily.
- Log CTRL-C message at zeroth indent level to make it easier to see in
the non-verbose output where it occurred.
If a target error occurs, check whether the target is unresponsive. If
it is, attempt to hard reset it if possible, or gracefully terminate
execution if not.