1
0
mirror of https://github.com/ARM-software/workload-automation.git synced 2024-10-06 10:51:13 +01:00
Commit Graph

73 Commits

Author SHA1 Message Date
Sergei Trofimov
b3de85455a Add support for Python 3
Add support for running under Python 3, while maintaining compatibility
with Python 2.

See http://python-future.org/compatible_idioms.html for more details
behind these changes.
2018-06-07 14:48:40 +01:00
Sergei Trofimov
a865d8f30c utils/serializer: add cpu_mask to POD types 2018-05-25 15:27:14 +01:00
Sergei Trofimov
4cc5aec39a utils/serializer: make is_pod() recursive
If the object being passed into is_pod() is iterable (and not a
string), recursively check that its elements are also PODs.
2018-05-25 10:21:06 +01:00
Marc Bonnici
c338962c48 utils/doc: Add missing newline when generating aliases docs
Ensure when extracting aliases documentation it is followed by 2
newlines so that it formats correctly.
2018-05-24 16:57:47 +01:00
Marc Bonnici
6cb7031713 wa/misc: Fix format_duration() to handle missing time
If a run is interrupted this can cause the time to be `None` causing an
error, if this is the case then set time to 0 and therefore return `N/A`.
2018-05-24 11:49:10 +01:00
Marc Bonnici
614a590b35 utils/counters: Remove duplicated counter
The implementation of the `counter` was duplicated in both
`wa.utils.counters` and `wa.utils.types.counter` the latter of which is
currently unsed, therefore port missing functionality to the former and
remove.
2018-05-24 11:49:10 +01:00
Sergei Trofimov
2a2eafa282 utils/doc: include plugin aliases in the docs
If a plugin defines aliases, include the aliases and the parameter
values they correspond to inside the docs.
2018-05-23 14:51:54 +01:00
Marc Bonnici
3efff81a5c utils/exec_control: Fix issue with once_per_instance decorator
Previously the `once_per_instance` used the output of `__repr__` to
identify the class of where the decorated method was called from. For
classes that override this method to produce the same output for
different instances of the same class this caused different instances to
be mistakenly treated as one. Now use the hash of the containing class
instead of the string representation and update the tests to catch this
error.
2018-05-22 17:18:40 +01:00
Marc Bonnici
670fe6fb83 wa/types: Add cpu_mask type
Add a cpu_mask type to allow for users to use a consistent way of representing a
cpus mask with properties to provide conversions between the various required
forms. The mask can be specified directly as a mask, as a list of cpus indexes
or a sysfs-style string.
Also add support for the type to be serialized/deserialzed for the json
and yaml serializers.
2018-05-14 17:07:49 +01:00
Sergei Trofimov
a2b0705ff0 processors: add cpustates
Add cpustates output processor. This is more-or-less a port of the
cpustates processor from WA, however there are some differences:

- Idle states are now tracked individually per-CPU. This will fix
  processing traces from targets that have different number of idle
  states on different clusters.
- Simplify the parameter list for report_power_stats:
    - Replace paths to individual report files with a path to a single
      directory. A subdirectory will be created under it which will
      contain all the reports.
    - Replace the individual bits bits of information about CPUs (core
      names, idle states, etc) with a list of CpuInfo objects.
- Clean up and simplify the code a bit:
    - Make all reports mandatory -- the marginal cost of generating
      an additional report is minimal compared to tracking power states
      in the first place.
    - Standardize the interface for Reporters and Reports.
- Rename some of the reports to something a bit more meaningful.
- The stand-alone command line interface is not ported for now, as it
  is now possible to run this offline on existing results using
  "wa process".
2018-05-14 17:05:06 +01:00
Sergei Trofimov
e7e272cd03 utils/trace_cmd: add trace_has_marker()
Add a function to check whether a trace file contains start/stop markers
(actually only check for start marker -- stop marker is assumed).
2018-05-14 17:05:06 +01:00
Sergei Trofimov
795b3485ce utils/trace_cmd: move params to __init__()
Move the check_for_markers and events parameters from parser() to
__init__(). These parameters control the behavior of the parser, and do
not relate to a particular trace file, so it makes more sense to have
them there.
2018-05-14 17:05:06 +01:00
Sergei Trofimov
1ffbce68c4 fw/execution: handle error on run initialize
- Use indentcontext inside initialize_run to make sure log level
  is dedented on error.
- Ensure Executor postamble always runs, event if runner errors.
- Fix format_duration() to handle sub-second timedeltas.
2018-05-10 11:49:30 +01:00
sergei Trofimov
ecc68ee367 utils/log: add indentcontext
Add indentcontext context manager to save having to add try/finally
clauses around indenting code just to ensure dedenting.
2018-05-09 15:06:31 +01:00
sergei Trofimov
c1a0f842d0 utils/log: maintain indent for buffered records
Ensure buffered records are indented properly by saving the indent level
at the time the record was generated as part of the record, and
preferring that over the current indent level inside LineFormatter.
2018-05-09 15:06:31 +01:00
sergei Trofimov
1e750c659b utils/log: add early record buffering
Buffer early log records until a log file becomes available, flush
records to said file afterwards. This ensures that early log output,
which, among other things, contains plugin discovery logging, is not
lost. Buffer capacity is configurable via an environment variable.
2018-05-09 15:06:31 +01:00
sergei Trofimov
aacba47f9a utils/types: toggle_set: add "~~" semantics
Add support for "~~" special value that indicates that, when merging
into another set, all values from that set should in fact be dropped.
Apart from the unique merge semantics, "~~" just like any other "~"
value.
2018-05-08 10:50:48 +01:00
sergei Trofimov
9093477f1b utils/types: fix toggle_set merge methods
- The merge_with and merge_into implementation was reversed. Fix this, so that
  it is implemented correctly, i.e. that
  	a.merge_with(b) === merge(a, b)
	a.merge_into(b) === merge(b, a)
- Ensure that in case of merge_into(), a toggle_set is always returned.
2018-05-08 10:50:48 +01:00
Marc Bonnici
a0dfac78c3 utils/doc: Add function to generate a line break 2018-04-30 09:14:22 +01:00
Marc Bonnici
98bed3822a utils/docs: Update format of parameter type rst output 2018-04-24 09:59:57 +01:00
Marc Bonnici
3ddb1272aa utils/doc: Add support for dicts to format literal
Now supports cleaner outputing of python dicts
2018-04-24 09:59:57 +01:00
Sergei Trofimov
67ea7c8ee1 utils/types: toggle_set: TypeError on string
Strings are iterable, so can be used to instantiate sets (resulting in a
set of chars). This is never what we want for toggle_set's though, and
may result in difficult-to-interpret errors when parsing configuration,
so raise a TypeError if attempting to create a toggle_set with a string.
2018-03-22 12:28:48 +00:00
Marc Bonnici
2371364ce4 utils/doc: Adds support for showing aliases when formatting parameters
Now displays all available local and global aliases when generating the
rst for a parameter.
2018-03-21 13:27:31 +00:00
Marc Bonnici
f5ed4e317a utils/misc: Fix missing import 2018-03-20 08:24:58 +00:00
Sergei Trofimov
fda418093d fw/execution: change how error events get added
Rather than relying on a custom Logger with a context to add events when
they are logged, have the runner register hooks for corresponding
signals that do that.

The issue with the previous approach is that the event ended up
being added too late -- after the current job was already cleared, so
all events were added at run level.

With the new approach, the event will be added when it is first logged,
while there is still a current job being set. There will be no
duplication for Exceptions being re-raised and handled at different
levels because log_error() will ensure that each Exception is logged
only once.
2018-03-08 11:18:12 +00:00
Sergei Trofimov
04ab336afc fw: fix error logging
- Keep track of logged exceptions inside log_error itself.
- signal: log the exception, if there is one  in the finally clause of
  the  signal wrapper; this will ensure that the error will be logged
  closer to the command that originated.
- entrypoint: use log.log_error for top-level error logging, rather than
  the  entrypoint logger directly; this will ensure that errors are not
  repeated unnecessarily.
- Log CTRL-C message at zeroth indent level to make it easier to see in
  the  non-verbose output where it occurred.
2018-03-08 11:18:12 +00:00
Sergei Trofimov
e3b3799737 utils/misc: remove unused to_identifier
This has been moved to devlib, but, it appears, never removed from WA.
2018-02-16 13:56:03 +00:00
Marc Bonnici
d9a7e1c475 utils/types: Add add method to toggle_set
Allows adding of elements that will respect the state of an existing
element in the toggle set.
2018-02-01 14:26:08 +00:00
Marc Bonnici
e5ed64d8d5 trace_cmd: Rename trace-cmd to trace_cmd
Both the instrument and utility functions were called `trace-cmd.py`.
`"-"`'s are not valid python identifiers therefore when trying to import
the `trace-cmd` module this causes an error.
2018-01-25 16:28:51 +00:00
Marc Bonnici
f26d819aad utils/exec_control: Fix once decorator implementation
Ensures that the once decorator does not affect classes in a parallel
inheritance hierarchy.
2018-01-15 15:42:26 +00:00
Marc Bonnici
950427f478 utils/revent: Refactor ReventRecording to avoid duplicate target calls
Changed location of revent binary on target to be a class attribute so that
it is shared across all instances. Also changed discovery method to a
static method which can be memoized to prevent the target being queried
unnecessarily.
2018-01-08 10:36:02 +00:00
Marc Bonnici
9832db9fa5 utils/doc: Rename get_rst_from_extension for consistency 2017-12-14 09:50:26 +00:00
Marc Bonnici
adb14442c8 utils/doc: Change get_params_rst to be more flexible
Instead of taking an extension as a parameter and assuming there it has
an attribute 'parameters' instead take a list of parameters directly
which allows the method to be used with parameter lists with different
names.
2017-12-14 09:50:26 +00:00
Brendan Jackman
aa2abd16a3 Run autoflake to remove unused imports & variables 2017-12-07 15:37:49 +00:00
Marc Bonnici
bc5b43df30 Utils/Android: Fix missing space in check
Logcat entries are split on a ':' followed by a space so ensure this
is present in the line instead of just a ':'. Some devices have entires
without the trailing space causing an error.
2017-11-24 16:38:01 +00:00
Sergei Trofimov
39e63f1358 utils/android: add a parser for Logcat
Add a simple streaming parser for Logcat logs. At the moment, only the
recent logcat format is supported.
2017-11-24 14:30:09 +00:00
Sergei Trofimov
2ff1a87e53 utils/types: fix toggle_set docstring
The docstring claimed that the toggle_set was "A list that...".
2017-11-03 17:33:32 +00:00
Brendan Jackman
4b7d43de9e utils/log: Convert exception to string before passing to Logger
Now that we add an event when logging errors, we need to avoid passing an
exception object directly to the logger. Otherwise we end up adding the
unserializable Exception object to the events list, which causes an error when
trying to write the output.
2017-10-23 17:25:15 +01:00
Brendan Jackman
b11bd6cf12 utils/log: Whitespace 2017-10-23 17:25:02 +01:00
Sergei Trofimov
47d883de09 utils/log: add ContextLogger
Add ContextLogger wich extends logging.Logger's  warning and error
methods to automatically add the message to the context's events (if a
context has been provided).
2017-10-23 12:28:29 +01:00
Sergei Trofimov
c93b1a0939 utils/log: adjust level for CGgroups logger
devlib's CGgroups module logs things at INFO level. The information it
logs is too detailed for what WA considers "INFO", and should be logged
at DEBUG level instead.
2017-10-06 15:04:02 +01:00
Sergei Trofimov
109490ec3e utils/types: make list_of classes serializable 2017-10-06 12:55:25 +01:00
Sergei Trofimov
7b1cc6fa4e utils/serializer: add OrderedDict to POD_TYPES
Allow OrderedDict to be serialized in the same way as regular dicts by
defining it as a Plain Old Data type.
2017-09-28 10:46:03 +01:00
Sergei Trofimov
0c40cdae18 tools/revent: improve recording + add prebuilts
The begining and end of recording timestamps are now stored as part of
the recording. This allows to correctly replay recordings where the
first event occurs some time after the recording started, or the last
event some time before recording ended.

Add pre-built revent binaries for the armeabi and arm64 architectures.
2017-09-27 10:34:00 +01:00
Sergei Trofimov
9b6d4d9fa6 utils/revent: stop recording fix.
It seems that occasionally an inclomplete recording file is pulled. Add
a one-second (target time) delay to allow SIGINT to be properly handled
and recording to be updated, followed by a sync to ensure the file has
been written.
2017-09-27 10:32:19 +01:00
Sergei Trofimov
c29b1c8b23 utils/log: log_error marks Exception as logged.
Mark the Exception is logged inside log_error, to prevent it form being
logged repeatedly if re-raised.
2017-09-27 10:32:19 +01:00
Marc Bonnici
c5b1caa276 Utils/Enum: Adds a __str__ method to display enum levels 2017-08-16 17:35:19 +01:00
Marc Bonnici
2cc60b77cb Utils/Serializer: Adds serializer support for level type 2017-08-16 11:10:46 +01:00
Sergei Trofimov
3f87f3ab07 utils/misc: remove enum_metaclass
It is unused and has been obsoleted by enum type in utils/types.
2017-08-09 15:59:20 +01:00
Sergei Trofimov
a9959550af utils/types: better enum class member setting
- What used to be enum.values is now enum.levels.
- Add enum.names and enum.values that are lists of enum's levels' names
  and values respectively.
- Add a check on creation to make sure that provided level names do not
  conflict with the atomatically created members.
2017-08-09 15:59:20 +01:00