mirror of
https://github.com/ARM-software/workload-automation.git
synced 2025-11-04 09:02:12 +00:00
setrofim
@@ -1,6 +1,398 @@
|
|||||||
|
=================================
|
||||||
What's New in Workload Automation
|
What's New in Workload Automation
|
||||||
=================================
|
=================================
|
||||||
|
-------------
|
||||||
|
Version 2.4.0
|
||||||
|
-------------
|
||||||
|
|
||||||
|
Additions:
|
||||||
|
##########
|
||||||
|
|
||||||
|
Devices
|
||||||
|
~~~~~~~~
|
||||||
|
- ``gem5_linux`` and ``gem5_android``: Interfaces for Gem5 simulation
|
||||||
|
environment running Linux and Android respectively.
|
||||||
|
- ``XE503C1211``: Interface for Samsung XE503C12 Chromebooks.
|
||||||
|
- ``chromeos_test_image``: Chrome OS test image device. An off the shelf
|
||||||
|
device will not work with this device interface.
|
||||||
|
|
||||||
|
Instruments
|
||||||
|
~~~~~~~~~~~~
|
||||||
|
- ``freq_sweep``: Allows "sweeping" workloads across multiple CPU frequencies.
|
||||||
|
- ``screenon``: Ensures screen is on, before each iteration, or periodically
|
||||||
|
on Android devices.
|
||||||
|
- ``energy_model``: This instrument can be used to generate an energy model
|
||||||
|
for a device based on collected power and performance measurments.
|
||||||
|
- ``netstats``: Allows monitoring data sent/received by applications on an
|
||||||
|
Android device.
|
||||||
|
|
||||||
|
Modules
|
||||||
|
~~~~~~~
|
||||||
|
- ``cgroups``: Allows query and manipulation of cgroups controllers on a Linux
|
||||||
|
device. Currently, only cpusets controller is implemented.
|
||||||
|
- ``cpuidle``: Implements cpuidle state discovery, query and manipulation for
|
||||||
|
a Linux device. This replaces the more primitive get_cpuidle_states method
|
||||||
|
of LinuxDevice.
|
||||||
|
- ``cpufreq`` has now been split out into a device module
|
||||||
|
|
||||||
|
Reasource Getters
|
||||||
|
~~~~~~~~~~~~~~~~~
|
||||||
|
- ``http_assets``: Downloads resources from a web server.
|
||||||
|
|
||||||
|
Results Processors
|
||||||
|
~~~~~~~~~~~~~~~~~~~
|
||||||
|
- ``ipynb_exporter``: Generates an IPython notebook from a template with the
|
||||||
|
results and runs it.
|
||||||
|
- ``notify``: Displays a desktop notification when a run finishes
|
||||||
|
(Linux only).
|
||||||
|
- ``cpustates``: Processes power ftrace to produce CPU state and parallelism
|
||||||
|
stats. There is also a script to invoke this outside of WA.
|
||||||
|
|
||||||
|
Workloads
|
||||||
|
~~~~~~~~~
|
||||||
|
- ``telemetry``: Executes Google's Telemetery benchmarking framework
|
||||||
|
- ``hackbench``: Hackbench runs tests on the Linux scheduler
|
||||||
|
- ``ebizzy``: This workload resembles common web server application workloads.
|
||||||
|
- ``power_loadtest``: Continuously cycles through a set of browser-based
|
||||||
|
activities and monitors battery drain on a device (part of ChromeOS autotest
|
||||||
|
suite).
|
||||||
|
- ``rt-app``: Simulates configurable real-time periodic load.
|
||||||
|
- ``linpack-cli``: Command line version of linpack benchmark.
|
||||||
|
- ``lmbench``: A suite of portable ANSI/C microbenchmarks for UNIX/POSIX.
|
||||||
|
- ``stream``: Measures memory bandwidth.
|
||||||
|
- ``iozone``: Runs a series of disk I/O performance tests.
|
||||||
|
- ``androbench``: Measures the storage performance of device.
|
||||||
|
- ``autotest``: Executes tests from ChromeOS autotest suite.
|
||||||
|
|
||||||
|
Framework
|
||||||
|
~~~~~~~~~
|
||||||
|
- ``wlauto.utils``:
|
||||||
|
- Added ``trace_cmd``, a generic trace-cmd paraser.
|
||||||
|
- Added ``UbootMenu``, allows navigating Das U-boot menu over serial.
|
||||||
|
- ``wlauto.utils.types``:
|
||||||
|
- ``caseless_string``: Behaves exactly like a string, except this ignores
|
||||||
|
case in comparisons. It does, however, preserve case.
|
||||||
|
- ``list_of``: allows dynamic generation of type-safe list types based on
|
||||||
|
an existing type.
|
||||||
|
- ``arguments``: represents arguments that are passed on a command line to
|
||||||
|
an application.
|
||||||
|
- ``list-or``: allows dynamic generation of types that accept either a base
|
||||||
|
type or a list of base type. Using this ``list_or_integer``,
|
||||||
|
``list_or_number`` and ``list_or_bool`` were also added.
|
||||||
|
- ``wlauto.core.configuration.WorkloadRunSpec``:
|
||||||
|
- ``copy``: Allows making duplicates of ``WorkloadRunSpec``'s
|
||||||
|
- ``wlatuo.utils.misc``:
|
||||||
|
- ``list_to_ranges`` and ``ranges_to_list``: convert between lists of
|
||||||
|
integers and corresponding range strings, e.g. between [0,1,2,4] and
|
||||||
|
'0-2,4'
|
||||||
|
- ``list_to_mask`` and ``mask_to_list``: convert between lists of integers
|
||||||
|
and corresponding integer masks, e.g. between [0,1,2,4] and 0x17
|
||||||
|
- ``wlauto.instrumentation``:
|
||||||
|
- ``instrument_is_enabled``: Returns whether or not an instrument is
|
||||||
|
enabled for the current job.
|
||||||
|
- ``wlauto.core.result``:
|
||||||
|
- Added "classifiers" field to Metric objects. This is a dict mapping
|
||||||
|
classifier names (arbitrary strings) to corresponding values for that
|
||||||
|
specific metrics. This is to allow extensions to add extension-specific
|
||||||
|
annotations to metric that could be handled in a generic way (e.g. by
|
||||||
|
result processors). They can also be set in agendas.
|
||||||
|
- Failed jobs will now be automatically retired
|
||||||
|
- Implemented dynamic device modules that may be loaded automatically on
|
||||||
|
device initialization if the device supports them.
|
||||||
|
- Added support for YAML configs.
|
||||||
|
- Added ``initialze`` and ``finalize`` methods to workloads.
|
||||||
|
- ``wlauto.core.ExecutionContext``:
|
||||||
|
- Added ``job_status`` property that returns the status of the currently
|
||||||
|
running job.
|
||||||
|
|
||||||
|
Fixes/Improvements
|
||||||
|
##################
|
||||||
|
|
||||||
|
Devices
|
||||||
|
~~~~~~~~
|
||||||
|
- ``tc2``: Workaround for buffer overrun when loading large initrd blob.
|
||||||
|
- ``juno``:
|
||||||
|
- UEFI config can now be specified as a parameter.
|
||||||
|
- Adding support for U-Boot booting.
|
||||||
|
- No longer auto-disconnects ADB at the end of a run.
|
||||||
|
- Added ``actually_disconnect`` to restore old disconnect behaviour
|
||||||
|
- Now passes ``video`` command line to Juno kernel to work around a known
|
||||||
|
issue where HDMI loses sync with monitors.
|
||||||
|
- Fixed flashing.
|
||||||
|
|
||||||
|
Instruments
|
||||||
|
~~~~~~~~~~~
|
||||||
|
- ``trace_cmd``:
|
||||||
|
- Fixed ``buffer_size_file`` for non-Android devices
|
||||||
|
- Reduce starting priority.
|
||||||
|
- Now handles trace headers and thread names with spaces
|
||||||
|
- ``energy_probe``: Added ``device_entry`` parameter.
|
||||||
|
- ``hwmon``:
|
||||||
|
- Sensor discovery is now done only at the start of a run.
|
||||||
|
- Now prints both before/after and mean temperatures.
|
||||||
|
- ``daq``:
|
||||||
|
- Now reports energy
|
||||||
|
- Fixed file descriptor leak
|
||||||
|
- ``daq_power.csv`` now matches the order of labels (if specified).
|
||||||
|
- Added ``gpio_sync``. When enabled, this wil cause the instrument to
|
||||||
|
insert a marker into ftrace, while at the same time setting a GPIO pin
|
||||||
|
high.
|
||||||
|
- Added ``negative_values`` parameter. which can be used to specify how
|
||||||
|
negative values in the samples should be handled.
|
||||||
|
- Added ``merge_channels`` parameter. When set DAQ channel will be summed
|
||||||
|
together.
|
||||||
|
- Workload labels, rather than names, are now used in the "workload"
|
||||||
|
column.
|
||||||
|
- ``cpufreq``:
|
||||||
|
- Fixes missing directories problem.
|
||||||
|
- Refined the availability check not to rely on the top-level cpu/cpufreq
|
||||||
|
directory
|
||||||
|
- Now handles non-integer output in ``get_available_frequencies``.
|
||||||
|
- ``sysfs_extractor``:
|
||||||
|
- No longer raises an error when both device and host paths are empty.
|
||||||
|
- Fixed pulled files verification.
|
||||||
|
- ``perf``:
|
||||||
|
- Updated binaries.
|
||||||
|
- Added option to force install.
|
||||||
|
- ``killall`` is now run as root on rooted Android devices.
|
||||||
|
- ``fps``:
|
||||||
|
- now generates detailed FPS traces as well as report average FPS.
|
||||||
|
- Updated jank calcluation to only count "large" janks.
|
||||||
|
- Now filters out bogus ``actual-present`` times and ignore janks above
|
||||||
|
``PAUSE_LATENCY``
|
||||||
|
- ``delay``:
|
||||||
|
- Added ``fixed_before_start`` parameter.
|
||||||
|
- Changed existing ``*_between_specs`` and ``*_between_iterations``
|
||||||
|
callbacks to be ``very_slow``
|
||||||
|
- ``streamline``:
|
||||||
|
- Added Linux support
|
||||||
|
- ``gatord`` is now only started once at the start of the run.
|
||||||
|
|
||||||
|
modules
|
||||||
|
~~~~~~~
|
||||||
|
- ``flashing``:
|
||||||
|
- Fixed vexpress flashing
|
||||||
|
- Added an option to keep UEFI entry
|
||||||
|
|
||||||
|
Result Processors
|
||||||
|
~~~~~~~~~~~~~~~~~
|
||||||
|
- ``cpustate``:
|
||||||
|
- Now generates a timeline csv as well as stats.
|
||||||
|
- Adding ID to overall cpustate reports.
|
||||||
|
- ``csv``: (partial) ``results.csv`` will now be written after each iteration
|
||||||
|
rather than at the end of the run.
|
||||||
|
|
||||||
|
Workloads
|
||||||
|
~~~~~~~~~
|
||||||
|
- ``glb_corporate``: clears logcat to prevent getting results from previous
|
||||||
|
run.
|
||||||
|
- ``sysbench``:
|
||||||
|
- Updated sysbench binary to a statically linked verison
|
||||||
|
- Added ``file_test_mode parameter`` - this is a mandatory argumet if
|
||||||
|
``test`` is ``"fileio"``.
|
||||||
|
- Added ``cmd_params`` parameter to pass options directily to sysbench
|
||||||
|
invocation.
|
||||||
|
- Removed Android browser launch and shutdown from workload (now runs on
|
||||||
|
both Linux and Android).
|
||||||
|
- Now works with unrooted devices.
|
||||||
|
- Added the ability to run based on time.
|
||||||
|
- Added a parameter to taskset to specific core(s).
|
||||||
|
- Added ``threads`` parameter to be consistent with dhrystone.
|
||||||
|
- Fixed case where default ``timeout`` < ``max_time``.
|
||||||
|
- ``Dhrystone``:
|
||||||
|
- added ``taskset_mask`` parameter to allow pinning to specific cores.
|
||||||
|
- Now kills any running instances during setup (also handles CTRL-C).
|
||||||
|
- ``sysfs_extractor``: Added parameter to explicitly enable/disable tempfs
|
||||||
|
caching.
|
||||||
|
- ``antutu``:
|
||||||
|
- Fixed multi-``times`` playback for v5.
|
||||||
|
- Updated result parsing to handle Android M logcat output.
|
||||||
|
- ``geekbench``: Increased timout to cater for slower devices.
|
||||||
|
- ``idle``: Now works on Linux devices.
|
||||||
|
- ``manhattan``: Added ``run_timemout`` parameter.
|
||||||
|
- ``bbench``: Now works when binaries_directory is not in path.
|
||||||
|
- ``nemamark``: Made duration configurable.
|
||||||
|
|
||||||
|
Framework
|
||||||
|
~~~~~~~~~~
|
||||||
|
- ``BaseLinuxDevice``:
|
||||||
|
- Now checks that at least one core is enabled on another cluster before
|
||||||
|
attempting to set number of cores on a cluster to ``0``.
|
||||||
|
- No longer uses ``sudo`` if already logged in as ``root``.
|
||||||
|
- Now saves ``dumpsys window`` output to the ``__meta`` directory.
|
||||||
|
- Now takes ``password_prompt`` as a parameter for devices with a non
|
||||||
|
standard ``sudo`` password prompt.
|
||||||
|
- No longer raises an error if ``keyfile`` or ``password`` are not
|
||||||
|
provided when they are not necessary.
|
||||||
|
- Added new cpufreq APIs:
|
||||||
|
- ``core`` APIs take a core name as the parameter (e.g. "a15")
|
||||||
|
- ``cluster`` APIs take a numeric cluster ID (eg. 0)
|
||||||
|
- ``cpu`` APIs take a cpufreq cpu ID as a parameter.
|
||||||
|
- ``set_cpu_frequency`` now has a ``exact`` parameter. When true (the
|
||||||
|
default) it will produce an error when the specified frequency is not
|
||||||
|
supported by the cpu, otherwise cpufreq will decide what to do.
|
||||||
|
- Added ``{core}_frequency`` runtime parameter to set cluster frequency.
|
||||||
|
- Added ``abi`` property.
|
||||||
|
- ``get_properties`` moved from ``LinuxDevice``, meaning ``AndroidDevice``
|
||||||
|
will try to pull the same files. Added more paths to pull by default
|
||||||
|
too.
|
||||||
|
- fixed ``list_file_systems`` for Android M and Linux devices.
|
||||||
|
- Now sets ``core_clusters`` from ``core_names`` if not explicitly
|
||||||
|
specified.
|
||||||
|
- Added ``invoke`` method that allows invoking an executable on the device
|
||||||
|
under controlled contions (e.g. within a particular directory, or
|
||||||
|
taskset to specific CPUs).
|
||||||
|
- No longer attempts to ``get_sysfile_value()`` as root on unrooted
|
||||||
|
devices.
|
||||||
|
- ``LinuxDevice``:
|
||||||
|
- Now creates ``binaries_directory`` path if it doesn't exist.
|
||||||
|
- Fixed device reset
|
||||||
|
- Fixed ``file_exists``
|
||||||
|
- implemented ``get_pid_of()`` and ``ps()``. Existing implementation
|
||||||
|
relied on Android version of ps.
|
||||||
|
- ``listdir`` will now return an empty list for an empty directory
|
||||||
|
instead of a list containing a single empty string.
|
||||||
|
- ``AndroidDevice``:
|
||||||
|
- Executable (un)installation now works on unrooted devices.
|
||||||
|
- Now takes into account ``binar_directory`` when setting up busybox path.
|
||||||
|
- update ``android_prompt`` so that it works even if is not ``"/"``
|
||||||
|
- ``adb_connect``: do not assume port 5555 anymore.
|
||||||
|
- Now always deploys busybox on rooted devices.
|
||||||
|
- Added ``swipe_to_unlock`` method.
|
||||||
|
- Fixed initialization of ``~/.workload_automation.``.
|
||||||
|
- Fixed replaying events using revent on 64 bit platforms.
|
||||||
|
- Improved error repoting when loading extensions.
|
||||||
|
- ``result`` objects now track their output directories.
|
||||||
|
- ``context.result`` will not result in ``context.run_result`` when not
|
||||||
|
executing a job.
|
||||||
|
- ``wlauto.utils.ssh``:
|
||||||
|
- Fixed key-based authentication.
|
||||||
|
- Fixed carriage return stripping in ssh.
|
||||||
|
- Now takes ``password_prompt`` as a parameter for non standard ``sudo``
|
||||||
|
password prompts.
|
||||||
|
- Now with 100% more thread safety!
|
||||||
|
- If a timeout condition is hit, ^C is now sent to kill the current
|
||||||
|
foreground process and make the shell available for subsequent commands.
|
||||||
|
- More robust ``exit_code`` handling for ssh interface
|
||||||
|
- Now attempts to deal with dropped connections
|
||||||
|
- Fixed error reporting on failed exit code extraction.
|
||||||
|
- Now handles backspaces in serial output
|
||||||
|
- Added ``port`` argument for telnet connections.
|
||||||
|
- Now allows telnet connections without a password.
|
||||||
|
- Fixed config processing for extensions with non-identifier names.
|
||||||
|
- Fixed ``get_meansd`` for numbers < 1
|
||||||
|
- ``wlatuo.utils.ipython``:
|
||||||
|
- Now supports old versions of IPython
|
||||||
|
- Updated version check to only initialize ipython utils if version is
|
||||||
|
< 4.0.0. Version 4.0.0 changes API and breaks WA's usage of it.
|
||||||
|
- Added ``ignore`` parameter to ``check_output``
|
||||||
|
- Agendas:
|
||||||
|
- Now raise an error if an agenda contains duplicate keys
|
||||||
|
- Now raise an error if config section in an agenda is not dict-like
|
||||||
|
- Now properly handles ``core_names`` and ``core_clusters``
|
||||||
|
- When merging list parameters from different sources, duplicates are no
|
||||||
|
longer removed.
|
||||||
|
- The ``INITIAL_BOOT`` signal is now sent went performing a hard reset during
|
||||||
|
intial boot
|
||||||
|
- updated ``ExecutionContext`` to keep a reference to the ``runner``. This
|
||||||
|
will enable Extenstions to do things like modify the job queue.
|
||||||
|
- Parameter now automatically convert int and boot kinds to integer and
|
||||||
|
boolean respectively, this behavior can be supressed by specifying
|
||||||
|
``convert_types``=``False`` when defining the parameter.
|
||||||
|
- Fixed resource resolution when dependency location does not exist.
|
||||||
|
- All device ``push`` and ``pull`` commands now raise ``DeviceError`` if they
|
||||||
|
didn't succeed.
|
||||||
|
- Fixed showing Parameter default of ``False`` for boolean values.
|
||||||
|
- Updated csv result processor with the option to use classifiers to
|
||||||
|
add columns to ``results.csv``.
|
||||||
|
- ``wlauto.utils.formatter``: Fix terminal size discovery.
|
||||||
|
- The extension loader will now follow symlinks.
|
||||||
|
- Added arm64-v8a to ABI map
|
||||||
|
- WA now reports syntax errors in a more informative way.
|
||||||
|
- Resource resolver: now prints the path of the found resource to the log.
|
||||||
|
- Resource getter: look for executable in the bin/ directory under resource
|
||||||
|
owner's dependencies directory as well as general dependencies bin.
|
||||||
|
- ``GamingWorkload``:
|
||||||
|
- Added an option to prevent clearing of package data before execution.
|
||||||
|
- Added the ability to override the timeout of deploying the assets
|
||||||
|
tarball.
|
||||||
|
- ``ApkWorkload``: Added an option to skip host-side APK check entirely.
|
||||||
|
- ``utils.misc.normalize``: only normalize string keys.
|
||||||
|
- Better error reporting for subprocess.CalledProcessError
|
||||||
|
- ``boolean`` now interprets ``'off'`` as ``False``
|
||||||
|
- ``wlauto.utils.uefi``: Added support for debug builds.
|
||||||
|
- ``wlauto.utils.serial_port``: Now supports fdexpect versions > 4.0.0
|
||||||
|
- Semanatics for ``initialize``/``finalize`` for *all* Extensions are changed
|
||||||
|
so that now they will always run at most once per run. They will not be
|
||||||
|
executed twice even if invoked via instances of different subclasses (if
|
||||||
|
those subclasses defined their own verions, then their versions will be
|
||||||
|
invoked once each, but the base version will only get invoked once).
|
||||||
|
- Pulling entries from procfs does not work on some platforms. WA now tries
|
||||||
|
to cat the contents of a property_file and write it to a output file on the
|
||||||
|
host.
|
||||||
|
|
||||||
|
Documentation
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
- ``installation``:
|
||||||
|
- Added ``post install`` section which lists workloads that require
|
||||||
|
additional external dependencies.
|
||||||
|
- Added the ``uninstall`` and ``upgrade`` commands for users to remove or
|
||||||
|
upgrade Workload Automation.
|
||||||
|
- Added documentation explaining how to use ``remote_assets_path``
|
||||||
|
setting.
|
||||||
|
- Added warning about potential permission issues with pip.
|
||||||
|
- ``quickstart``: Added steps for setting up WA to run on Linux devices.
|
||||||
|
- ``device_setup``: fixed ``generic_linux`` ``device_config`` example.
|
||||||
|
- ``contributing``: Clarified style guidelines
|
||||||
|
- ``daq_device_setup``: Added an illustration for DAQ wiring.
|
||||||
|
- ``writing_extensions``: Documented the Workload initialize and finalize
|
||||||
|
methods.
|
||||||
|
- Added descriptions to extension that didn't have one.
|
||||||
|
|
||||||
|
Other
|
||||||
|
~~~~~
|
||||||
|
- ``daq_server``:
|
||||||
|
- Fixed showing available devices.
|
||||||
|
- Now works with earlier versions of the DAQmx driver.thus you can now run
|
||||||
|
the server on Linux systems.
|
||||||
|
- DAQ error messages are now properly propaged to the client.
|
||||||
|
- Server will now periodically clean up uncollected files.
|
||||||
|
- fixed not being able to resolve IP address for hostname
|
||||||
|
(report "localhost" in that case).
|
||||||
|
- Works with latest version of twisted.
|
||||||
|
- ``setup.py``: Fixed paths to work with Mac OS X.
|
||||||
|
- ``summary_csv`` is no longer enabled by default.
|
||||||
|
- ``status`` result processor is now enabled by default.
|
||||||
|
- Commands:
|
||||||
|
- ``show``:
|
||||||
|
- Now shows what platform extensions support.
|
||||||
|
- Will no longer try to use a pager if ``PAGER=''`` in the environment.
|
||||||
|
- ``list``:
|
||||||
|
- Added ``"-p"`` option to filter results by supported platforms.
|
||||||
|
- Added ``"--packaged-only"`` option to only list extensions packaged
|
||||||
|
with WA.
|
||||||
|
- ``run``: Added ``"--disable"`` option to diable instruments.
|
||||||
|
- ``create``:
|
||||||
|
- Added ``agenda`` sub-command to generate agendas for a set of
|
||||||
|
extensions.
|
||||||
|
- ``create workload`` now gives more informative errors if Android SDK
|
||||||
|
installed but no platform has been downloaded.
|
||||||
|
|
||||||
|
Incompatible changes
|
||||||
|
####################
|
||||||
|
|
||||||
|
Framework
|
||||||
|
~~~~~~~~~
|
||||||
|
- ``BaseLinuxDevice``:
|
||||||
|
- Renamed ``active_cpus`` to ``online_cpus``
|
||||||
|
- Renamed ``get_cluster_cpu`` to ``get_cluster_active_cpu``
|
||||||
|
- Renamed ``get_core_cpu`` to ``get_core_online_cpu``
|
||||||
|
- All extension's ``initialize`` function now takes one (and only one)
|
||||||
|
parameter, ``context``.
|
||||||
|
- ``wlauto.core.device``: Removed ``init`` function. Replaced with
|
||||||
|
``initialize``
|
||||||
|
|
||||||
|
-------------
|
||||||
Version 2.3.0
|
Version 2.3.0
|
||||||
-------------
|
-------------
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user