workload-automation/doc/source/changes.rst

=================================
What's New in Workload Automation
=================================
-------------
Version 2.4.0
-------------

Additions:
##########

Devices
~~~~~~~~
- ``gem5_linux`` and ``gem5_android``: Interfaces for Gem5 simulation
  environment running Linux and Android respectively.
- ``XE503C1211``: Interface for Samsung XE503C12 Chromebooks.
- ``chromeos_test_image``: Chrome OS test image device. An off the shelf
  device will not work with this device interface.

Instruments
~~~~~~~~~~~~
- ``freq_sweep``: Allows "sweeping" workloads across multiple CPU frequencies.
- ``screenon``: Ensures screen is on, before each iteration, or periodically
  on Android devices.
- ``energy_model``: This instrument can be used to generate an energy model
  for a device based on collected power and performance measurments.
- ``netstats``:  Allows monitoring data sent/received by applications on an
  Android device.

Modules
~~~~~~~
- ``cgroups``: Allows query and manipulation of cgroups controllers on a Linux
  device. Currently, only cpusets controller is implemented.
- ``cpuidle``: Implements cpuidle state discovery, query and manipulation for
  a Linux device. This replaces the more primitive get_cpuidle_states method
  of LinuxDevice.
- ``cpufreq`` has now been split out into a device module

Reasource Getters
~~~~~~~~~~~~~~~~~
- ``http_assets``:  Downloads resources from a web server.

Results Processors
~~~~~~~~~~~~~~~~~~~
- ``ipynb_exporter``: Generates an IPython notebook from a template with the
  results and runs it.
- ``notify``: Displays a desktop notification when a run finishes
  (Linux only).
- ``cpustates``: Processes power ftrace to produce CPU state and parallelism
  stats. There is also a script to invoke this outside of WA.

Workloads
~~~~~~~~~
- ``telemetry``: Executes Google's Telemetery benchmarking framework
- ``hackbench``: Hackbench runs tests on the Linux scheduler
- ``ebizzy``: This workload resembles common web server application workloads.
- ``power_loadtest``: Continuously cycles through a set of browser-based
  activities and monitors battery drain on a device (part of ChromeOS autotest
  suite).
- ``rt-app``: Simulates configurable real-time periodic load.
- ``linpack-cli``:  Command line version of linpack benchmark.
- ``lmbench``: A suite of portable ANSI/C microbenchmarks for UNIX/POSIX.
- ``stream``: Measures memory bandwidth.
- ``iozone``: Runs a series of disk I/O performance tests.
- ``androbench``:  Measures the storage performance of device.
- ``autotest``:  Executes tests from ChromeOS autotest suite.

Framework
~~~~~~~~~
- ``wlauto.utils``:
   - Added ``trace_cmd``, a generic trace-cmd paraser.
   - Added ``UbootMenu``, allows navigating Das U-boot menu over serial.
- ``wlauto.utils.types``:
   - ``caseless_string``: Behaves exactly like a string, except this ignores
     case in comparisons. It does, however, preserve case.
   - ``list_of``: allows dynamic generation of type-safe list types based on
     an existing type.
   - ``arguments``: represents arguments that are passed on a command line to
     an application.
   - ``list-or``: allows dynamic generation of types that accept either a base
     type or a list of base type. Using this ``list_or_integer``,
     ``list_or_number`` and ``list_or_bool`` were also added.
- ``wlauto.core.configuration.WorkloadRunSpec``:
   - ``copy``: Allows making duplicates of ``WorkloadRunSpec``'s
- ``wlatuo.utils.misc``:
   - ``list_to_ranges`` and ``ranges_to_list``: convert between lists of
     integers and corresponding range strings, e.g. between [0,1,2,4] and
     '0-2,4'
   - ``list_to_mask`` and ``mask_to_list``: convert between lists of integers
     and corresponding integer masks, e.g. between [0,1,2,4] and 0x17
- ``wlauto.instrumentation``:
   - ``instrument_is_enabled``: Returns whether or not an instrument is
     enabled for the current job.
- ``wlauto.core.result``:
   - Added "classifiers" field to Metric objects. This is a dict mapping
     classifier names (arbitrary strings) to corresponding values for that
     specific metrics. This is to allow plugins to add plugin-specific
     annotations to metric that could be handled in a generic way (e.g. by
     result processors). They can also be set in agendas.
- Failed jobs will now be automatically retired
- Implemented dynamic device modules that may be loaded automatically on
  device initialization if the device supports them.
- Added support for YAML configs.
- Added ``initialze`` and ``finalize`` methods to workloads.
- ``wlauto.core.ExecutionContext``:
   - Added ``job_status`` property that returns the status of the currently
     running job.

Fixes/Improvements
##################

Devices
~~~~~~~~
- ``tc2``: Workaround for buffer overrun when loading large initrd blob.
- ``juno``:
     - UEFI config can now be specified as a parameter.
     - Adding support for U-Boot booting.
     - No longer auto-disconnects ADB at the end of a run.
     - Added ``actually_disconnect`` to restore old disconnect behaviour
     - Now passes ``video`` command line to Juno kernel to work around a known
       issue where HDMI loses sync with monitors.
     - Fixed flashing.

Instruments
~~~~~~~~~~~
- ``trace_cmd``:
     - Fixed ``buffer_size_file`` for non-Android devices
     - Reduce starting priority.
     - Now handles trace headers and thread names with spaces
- ``energy_probe``: Added ``device_entry`` parameter.
- ``hwmon``:
     - Sensor discovery is now done only at the start of a run.
     - Now prints both before/after and mean temperatures.
- ``daq``:
     - Now reports energy
     - Fixed file descriptor leak
     - ``daq_power.csv`` now matches the order of labels (if specified).
     - Added ``gpio_sync``. When enabled, this wil cause the instrument to
       insert a marker into ftrace, while at the same time setting a GPIO pin
       high.
     - Added ``negative_values`` parameter. which can be used to specify how
       negative values in the samples should be handled.
     - Added ``merge_channels`` parameter. When set DAQ channel will be summed
       together.
     - Workload labels, rather than names, are now used in the "workload"
       column.
- ``cpufreq``:
     - Fixes missing directories problem.
     - Refined the availability check not to rely on the top-level cpu/cpufreq
       directory
     - Now handles non-integer output in ``get_available_frequencies``.
- ``sysfs_extractor``:
     - No longer raises an error when both device and host paths are empty.
     - Fixed pulled files verification.
- ``perf``:
     - Updated binaries.
     - Added option to force install.
     - ``killall`` is now run as root on rooted Android devices.
- ``fps``:
     - now generates detailed FPS traces as well as report average FPS.
     - Updated jank calcluation to only count "large" janks.
     - Now filters out bogus ``actual-present`` times and ignore janks above
       ``PAUSE_LATENCY``
- ``delay``:
     - Added ``fixed_before_start`` parameter.
     - Changed existing ``*_between_specs`` and ``*_between_iterations``
       callbacks to be ``very_slow``
- ``streamline``:
     - Added Linux support
     - ``gatord`` is now only started once at the start of the run.

modules
~~~~~~~
- ``flashing``:
     - Fixed vexpress flashing
     - Added an option to keep UEFI entry

Result Processors
~~~~~~~~~~~~~~~~~
- ``cpustate``:
     - Now generates a timeline csv as well as stats.
     - Adding ID to overall cpustate reports.
- ``csv``: (partial) ``results.csv`` will now be written after each iteration
  rather than at the end of the run.

Workloads
~~~~~~~~~
- ``glb_corporate``: clears logcat to prevent getting results from previous
  run.
- ``sysbench``:
     - Updated sysbench binary to a statically linked verison
     - Added ``file_test_mode parameter`` - this is a mandatory argumet if
       ``test`` is ``"fileio"``.
     - Added ``cmd_params`` parameter to pass options directily to sysbench
       invocation.
     - Removed Android browser launch and shutdown from workload (now runs on
       both Linux and Android).
     - Now works with unrooted devices.
     - Added the ability to run based on time.
     - Added a parameter to taskset to specific core(s).
     - Added ``threads`` parameter to be consistent with dhrystone.
     - Fixed case where default ``timeout`` < ``max_time``.
- ``Dhrystone``:
     - added ``taskset_mask`` parameter to allow pinning to specific cores.
     - Now kills any running instances during setup (also handles CTRL-C).
- ``sysfs_extractor``: Added parameter to explicitly enable/disable tempfs
  caching.
- ``antutu``:
     - Fixed multi-``times`` playback for v5.
     - Updated result parsing to handle Android M logcat output.
- ``geekbench``: Increased timout to cater for slower devices.
- ``idle``: Now works on Linux devices.
- ``manhattan``: Added ``run_timemout`` parameter.
- ``bbench``: Now works when binaries_directory is not in path.
- ``nemamark``: Made duration configurable.

Framework
~~~~~~~~~~
- ``BaseLinuxDevice``:
     - Now checks that at least one core is enabled on another cluster before
       attempting to set number of cores on a cluster to ``0``.
     - No longer uses ``sudo`` if already logged in as ``root``.
     - Now saves ``dumpsys window`` output to the ``__meta`` directory.
     - Now takes ``password_prompt`` as a parameter for devices with a non
       standard ``sudo`` password prompt.
     - No longer raises an error if ``keyfile`` or ``password`` are not
       provided when they are not necessary.
     - Added new cpufreq APIs:
        - ``core`` APIs take a core name as the parameter (e.g. "a15")
        - ``cluster`` APIs take a numeric cluster ID (eg. 0)
        - ``cpu`` APIs take a cpufreq cpu ID as a parameter.
     - ``set_cpu_frequency`` now has a ``exact`` parameter. When true (the
       default) it will produce an error when the specified frequency is not
       supported by the cpu, otherwise cpufreq will decide what to do.
     - Added ``{core}_frequency`` runtime parameter to set cluster frequency.
     - Added ``abi`` property.
     - ``get_properties`` moved from ``LinuxDevice``, meaning ``AndroidDevice``
       will try to pull the same files. Added more paths to pull by default
       too.
     - fixed ``list_file_systems`` for Android M and Linux devices.
     - Now sets ``core_clusters`` from ``core_names`` if not explicitly
       specified.
     - Added ``invoke`` method that allows invoking an executable on the device
       under controlled contions (e.g. within a particular directory, or
       taskset to specific CPUs).
     - No longer attempts to ``get_sysfile_value()`` as root on unrooted
       devices.
- ``LinuxDevice``:
     - Now creates ``binaries_directory`` path if it doesn't exist.
     - Fixed device reset
     - Fixed ``file_exists``
     - implemented ``get_pid_of()`` and ``ps()``. Existing implementation
       relied on Android version of ps.
     - ``listdir`` will now return an empty list for an empty directory
       instead of a list containing a single empty string.
- ``AndroidDevice``:
     - Executable (un)installation now works on unrooted devices.
     - Now takes into account ``binar_directory`` when setting up busybox path.
     - update ``android_prompt`` so that it works even if is not ``"/"``
     - ``adb_connect``: do not assume port 5555 anymore.
     - Now always deploys busybox on rooted devices.
     - Added ``swipe_to_unlock`` method.
- Fixed initialization of ``~/.workload_automation.``.
- Fixed replaying events using revent on 64 bit platforms.
- Improved error repoting when loading plugins.
- ``result`` objects now track their output directories.
- ``context.result`` will not result in ``context.run_result`` when not
  executing a job.
- ``wlauto.utils.ssh``:
     - Fixed key-based authentication.
     - Fixed carriage return stripping in ssh.
     - Now takes ``password_prompt`` as a parameter for non standard ``sudo``
       password prompts.
     - Now with 100% more thread safety!
     - If a timeout condition is hit, ^C is now sent to kill the current
       foreground process and make the shell available for subsequent commands.
     - More robust ``exit_code`` handling for ssh interface
     - Now attempts to deal with dropped connections
     - Fixed error reporting on failed exit code extraction.
     - Now handles backspaces in serial output
     - Added ``port`` argument for telnet connections.
     - Now allows telnet connections without a password.
- Fixed config processing for plugins with non-identifier names.
- Fixed ``get_meansd`` for numbers < 1
- ``wlatuo.utils.ipython``:
     - Now supports old versions of IPython
     - Updated version check to only initialize ipython utils if version is
       < 4.0.0. Version 4.0.0 changes API and breaks WA's usage of it.
- Added ``ignore`` parameter to ``check_output``
- Agendas:
     - Now raise an error if an agenda contains duplicate keys
     - Now raise an error if config section in an agenda is not dict-like
     - Now properly handles ``core_names`` and ``core_clusters``
     - When merging list parameters from different sources, duplicates are no
       longer removed.
- The ``INITIAL_BOOT`` signal is now sent went performing a hard reset during
  intial boot
- updated ``ExecutionContext`` to keep a reference to the ``runner``. This
  will enable Extenstions to do things like modify the job queue.
- Parameter now automatically convert int and boot kinds to integer and
  boolean respectively, this behavior can be supressed by specifying
  ``convert_types``=``False`` when defining the parameter.
- Fixed resource resolution when dependency location does not exist.
- All device ``push`` and ``pull`` commands now raise ``DeviceError`` if they
  didn't succeed.
- Fixed showing Parameter default of ``False`` for boolean values.
- Updated csv result processor with the option to use classifiers to
  add columns to ``results.csv``.
- ``wlauto.utils.formatter``: Fix terminal size discovery.
- The plugin loader will now follow symlinks.
- Added arm64-v8a to ABI map
- WA now reports syntax errors in a more informative way.
- Resource resolver: now prints the path of the found resource to the log.
- Resource getter: look for executable in the bin/ directory under resource
  owner's dependencies directory as well as general dependencies bin.
- ``GamingWorkload``:
     - Added an option to prevent clearing of package data before execution.
     - Added the ability to override the timeout of deploying the assets
       tarball.
- ``ApkWorkload``: Added an option to skip host-side APK check entirely.
- ``utils.misc.normalize``: only normalize string keys.
- Better error reporting for subprocess.CalledProcessError
- ``boolean`` now interprets ``'off'`` as ``False``
- ``wlauto.utils.uefi``: Added support for debug builds.
- ``wlauto.utils.serial_port``: Now supports fdexpect versions > 4.0.0
- Semanatics for ``initialize``/``finalize`` for *all* Plugins are changed
  so that now they will always run at most once per run. They will not be
  executed twice even if invoked via instances of different subclasses (if
  those subclasses defined their own verions, then their versions will be
  invoked once each, but the base version will only get invoked once).
- Pulling entries from procfs does not work on some platforms. WA now tries
  to cat the contents of a property_file and write it to a output file on the
  host.

Documentation
~~~~~~~~~~~~~
- ``installation``:
     - Added ``post install`` section which lists workloads that require
       additional external dependencies.
     - Added the ``uninstall`` and ``upgrade`` commands for users to remove or
       upgrade Workload Automation.
     - Added documentation explaining how to use ``remote_assets_path``
       setting.
     - Added warning about potential permission issues with pip.
- ``quickstart``: Added steps for setting up WA to run on Linux devices.
- ``device_setup``: fixed ``generic_linux`` ``device_config`` example.
- ``contributing``: Clarified style guidelines
- ``daq_device_setup``: Added an illustration for DAQ wiring.
- ``writing_plugins``: Documented the Workload initialize and finalize
  methods.
- Added descriptions to plugin that didn't have one.

Other
~~~~~
- ``daq_server``:
     - Fixed showing available devices.
     - Now works with earlier versions of the DAQmx driver.thus you can now run
       the server on Linux systems.
     - DAQ error messages are now properly propaged to the client.
     - Server will now periodically clean up uncollected files.
     - fixed not being able to resolve IP address for hostname
       (report "localhost" in that case).
     - Works with latest version of twisted.
- ``setup.py``: Fixed paths to work with Mac OS X.
- ``summary_csv`` is no longer enabled by default.
- ``status`` result processor is now enabled by default.
- Commands:
     - ``show``:
         - Now shows what platform plugins support.
         - Will no longer try to use a pager if ``PAGER=''`` in the environment.
     - ``list``:
         - Added ``"-p"`` option to filter results by supported platforms.
         - Added ``"--packaged-only"`` option to only list plugins packaged
           with WA.
     - ``run``: Added ``"--disable"`` option to diable instruments.
     - ``create``:
         - Added ``agenda`` sub-command to generate agendas for a set of
           plugins.
         - ``create workload`` now gives more informative errors if Android SDK
           installed but no platform has been downloaded.

Incompatible changes
####################

Framework
~~~~~~~~~
- ``BaseLinuxDevice``:
     - Renamed ``active_cpus`` to ``online_cpus``
     - Renamed ``get_cluster_cpu`` to ``get_cluster_active_cpu``
     - Renamed ``get_core_cpu`` to ``get_core_online_cpu``
- All plugin's ``initialize`` function now takes one (and only one)
  parameter, ``context``.
- ``wlauto.core.device``: Removed ``init`` function. Replaced with
  ``initialize``

-------------
Version 2.3.0
-------------

- First publicly-released version.