Updated change log

Initial update of change log, this will probably need a lot of tidy up.

doc/source/changes.rst

@@ -1,6 +1,398 @@
+=================================
 What's New in Workload Automation
 =================================
+-------------
+Version 2.4.0
+-------------
 
+Additions:
+##########
+
+Devices
+~~~~~~~~
+- ``gem5_linux`` and ``gem5_android``: Interfaces for Gem5 simulation
+  environment running Linux and Android respectively.
+- ``XE503C1211``: Interface for Samsung XE503C12 Chromebooks.
+- ``chromeos_test_image``: Chrome OS test image device. An off the shelf
+  device will not work with this device interface.
+
+Instruments
+~~~~~~~~~~~~
+- ``freq_sweep``: Allows "sweeping" workloads across multiple CPU frequencies.
+- ``screenon``: Ensures screen is on, before each iteration, or periodically
+  on Android devices.
+- ``energy_model``: This instrument can be used to generate an energy model
+  for a device based on collected power and performance measurments.
+- ``netstats``:  Allows monitoring data sent/received by applications on an
+  Android device.
+
+Modules
+~~~~~~~
+- ``cgroups``: Allows query and manipulation of cgroups controllers on a Linux
+  device. Currently, only cpusets controller is implemented.
+- ``cpuidle``: Implements cpuidle state discovery, query and manipulation for
+  a Linux device. This replaces the more primitive get_cpuidle_states method
+  of LinuxDevice.
+- ``cpufreq`` has now been split out into a device module
+
+Reasource Getters
+~~~~~~~~~~~~~~~~~
+- ``http_assets``:  Downloads resources from a web server.
+
+Results Processors
+~~~~~~~~~~~~~~~~~~~
+- ``ipynb_exporter``: Generates an IPython notebook from a template with the
+  results and runs it.
+- ``notify``: Displays a desktop notification when a run finishes
+  (Linux only).
+- ``cpustates``: Processes power ftrace to produce CPU state and parallelism
+  stats. There is also a script to invoke this outside of WA.
+
+Workloads
+~~~~~~~~~
+- ``telemetry``: Executes Google's Telemetery benchmarking framework
+- ``hackbench``: Hackbench runs tests on the Linux scheduler
+- ``ebizzy``: This workload resembles common web server application workloads.
+- ``power_loadtest``: Continuously cycles through a set of browser-based
+  activities and monitors battery drain on a device (part of ChromeOS autotest
+  suite).
+- ``rt-app``: Simulates configurable real-time periodic load.
+- ``linpack-cli``:  Command line version of linpack benchmark.
+- ``lmbench``: A suite of portable ANSI/C microbenchmarks for UNIX/POSIX.
+- ``stream``: Measures memory bandwidth.
+- ``iozone``: Runs a series of disk I/O performance tests.
+- ``androbench``:  Measures the storage performance of device.
+- ``autotest``:  Executes tests from ChromeOS autotest suite.
+
+Framework
+~~~~~~~~~
+- ``wlauto.utils``:
+   - Added ``trace_cmd``, a generic trace-cmd paraser.
+   - Added ``UbootMenu``, allows navigating Das U-boot menu over serial.
+- ``wlauto.utils.types``:
+   - ``caseless_string``: Behaves exactly like a string, except this ignores
+     case in comparisons. It does, however, preserve case.
+   - ``list_of``: allows dynamic generation of type-safe list types based on
+     an existing type.
+   - ``arguments``: represents arguments that are passed on a command line to
+     an application.
+   - ``list-or``: allows dynamic generation of types that accept either a base
+     type or a list of base type. Using this ``list_or_integer``,
+     ``list_or_number`` and ``list_or_bool`` were also added.
+- ``wlauto.core.configuration.WorkloadRunSpec``:
+   - ``copy``: Allows making duplicates of ``WorkloadRunSpec``'s
+- ``wlatuo.utils.misc``:
+   - ``list_to_ranges`` and ``ranges_to_list``: convert between lists of
+     integers and corresponding range strings, e.g. between [0,1,2,4] and
+     '0-2,4'
+   - ``list_to_mask`` and ``mask_to_list``: convert between lists of integers
+     and corresponding integer masks, e.g. between [0,1,2,4] and 0x17
+- ``wlauto.instrumentation``:
+   - ``instrument_is_enabled``: Returns whether or not an instrument is
+     enabled for the current job.
+- ``wlauto.core.result``:
+   - Added "classifiers" field to Metric objects. This is a dict mapping
+     classifier names (arbitrary strings) to corresponding values for that
+     specific metrics. This is to allow extensions to add extension-specific
+     annotations to metric that could be handled in a generic way (e.g. by
+     result processors). They can also be set in agendas.
+- Failed jobs will now be automatically retired
+- Implemented dynamic device modules that may be loaded automatically on
+  device initialization if the device supports them.
+- Added support for YAML configs.
+- Added ``initialze`` and ``finalize`` methods to workloads.
+- ``wlauto.core.ExecutionContext``:
+   - Added ``job_status`` property that returns the status of the currently
+     running job.
+
+Fixes/Improvements
+##################
+
+Devices
+~~~~~~~~
+- ``tc2``: Workaround for buffer overrun when loading large initrd blob.
+- ``juno``:
+     - UEFI config can now be specified as a parameter.
+     - Adding support for U-Boot booting.
+     - No longer auto-disconnects ADB at the end of a run.
+     - Added ``actually_disconnect`` to restore old disconnect behaviour
+     - Now passes ``video`` command line to Juno kernel to work around a known
+       issue where HDMI loses sync with monitors.
+     - Fixed flashing.
+
+Instruments
+~~~~~~~~~~~
+- ``trace_cmd``:
+     - Fixed ``buffer_size_file`` for non-Android devices
+     - Reduce starting priority.
+     - Now handles trace headers and thread names with spaces
+- ``energy_probe``: Added ``device_entry`` parameter.
+- ``hwmon``:
+     - Sensor discovery is now done only at the start of a run.
+     - Now prints both before/after and mean temperatures.
+- ``daq``:
+     - Now reports energy
+     - Fixed file descriptor leak
+     - ``daq_power.csv`` now matches the order of labels (if specified).
+     - Added ``gpio_sync``. When enabled, this wil cause the instrument to
+       insert a marker into ftrace, while at the same time setting a GPIO pin
+       high.
+     - Added ``negative_values`` parameter. which can be used to specify how
+       negative values in the samples should be handled.
+     - Added ``merge_channels`` parameter. When set DAQ channel will be summed
+       together.
+     - Workload labels, rather than names, are now used in the "workload"
+       column.
+- ``cpufreq``:
+     - Fixes missing directories problem.
+     - Refined the availability check not to rely on the top-level cpu/cpufreq
+       directory
+     - Now handles non-integer output in ``get_available_frequencies``.
+- ``sysfs_extractor``:
+     - No longer raises an error when both device and host paths are empty.
+     - Fixed pulled files verification.
+- ``perf``:
+     - Updated binaries.
+     - Added option to force install.
+     - ``killall`` is now run as root on rooted Android devices.
+- ``fps``:
+     - now generates detailed FPS traces as well as report average FPS.
+     - Updated jank calcluation to only count "large" janks.
+     - Now filters out bogus ``actual-present`` times and ignore janks above
+       ``PAUSE_LATENCY``
+- ``delay``:
+     - Added ``fixed_before_start`` parameter.
+     - Changed existing ``*_between_specs`` and ``*_between_iterations``
+       callbacks to be ``very_slow``
+- ``streamline``:
+     - Added Linux support
+     - ``gatord`` is now only started once at the start of the run.
+
+modules
+~~~~~~~
+- ``flashing``:
+     - Fixed vexpress flashing
+     - Added an option to keep UEFI entry
+
+Result Processors
+~~~~~~~~~~~~~~~~~
+- ``cpustate``:
+     - Now generates a timeline csv as well as stats.
+     - Adding ID to overall cpustate reports.
+- ``csv``: (partial) ``results.csv`` will now be written after each iteration
+  rather than at the end of the run.
+
+Workloads
+~~~~~~~~~
+- ``glb_corporate``: clears logcat to prevent getting results from previous
+  run.
+- ``sysbench``:
+     - Updated sysbench binary to a statically linked verison
+     - Added ``file_test_mode parameter`` - this is a mandatory argumet if
+       ``test`` is ``"fileio"``.
+     - Added ``cmd_params`` parameter to pass options directily to sysbench
+       invocation.
+     - Removed Android browser launch and shutdown from workload (now runs on
+       both Linux and Android).
+     - Now works with unrooted devices.
+     - Added the ability to run based on time.
+     - Added a parameter to taskset to specific core(s).
+     - Added ``threads`` parameter to be consistent with dhrystone.
+     - Fixed case where default ``timeout`` < ``max_time``.
+- ``Dhrystone``:
+     - added ``taskset_mask`` parameter to allow pinning to specific cores.
+     - Now kills any running instances during setup (also handles CTRL-C).
+- ``sysfs_extractor``: Added parameter to explicitly enable/disable tempfs
+  caching.
+- ``antutu``:
+     - Fixed multi-``times`` playback for v5.
+     - Updated result parsing to handle Android M logcat output.
+- ``geekbench``: Increased timout to cater for slower devices.
+- ``idle``: Now works on Linux devices.
+- ``manhattan``: Added ``run_timemout`` parameter.
+- ``bbench``: Now works when binaries_directory is not in path.
+- ``nemamark``: Made duration configurable.
+
+Framework
+~~~~~~~~~~
+- ``BaseLinuxDevice``:
+     - Now checks that at least one core is enabled on another cluster before
+       attempting to set number of cores on a cluster to ``0``.
+     - No longer uses ``sudo`` if already logged in as ``root``.
+     - Now saves ``dumpsys window`` output to the ``__meta`` directory.
+     - Now takes ``password_prompt`` as a parameter for devices with a non
+       standard ``sudo`` password prompt.
+     - No longer raises an error if ``keyfile`` or ``password`` are not
+       provided when they are not necessary.
+     - Added new cpufreq APIs:
+        - ``core`` APIs take a core name as the parameter (e.g. "a15")
+        - ``cluster`` APIs take a numeric cluster ID (eg. 0)
+        - ``cpu`` APIs take a cpufreq cpu ID as a parameter.
+     - ``set_cpu_frequency`` now has a ``exact`` parameter. When true (the
+       default) it will produce an error when the specified frequency is not
+       supported by the cpu, otherwise cpufreq will decide what to do.
+     - Added ``{core}_frequency`` runtime parameter to set cluster frequency.
+     - Added ``abi`` property.
+     - ``get_properties`` moved from ``LinuxDevice``, meaning ``AndroidDevice``
+       will try to pull the same files. Added more paths to pull by default
+       too.
+     - fixed ``list_file_systems`` for Android M and Linux devices.
+     - Now sets ``core_clusters`` from ``core_names`` if not explicitly
+       specified.
+     - Added ``invoke`` method that allows invoking an executable on the device
+       under controlled contions (e.g. within a particular directory, or
+       taskset to specific CPUs).
+     - No longer attempts to ``get_sysfile_value()`` as root on unrooted
+       devices.
+- ``LinuxDevice``:
+     - Now creates ``binaries_directory`` path if it doesn't exist.
+     - Fixed device reset
+     - Fixed ``file_exists``
+     - implemented ``get_pid_of()`` and ``ps()``. Existing implementation
+       relied on Android version of ps.
+     - ``listdir`` will now return an empty list for an empty directory
+       instead of a list containing a single empty string.
+- ``AndroidDevice``:
+     - Executable (un)installation now works on unrooted devices.
+     - Now takes into account ``binar_directory`` when setting up busybox path.
+     - update ``android_prompt`` so that it works even if is not ``"/"``
+     - ``adb_connect``: do not assume port 5555 anymore.
+     - Now always deploys busybox on rooted devices.
+     - Added ``swipe_to_unlock`` method.
+- Fixed initialization of ``~/.workload_automation.``.
+- Fixed replaying events using revent on 64 bit platforms.
+- Improved error repoting when loading extensions.
+- ``result`` objects now track their output directories.
+- ``context.result`` will not result in ``context.run_result`` when not
+  executing a job.
+- ``wlauto.utils.ssh``:
+     - Fixed key-based authentication.
+     - Fixed carriage return stripping in ssh.
+     - Now takes ``password_prompt`` as a parameter for non standard ``sudo``
+       password prompts.
+     - Now with 100% more thread safety!
+     - If a timeout condition is hit, ^C is now sent to kill the current
+       foreground process and make the shell available for subsequent commands.
+     - More robust ``exit_code`` handling for ssh interface
+     - Now attempts to deal with dropped connections
+     - Fixed error reporting on failed exit code extraction.
+     - Now handles backspaces in serial output
+     - Added ``port`` argument for telnet connections.
+     - Now allows telnet connections without a password.
+- Fixed config processing for extensions with non-identifier names.
+- Fixed ``get_meansd`` for numbers < 1
+- ``wlatuo.utils.ipython``:
+     - Now supports old versions of IPython
+     - Updated version check to only initialize ipython utils if version is
+       < 4.0.0. Version 4.0.0 changes API and breaks WA's usage of it.
+- Added ``ignore`` parameter to ``check_output``
+- Agendas:
+     - Now raise an error if an agenda contains duplicate keys
+     - Now raise an error if config section in an agenda is not dict-like
+     - Now properly handles ``core_names`` and ``core_clusters``
+     - When merging list parameters from different sources, duplicates are no
+       longer removed.
+- The ``INITIAL_BOOT`` signal is now sent went performing a hard reset during
+  intial boot
+- updated ``ExecutionContext`` to keep a reference to the ``runner``. This
+  will enable Extenstions to do things like modify the job queue.
+- Parameter now automatically convert int and boot kinds to integer and
+  boolean respectively, this behavior can be supressed by specifying
+  ``convert_types``=``False`` when defining the parameter.
+- Fixed resource resolution when dependency location does not exist.
+- All device ``push`` and ``pull`` commands now raise ``DeviceError`` if they
+  didn't succeed.
+- Fixed showing Parameter default of ``False`` for boolean values.
+- Updated csv result processor with the option to use classifiers to
+  add columns to ``results.csv``.
+- ``wlauto.utils.formatter``: Fix terminal size discovery.
+- The extension loader will now follow symlinks.
+- Added arm64-v8a to ABI map
+- WA now reports syntax errors in a more informative way.
+- Resource resolver: now prints the path of the found resource to the log.
+- Resource getter: look for executable in the bin/ directory under resource
+  owner's dependencies directory as well as general dependencies bin.
+- ``GamingWorkload``:
+     - Added an option to prevent clearing of package data before execution.
+     - Added the ability to override the timeout of deploying the assets
+       tarball.
+- ``ApkWorkload``: Added an option to skip host-side APK check entirely.
+- ``utils.misc.normalize``: only normalize string keys.
+- Better error reporting for subprocess.CalledProcessError
+- ``boolean`` now interprets ``'off'`` as ``False``
+- ``wlauto.utils.uefi``: Added support for debug builds.
+- ``wlauto.utils.serial_port``: Now supports fdexpect versions > 4.0.0
+- Semanatics for ``initialize``/``finalize`` for *all* Extensions are changed
+  so that now they will always run at most once per run. They will not be
+  executed twice even if invoked via instances of different subclasses (if
+  those subclasses defined their own verions, then their versions will be
+  invoked once each, but the base version will only get invoked once).
+- Pulling entries from procfs does not work on some platforms. WA now tries
+  to cat the contents of a property_file and write it to a output file on the
+  host.
+
+Documentation
+~~~~~~~~~~~~~
+- ``installation``:
+     - Added ``post install`` section which lists workloads that require
+       additional external dependencies.
+     - Added the ``uninstall`` and ``upgrade`` commands for users to remove or
+       upgrade Workload Automation.
+     - Added documentation explaining how to use ``remote_assets_path``
+       setting.
+     - Added warning about potential permission issues with pip.
+- ``quickstart``: Added steps for setting up WA to run on Linux devices.
+- ``device_setup``: fixed ``generic_linux`` ``device_config`` example.
+- ``contributing``: Clarified style guidelines
+- ``daq_device_setup``: Added an illustration for DAQ wiring.
+- ``writing_extensions``: Documented the Workload initialize and finalize
+  methods.
+- Added descriptions to extension that didn't have one.
+
+Other
+~~~~~
+- ``daq_server``:
+     - Fixed showing available devices.
+     - Now works with earlier versions of the DAQmx driver.thus you can now run
+       the server on Linux systems.
+     - DAQ error messages are now properly propaged to the client.
+     - Server will now periodically clean up uncollected files.
+     - fixed not being able to resolve IP address for hostname
+       (report "localhost" in that case).
+     - Works with latest version of twisted.
+- ``setup.py``: Fixed paths to work with Mac OS X.
+- ``summary_csv`` is no longer enabled by default.
+- ``status`` result processor is now enabled by default.
+- Commands:
+     - ``show``:
+         - Now shows what platform extensions support.
+         - Will no longer try to use a pager if ``PAGER=''`` in the environment.
+     - ``list``:
+         - Added ``"-p"`` option to filter results by supported platforms.
+         - Added ``"--packaged-only"`` option to only list extensions packaged
+           with WA.
+     - ``run``: Added ``"--disable"`` option to diable instruments.
+     - ``create``:
+         - Added ``agenda`` sub-command to generate agendas for a set of
+           extensions.
+         - ``create workload`` now gives more informative errors if Android SDK
+           installed but no platform has been downloaded.
+
+Incompatible changes
+####################
+
+Framework
+~~~~~~~~~
+- ``BaseLinuxDevice``:
+     - Renamed ``active_cpus`` to ``online_cpus``
+     - Renamed ``get_cluster_cpu`` to ``get_cluster_active_cpu``
+     - Renamed ``get_core_cpu`` to ``get_core_online_cpu``
+- All extension's ``initialize`` function now takes one (and only one)
+  parameter, ``context``.
+- ``wlauto.core.device``: Removed ``init`` function. Replaced with
+  ``initialize``
+
+-------------
 Version 2.3.0
 -------------