diff --git a/doc/source/changes.rst b/doc/source/changes.rst index 9d1dd58d..6bc3fba2 100644 --- a/doc/source/changes.rst +++ b/doc/source/changes.rst @@ -1,6 +1,398 @@ +================================= What's New in Workload Automation ================================= +------------- +Version 2.4.0 +------------- +Additions: +########## + +Devices +~~~~~~~~ +- ``gem5_linux`` and ``gem5_android``: Interfaces for Gem5 simulation + environment running Linux and Android respectively. +- ``XE503C1211``: Interface for Samsung XE503C12 Chromebooks. +- ``chromeos_test_image``: Chrome OS test image device. An off the shelf + device will not work with this device interface. + +Instruments +~~~~~~~~~~~~ +- ``freq_sweep``: Allows "sweeping" workloads across multiple CPU frequencies. +- ``screenon``: Ensures screen is on, before each iteration, or periodically + on Android devices. +- ``energy_model``: This instrument can be used to generate an energy model + for a device based on collected power and performance measurments. +- ``netstats``: Allows monitoring data sent/received by applications on an + Android device. + +Modules +~~~~~~~ +- ``cgroups``: Allows query and manipulation of cgroups controllers on a Linux + device. Currently, only cpusets controller is implemented. +- ``cpuidle``: Implements cpuidle state discovery, query and manipulation for + a Linux device. This replaces the more primitive get_cpuidle_states method + of LinuxDevice. +- ``cpufreq`` has now been split out into a device module + +Reasource Getters +~~~~~~~~~~~~~~~~~ +- ``http_assets``: Downloads resources from a web server. + +Results Processors +~~~~~~~~~~~~~~~~~~~ +- ``ipynb_exporter``: Generates an IPython notebook from a template with the + results and runs it. +- ``notify``: Displays a desktop notification when a run finishes + (Linux only). +- ``cpustates``: Processes power ftrace to produce CPU state and parallelism + stats. There is also a script to invoke this outside of WA. + +Workloads +~~~~~~~~~ +- ``telemetry``: Executes Google's Telemetery benchmarking framework +- ``hackbench``: Hackbench runs tests on the Linux scheduler +- ``ebizzy``: This workload resembles common web server application workloads. +- ``power_loadtest``: Continuously cycles through a set of browser-based + activities and monitors battery drain on a device (part of ChromeOS autotest + suite). +- ``rt-app``: Simulates configurable real-time periodic load. +- ``linpack-cli``: Command line version of linpack benchmark. +- ``lmbench``: A suite of portable ANSI/C microbenchmarks for UNIX/POSIX. +- ``stream``: Measures memory bandwidth. +- ``iozone``: Runs a series of disk I/O performance tests. +- ``androbench``: Measures the storage performance of device. +- ``autotest``: Executes tests from ChromeOS autotest suite. + +Framework +~~~~~~~~~ +- ``wlauto.utils``: + - Added ``trace_cmd``, a generic trace-cmd paraser. + - Added ``UbootMenu``, allows navigating Das U-boot menu over serial. +- ``wlauto.utils.types``: + - ``caseless_string``: Behaves exactly like a string, except this ignores + case in comparisons. It does, however, preserve case. + - ``list_of``: allows dynamic generation of type-safe list types based on + an existing type. + - ``arguments``: represents arguments that are passed on a command line to + an application. + - ``list-or``: allows dynamic generation of types that accept either a base + type or a list of base type. Using this ``list_or_integer``, + ``list_or_number`` and ``list_or_bool`` were also added. +- ``wlauto.core.configuration.WorkloadRunSpec``: + - ``copy``: Allows making duplicates of ``WorkloadRunSpec``'s +- ``wlatuo.utils.misc``: + - ``list_to_ranges`` and ``ranges_to_list``: convert between lists of + integers and corresponding range strings, e.g. between [0,1,2,4] and + '0-2,4' + - ``list_to_mask`` and ``mask_to_list``: convert between lists of integers + and corresponding integer masks, e.g. between [0,1,2,4] and 0x17 +- ``wlauto.instrumentation``: + - ``instrument_is_enabled``: Returns whether or not an instrument is + enabled for the current job. +- ``wlauto.core.result``: + - Added "classifiers" field to Metric objects. This is a dict mapping + classifier names (arbitrary strings) to corresponding values for that + specific metrics. This is to allow extensions to add extension-specific + annotations to metric that could be handled in a generic way (e.g. by + result processors). They can also be set in agendas. +- Failed jobs will now be automatically retired +- Implemented dynamic device modules that may be loaded automatically on + device initialization if the device supports them. +- Added support for YAML configs. +- Added ``initialze`` and ``finalize`` methods to workloads. +- ``wlauto.core.ExecutionContext``: + - Added ``job_status`` property that returns the status of the currently + running job. + +Fixes/Improvements +################## + +Devices +~~~~~~~~ +- ``tc2``: Workaround for buffer overrun when loading large initrd blob. +- ``juno``: + - UEFI config can now be specified as a parameter. + - Adding support for U-Boot booting. + - No longer auto-disconnects ADB at the end of a run. + - Added ``actually_disconnect`` to restore old disconnect behaviour + - Now passes ``video`` command line to Juno kernel to work around a known + issue where HDMI loses sync with monitors. + - Fixed flashing. + +Instruments +~~~~~~~~~~~ +- ``trace_cmd``: + - Fixed ``buffer_size_file`` for non-Android devices + - Reduce starting priority. + - Now handles trace headers and thread names with spaces +- ``energy_probe``: Added ``device_entry`` parameter. +- ``hwmon``: + - Sensor discovery is now done only at the start of a run. + - Now prints both before/after and mean temperatures. +- ``daq``: + - Now reports energy + - Fixed file descriptor leak + - ``daq_power.csv`` now matches the order of labels (if specified). + - Added ``gpio_sync``. When enabled, this wil cause the instrument to + insert a marker into ftrace, while at the same time setting a GPIO pin + high. + - Added ``negative_values`` parameter. which can be used to specify how + negative values in the samples should be handled. + - Added ``merge_channels`` parameter. When set DAQ channel will be summed + together. + - Workload labels, rather than names, are now used in the "workload" + column. +- ``cpufreq``: + - Fixes missing directories problem. + - Refined the availability check not to rely on the top-level cpu/cpufreq + directory + - Now handles non-integer output in ``get_available_frequencies``. +- ``sysfs_extractor``: + - No longer raises an error when both device and host paths are empty. + - Fixed pulled files verification. +- ``perf``: + - Updated binaries. + - Added option to force install. + - ``killall`` is now run as root on rooted Android devices. +- ``fps``: + - now generates detailed FPS traces as well as report average FPS. + - Updated jank calcluation to only count "large" janks. + - Now filters out bogus ``actual-present`` times and ignore janks above + ``PAUSE_LATENCY`` +- ``delay``: + - Added ``fixed_before_start`` parameter. + - Changed existing ``*_between_specs`` and ``*_between_iterations`` + callbacks to be ``very_slow`` +- ``streamline``: + - Added Linux support + - ``gatord`` is now only started once at the start of the run. + +modules +~~~~~~~ +- ``flashing``: + - Fixed vexpress flashing + - Added an option to keep UEFI entry + +Result Processors +~~~~~~~~~~~~~~~~~ +- ``cpustate``: + - Now generates a timeline csv as well as stats. + - Adding ID to overall cpustate reports. +- ``csv``: (partial) ``results.csv`` will now be written after each iteration + rather than at the end of the run. + +Workloads +~~~~~~~~~ +- ``glb_corporate``: clears logcat to prevent getting results from previous + run. +- ``sysbench``: + - Updated sysbench binary to a statically linked verison + - Added ``file_test_mode parameter`` - this is a mandatory argumet if + ``test`` is ``"fileio"``. + - Added ``cmd_params`` parameter to pass options directily to sysbench + invocation. + - Removed Android browser launch and shutdown from workload (now runs on + both Linux and Android). + - Now works with unrooted devices. + - Added the ability to run based on time. + - Added a parameter to taskset to specific core(s). + - Added ``threads`` parameter to be consistent with dhrystone. + - Fixed case where default ``timeout`` < ``max_time``. +- ``Dhrystone``: + - added ``taskset_mask`` parameter to allow pinning to specific cores. + - Now kills any running instances during setup (also handles CTRL-C). +- ``sysfs_extractor``: Added parameter to explicitly enable/disable tempfs + caching. +- ``antutu``: + - Fixed multi-``times`` playback for v5. + - Updated result parsing to handle Android M logcat output. +- ``geekbench``: Increased timout to cater for slower devices. +- ``idle``: Now works on Linux devices. +- ``manhattan``: Added ``run_timemout`` parameter. +- ``bbench``: Now works when binaries_directory is not in path. +- ``nemamark``: Made duration configurable. + +Framework +~~~~~~~~~~ +- ``BaseLinuxDevice``: + - Now checks that at least one core is enabled on another cluster before + attempting to set number of cores on a cluster to ``0``. + - No longer uses ``sudo`` if already logged in as ``root``. + - Now saves ``dumpsys window`` output to the ``__meta`` directory. + - Now takes ``password_prompt`` as a parameter for devices with a non + standard ``sudo`` password prompt. + - No longer raises an error if ``keyfile`` or ``password`` are not + provided when they are not necessary. + - Added new cpufreq APIs: + - ``core`` APIs take a core name as the parameter (e.g. "a15") + - ``cluster`` APIs take a numeric cluster ID (eg. 0) + - ``cpu`` APIs take a cpufreq cpu ID as a parameter. + - ``set_cpu_frequency`` now has a ``exact`` parameter. When true (the + default) it will produce an error when the specified frequency is not + supported by the cpu, otherwise cpufreq will decide what to do. + - Added ``{core}_frequency`` runtime parameter to set cluster frequency. + - Added ``abi`` property. + - ``get_properties`` moved from ``LinuxDevice``, meaning ``AndroidDevice`` + will try to pull the same files. Added more paths to pull by default + too. + - fixed ``list_file_systems`` for Android M and Linux devices. + - Now sets ``core_clusters`` from ``core_names`` if not explicitly + specified. + - Added ``invoke`` method that allows invoking an executable on the device + under controlled contions (e.g. within a particular directory, or + taskset to specific CPUs). + - No longer attempts to ``get_sysfile_value()`` as root on unrooted + devices. +- ``LinuxDevice``: + - Now creates ``binaries_directory`` path if it doesn't exist. + - Fixed device reset + - Fixed ``file_exists`` + - implemented ``get_pid_of()`` and ``ps()``. Existing implementation + relied on Android version of ps. + - ``listdir`` will now return an empty list for an empty directory + instead of a list containing a single empty string. +- ``AndroidDevice``: + - Executable (un)installation now works on unrooted devices. + - Now takes into account ``binar_directory`` when setting up busybox path. + - update ``android_prompt`` so that it works even if is not ``"/"`` + - ``adb_connect``: do not assume port 5555 anymore. + - Now always deploys busybox on rooted devices. + - Added ``swipe_to_unlock`` method. +- Fixed initialization of ``~/.workload_automation.``. +- Fixed replaying events using revent on 64 bit platforms. +- Improved error repoting when loading extensions. +- ``result`` objects now track their output directories. +- ``context.result`` will not result in ``context.run_result`` when not + executing a job. +- ``wlauto.utils.ssh``: + - Fixed key-based authentication. + - Fixed carriage return stripping in ssh. + - Now takes ``password_prompt`` as a parameter for non standard ``sudo`` + password prompts. + - Now with 100% more thread safety! + - If a timeout condition is hit, ^C is now sent to kill the current + foreground process and make the shell available for subsequent commands. + - More robust ``exit_code`` handling for ssh interface + - Now attempts to deal with dropped connections + - Fixed error reporting on failed exit code extraction. + - Now handles backspaces in serial output + - Added ``port`` argument for telnet connections. + - Now allows telnet connections without a password. +- Fixed config processing for extensions with non-identifier names. +- Fixed ``get_meansd`` for numbers < 1 +- ``wlatuo.utils.ipython``: + - Now supports old versions of IPython + - Updated version check to only initialize ipython utils if version is + < 4.0.0. Version 4.0.0 changes API and breaks WA's usage of it. +- Added ``ignore`` parameter to ``check_output`` +- Agendas: + - Now raise an error if an agenda contains duplicate keys + - Now raise an error if config section in an agenda is not dict-like + - Now properly handles ``core_names`` and ``core_clusters`` + - When merging list parameters from different sources, duplicates are no + longer removed. +- The ``INITIAL_BOOT`` signal is now sent went performing a hard reset during + intial boot +- updated ``ExecutionContext`` to keep a reference to the ``runner``. This + will enable Extenstions to do things like modify the job queue. +- Parameter now automatically convert int and boot kinds to integer and + boolean respectively, this behavior can be supressed by specifying + ``convert_types``=``False`` when defining the parameter. +- Fixed resource resolution when dependency location does not exist. +- All device ``push`` and ``pull`` commands now raise ``DeviceError`` if they + didn't succeed. +- Fixed showing Parameter default of ``False`` for boolean values. +- Updated csv result processor with the option to use classifiers to + add columns to ``results.csv``. +- ``wlauto.utils.formatter``: Fix terminal size discovery. +- The extension loader will now follow symlinks. +- Added arm64-v8a to ABI map +- WA now reports syntax errors in a more informative way. +- Resource resolver: now prints the path of the found resource to the log. +- Resource getter: look for executable in the bin/ directory under resource + owner's dependencies directory as well as general dependencies bin. +- ``GamingWorkload``: + - Added an option to prevent clearing of package data before execution. + - Added the ability to override the timeout of deploying the assets + tarball. +- ``ApkWorkload``: Added an option to skip host-side APK check entirely. +- ``utils.misc.normalize``: only normalize string keys. +- Better error reporting for subprocess.CalledProcessError +- ``boolean`` now interprets ``'off'`` as ``False`` +- ``wlauto.utils.uefi``: Added support for debug builds. +- ``wlauto.utils.serial_port``: Now supports fdexpect versions > 4.0.0 +- Semanatics for ``initialize``/``finalize`` for *all* Extensions are changed + so that now they will always run at most once per run. They will not be + executed twice even if invoked via instances of different subclasses (if + those subclasses defined their own verions, then their versions will be + invoked once each, but the base version will only get invoked once). +- Pulling entries from procfs does not work on some platforms. WA now tries + to cat the contents of a property_file and write it to a output file on the + host. + +Documentation +~~~~~~~~~~~~~ +- ``installation``: + - Added ``post install`` section which lists workloads that require + additional external dependencies. + - Added the ``uninstall`` and ``upgrade`` commands for users to remove or + upgrade Workload Automation. + - Added documentation explaining how to use ``remote_assets_path`` + setting. + - Added warning about potential permission issues with pip. +- ``quickstart``: Added steps for setting up WA to run on Linux devices. +- ``device_setup``: fixed ``generic_linux`` ``device_config`` example. +- ``contributing``: Clarified style guidelines +- ``daq_device_setup``: Added an illustration for DAQ wiring. +- ``writing_extensions``: Documented the Workload initialize and finalize + methods. +- Added descriptions to extension that didn't have one. + +Other +~~~~~ +- ``daq_server``: + - Fixed showing available devices. + - Now works with earlier versions of the DAQmx driver.thus you can now run + the server on Linux systems. + - DAQ error messages are now properly propaged to the client. + - Server will now periodically clean up uncollected files. + - fixed not being able to resolve IP address for hostname + (report "localhost" in that case). + - Works with latest version of twisted. +- ``setup.py``: Fixed paths to work with Mac OS X. +- ``summary_csv`` is no longer enabled by default. +- ``status`` result processor is now enabled by default. +- Commands: + - ``show``: + - Now shows what platform extensions support. + - Will no longer try to use a pager if ``PAGER=''`` in the environment. + - ``list``: + - Added ``"-p"`` option to filter results by supported platforms. + - Added ``"--packaged-only"`` option to only list extensions packaged + with WA. + - ``run``: Added ``"--disable"`` option to diable instruments. + - ``create``: + - Added ``agenda`` sub-command to generate agendas for a set of + extensions. + - ``create workload`` now gives more informative errors if Android SDK + installed but no platform has been downloaded. + +Incompatible changes +#################### + +Framework +~~~~~~~~~ +- ``BaseLinuxDevice``: + - Renamed ``active_cpus`` to ``online_cpus`` + - Renamed ``get_cluster_cpu`` to ``get_cluster_active_cpu`` + - Renamed ``get_core_cpu`` to ``get_core_online_cpu`` +- All extension's ``initialize`` function now takes one (and only one) + parameter, ``context``. +- ``wlauto.core.device``: Removed ``init`` function. Replaced with + ``initialize`` + +------------- Version 2.3.0 -------------