Merge pull request #325 from marcbonnici/documentation

Documentation and get-assets Update

doc/source/changes.rst

@@ -1,6 +1,228 @@
 =================================
 What's New in Workload Automation
 =================================
+-------------
+Version 2.6.0
+-------------
+
+ .. note:: Users who are currently using the GitHub master version of WA should
+           uninstall the existing version before upgrading to avoid potential issues.
+
+Additions:
+##########
+
+Workloads
+~~~~~~~~~
+- ``AdobeReader``: A workload that carries out following typical productivity
+  tasks. These include opening a file, performing various gestures and
+  zooms on screen and searching for a predefined set of strings.
+- ``octaned8``: A workload to run the binary (non-browser) version of the JS
+  benchmark Octane.
+- ``GooglePlayBooks``: A workload to perform standard productivity tasks with
+  Google Play Books. This workload performs various tasks, such as searching
+  for a book title online, browsing through a book, adding and removing notes,
+  word searching, and querying information about the book.
+- ``GooglePhotos``: A workload to perform standard productivity tasks with
+  Google Photos. Carries out various tasks, such as browsing images,
+  performing zooms, and post-processing the image.
+- ``GoogleSlides``: Carries out various tasks, such as creating a new
+  presentation, adding text, images, and shapes, as well as basic editing and
+  playing a slideshow.
+- ``Youtube``: The workload plays a video, determined by the ``video_source``
+  parameter. While the video is playing, some common actions such as video
+  seeking, pausing playback and navigating the comments section are performed.
+- ``Skype``: Replacement for the ``skypevideo`` workload. Logs into Skype
+  and initiates a voice or video call with a contact.
+
+Framework
+~~~~~~~~~
+- ``AndroidUxPerfWorkload``: Added a new workload class to encapsulate
+  functionality common to all uxperf workloads.
+- ``UxPerfUiAutomation``: Added class which contains methods specific to
+  UX performance
+  testing.
+- ``get-assets``: Added new script and command to retrieve external assets
+  for workloads
+
+Results Processors
+~~~~~~~~~~~~~~~~~~~
+- ``uxperf``: Parses device logcat for `UX_PERF` markers to produce performance
+  metrics for workload actions using specified instrumentation.
+
+Other
+~~~~~
+- ``State Detection``: Added feature to use visual state detection to
+  verify the state of a workload after setup and run.
+
+
+Fixes/Improvements:
+###################
+
+Documentation
+~~~~~~~~~~~~~~
+- ``Revent``: Added file structure to the documentation.
+- Clarified documentation regarding binary dependencies.
+- Updated documentation with ``create`` and ``get-assets`` commands.
+
+Instruments
+~~~~~~~~~~~~
+- ``sysfs_extractor``: Fixed error when `tar.gz` file already existed on device,
+  now overwrites.
+- ``cpufreq``: Fixed error when `tar.gz` file already existed on device, now
+  overwrites.
+- ``file-poller``:
+    - Improved csv output.
+    - Added error checking and reporting.
+    - Changed ``files`` to be a mandatory parameter.
+- ``fps``:
+    - Added a new parameter to fps instrument to specify the time period between
+      calls to ``dumpsys SurfaceFlinger --latency`` when collecting frame data.
+    - Added gfxinfo methods to obtain fps stats. Auto detects and uses appropriate
+      method via android version of device.
+    - Fixed issue with regex.
+    - Now handles empty frames correctly.
+- ``energy_model``: Ensures that the ``ui`` runtime parameter is only set for
+  ChromeOS devices.
+- ``ftrace``: Added support to handle traces collected by both WA and devlib.
+- ``Perf``: Updated 32bit binary file for little endian devices.
+
+Resource Getters
+~~~~~~~~~~~~~~~~
+- ``http_getter``: Now used to try and find executables files from a
+  provided ``remove_assets_url``.
+
+Result Processors
+~~~~~~~~~~~~~~~~~
+- ``cpu_states``: Fixes using stand-alone script with timeline option.
+
+Workloads
+~~~~~~~~~
+- ``antutu``: Fixed setting permissions of ``FINE_LOCATION`` on some devices.
+- ``bbench`` Fixed handling of missing results.
+- ``camerarecord``:
+    - Added frame stats collection through dumpsys gfxinfo.
+    - Added possibility to select slow_motion recording mode.
+- ``Geekbench``:
+    - Fixed output file listing causing pull failure.
+    - Added support for Geekbench 4.
+- ``recentfling``:
+    - Fixed issue when binaries were not uninstalled correctly.
+    - Scripts are now deployed via ``install()`` to ensure they are executable.
+    - Fixed handling of when a PID file is deleted before reaching processing
+      results stage.
+    - Added parameter to not start any apps before flinging.
+- ``rt-app``: Added camera recorder simulation.
+- ``sysbench``: Added arm64 binary.
+- ``Vellamo``: Fixed capitalization in part of UIAutomation to prevent
+  potential issues.
+- ``Spec2000``: Now uses WA deployed version of busybox.
+- ``NetStat``: Updated to support new default logcat format in Android 6.
+- ``Dex2oat``: Now uses root if available.
+
+Framework
+~~~~~~~~~
+- ``adb_shell``:
+    - Fixed issue when using single quoted command with ``adb_shell``.
+    - Correctly forward stderror to the caller for newer version of adb.
+- ``revent``
+    - Added ``-S`` argument to "record" command to automatically record a
+      screen capture after a recording is completed.
+    - Fixed issue with multiple iterations of a revent workload.
+    - Added ``-s`` option to executable to allow waiting on stdin.
+    - Removed timeout in command as ``-s`` is specified.
+    - Revent recordings can now be parsed and used within WA.
+    - Fixed issue when some recordings wouldn't be retrieved correctly.
+    - Timeout is now based on recording duration.
+    - Added `magic` and file version to revent files. Revent files should now
+      start with ``REVENT`` followed by the file format version.
+    - Added support for gamepad recording. This type of recording contains
+      only the events from a gamepad device (which is automatically
+      identified).
+    - A ``mode`` field has been added to the recording format to help
+      distinguish between the normal and gamepad recording types.
+    - Added ``-g`` option to ``record`` command to expose the gamepad recording
+      mode.
+    - The structure of revent code has undergone a major overhaul to improve
+      maintainability and robustness.
+    - More detailed ``info`` command output.
+    - Updated Makefile to support debug/production builds.
+- ``Android API``: Upgraded Android API level from 17 to 18.
+- ``uiautomator``: The window hierarchy is now dumped to a file when WA fails
+  on android devices.
+- ``AndroidDevice``:
+    - Added support for downgrading when installing an APK.
+    - Added a ``broadcast_media_mounted`` method to force a re-index of the
+      mediaserver cache for a specified directory.
+    - Now correctly handles ``None`` output for ``get_pids_of()`` when there are no
+      running processes with the specified name.
+    - Renamed the capture method from ``capture_view_hierachy`` to
+      ``capture_ui_hierarchy``.
+    - Changed the file extension of the capture file to ``.uix``
+    - Added ``-rf`` to delete_files to be consistent with ``LinuxDevice``.
+- ``LinuxDevice``: Now ensures output from both stdout and etderr is propagated in
+  the event of a DeviceError.
+- ``APKWorkload``:
+    - Now ensure APKs are replaced properly when reinstalling.
+    - Now checks APK version and ABI when installing.
+    - Fixed error on some devices when trying to grant permissions that were
+      already granted.
+    - Fixed some permissions not being granted.
+    - Now allows disabling the main activity launch in setup (required for some
+      apps).
+    - Added parameter to clear data on reset (default behaviour unchanged).
+    - Ignores exception for non-fatal permission grant failure.
+    - Fixed issue of multiple versions of the same workload failing to find their APK.
+    - Added method to ensure a valid apk version is used within a workload.
+    - Updated how APK resolution is performed to maximise likelihood of
+      a workload running.
+    - When ``check_apk`` is ``True`` will prefer host APK and if no suitable APK
+      is found, will use target APK if the correct version is present. When ``False``
+      will prefer target apk if it is a valid version otherwise will fallback to
+      host APK.
+- ``RunConfiguration``: Fixed disabling of instruments in workload specs.
+- ``Devices``:
+    - Added network connectivity check for devices.
+    - Subclasses can now set ``requires_network`` to ``True`` and network
+      connectivity check will be performed during ``setup()``.
+- ``Workloads``:
+    - Added network check methods.
+    - Fixed versions to be backwards compatible.
+    - Updated workload versions to match APK files.
+    - Fixed issues with calling super.
+- ``Assets``: Added script to retrieve external assets for workloads.
+- ``Execution``: Added a ``clean_up`` global config option to delete WA files from
+  devices.
+- ``Runner``: No longer takes a screenshot or dump of UI hierarchy for some errors when
+  unnecessary, e.g. host errors.
+- ``core``: Constraints and allowed values are now checked when set instead of
+  when validating.
+- ``FpsProcessor``:
+    - Added requirement on ``filtered_vsyncs_to_compose`` for ``total_vsync metric``.
+    - Removed misleading comment in class description.
+- ``BaseUiAutomation``: Added new Marker API so workloads generate start and end
+  markers with a string name.
+- ``AndroidUiAutoBenchmark``: Automatically checks for known package versions
+  that don't work well with AndroidUiAutoBenchmark workloads.
+
+Other
+~~~~~
+- Updated setup.py url to be a valid URI.
+- Fixed workload name in big.Little sample agenda.
+
+Incompatible changes
+####################
+
+Framework
+~~~~~~~~~
+- ``check_abi``: Now renamed to ``exact_abi``, is used to ensure that if enabled,
+  only an apk containing no native code or code designed for the devices primary
+  abi is use.
+- ``AndroidDevice``: Renamed ``supported_eabis`` property to ``supported_abis``
+  to be consistent with linux devices.
+
+Workloads
+~~~~~~~~~~
+- ``skypevideo``: Workload removed and replaced with ``skype`` workload.
 
 -------------
 Version 2.5.0

doc/source/invocation.rst

@@ -1,4 +1,5 @@
 .. _invocation:
+.. highlight:: none
 
 ========
 Commands
@@ -15,7 +16,7 @@ Individual sub-commands are discussed in detail below.
 run
 ---
 
-The most common sub-command you will use is ``run``. This will run specfied
+The most common sub-command you will use is ``run``. This will run specified
 workload(s) and process resulting output. This takes a single mandatory
 argument that specifies what you want WA to run. This could be either a
 workload name, or a path  to an "agenda" file that allows to specify multiple
@@ -24,7 +25,7 @@ section for details). Executing ::
 
         wa run -h
 
-Will display help for this subcommand that will look somehtign like this::
+Will display help for this subcommand that will look something like this::
 
         usage: run [-d DIR] [-f] AGENDA
 
@@ -81,6 +82,70 @@ agenda file used to run the workloads along with any other device-specific
 configuration files used during execution.
 
 
+create
+------
+
+This can be used to create various WA-related objects, currently workloads, packages and agendas.
+The full set of options for this command are::
+
+    usage: wa create [-h] [-c CONFIG] [-v] [--debug] [--version]
+                     {workload,package,agenda} ...
+
+    positional arguments:
+      {workload,package,agenda}
+        workload            Create a new workload. By default, a basic workload
+                            template will be used but you can use options to
+                            specify a different template.
+        package             Create a new empty Python package for WA extensions.
+                            On installation, this package will "advertise" itself
+                            to WA so that Extensions with in it will be loaded by
+                            WA when it runs.
+        agenda              Create an agenda whit the specified extensions
+                            enabled. And parameters set to their default values.
+
+    optional arguments:
+      -h, --help            show this help message and exit
+      -c CONFIG, --config CONFIG
+                            specify an additional config.py
+      -v, --verbose         The scripts will produce verbose output.
+      --debug               Enable debug mode. Note: this implies --verbose.
+      --version             show program's version number and exit
+
+Use "wa create <object> -h" to see all the object-specific arguments. For example::
+
+        wa create agenda -h
+
+will display the relevant options that can be used to create an agenda. 
+
+get-assets
+----------
+
+This command can download external extension dependencies used by Workload Automation.
+It can be used to download assets for all available extensions or those specificity listed. 
+The full set of options for this command are::
+
+    usage: wa get-assets [-h] [-c CONFIG] [-v] [--debug] [--version] [-f]
+                         [--url URL] (-a | -e EXT [EXT ...])
+
+    optional arguments:
+      -h, --help            show this help message and exit
+      -c CONFIG, --config CONFIG
+                            specify an additional config.py
+      -v, --verbose         The scripts will produce verbose output.
+      --debug               Enable debug mode. Note: this implies --verbose.
+      --version             show program's version number and exit
+      -f, --force           Always fetch the assets, even if matching versions
+                            exist in local cache.
+      --url URL             The location from which to download the files. If not
+                            provided, config setting ``remote_assets_url`` will be
+                            used if available, else uses the default
+                            REMOTE_ASSETS_URL parameter in the script.
+      -a, --all             Download assets for all extensions found in the index.
+                            Cannot be used with -e.
+      -e EXT [EXT ...]      One or more extensions whose assets to download.
+                            Cannot be used with --all.
+
+
 list
 ----
 

wlauto/commands/get_assets.py

@@ -22,7 +22,7 @@ from wlauto import File, ExtensionLoader, Command, settings
 from wlauto.core.extension import Extension
 
 
-REMOTE_ASSETS_URL = 'https://github.com/ARM-software/wa-assets/raw/master/dependencies'
+REMOTE_ASSETS_URL = 'https://github.com/ARM-software/workload-automation-assets/raw/master/dependencies'
 
 
 class GetAssetsCommand(Command):

wlauto/instrumentation/freqsweep/__init__.py

@@ -55,6 +55,7 @@ class FreqSweep(Instrument):
                    - label: Workload specs will be named
                      `{spec id}_{label}_{frequency}`. If a label is not
                      provided it will be named `sweep{sweep No.}`
+
                   Example sweep specification: ::
 
                       freq_sweep:

wlauto/workloads/adobereader/__init__.py

@@ -33,6 +33,7 @@ class AdobeReader(AndroidUxPerfWorkload):
     The Adobe Reader workflow carries out the following typical productivity tasks.
 
     Test description:
+
     1. Open a local file on the device
     2. Gestures test:
         2.1. Swipe down across the central 50% of the screen in 200 x 5ms steps

wlauto/workloads/googlephotos/__init__.py

@@ -37,6 +37,7 @@ class Googlephotos(AndroidUxPerfWorkload):
     various tasks, such as browsing images, performing zooms, and post-processing the image.
 
     Test description:
+
     1. Four images are copied to the device
     2. The application is started in offline access mode
     3. Gestures are performed to pinch zoom in and out of the selected image

wlauto/workloads/googleslides/__init__.py

@@ -48,6 +48,7 @@ class GoogleSlides(AndroidUxPerfWorkload):
     requires an image file specified by the param ``test_image`` to be copied onto the device.
 
     Test description:
+
     1. Start the app and skip the welcome screen. Dismiss the work offline banner if present.
     2. Go to the app settings page and enables PowerPoint compatibility mode. This allows
        PowerPoint files to be created inside Google Slides.
@@ -62,6 +63,7 @@ class GoogleSlides(AndroidUxPerfWorkload):
     file to be copied is given by ``test_file``.
 
     Test description:
+
     1. From the documents list (following the create test), open the specified PowerPoint
        by navigating into device storage and wait for it to be loaded.
     2. A navigation test is performed while the file is in editing mode (i.e. not slideshow).

wlauto/workloads/hwuitest/__init__.py

@@ -40,6 +40,7 @@ class HWUITest(Workload):
                   allowed_values=["shadowgrid", "rectgrid", "oval"],
                   description="""
                   The test to run:
+
                      - ``'shadowgrid'``: creates a grid of rounded rects that
                        cast shadows, high CPU & GPU load
                      - ``'rectgrid'``: creates a grid of 1x1 rects

wlauto/workloads/rt_app/__init__.py

@@ -49,13 +49,13 @@ class RtApp(Workload):
     The load is described using JSON-like config files. Below are a couple of simple
     examples.
 
+
+    Simple use case which creates a thread that run 1ms then sleep 9ms
+    until the use case is stopped with Ctrl+C:
+
     .. code-block:: json
 
         {
-            /*
-            * Simple use case which creates a thread that run 1ms then sleep 9ms
-            * until the use case is stopped with Ctrl+C
-            */
             "tasks" : {
                 "thread0" : {
                     "loop" : -1,
@@ -76,13 +76,13 @@ class RtApp(Workload):
             }
         }
 
+
+    Simple use case with 2 threads that runs for 10 ms and wake up each
+    other until the use case is stopped with Ctrl+C
+
     .. code-block:: json
 
         {
-            /*
-            * Simple use case with 2 threads that runs for 10 ms and wake up each
-            * other until the use case is stopped with Ctrl+C
-            */
             "tasks" : {
                 "thread0" : {
                     "loop" : -1,

wlauto/workloads/youtube/__init__.py

@@ -35,6 +35,7 @@ class Youtube(AndroidUxPerfWorkload):
     Test description:
     The ``video_source`` parameter determines where the video to be played will be found
     in the app. Possible values are ``search``, ``home``, ``my_videos``, and ``trending``.
+
     -A. search - Goes to the search view, does a search for the given term, and plays the
         first video in the results. The parameter ``search_term`` must also be provided
         in the agenda for this to work. This is the default mode.
@@ -46,6 +47,7 @@ class Youtube(AndroidUxPerfWorkload):
         video in the trending videos list.
 
     For the selected video source, the following test steps are performed:
+
     1.  Navigate to the general app settings page to disable autoplay. This improves test
         stability and predictability by preventing screen transition to load a new video
         while in the middle of the test.