sascha/workload-automation
1
0
Fork 0
mirror of https://github.com/ARM-software/workload-automation.git synced 2025-01-18 20:11:20 +00:00
Code Issues Releases Wiki Activity

Merge pull request #325 from marcbonnici/documentation

Browse Source 
Documentation and get-assets Update
This commit is contained in:
marcbonnici 2016-12-23 14:13:30 +00:00 committed by GitHub
commit d5c888cc90
13 changed files with 326 additions and 31 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

222
doc/source/changes.rst
View File

@ -1,6 +1,228 @@
=================================
What's New in Workload Automation
=================================
-------------
Version 2.6.0
-------------
.. note:: Users who are currently using the GitHub master version of WA should
uninstall the existing version before upgrading to avoid potential issues.
Additions:
##########
Workloads
~~~~~~~~~
- ``AdobeReader``: A workload that carries out following typical productivity
tasks. These include opening a file, performing various gestures and
zooms on screen and searching for a predefined set of strings.
- ``octaned8``: A workload to run the binary (non-browser) version of the JS
benchmark Octane.
- ``GooglePlayBooks``: A workload to perform standard productivity tasks with
Google Play Books. This workload performs various tasks, such as searching
for a book title online, browsing through a book, adding and removing notes,
word searching, and querying information about the book.
- ``GooglePhotos``: A workload to perform standard productivity tasks with
Google Photos. Carries out various tasks, such as browsing images,
performing zooms, and post-processing the image.
- ``GoogleSlides``: Carries out various tasks, such as creating a new
presentation, adding text, images, and shapes, as well as basic editing and
playing a slideshow.
- ``Youtube``: The workload plays a video, determined by the ``video_source``
parameter. While the video is playing, some common actions such as video
seeking, pausing playback and navigating the comments section are performed.
- ``Skype``: Replacement for the ``skypevideo`` workload. Logs into Skype
and initiates a voice or video call with a contact.
Framework
~~~~~~~~~
- ``AndroidUxPerfWorkload``: Added a new workload class to encapsulate
functionality common to all uxperf workloads.
- ``UxPerfUiAutomation``: Added class which contains methods specific to
UX performance
testing.
- ``get-assets``: Added new script and command to retrieve external assets
for workloads
Results Processors
~~~~~~~~~~~~~~~~~~~
- ``uxperf``: Parses device logcat for `UX_PERF` markers to produce performance
metrics for workload actions using specified instrumentation.
Other
~~~~~
- ``State Detection``: Added feature to use visual state detection to
verify the state of a workload after setup and run.
Fixes/Improvements:
###################
Documentation
~~~~~~~~~~~~~~
- ``Revent``: Added file structure to the documentation.
- Clarified documentation regarding binary dependencies.
- Updated documentation with ``create`` and ``get-assets`` commands.
Instruments
~~~~~~~~~~~~
- ``sysfs_extractor``: Fixed error when `tar.gz` file already existed on device,
now overwrites.
- ``cpufreq``: Fixed error when `tar.gz` file already existed on device, now
overwrites.
- ``file-poller``:
- Improved csv output.
- Added error checking and reporting.
- Changed ``files`` to be a mandatory parameter.
- ``fps``:
- Added a new parameter to fps instrument to specify the time period between
calls to ``dumpsys SurfaceFlinger --latency`` when collecting frame data.
- Added gfxinfo methods to obtain fps stats. Auto detects and uses appropriate
method via android version of device.
- Fixed issue with regex.
- Now handles empty frames correctly.
- ``energy_model``: Ensures that the ``ui`` runtime parameter is only set for
ChromeOS devices.
- ``ftrace``: Added support to handle traces collected by both WA and devlib.
- ``Perf``: Updated 32bit binary file for little endian devices.
Resource Getters
~~~~~~~~~~~~~~~~
- ``http_getter``: Now used to try and find executables files from a
provided ``remove_assets_url``.
Result Processors
~~~~~~~~~~~~~~~~~
- ``cpu_states``: Fixes using stand-alone script with timeline option.
Workloads
~~~~~~~~~
- ``antutu``: Fixed setting permissions of ``FINE_LOCATION`` on some devices.
- ``bbench`` Fixed handling of missing results.
- ``camerarecord``:
- Added frame stats collection through dumpsys gfxinfo.
- Added possibility to select slow_motion recording mode.
- ``Geekbench``:
- Fixed output file listing causing pull failure.
- Added support for Geekbench 4.
- ``recentfling``:
- Fixed issue when binaries were not uninstalled correctly.
- Scripts are now deployed via ``install()`` to ensure they are executable.
- Fixed handling of when a PID file is deleted before reaching processing
results stage.
- Added parameter to not start any apps before flinging.
- ``rt-app``: Added camera recorder simulation.
- ``sysbench``: Added arm64 binary.
- ``Vellamo``: Fixed capitalization in part of UIAutomation to prevent
potential issues.
- ``Spec2000``: Now uses WA deployed version of busybox.
- ``NetStat``: Updated to support new default logcat format in Android 6.
- ``Dex2oat``: Now uses root if available.
Framework
~~~~~~~~~
- ``adb_shell``:
- Fixed issue when using single quoted command with ``adb_shell``.
- Correctly forward stderror to the caller for newer version of adb.
- ``revent``
- Added ``-S`` argument to "record" command to automatically record a
screen capture after a recording is completed.
- Fixed issue with multiple iterations of a revent workload.
- Added ``-s`` option to executable to allow waiting on stdin.
- Removed timeout in command as ``-s`` is specified.
- Revent recordings can now be parsed and used within WA.
- Fixed issue when some recordings wouldn't be retrieved correctly.
- Timeout is now based on recording duration.
- Added `magic` and file version to revent files. Revent files should now
start with ``REVENT`` followed by the file format version.
- Added support for gamepad recording. This type of recording contains
only the events from a gamepad device (which is automatically
identified).
- A ``mode`` field has been added to the recording format to help
distinguish between the normal and gamepad recording types.
- Added ``-g`` option to ``record`` command to expose the gamepad recording
mode.
- The structure of revent code has undergone a major overhaul to improve
maintainability and robustness.
- More detailed ``info`` command output.
- Updated Makefile to support debug/production builds.
- ``Android API``: Upgraded Android API level from 17 to 18.
- ``uiautomator``: The window hierarchy is now dumped to a file when WA fails
on android devices.
- ``AndroidDevice``:
- Added support for downgrading when installing an APK.
- Added a ``broadcast_media_mounted`` method to force a re-index of the
mediaserver cache for a specified directory.
- Now correctly handles ``None`` output for ``get_pids_of()`` when there are no
running processes with the specified name.
- Renamed the capture method from ``capture_view_hierachy`` to
``capture_ui_hierarchy``.
- Changed the file extension of the capture file to ``.uix``
- Added ``-rf`` to delete_files to be consistent with ``LinuxDevice``.
- ``LinuxDevice``: Now ensures output from both stdout and etderr is propagated in
the event of a DeviceError.
- ``APKWorkload``:
- Now ensure APKs are replaced properly when reinstalling.
- Now checks APK version and ABI when installing.
- Fixed error on some devices when trying to grant permissions that were
already granted.
- Fixed some permissions not being granted.
- Now allows disabling the main activity launch in setup (required for some
apps).
- Added parameter to clear data on reset (default behaviour unchanged).
- Ignores exception for non-fatal permission grant failure.
- Fixed issue of multiple versions of the same workload failing to find their APK.
- Added method to ensure a valid apk version is used within a workload.
- Updated how APK resolution is performed to maximise likelihood of
a workload running.
- When ``check_apk`` is ``True`` will prefer host APK and if no suitable APK
is found, will use target APK if the correct version is present. When ``False``
will prefer target apk if it is a valid version otherwise will fallback to
host APK.
- ``RunConfiguration``: Fixed disabling of instruments in workload specs.
- ``Devices``:
- Added network connectivity check for devices.
- Subclasses can now set ``requires_network`` to ``True`` and network
connectivity check will be performed during ``setup()``.
- ``Workloads``:
- Added network check methods.
- Fixed versions to be backwards compatible.
- Updated workload versions to match APK files.
- Fixed issues with calling super.
- ``Assets``: Added script to retrieve external assets for workloads.
- ``Execution``: Added a ``clean_up`` global config option to delete WA files from
devices.
- ``Runner``: No longer takes a screenshot or dump of UI hierarchy for some errors when
unnecessary, e.g. host errors.
- ``core``: Constraints and allowed values are now checked when set instead of
when validating.
- ``FpsProcessor``:
- Added requirement on ``filtered_vsyncs_to_compose`` for ``total_vsync metric``.
- Removed misleading comment in class description.
- ``BaseUiAutomation``: Added new Marker API so workloads generate start and end
markers with a string name.
- ``AndroidUiAutoBenchmark``: Automatically checks for known package versions
that don't work well with AndroidUiAutoBenchmark workloads.
Other
~~~~~
- Updated setup.py url to be a valid URI.
- Fixed workload name in big.Little sample agenda.
Incompatible changes
####################
Framework
~~~~~~~~~
- ``check_abi``: Now renamed to ``exact_abi``, is used to ensure that if enabled,
only an apk containing no native code or code designed for the devices primary
abi is use.
- ``AndroidDevice``: Renamed ``supported_eabis`` property to ``supported_abis``
to be consistent with linux devices.
Workloads
~~~~~~~~~~
- ``skypevideo``: Workload removed and replaced with ``skype`` workload.
-------------
Version 2.5.0

81
doc/source/invocation.rst
View File

@ -1,4 +1,5 @@
.. _invocation:
.. highlight:: none
========
Commands
@ -15,7 +16,7 @@ Individual sub-commands are discussed in detail below.
run
---
The most common sub-command you will use is ``run``. This will run specfied
The most common sub-command you will use is ``run``. This will run specified
workload(s) and process resulting output. This takes a single mandatory
argument that specifies what you want WA to run. This could be either a
workload name, or a path to an "agenda" file that allows to specify multiple
@ -24,7 +25,7 @@ section for details). Executing ::
wa run -h
Will display help for this subcommand that will look somehtign like this::
Will display help for this subcommand that will look something like this::
usage: run [-d DIR] [-f] AGENDA
@ -47,13 +48,13 @@ Will display help for this subcommand that will look somehtign like this::
--debug Enable debug mode. Note: this implies --verbose.
-d DIR, --output-directory DIR
Specify a directory where the output will be
generated. If the directoryalready exists, the script
generated. If the directory already exists, the script
will abort unless -f option (see below) is used,in
which case the contents of the directory will be
overwritten. If this optionis not specified, then
overwritten. If this option is not specified, then
wa_output will be used instead.
-f, --force Overwrite output directory if it exists. By default,
the script will abort in thissituation to prevent
the script will abort in this situation to prevent
accidental data loss.
-i ID, --id ID Specify a workload spec ID from an agenda to run. If
this is specified, only that particular spec will be
@ -81,10 +82,74 @@ agenda file used to run the workloads along with any other device-specific
configuration files used during execution.
create
------
This can be used to create various WA-related objects, currently workloads, packages and agendas.
The full set of options for this command are::
usage: wa create [-h] [-c CONFIG] [-v] [--debug] [--version]
{workload,package,agenda} ...
positional arguments:
{workload,package,agenda}
workload Create a new workload. By default, a basic workload
template will be used but you can use options to
specify a different template.
package Create a new empty Python package for WA extensions.
On installation, this package will "advertise" itself
to WA so that Extensions with in it will be loaded by
WA when it runs.
agenda Create an agenda whit the specified extensions
enabled. And parameters set to their default values.
optional arguments:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
specify an additional config.py
-v, --verbose The scripts will produce verbose output.
--debug Enable debug mode. Note: this implies --verbose.
--version show program's version number and exit
Use "wa create <object> -h" to see all the object-specific arguments. For example::
wa create agenda -h
will display the relevant options that can be used to create an agenda.
get-assets
----------
This command can download external extension dependencies used by Workload Automation.
It can be used to download assets for all available extensions or those specificity listed.
The full set of options for this command are::
usage: wa get-assets [-h] [-c CONFIG] [-v] [--debug] [--version] [-f]
[--url URL] (-a | -e EXT [EXT ...])
optional arguments:
-h, --help show this help message and exit
-c CONFIG, --config CONFIG
specify an additional config.py
-v, --verbose The scripts will produce verbose output.
--debug Enable debug mode. Note: this implies --verbose.
--version show program's version number and exit
-f, --force Always fetch the assets, even if matching versions
exist in local cache.
--url URL The location from which to download the files. If not
provided, config setting ``remote_assets_url`` will be
used if available, else uses the default
REMOTE_ASSETS_URL parameter in the script.
-a, --all Download assets for all extensions found in the index.
Cannot be used with -e.
-e EXT [EXT ...] One or more extensions whose assets to download.
Cannot be used with --all.
list
----
This lists all extensions of a particular type. For example ::
This lists all extensions of a particular type. For example::
wa list workloads
@ -97,11 +162,11 @@ show
This will show detailed information about an extension, including more in-depth
description and any parameters/configuration that are available. For example
executing ::
executing::
wa show andebench
will produce something like ::
will produce something like::
andebench

8
doc/source/writing_extensions.rst
View File

@ -161,8 +161,8 @@ In order for the binary to be obtained in this way, it must be stored in one of
the locations scanned by the resource resolver in a directry structure
``<root>/bin/<abi>/<binary>`` (where ``root`` is the base resource location to
be searched, e.g. ``~/.workload_automation/depencencies/<extension name>``, and
``<abi>`` is the ABI for which the exectuable has been compiled, as returned by
``self.device.abi``).
``<abi>`` is the ABI for which the exectuable has been compiled, as returned by
``self.device.abi``).
Once the path to the host-side binary has been obtained, it may be deployed using
one of two methods of a ``Device`` instace -- ``install`` or ``install_if_needed``.
@ -182,8 +182,8 @@ WA and will not try to re-install.
Both of the above methods will return the path to the installed binary on the
device. The executable should be invoked *only* via that path; do **not** assume
that it will be in ``PATH`` on the target (or that the executable with the same
name in ``PATH`` is the version deployed by WA.
that it will be in ``PATH`` on the target (or that the executable with the same
name in ``PATH`` is the version deployed by WA.
.. code:: python

2
wlauto/commands/get-assets.py → wlauto/commands/get_assets.py
View File

@ -22,7 +22,7 @@ from wlauto import File, ExtensionLoader, Command, settings
from wlauto.core.extension import Extension
REMOTE_ASSETS_URL = 'https://github.com/ARM-software/wa-assets/raw/master/dependencies'
REMOTE_ASSETS_URL = 'https://github.com/ARM-software/workload-automation-assets/raw/master/dependencies'
class GetAssetsCommand(Command):

1
wlauto/instrumentation/freqsweep/__init__.py
View File

@ -55,6 +55,7 @@ class FreqSweep(Instrument):
- label: Workload specs will be named
`{spec id}_{label}_{frequency}`. If a label is not
provided it will be named `sweep{sweep No.}`
Example sweep specification: ::
freq_sweep:

1
wlauto/workloads/adobereader/__init__.py
View File

@ -33,6 +33,7 @@ class AdobeReader(AndroidUxPerfWorkload):
The Adobe Reader workflow carries out the following typical productivity tasks.
Test description:
1. Open a local file on the device
2. Gestures test:
2.1. Swipe down across the central 50% of the screen in 200 x 5ms steps

2
wlauto/workloads/geekbench/__init__.py
View File

@ -94,7 +94,7 @@ class Geekbench(AndroidUiAutoBenchmark):
description=('If ``True`` the results file will not be pulled from the devices '
'/data/data/com.primatelabs.geekbench folder. This allows the '
'workload to be run on unrooted devices and the results extracted '
'manually later. ' )),
'manually later.')),
]
@property

1
wlauto/workloads/googlephotos/__init__.py
View File

@ -37,6 +37,7 @@ class Googlephotos(AndroidUxPerfWorkload):
various tasks, such as browsing images, performing zooms, and post-processing the image.
Test description:
1. Four images are copied to the device
2. The application is started in offline access mode
3. Gestures are performed to pinch zoom in and out of the selected image

18
wlauto/workloads/googleplaybooks/__init__.py
View File

@ -33,15 +33,15 @@ class Googleplaybooks(AndroidUxPerfWorkload):
and querying information about the book.
Test description:
1. Open Google Play Books application
2. Dismisses sync operation (if applicable)
3. Searches for a book title
4. Adds books to library if not already present
5. Opens 'My Library' contents
6. Opens selected book
7. Gestures are performed to swipe between pages and pinch zoom in and out of a page
8. Selects a specified chapter based on page number from the navigation view
9. Selects a word in the centre of screen and adds a test note to the page
1. Open Google Play Books application
2. Dismisses sync operation (if applicable)
3. Searches for a book title
4. Adds books to library if not already present
5. Opens 'My Library' contents
6. Opens selected book
7. Gestures are performed to swipe between pages and pinch zoom in and out of a page
8. Selects a specified chapter based on page number from the navigation view
9. Selects a word in the centre of screen and adds a test note to the page
10. Removes the test note from the page (clean up)
11. Searches for the number of occurrences of a common word throughout the book
12. Switches page styles from 'Day' to 'Night' to 'Sepia' and back to 'Day'

2
wlauto/workloads/googleslides/__init__.py
View File

@ -48,6 +48,7 @@ class GoogleSlides(AndroidUxPerfWorkload):
requires an image file specified by the param ``test_image`` to be copied onto the device.
Test description:
1. Start the app and skip the welcome screen. Dismiss the work offline banner if present.
2. Go to the app settings page and enables PowerPoint compatibility mode. This allows
PowerPoint files to be created inside Google Slides.
@ -62,6 +63,7 @@ class GoogleSlides(AndroidUxPerfWorkload):
file to be copied is given by ``test_file``.
Test description:
1. From the documents list (following the create test), open the specified PowerPoint
by navigating into device storage and wait for it to be loaded.
2. A navigation test is performed while the file is in editing mode (i.e. not slideshow).

1
wlauto/workloads/hwuitest/__init__.py
View File

@ -40,6 +40,7 @@ class HWUITest(Workload):
allowed_values=["shadowgrid", "rectgrid", "oval"],
description="""
The test to run:
- ``'shadowgrid'``: creates a grid of rounded rects that
cast shadows, high CPU & GPU load
- ``'rectgrid'``: creates a grid of 1x1 rects

16
wlauto/workloads/rt_app/__init__.py
View File

@ -49,13 +49,13 @@ class RtApp(Workload):
The load is described using JSON-like config files. Below are a couple of simple
examples.
Simple use case which creates a thread that run 1ms then sleep 9ms
until the use case is stopped with Ctrl+C:
.. code-block:: json
{
/*
* Simple use case which creates a thread that run 1ms then sleep 9ms
* until the use case is stopped with Ctrl+C
*/
"tasks" : {
"thread0" : {
"loop" : -1,
@ -76,13 +76,13 @@ class RtApp(Workload):
}
}
Simple use case with 2 threads that runs for 10 ms and wake up each
other until the use case is stopped with Ctrl+C
.. code-block:: json
{
/*
* Simple use case with 2 threads that runs for 10 ms and wake up each
* other until the use case is stopped with Ctrl+C
*/
"tasks" : {
"thread0" : {
"loop" : -1,

2
wlauto/workloads/youtube/__init__.py
View File

@ -35,6 +35,7 @@ class Youtube(AndroidUxPerfWorkload):
Test description:
The ``video_source`` parameter determines where the video to be played will be found
in the app. Possible values are ``search``, ``home``, ``my_videos``, and ``trending``.
-A. search - Goes to the search view, does a search for the given term, and plays the
first video in the results. The parameter ``search_term`` must also be provided
in the agenda for this to work. This is the default mode.
@ -46,6 +47,7 @@ class Youtube(AndroidUxPerfWorkload):
video in the trending videos list.
For the selected video source, the following test steps are performed:
1. Navigate to the general app settings page to disable autoplay. This improves test
stability and predictability by preventing screen transition to load a new video
while in the middle of the test.