diff --git a/doc/source/changes.rst b/doc/source/changes.rst index 38d8ae81..a17c2ffd 100644 --- a/doc/source/changes.rst +++ b/doc/source/changes.rst @@ -16,8 +16,10 @@ changes moving from WA2 to WA3. config file and will be performed upon first invocation of WA3. - The "config" and "global" sections in an agenda are not interchangeable so can all be specified in a "config" section. - "Results Processors" are now known as "Output Processors" and can now be ran offline. - - "Instrumentation" is now know as "Instruments" for more consistent naming. - - "Both "Output Processor" and "Instrument" configuration has been merged into "Augmentations" (support for the old naming schemes have been retained for backwards compatibility) + - "Instrumentation" is now known as "Instruments" for more consistent naming. + - "Both "Output Processor" and "Instrument" configuration have been merged + into "Augmentations" (support for the old naming schemes have been + retained for backwards compatibility) - New features: diff --git a/doc/source/developer_reference.rst b/doc/source/developer_reference.rst index 1ca32b60..88b0a558 100644 --- a/doc/source/developer_reference.rst +++ b/doc/source/developer_reference.rst @@ -13,6 +13,12 @@ Developer Reference .. include:: developer_reference/output_processing_api.rst .. include:: developer_reference/writing_plugins.rst + +----------------- + .. include:: developer_reference/contributing.rst + +----------------- + .. include:: developer_reference/revent.rst diff --git a/doc/source/developer_reference/contributing.rst b/doc/source/developer_reference/contributing.rst index 8e0c5bc3..758c5b16 100644 --- a/doc/source/developer_reference/contributing.rst +++ b/doc/source/developer_reference/contributing.rst @@ -1,7 +1,7 @@ Contributing Code ================= -We welcome code contributions via GitHub pull requests.To help with +We welcome code contributions via GitHub pull requests. To help with maintainability of the code line we ask that the code uses a coding style consistent with the rest of WA code. Briefly, it is @@ -16,7 +16,7 @@ consistent with the rest of WA code. Briefly, it is "stats" for "statistics", "config" for "configuration", etc are OK). Do *not* use Hungarian notation (so prefer ``birth_date`` over ``dtBirth``). -New extensions should also follow implementation guidelines specified in +New extensions should also follow implementation guidelines specified in the :ref:`writing-plugins` section of the documentation. We ask that the following checks are performed on the modified code prior to diff --git a/doc/source/developer_reference/output_processing_api.rst b/doc/source/developer_reference/output_processing_api.rst index 992d0e6b..937eab18 100644 --- a/doc/source/developer_reference/output_processing_api.rst +++ b/doc/source/developer_reference/output_processing_api.rst @@ -21,7 +21,7 @@ directory we can initialize a ``RunOutput`` as follows: -From here we can retrieve different information about the run. For example if we +From here we can retrieve various information about the run. For example if we want to see what the overall status of the run was, along with the runtime parameters and the metrics recorded from the first job was we can do the following: diff --git a/doc/source/developer_reference/revent.rst b/doc/source/developer_reference/revent.rst index 051a7c3e..faea285c 100644 --- a/doc/source/developer_reference/revent.rst +++ b/doc/source/developer_reference/revent.rst @@ -26,8 +26,8 @@ Only the run stage is mandatory, the remaining stages will be replayed if a recording is present otherwise no actions will be performed for that particular stage. -For instance, to add a custom revent files for a device named mydevice and -a workload name myworkload, you need to add the revent files to the directory +For instance, to add a custom revent files for a device named "mydevice" and +a workload name "myworkload", you need to add the revent files to the directory ``/home/$WA_USER_HOME/dependencies/myworkload/revent_files`` creating it if necessary. :: diff --git a/doc/source/developer_reference/writing_plugins.rst b/doc/source/developer_reference/writing_plugins.rst index 1de1d329..bc766a91 100644 --- a/doc/source/developer_reference/writing_plugins.rst +++ b/doc/source/developer_reference/writing_plugins.rst @@ -4,7 +4,7 @@ Writing Plugins ================ -Workload Automation offers several plugin points (or plugin types).The most +Workload Automation offers several plugin points (or plugin types). The most interesting of these are :workloads: These are the tasks that get executed and measured on the device. These @@ -14,24 +14,24 @@ interesting of these are physical device would require its own interface class (though some functionality may be reused by subclassing from an existing base). :instruments: Instruments allow collecting additional data from workload execution (e.g. - system traces). Instruments are not specific to a particular Workload. Instruments + system traces). Instruments are not specific to a particular workload. Instruments can hook into any stage of workload execution. :output processors: These are used to format the results of workload execution once they have been collected. Depending on the callback used, these will run either after each - iteration or at the end of the run, after all of the results have been + iteration and/or at the end of the run, after all of the results have been collected. -You can create an plugin by subclassing the appropriate base class, defining -appropriate methods and attributes, and putting the .py file with the class into -the "plugins" subdirectory under ``~/.workload_automation`` (or equivalent) -where it will be automatically picked up by WA. +You can create a plugin by subclassing the appropriate base class, defining +appropriate methods and attributes, and putting the .py file containing the +class into the "plugins" subdirectory under ``~/.workload_automation`` (or +equivalent) where it will be automatically picked up by WA. Plugin Basics ----------------- +-------------- This sub-section covers things common to implementing plugins of all types. It -is recommended you familiarize yourself with the information here before +is recommended you familiarize yourself with the information here before proceeding onto guidance for specific plugin types. .. _context: @@ -280,7 +280,8 @@ As with other resources, host-side paths to the executable binary to be deployed should be obtained via the :ref:`resource resolver `. A special resource type, ``Executable`` is used to identify a binary to be deployed. This is similar to the regular ``File`` resource, however it takes an -additional parameter that specifies the ABI for which executable was compiled. +additional parameter that specifies the ABI for which the executable was +compiled for. In order for the binary to be obtained in this way, it must be stored in one of the locations scanned by the resource resolver in a directory structure @@ -304,7 +305,8 @@ binary has been previously deployed by WA and will not try to re-install. .. note:: Please also note that the check is done based solely on the binary name. - For more information please see: :func:`devlib.target.Target.install_if_needed` + For more information please see the devlib + `documentation `_. Both of the above methods will return the path to the installed binary on the target. The executable should be invoked *only* via that path; do **not** assume @@ -312,7 +314,7 @@ that it will be in ``PATH`` on the target (or that the executable with the same name in ``PATH`` is the version deployed by WA. For more information on how to implement this, please see the -:ref:`how to guide ` +:ref:`how to guide `. Deploying assets @@ -344,8 +346,8 @@ name Python identifier. kind - This is the type of the value of the parameter. This could be an - callable. Normally this should be a standard Python type, e.g. ``int`` + This is the type of the value of the parameter. This must be an + callable. Normally this should be a standard Python type, e.g. ``int`` or ``float``, or one the types defined in :mod:`wa.utils.types`. If not explicitly specified, this will default to ``str``. @@ -402,12 +404,12 @@ override Validation and cross-parameter constraints ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -An plugin will get validated at some point after constructions. When exactly +A plugin will get validated at some point after construction. When exactly this occurs depends on the plugin type, but it *will* be validated before it is used. You can implement ``validate`` method in your plugin (that takes no arguments -beyond the ``self``) to perform any additions *internal* validation in your +beyond the ``self``) to perform any additional *internal* validation in your plugin. By "internal", I mean that you cannot make assumptions about the surrounding environment (e.g. that the device has been initialized). @@ -476,7 +478,7 @@ notify the user. The exception would typically be :class:`ConfigError` or :class:`WorkloadError`/:class:`DeviceError`/:class:`InstrumentError`/:class:`OutputProcessorError`. All these errors are defined in :mod:`wa.framework.exception` module. -:class:`ConfigError` should be raised where there is a problem in configuration +A :class:`ConfigError` should be raised where there is a problem in configuration specified by the user (either through the agenda or config files). These errors are meant to be resolvable by simple adjustments to the configuration (and the error message should suggest what adjustments need to be made. For all other @@ -606,9 +608,9 @@ Workload Interface ^^^^^^^^^^^^^^^^^^^ The workload interface should be implemented as follows: - :name: This identifies the workload (e.g. it used to specify it in the - :ref:`agenda`. - :init_resources: This method may be optionally override to implement dynamic + :name: This identifies the workload (e.g. it is used to specify the workload + in the :ref:`agenda `). + :init_resources: This method may be optionally overridden to implement dynamic resource discovery for the workload. This method executes early on, before the device has been initialized, so it should only be used to initialize resources that do not @@ -661,7 +663,7 @@ track of the current execution state (such as the current workload, iteration number, etc), and contains, among other things, a :class:`wa.framework.output.JobOutput` instance that should be populated from the ``update_output`` method with the results of the execution. For more -information please see `the context`_ documentation.:: +information please see `the context`_ documentation. :: # ... @@ -697,7 +699,7 @@ The revent workload classes define the following interfaces:: The interface should be implemented as follows :name: This identifies the workload (e.g. it used to specify it in the - :ref:`agenda`. + :ref:`agenda `. :package_names: This is a list of the android application apk packages names that are required to run the workload. @@ -714,11 +716,11 @@ Automation, the methods of the new instrument will be found automatically and hooked up to the supported signals. Once a signal is broadcasted, the corresponding registered method is invoked. -Each method in Instrument must take two arguments, which are self and context. -Supported method and their corresponding signals can be found in the -:ref:`Signals Documentation ` To make implementations -easier and common, the basic steps to add new instrument is similar to the steps -to add new workload and an example can be found +Each method in ``Instrument`` must take two arguments, which are ``self`` and +``context``. Supported methods and their corresponding signals can be found in +the :ref:`Signals Documentation `. To make +implementations easier and common, the basic steps to add new instrument is +similar to the steps to add new workload and an example can be found :ref:`here `. .. _instrument-api: @@ -787,7 +789,7 @@ The full interface of WA instruments is shown below:: """ pass -This is similar to a Workload, except all methods are optional. In addition to +This is similar to a ``Workload``, except all methods are optional. In addition to the workload-like methods, instruments can define a number of other methods that will get invoked at various points during run execution. The most useful of which is perhaps ``initialize`` that gets invoked after the device has been @@ -861,11 +863,11 @@ Below is a simple instrument that measures the execution time of a workload:: self.start_time = None self.end_time = None - @fast + @very_fast def start(self, context): self.start_time = time.time() - @fast + @very_fast def stop(self, context): self.end_time = time.time() @@ -887,7 +889,7 @@ generating plots, etc. WA comes with a few output processors that output results in a few common formats (such as csv or JSON). You can add your own output processors by creating a Python file in -``~/.workload_automation/result_processors`` with a class that derives from +``~/.workload_automation/plugins`` with a class that derives from :class:`wa.OutputProcessor `, which has the following interface:: @@ -919,7 +921,7 @@ the following interface:: The method names should be fairly self-explanatory. The difference between -"process" and "export" methods is that export methods will be invoke after +"process" and "export" methods is that export methods will be invoked after process methods for all output processors have been generated. Process methods may generated additional artifacts (metrics, files, etc), while export methods should not -- the should only handle existing results (upload them to a @@ -927,8 +929,7 @@ database, archive on a filer, etc). The output object passed to job methods is an instance of :class:`wa.framework.output.JobOutput`, the output object passed to run methods -is an instance of :class:`wa.RunOutput `. Please -refer to their API documentation for details. +is an instance of :class:`wa.RunOutput `. Adding a Resource Getter @@ -987,7 +988,7 @@ should typically be registered as ``preferred``. You don't have to stick to standard priority levels (though you should, unless there is a good reason). Any integer is a valid priority. The standard priorities -range from 0 to 50 in increments of 10. +range from 0 to 40 in increments of 10. Example ^^^^^^^ @@ -1036,9 +1037,9 @@ The currently available platforms are: The currently available targets from devlib are: :linux: A device running a Linux based OS. - :android: A device running the Android OS. + :android: A device running Android OS. :local: Used to run locally on a linux based host. - :chromeos: A device running chromeos, supporting an android container if available. + :chromeos: A device running ChromeOS, supporting an android container if available. For an example of adding you own customized version of an existing devlib target, please see the how to section :ref:`Adding a Custom Target `. @@ -1113,7 +1114,7 @@ management tools, e.g. :: sudo pip install my_wa_exts-0.0.1.tar.gz As part of the installation process, the setup.py in the package, will write the -package's name into ``~/.workoad_automoation/packages``. This will tell WA that +package's name into ``~/.workoad_automation/packages``. This will tell WA that the package contains plugin and it will load them next time it runs. .. note:: There are no uninstall hooks in ``setuputils``, so if you ever diff --git a/doc/source/faq.rst b/doc/source/faq.rst index 21b0989a..94879265 100644 --- a/doc/source/faq.rst +++ b/doc/source/faq.rst @@ -37,8 +37,8 @@ install the application onto the device or source the apk and place into **Q:** I am trying to set a valid runtime parameters however I still receive the error ``"Unknown runtime parameter"`` ------------------------------------------------------------------------------------------------------------------------- -**A:** PLease ensure you have the corresponding module loaded on the device. -Please see :ref:`Runtime Parameters ` for the list of +**A:** Please ensure you have the corresponding module loaded on the device. +See :ref:`Runtime Parameters ` for the list of runtime parameters and their containing modules, and the appropriate section in :ref:`setting up a device ` for ensuring it is installed. diff --git a/doc/source/how_tos/developers/adding_plugins.rst b/doc/source/how_tos/developers/adding_plugins.rst index 0b8ac5d2..2fb707cf 100644 --- a/doc/source/how_tos/developers/adding_plugins.rst +++ b/doc/source/how_tos/developers/adding_plugins.rst @@ -11,7 +11,7 @@ then the ``initialize`` method should be decorated with the ``@once`` method which is decorated instead. Please note if doing this then any installed paths should be added as class attributes rather than instance variables. As a general rule if binaries are installed as part of ``initialize`` then they -should be installed in the complementary ``finalize`` method. +should be uninstalled in the complementary ``finalize`` method. Part of an example workload demonstrating this is shown below: @@ -41,11 +41,11 @@ Part of an example workload demonstrating this is shown below: Adding a Workload Examples ========================== -The easiest way to create a new workload is to use the create command. ``wa -create workload ``. This will use predefined templates to create a -workload based on the options that are supplied to be used as a starting point -for the workload. For more information on using the create workload command see -``wa create workload -h`` +The easiest way to create a new workload is to use the +:ref:`create ` command. ``wa create workload ``. This +will use predefined templates to create a workload based on the options that are +supplied to be used as a starting point for the workload. For more information +on using the create workload command see ``wa create workload -h`` The first thing to decide is the type of workload you want to create depending on the OS you will be using and the aim of the workload. The are currently 6 @@ -55,7 +55,7 @@ Once you have decided what type of workload you wish to choose this can be specified with ``-k `` followed by the workload name. This will automatically generate a workload in the your ``WA_CONFIG_DIR/plugins``. If you wish to specify a custom location this can be provided with ``-p -`` +`` Adding a Basic Workload Example -------------------------------- @@ -68,10 +68,9 @@ This will generate a very basic workload with dummy methods for the workload interface and it is left to the developer to add any required functionality to the workload. -This example shows a simple workload that times how long it takes to compress a -file of a particular size on the device, not all the methods are required to be -implements however as many as possible have been used to demonstrate their -purpose. +Not all the methods are required to be implemented, this example shows how a +subset might be used to implement a simple workload that times how long it takes +to compress a file of a particular size on the device. .. note:: This is intended as an example of how to implement the Workload @@ -92,7 +91,6 @@ purpose. This workload was created for illustration purposes only. It should not be used to collect actual measurements. - ''' parameters = [ @@ -137,9 +135,9 @@ purpose. This method is used to extract any results from the target for example we want to pull the file containing the timing information that we will use to generate metrics for our workload and then we - add this file as a artifact as a 'raw' file which once WA has - finished processing it will allow it to decide whether to keep the - file or not. + add this file as an artifact with a 'raw' kind, which means once WA + has finished processing it will allow it to decide whether to keep + the file or not. """ super(ZipTestWorkload, self).extract_results(context) # Pull the results file to the host @@ -149,9 +147,9 @@ purpose. def update_output(self, context): """ - In this method we can do any generation of metric that we wish to + In this method we can do any generation of metrics that we wish to for our workload. In this case we are going to simply convert the - times reported to seconds and add them as 'metrics' to WA which can + times reported into seconds and add them as 'metrics' to WA which can then be displayed to the user along with any others in a format dependant on which output processors they have enabled for the run. """ @@ -343,7 +341,8 @@ command:: This will generate a revent based workload you will end up with a very similar python file as to the one outlined in generating a :ref:`UiAutomator based -workload ` except without the java automation files. +workload ` however without the accompanying java +automation files. The main difference between the two is that this workload will subclass ``ApkReventWorkload`` instead of ``ApkUiautomatorWorkload`` as shown below. @@ -367,7 +366,7 @@ The main difference between the two is that this workload will subclass Adding an Instrument Example ============================= This is an example of how we would create a instrument which will trace device -errors. For more detailed information please see +errors using a custom "trace" binary file. For more detailed information please see :ref:`here `. The first thing to do is to subclass :class:`Instrument`, overwrite the variable name with what we want our instrument to be called and locate our binary for our instrument. @@ -388,7 +387,7 @@ We then declare and implement the required methods as detailed :ref:`here `. For the ``initialize`` method, we want to install the executable file to the target so we can use the target's ``install`` method which will try to copy the file to a location on the device that -supports execution, will change the file mode appropriately and return the +supports execution, change the file mode appropriately and return the file path on the target. :: def initialize(self, context): @@ -398,7 +397,7 @@ Then we implemented the start method, which will simply run the file to start tracing. Supposing that the call to this binary requires some overhead to begin collecting errors we might want to decorate the method with the ``@slow`` decorator to try and reduce the impact on other running instruments. For more -information on prioritization please see :ref:`here `:: +information on prioritization please see :ref:`here `. :: @slow def start(self, context): @@ -406,7 +405,7 @@ information on prioritization please see :ref:`here `:: Lastly, we need to stop tracing once the workload stops and this happens in the stop method, assuming stopping the collection also require some overhead we have -again decorated the method:: +again decorated the method. :: @slow def stop(self, context): @@ -425,9 +424,9 @@ example for trace data we will want to pull it to the device and add it as a Once we have retrieved the data we can now do any further processing and add any relevant :ref:`Metrics ` to the :ref:`context `. For this we -will use the the ``add_metric`` method and to add the instruments results to the -final result for that workload. The method can be passed 4 params, which are the -metric `key`, `value`, `unit` and `lower_is_better`. :: +will use the the ``add_metric`` method to add the results to the final output +for that workload. The method can be passed 4 params, which are the metric +`key`, `value`, `unit` and `lower_is_better`. :: def update_output(self, context): # parse the file if needs to be parsed, or add result directly to @@ -442,7 +441,7 @@ instruments and the code to clear these file goes in teardown method. :: def teardown(self, context): self.target.remove(os.path.join(self.target.working_directory, 'trace.txt')) -At the very end of the run we would want to uninstall the binary we deployed earlier :: +At the very end of the run we would want to uninstall the binary we deployed earlier. :: def finalize(self, context): self.target.uninstall(self.binary_name) @@ -499,7 +498,7 @@ Next we need to implement any relevant methods, (please see available methods). In this case we only want to implement the ``export_run_output`` method as we are not generating any new artifacts and we only care about the overall output rather than the individual job -outputs. And the implementation is very simple, it just loops through all +outputs. The implementation is very simple, it just loops through all the available metrics for all the available jobs and adds them to a list which is written to file and then added as an :ref:`artifact ` to the :ref:`context `. @@ -514,7 +513,7 @@ the :ref:`context `. class Table(OutputProcessor): name = 'table' - description = 'Generates a text file containing a column-aligned table with run results.' + description = 'Generates a text file containing a column-aligned table of run results.' def export_run_output(self, output, target_info): rows = [] @@ -527,7 +526,7 @@ the :ref:`context `. outfile = output.get_path('table.txt') with open(outfile, 'w') as wfh: write_table(rows, wfh) - output.add_artifact('results_table', 'table.txt, 'export') + output.add_artifact('results_table', 'table.txt', 'export') .. _adding-custom-target-example: diff --git a/doc/source/how_tos/users/agenda.rst b/doc/source/how_tos/users/agenda.rst index b986a9af..79d25ff7 100644 --- a/doc/source/how_tos/users/agenda.rst +++ b/doc/source/how_tos/users/agenda.rst @@ -347,8 +347,8 @@ Classifiers ------------ Classifiers can be used in 2 distinct ways, the first use is being supplied in -an agenda as a set of key-value pairs which can be used to help identify sub- -tests of a run, for example if you have multiple sections in your agenda running +an agenda as a set of key-value pairs which can be used to help identify sub-tests +of a run, for example if you have multiple sections in your agenda running your workloads at different frequencies you might want to set a classifier specifying which frequencies are being used. These can then be utilized later, for example with the ``csv`` :ref:`output processor ` with @@ -384,9 +384,9 @@ An example agenda is shown here: The other way that they can used is by being automatically added by some workloads to identify their results metrics and artifacts. For example some workloads perform multiple tests with the same execution run and therefore will -use metrics to differentiate between them, For example the ``recentfling`` -workload will use classifiers to distinguish between which loop a particular -result is for or whether it is an average across all loops ran. +use metrics to differentiate between them, e.g. the ``recentfling`` workload +will use classifiers to distinguish between which loop a particular result is +for or whether it is an average across all loops ran. The output from the agenda above will produce a csv file similar to what is shown below. Some columns have been omitted for clarity however as can been seen @@ -500,7 +500,7 @@ turn override global settings. -Augmentationts +Augmentations -------------- Augmentations are plugins that augment the execution of workload jobs with @@ -509,7 +509,7 @@ metrics and/or artifacts, such as traces or logs. There are two types of augmentations: Instruments - These "instrument" a WA run in order to change it's behavior (e.g. + These "instrument" a WA run in order to change it's behaviour (e.g. introducing delays between successive job executions), or collect additional measurements (e.g. energy usage). Some instruments may depend on particular features being enabled on the target (e.g. cpufreq), or @@ -677,9 +677,11 @@ Each workload will be run in two configurations: once, to collect energy measurements, and once to collect thermal data and kernel trace. Trace can give insight into why a workload is using more or less energy than expected, but it can be relatively intrusive and might impact absolute energy and performance -metrics, which is why it is collected separately. classifiers_ are used to +metrics, which is why it is collected separately. Classifiers_ are used to separate metrics from the two configurations in the results. +.. _other-agenda-configuration: + Other Configuration ------------------- diff --git a/doc/source/how_tos/users/device_setup.rst b/doc/source/how_tos/users/device_setup.rst index 76b3fcb4..7ea22771 100644 --- a/doc/source/how_tos/users/device_setup.rst +++ b/doc/source/how_tos/users/device_setup.rst @@ -20,12 +20,11 @@ Android General Device Setup ^^^^^^^^^^^^^^^^^^^^ -You can specify the device interface by setting ``device`` setting in -``~/.workload_automation/config.yaml``. Available interfaces can be viewed by -running ``wa list targets`` command. If you don't see your specific platform -listed (which is likely unless you're using one of the Arm-supplied platforms), then -you should use ``generic_android`` interface (this is set in the config by -default). +You can specify the device interface by setting ``device`` setting in a +``config`` file or section. Available interfaces can be viewed by running ``wa +list targets`` command. If you don't see your specific platform listed (which is +likely unless you're using one of the Arm-supplied platforms), then you should +use ``generic_android`` interface (this is what is used by the default config). .. code-block:: yaml @@ -38,7 +37,8 @@ common parameters you might want to change are outlined below. .. confval:: device If you have multiple Android devices connected to the host machine, you will - need to set this to indicate to WA which device you want it to use. + need to set this to indicate to WA which device you want it to use. The will + be the adb name the is displayed when running ``adb devices`` .. confval:: working_directory @@ -81,7 +81,7 @@ A typical ``device_config`` inside ``config.yaml`` may look something like device_config: device: 0123456789ABCDEF - # ... + # ... or a more specific config could be be @@ -172,12 +172,11 @@ Linux General Device Setup ^^^^^^^^^^^^^^^^^^^^ -You can specify the device interface by setting ``device`` setting in -``~/.workload_automation/config.yaml``. Available interfaces can be viewed by -running ``wa list targets`` command. If you don't see your specific platform -listed (which is likely unless you're using one of the Arm-supplied platforms), then -you should use ``generic_linux`` interface (this is set in the config by -default). +You can specify the device interface by setting ``device`` setting in a +``config`` file or section. Available interfaces can be viewed by running +``wa list targets`` command. If you don't see your specific platform listed +(which is likely unless you're using one of the Arm-supplied platforms), then +you should use ``generic_linux`` interface. .. code-block:: yaml @@ -239,25 +238,24 @@ Chrome OS General Device Setup ^^^^^^^^^^^^^^^^^^^^ -You can specify the device interface by setting ``device`` setting in -``~/.workload_automation/config.yaml``. Available interfaces can be viewed by +You can specify the device interface by setting ``device`` setting in a +``config`` file or section. Available interfaces can be viewed by running ``wa list targets`` command. If you don't see your specific platform listed (which is likely unless you're using one of the Arm-supplied platforms), then -you should use ``generic_chromeos`` interface (this is set in the config by -default). +you should use ``generic_chromeos`` interface. .. code-block:: yaml device: generic_chromeos The device interface may be configured through ``device_config`` setting, who's -value is a ``dict`` mapping setting names to their values. The chrome os target +value is a ``dict`` mapping setting names to their values. The ChromeOS target is essentially the same as a linux device and requires a similar setup, however it also optionally supports connecting to an android container running on the -device which will be automatically deted if present. If the device supports +device which will be automatically detected if present. If the device supports android applications then the android configuration is also supported. In order -to support this then WA will open 2 connections to the device, one via SSH to -the main ChomeOS OS and another via ADB to the android container where a limited +to support this WA will open 2 connections to the device, one via SSH to +the main OS and another via ADB to the android container where a limited subset of functionality can be performed. In order to distinguish between the two connections some of the android specific diff --git a/doc/source/how_tos/users/revent.rst b/doc/source/how_tos/users/revent.rst index c0d75533..05b23fbe 100644 --- a/doc/source/how_tos/users/revent.rst +++ b/doc/source/how_tos/users/revent.rst @@ -94,7 +94,7 @@ command with an example output shown below:: INFO Recording(s) are available at: '$WA_USER_DIRECTORY/dependencies/angrybirds_rio/revent_files' Once you have made your desired recordings, you can either manually playback -individual recordings using the :ref:`reply ` command or, with +individual recordings using the :ref:`replay ` command or, with the recordings in the appropriate dependencies location, simply run the workload using the :ref:`run ` command and then all the available recordings will be played back automatically. diff --git a/doc/source/installation.rst b/doc/source/installation.rst index 1268f7f3..0e007464 100644 --- a/doc/source/installation.rst +++ b/doc/source/installation.rst @@ -227,7 +227,7 @@ which contains addional information about how to build and to use the file. Some WA plugins have additional dependencies that need to be satisfied before they can be used. Not all of these can be provided with WA and so will need to be supplied by the user. They should be placed into -``~/.workload_uatomation/dependencies/`` so that WA can find +``~/.workload_automation/dependencies/`` so that WA can find them (you may need to create the directory if it doesn't already exist). You only need to provide the dependencies for workloads you want to use. @@ -237,16 +237,16 @@ APK Files --------- APKs are application packages used by Android. These are necessary to install on -a device when running an :ref:`ApkWorkload ` or derivative. PLease -see the workload description using the :ref:`show command ` to see +a device when running an :ref:`ApkWorkload ` or derivative. Please +see the workload description using the :ref:`show ` command to see which version of the apk the UI automation has been tested with and place the -apk in the corresponding Automation may also work with other versions -(especially if it's only a minor or revision difference -- major version -differences are more likely to contain incompatible UI changes) but this has not -been tested. As a general rule we do not guarantee support for the latest -version of an app and they are updated on as needed basis. We do however attempt -to support backwards compatibility with previous major releases however beyond -this support will likely be dropped. +apk in the corresponding workloads dependency folder. Automation may also work +with other versions (especially if it's only a minor or revision difference -- +major version differences are more likely to contain incompatible UI changes) +but this has not been tested. As a general rule we do not guarantee support for +the latest version of an app and they are updated on an as needed basis. We do +however attempt to support backwards compatibility with previous major releases +however beyond this support will likely be dropped. Gaming Workloads diff --git a/doc/source/migration_guide.rst b/doc/source/migration_guide.rst index 98a5f578..a15a137f 100644 --- a/doc/source/migration_guide.rst +++ b/doc/source/migration_guide.rst @@ -15,12 +15,12 @@ Configuration Default configuration file change ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Instead of the standard ``config.py`` file located at ``$WA_USER_HOME/config.py` -WA not use a ``confg.yaml`` files which is written in the YAML format instead of -python. Additionally upon first invocation WA3 will automatically try and detect -whether a WA2 config file is present and convert it to use the new WA3 format. -During this process any know parameter name changes should be detected and -updated accordingly. +Instead of the standard ``config.py`` file located at +``$WA_USER_HOME/config.py`` WA now uses a ``confg.yaml`` file (at the same +location) which is written in the YAML format instead of python. Additionally +upon first invocation WA3 will automatically try and detect whether a WA2 config +file is present and convert it to use the new WA3 format. During this process +any known parameter name changes should be detected and updated accordingly. Plugin Changes ^^^^^^^^^^^^^^^ @@ -78,7 +78,7 @@ the new parameter names should be preferred: - The workload parameter :confval:`check_apk` has been renamed to :confval:`prefer_host_package` to be more explicit in it's functionality to indicated - whether a package on the target for the host should have priority when + whether a package on the target or the host should have priority when searching for a suitable package. Individual workload parameters have been attempted to be standardized for the @@ -110,7 +110,7 @@ moved into a ``__failed`` subdirectory to allow for quicker analysis. Output API ~~~~~~~~~~ There is now an Output API which can be used to more easily post process the -output from a workload. For more information please see the +output from a run. For more information please see the :ref:`Output API ` documentation. @@ -155,15 +155,15 @@ Workloads Python Workload Structure ^^^^^^^^^^^^^^^^^^^^^^^^^^ -The ``update_results`` method has been split out into 2 stages. There is now -``extract_results`` and ``update_output`` which should be used for extracting -any results from the target back to the host system and to update the output -with any metrics or artefacts for the specific workload iteration respectively. +- The ``update_results`` method has been split out into 2 stages. There is now + ``extract_results`` and ``update_output`` which should be used for extracting + any results from the target back to the host system and to update the output + with any metrics or artefacts for the specific workload iteration respectively. -WA now features :ref:`decorators ` which can be used to allow for more efficient -binary deployment and that they are only installed to the device once per run. For -more information of implementing this please see -:ref:`deploying executables to a target `. +- WA now features :ref:`decorators ` which can be used to allow for more efficient + binary deployment and that they are only installed to the device once per run. For + more information of implementing this please see + :ref:`deploying executables to a target `. APK Functionality @@ -176,27 +176,25 @@ available as the apk attribute of the workload. This means that for example UiAutomator Java Structure ^^^^^^^^^^^^^^^^^^^^^^^^^^ Instead of a single ``runUiAutomation`` method to perform all of the UiAutomation, -the structure has been refactored into 4 methods that can optionally be overridden. +the structure has been refactored into 5 methods that can optionally be overridden. The available methods are ``initialize``, ``setup``, ``runWorkload``, ``extactResults`` and ``teardown`` to better mimic the different stages in the python workload. -``initialize`` should have the ``@Before`` tag attached to the method which will cause it -to be ran before each of the stages of the workload. This method should be used to retrieve -and set any relevant parameters required during the workload. + - ``initialize`` should be used to retrieve + and set any relevant parameters required during the workload. + - ``setup`` should be used to perform any setup required for the workload, for + example dismissing popups or configuring and required settings. + - ``runWorkload`` should be used to perform the actual measurable work of the workload. + - ``extractResults`` should be used to extract any relevant results from the + target after the workload has been completed. + - ``teardown`` should be used to perform any final clean up of the workload on the target. -The remaining method all have the ``@Test`` tag attached to the method to indicate that this -is a test stage that should be called at the appropriate time. - -``setup`` should be used to perform any setup required for the workload, for -example dismissing popups or configuring and required settings. - -``runWorkload`` should be used to perform the actual measurable work of the workload. - -``extractResults`` should be used to extract any relevant results from the -target after the workload has been completed. - -``teardown`` should be used to perform any final clean up of the workload on the target. +.. note:: The ``initialize`` method should have the ``@Before`` tag attached + to the method which will cause it to be ran before each of the stages of + the workload. The remaining method should all have the ``@Test`` tag + attached to the method to indicate that this is a test stage that should be + called at the appropriate time. GUI Functionality ^^^^^^^^^^^^^^^^^ @@ -208,28 +206,26 @@ you wish to pass parameters to a UiAuotmator workload you will now need to use Attributes ^^^^^^^^^^ -The ``device`` attribute of the workload is now a devlib ``target``. Some of the -command names remain the same, however there will be differences. The API can be -found here: http://devlib.readthedocs.io/en/latest/target.html however some of -the more common changes can be found below: +- The old ``package`` attribute has been replaced by ``package_names`` which + expects a list of strings which allows for multiple package names to be + specified if required. It is also no longer required to explicitly state the + launch-able activity, this will be automatically discovered from the apk so this + workload attribute can be removed. + +- The ``device`` attribute of the workload is now a devlib ``target``. Some of the + command names remain the same, however there will be differences. The API can be + found at http://devlib.readthedocs.io/en/latest/target.html however some of + the more common changes can be found below: -+----------------------------------------------+---------------------------------+ -| Original Method | New Method | -+----------------------------------------------+---------------------------------+ -|``self.device.pull_file(file)`` | ``self.target.pull(file)`` | -+----------------------------------------------+---------------------------------+ -|``self.device.push_file(file)`` | ``self.target.push(file)`` | -+----------------------------------------------+---------------------------------+ -|``self.device.install_executable(file)`` | ``self.target.install(file)`` | -+----------------------------------------------+---------------------------------+ -|``self.device.execute(cmd, background=True)`` | ``self.target.background(cmd)``| -+----------------------------------------------+---------------------------------+ - - -The old ``package`` attribute has been replaced by ``package_names`` which -expects a list of strings which allows for multiple package names to be -specified if required. It is also no longer required to explicitly state the -launch-able activity, this will be automatically discovered from the apk so this -workload attribute can be removed. - + +----------------------------------------------+---------------------------------+ + | Original Method | New Method | + +----------------------------------------------+---------------------------------+ + |``self.device.pull_file(file)`` | ``self.target.pull(file)`` | + +----------------------------------------------+---------------------------------+ + |``self.device.push_file(file)`` | ``self.target.push(file)`` | + +----------------------------------------------+---------------------------------+ + |``self.device.install_executable(file)`` | ``self.target.install(file)`` | + +----------------------------------------------+---------------------------------+ + |``self.device.execute(cmd, background=True)`` | ``self.target.background(cmd)``| + +----------------------------------------------+---------------------------------+ diff --git a/doc/source/user_guide.rst b/doc/source/user_guide.rst index 61ed763c..52bcd882 100644 --- a/doc/source/user_guide.rst +++ b/doc/source/user_guide.rst @@ -67,7 +67,7 @@ List Command ============ In order to get started with using WA we first we need to find -out what is available to use. In order to do this we can use the "list" +out what is available to use. In order to do this we can use the :ref:`list ` command followed by the type of plugin that you wish to see. For example to see what workloads are available along with a short description @@ -97,7 +97,9 @@ Which will give an output in the format of: The same syntax can be used to display ``commands``, ``energy_instrument_backends``, ``instruments``, ``output_processors``, -``resource_getters`` and ``targets``. Alternatively please see the +``resource_getters``, ``targets``. Once you have found the plugin you are +looking for you can use the :ref:`show ` command to display more +detailed information. Alternatively please see the :ref:`Plugin Reference ` for an online version. Show Command @@ -141,13 +143,14 @@ configuration file which will in turn overwrite the default configuration file. Android ------- -By default, the device is set to 'generic_android'. WA is configured to work -with a generic Android device through ``adb``. If you only have one device listed -when you execute ``adb devices``, and your device has a standard Android -configuration, then no extra configuration is required. +By default, the device WA will use is set to 'generic_android'. WA is configured +to work with a generic Android device through ``adb``. If you only have one +device listed when you execute ``adb devices``, and your device has a standard +Android configuration, then no extra configuration is required. -However, if your device is connected via network, you will have to manually execute -``adb connect `` so that it appears in the device listing. +However, if your device is connected via network, you will have to manually +execute ``adb connect `` (or specify this in your +:ref:`agenda `) so that it appears in the device listing. If you have multiple devices connected, you will need to tell WA which one you want it to use. You can do that by setting ``device`` in the device_config section. diff --git a/doc/source/user_reference.rst b/doc/source/user_reference.rst index e2365aa3..a0dd9f35 100644 --- a/doc/source/user_reference.rst +++ b/doc/source/user_reference.rst @@ -12,5 +12,14 @@ User Reference --------------------------------------------------------------- .. include:: user_reference/configuration.rst + +------------------- + .. include:: user_reference/invocation.rst + +------------------- + .. include:: user_reference/output_directory.rst + + + diff --git a/doc/source/user_reference/invocation.rst b/doc/source/user_reference/invocation.rst index 7679fcd6..6c1762e2 100644 --- a/doc/source/user_reference/invocation.rst +++ b/doc/source/user_reference/invocation.rst @@ -16,16 +16,18 @@ Individual sub-commands are discussed in detail below. Run --- -The most common sub-command you will use is ``run``. This will run specified -workload(s) and process resulting output. This takes a single mandatory -argument that specifies what you want WA to run. This could be either a -workload name, or a path to an "agenda" file that allows to specify multiple -workloads as well as a lot additional configuration (see :ref:`agenda` -section for details). Executing :: +The most common sub-command you will use is ``run``. This will run the specified +workload(s) and process its resulting output. This takes a single mandatory +argument which specifies what you want WA to run. This could be either a workload +name, or a path to an agenda" file that allows to specify multiple workloads as +well as a lot additional configuration (see :ref:`agenda` section for details). +Executing :: wa run -h -Will display help for this subcommand that will look something like this:: +Will display help for this subcommand that will look something like this: + +.. code-block:: none usage: wa run [-h] [-c CONFIG] [-v] [--version] [-d DIR] [-f] [-i ID] [--disable INSTRUMENT] @@ -130,7 +132,7 @@ will produce something like: :: from the website: The whole idea behind this application is to use the same Pi calculation - algorithm on every Android Device and check how fast that proccess is. + algorithm on every Android Device and check how fast that process is. Better calculation times, conclude to faster Android devices. This way you can also check how lightweight your custom made Android build is. Or not. @@ -185,9 +187,9 @@ will produce something like: :: markers_enabled : boolean If set to ``True``, workloads will insert markers into logs - at various points during execution. These markes may be used + at various points during execution. These markers may be used by other plugins or post-processing scripts to provide - measurments or statistics for specific parts of the workload + measurements or statistics for specific parts of the workload execution. .. note:: You can also use this command to view global settings by using ``wa show settings`` @@ -202,9 +204,8 @@ This aids in the creation of new WA-related objects for example agendas and work For more detailed information on creating workloads please see the :ref:`adding a workload ` section for more details. -agendas: As an example to create an agenda that will run the dhrystone and memcpy workloads -that will use the status and hwmon augumentations, run each test 3 times and save +that will use the status and hwmon augmentations, run each test 3 times and save into the file ``my_agenda.yaml`` the following command can be used:: wa create agenda dhrystone memcpy status hwmon -i 3 -o my_agenda.yaml @@ -271,7 +272,7 @@ using the following command:: Record ------ -This command simiplifies the process of recording revent files. It will +This command simplifies the process of recording revent files. It will automatically deploy revent and has options to automatically open apps and record specified stages of a workload. Revent allows you to record raw inputs such as screen swipes or button presses. This can be useful for recording inputs @@ -285,14 +286,14 @@ devices it is obtained from ``/proc/device-tree/model``. - suffix is used by WA to determine which part of the app execution the recording is for, currently these are either ``setup``, ``run``, ``extract_results`` or ``teardown``. All stages except ``run`` are optional for playback and to specify which stages -shoule be recorded the ``-s``, ``-r``, ``-e`` or ``-t`` arguments respectively, +should be recorded the ``-s``, ``-r``, ``-e`` or ``-t`` arguments respectively, or optionally ``-a`` to indicate all stages should be recorded. The full set of options for this command are:: usage: wa record [-h] [-c CONFIG] [-v] [--version] [-d DEVICE] [-o FILE] [-s] - [-e] [-t] [-a] [-C] [-p PACKAGE | -w WORKLOAD] + [-r] [-e] [-t] [-a] [-C] [-p PACKAGE | -w WORKLOAD] optional arguments: -h, --help show this help message and exit @@ -307,8 +308,8 @@ The full set of options for this command are:: -o FILE, --output FILE Specify the output file -s, --setup Record a recording for setup stage - -e, --extract_results - Record a recording for extract_results stage + -r, --run Record a recording for run stage + -e, --extract_results Record a recording for extract_results stage -t, --teardown Record a recording for teardown stage -a, --all Record recordings for available stages -C, --clear Clear app cache before launching it @@ -324,8 +325,9 @@ For more information please see :ref:`Revent Recording `. Replay ------ -Along side ``record`` wa also has a command to playback a single recorded revent file. -It behaves similar to the ``record`` command taking a subset of the same options allowing you to automatically launch a package on the device :: +Alongside ``record`` wa also has a command to playback a single recorded revent +file. It behaves similar to the ``record`` command taking a subset of the same +options allowing you to automatically launch a package on the device :: usage: wa replay [-h] [-c CONFIG] [-v] [--debug] [--version] [-p PACKAGE] [-C] revent diff --git a/doc/source/user_reference/runtime_parameters.rst b/doc/source/user_reference/runtime_parameters.rst index ae59378a..f1ca0541 100644 --- a/doc/source/user_reference/runtime_parameters.rst +++ b/doc/source/user_reference/runtime_parameters.rst @@ -14,22 +14,27 @@ Example ^^^^^^^ Say we want to perform an experiment on an Android big.LITTLE devices to compare the power consumption between the big and LITTLE clusters running the dhrystone -workload. Assuming we have additional instrumentation active for this device -that we can measure the power the device is consuming, to reduce external -factors we want to ensure that the device has its screen turned off and that it -is in airplane mode turned on for both tests. We will then run 2 :ref:`sections -` will each enable a single cluster on the device, set the cores to their -maximum frequency and disable all available idle states. +and benchmarkpi workloads. Assuming we have additional instrumentation active +for this device that can measure the power the device is consuming, to reduce +external factors we want to ensure that the device is in airplane mode turned on +for all our tests and the screen is off only for our dhrystone run. We will then +run 2 :ref:`sections ` will each enable a single cluster on the +device, set the cores to their maximum frequency and disable all available idle +states. .. code-block:: yaml + config: + runtime_parameters: + airplane_mode: true #.. workloads: - name: dhrystone iterations: 1 runtime_parameters: - airplane_mode: true screen_on: false + - name: benchmarkpi + iterations: 1 sections: - id: LITTLES runtime_parameters: @@ -167,8 +172,9 @@ CPUIdle ^^^^^^^ :idle_states: A ``string`` or list of strings which can be used to specify what - idles states should be enabled for all cores if there are common frequencies - available. 'all' and 'none' are also valid entries as a shorthand + idles states should be enabled for all cores if there are common + idle states available. 'all' and 'none' are also valid entries as a + shorthand :_idle_states: A ``string`` or list of strings which can be used to specify what idles states should be enabled for cores of a particular type @@ -200,7 +206,7 @@ Android Specific Runtime Parameters entries are ``NATURAL``, ``LEFT``, ``INVERTED``, ``RIGHT``. :screen_on: A ``boolean`` to specify whether the devices screen should be - turned on. Defaults to ``true``. + turned on. Defaults to ``True``. .. _setting-sysfiles: @@ -209,18 +215,19 @@ Setting Sysfiles In order to perform additional configuration of a target the ``sysfile_values`` runtime parameter can be used. The value for this parameter is a mapping (an associative array, in YAML) of file paths onto values that should be written -into those files. sysfile_values is the only runtime parameter that is available -for any (Linux) device. Other runtime parameters will depend on the specifics of -the device used (e.g. its CPU cores configuration) as detailed above. +into those files. ``sysfile_values`` is the only runtime parameter that is +available for any (Linux) device. Other runtime parameters will depend on the +specifics of the device used (e.g. its CPU cores configuration) as detailed +above. -.. note:: By default WA will attempt to verify that the sysfile value was +.. note:: By default WA will attempt to verify that the any sysfile values were written correctly by reading the node back and comparing the two values. If you do not wish this check to happen, for example the node you are writing to is write only, you can append an ``!`` to the file path to disable this verification. For example the following configuration could be used to enable and verify that cpu0 -is online, however will not attempt to check that it's governor have been set to +is online, however will not attempt to check that its governor have been set to userspace:: - name: dhrystone diff --git a/wa/framework/configuration/core.py b/wa/framework/configuration/core.py index 5fbf407f..8eee49d8 100644 --- a/wa/framework/configuration/core.py +++ b/wa/framework/configuration/core.py @@ -652,7 +652,7 @@ class RunConfiguration(Configuration): The maximum number of times failed jobs will be retried before giving up. If not set. - .. note:: this number does not include the original attempt + .. note:: This number does not include the original attempt ''', ), ConfigurationPoint( diff --git a/wa/workloads/geekbench/__init__.py b/wa/workloads/geekbench/__init__.py index 04639bc7..0adad307 100644 --- a/wa/workloads/geekbench/__init__.py +++ b/wa/workloads/geekbench/__init__.py @@ -37,8 +37,8 @@ class Geekbench(ApkUiautoWorkload): Geekbench scores are calibrated against a baseline score of 1,000 (which is the score of a single-processor Power Mac G5 @ 1.6GHz). Higher scores are better, with double the score indicating double the performance. - The benchmarks fall into one of four categories: + The benchmarks fall into one of four categories: - integer performance. - floating point performance. - memory performance.