mirror of
				https://github.com/ARM-software/workload-automation.git
				synced 2025-10-29 22:24:51 +00:00 
			
		
		
		
	doc/agenda: Add documentation about classifiers
This commit is contained in:
		| @@ -142,9 +142,9 @@ to 15 million. You can specify this using dhrystone's parameters: | ||||
|  | ||||
| In addition to configuring the workload itself, we can also specify | ||||
| configuration for the underlying device which can be done by setting runtime | ||||
| parameters in the workload spec. Explict runtime paremeters have been exposed for | ||||
| parameters in the workload spec. Explicit runtime parameters have been exposed for | ||||
| configuring cpufreq, hotplug and cpuidle. For more detailed information on Runtime | ||||
| Parameters see the :ref:`runtime parmeters <runtime-parameters>` section. For | ||||
| Parameters see the :ref:`runtime parameters <runtime-parameters>` section. For | ||||
| example, suppose we want to ensure the maximum score for our benchmarks, at the | ||||
| expense of power consumption so we want to set the cpufreq governor to | ||||
| "performance" and enable all of the cpus on the device, (assuming there are 8 | ||||
| @@ -343,6 +343,90 @@ parameters used. | ||||
|                   name: cyclictest | ||||
|                   iterations: 10 | ||||
|  | ||||
| .. _classifiers: | ||||
|  | ||||
| Classifiers | ||||
| ------------ | ||||
|  | ||||
| Classifiers can be used in 2 distinct ways, the first use is being supplied in | ||||
| an agenda as a set of key-value pairs which can be used to help identify sub- | ||||
| tests of a run, for example if you have multiple sections in your agenda running | ||||
| your workloads at different frequencies you might want to set a classifier | ||||
| specifying which frequencies are being used. These can then be utilized later, | ||||
| for example with the ``csv`` :ref:`output processor <output-processors>` with | ||||
| ``use_all_classifiers`` set to ``True`` and this will add additional columns to | ||||
| the output file for each of the classifier keys that have been specified | ||||
| allowing for quick comparison. | ||||
|  | ||||
| An example agenda is shown here: | ||||
|  | ||||
| .. code-block:: yaml | ||||
|  | ||||
|         config: | ||||
|             augmentations: | ||||
|                 - csv | ||||
|             iterations: 1 | ||||
|             device: generic_android | ||||
|             csv: | ||||
|                 use_all_classifiers: True | ||||
|         sections: | ||||
|             - id: max_speed | ||||
|               runtime_parameters: | ||||
|                   frequency: 1700000 | ||||
|               classifiers: | ||||
|                   freq: 1700000 | ||||
|             - id: min_speed | ||||
|               runtime_parameters: | ||||
|                   frequency: 200000 | ||||
|               classifiers: | ||||
|                   freq: 200000 | ||||
|         workloads: | ||||
|         -   name: recentfling | ||||
|  | ||||
| The other way that they can used is by being automatically added by some | ||||
| workloads to identify their results metrics and artifacts. For example some | ||||
| workloads perform multiple tests with the same execution run and therefore will | ||||
| use metrics to differentiate between them, For example the ``recentfling`` | ||||
| workload will use classifiers to distinguish between which loop a particular | ||||
| result is for or whether it is an average across all loops ran. | ||||
|  | ||||
| The output from the agenda above will produce a csv file similar to what is | ||||
| shown below. Some columns have been omitted for clarity however as can been seen | ||||
| the custom **frequency** classifier column has been added and populated, along | ||||
| with the **loop** classifier added by the workload. | ||||
|  | ||||
| :: | ||||
|  | ||||
|  id              | workload      | metric                    | freq      | loop    | value ‖ | ||||
|  max_speed-wk1   | recentfling   | 90th Percentile           | 1700000   | 1       | 8     ‖ | ||||
|  max_speed-wk1   | recentfling   | 95th Percentile           | 1700000   | 1       | 9     ‖ | ||||
|  max_speed-wk1   | recentfling   | 99th Percentile           | 1700000   | 1       | 16    ‖ | ||||
|  max_speed-wk1   | recentfling   | Jank                      | 1700000   | 1       | 11    ‖ | ||||
|  max_speed-wk1   | recentfling   | Jank%                     | 1700000   | 1       | 1     ‖ | ||||
|  # ... | ||||
|  max_speed-wk1   | recentfling   | Jank                      | 1700000   | 3       | 1     ‖ | ||||
|  max_speed-wk1   | recentfling   | Jank%                     | 1700000   | 3       | 0     ‖ | ||||
|  max_speed-wk1   | recentfling   | Average 90th Percentqile  | 1700000   | Average | 7     ‖ | ||||
|  max_speed-wk1   | recentfling   | Average 95th Percentile   | 1700000   | Average | 8     ‖ | ||||
|  max_speed-wk1   | recentfling   | Average 99th Percentile   | 1700000   | Average | 14    ‖ | ||||
|  max_speed-wk1   | recentfling   | Average Jank              | 1700000   | Average | 6     ‖ | ||||
|  max_speed-wk1   | recentfling   | Average Jank%             | 1700000   | Average | 0     ‖ | ||||
|  min_speed-wk1   | recentfling   | 90th Percentile           | 200000    | 1       | 7     ‖ | ||||
|  min_speed-wk1   | recentfling   | 95th Percentile           | 200000    | 1       | 8     ‖ | ||||
|  min_speed-wk1   | recentfling   | 99th Percentile           | 200000    | 1       | 14    ‖ | ||||
|  min_speed-wk1   | recentfling   | Jank                      | 200000    | 1       | 5     ‖ | ||||
|  min_speed-wk1   | recentfling   | Jank%                     | 200000    | 1       | 0     ‖ | ||||
|  # ... | ||||
|  min_speed-wk1   | recentfling   | Jank                      | 200000    | 3       | 5     ‖ | ||||
|  min_speed-wk1   | recentfling   | Jank%                     | 200000    | 3       | 0     ‖ | ||||
|  min_speed-wk1   | recentfling   | Average 90th Percentile   | 200000    | Average | 7     ‖ | ||||
|  min_speed-wk1   | recentfling   | Average 95th Percentile   | 200000    | Average | 8     ‖ | ||||
|  min_speed-wk1   | recentfling   | Average 99th Percentile   | 200000    | Average | 13    ‖ | ||||
|  min_speed-wk1   | recentfling   | Average Jank              | 200000    | Average | 4     ‖ | ||||
|  min_speed-wk1   | recentfling   | Average Jank%             | 200000    | Average | 0     ‖ | ||||
|  | ||||
|  | ||||
|  | ||||
| .. _sections: | ||||
|  | ||||
| Sections | ||||
|   | ||||
		Reference in New Issue
	
	Block a user