Before exporting output to ensure that we are still connected to the
database. The connection may be dropped so reconnect if necessary, this
is a more of an issue with longer running jobs.
To provide the same user experience of accessing a directory
artifact from a standard `wa_output` when attempting to retrieve the
path of the artifact extract the stored tar file and extract it to a
temporary location on the host returning the path.
Reflecting the addition to being able to store directories as Artifacts
enable uploading of a directory as a compressed tar file rather than
storing the file directly.
In the case of an empty iterable an empty string would be returned
however this was not an valid value so ensure that the brackets are
always inserted into the output.
Check what the default encoding for the system is set to. If this is not
configured to use 'UTF-8', log a warning to the user as this is known
to cause issues when attempting to parse none ascii files during operation.
Previously the `setScreenOrientation` function only accepted relative
orientations, this causes issue when attempt to use across tablets and
phones with different natural orientations. Now take into account the
current orientation and screen resolution to allow specifying portrait vs
landscape across different types of devices.
Ensure that the appropriate error message is returned to the user to
outline what caused the version matching to fail.
Additionally fix the case where if specifying a package name directly
the version matching result would be ignored.
`ApkRevent` workloads should be able to use the same Apk selection
criteria as `ApkWorkloads` therefore rely on the superclass to
instantiate the `PackageHandler`.
In additional to being able to specify whether the APK should be
uninstalled as part of a `APKWorkload`s teardown add the `uninstall`
parameter to the base `workload` class in order to specify whether any
binaries installed for a workload should be uninstalled again.
Add a workload that launches UIBenchJankTests. This differs from the
UIBench application as it adds automation and instrumentation to that
APK. This therefore requires a different implementation than classical
ApkWorkloads as 2 APKs are required (UIBench and UIBenchJankTests) and
the main APK is invoked through `am instrument` (as opposed to `am
start`).
Add support for Android applications that are invoked through `am
instrument` (as opposed to `am start`) _i.e._ that have been
instrumented. See AOSP `/platform_testing/tests/` for examples of such
applications.
If no jobs have been generated that are available for running then WA
will crash when trying to access the job queue. Add an explicit check to
ensure that a sensible error is raised in this case, for example if
attempting to run a specific job ID that is not found.
If the activity field of an instance of ApkWorkload does not the '.'
character, it is assumed that it is in the Java namespace of the
application. This is similar to how activities can be referred to with
relative paths:
com.domain.app/.activity
instead of
com.domain.app/com.domain.app.activity
Include missing `flatten_mapping` call in our implementation of
`construct_mapping`. This is performed by a subclass in the default
implementation which was missing in our previous fix.
The workload could attempt to use the version attribute before it was
discovered to assess the workload activity causing an error however the
whole process can be simplified using newer discovery features.
When attempting to access the message of a exception check not only that
e.args is populated, but also that e.args[0] actually contains
something, before defaulting to str(e).
Update checking of attributes to allow for empty structures as they can
be set to empty lists etc. and therefore should not be checking if
explicitly `None`.
In newer versions of PyYAML we need to manually specify the `Loader` to
be used as per https://msg.pyyaml.org/load.
`FullLoader` is now the default loader which attempts to avoid arbitrary
code execution, however if we are running an older version where this is
not available default back to the original Loader.
Due to mismatches in WA and devlib versions this previous upgrade method
could have been trigger before it was needed and would not be called a
second time. Now we can be sure that WA and devlib are updated together
bump the version number again to ensure the upgrade method is called a
second time to ensure the POD is upgraded correctly.
To ensure that a compatible version of devlib is installed on the system
keep track of the version of devlib that is required by WA and provide a
more useful error message if this is not satisfied.
Some tablet devices use an alternate tab switching method due to the
larger screen space. Add support for adding new tabs via the menu
instead of via the tab switcher.
No longer specify a default version to allow any available apks to be
detected and then choose the appropriate automation based on the
detected version.
Refactor to support new supported_versions attribute and since APK
resolution needs to have happened before setting uiauto parameter
move assignments to ``initialize``.
Allow for specifying a list of supported APK versions for a workload. If
a specific version is no specified then attempt to a resolve any valid
version for the workload.
If a workloads `version` attribute is not set, and an APK file is
found, use this as the version number. This allows for workloads to not
specify a default version via parameters and for an available APK to be
automatically chosen.
The Devlib KernelConfig object was refactored in commit
f65130b7c7
therefore update the way KernelConfig objects are deserialized to reflect the new
implementation and provide a conversion for PODs.
Allow for easy configuring of a view for a particular workload as this
can vary depending on the device which can be used when using certain
instruments for example `fps`.
- cross-compiled revent binaries to match latest version (with recording timestamp fix f64aaf6 on 12 Oct 2018)
toolchains used:
gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabi
gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu
- fixes error in utils/revent.py when reading timestamps from recordings made with previous wa revent binaries
- Move the slide editing test into the main runWorkload instead of
setup.
- On some devices the folder picker has changed layout so add support for
navigating.
- Add support for differently capitalized splash buttons.
- Add workaround for adding a new slide if click the button doesn't work
the first time.
- Fix formatting
- Skip teardown automation if elements are not present on some devices
instead of failing the workload.
- Give extra time for start button to appear as some devices can be slow
to load.
Add additional check for text based directions button as id can be
missing on some devices and allow for skipping the view steps stage for
large screen devices which do not require this step.
In commit bb282eb19c devlibs
`escape_double_quotes` method was retired in favour of the `pipes.quote`
method however this does not format correctly for this purpose therefore
revert back to the original escaping method.
Previously the `Status` Enum was converted to a string as part of
serialization however now use the Enum `to_pod` method and make the
respective changes for de-serialization.
Update the postgres database schema:
- Rename "resourcegetters" schema to "resource_getters" for
consistency
- Rename "retreies" colum to "retry" to better relflect it purpose
- Store additional information including:
- POD serialization data
- Missing target information
- JSON formatted runstate
Ensure that Run structures now have serialization versions.
Also fix serialization/de-serialization of `Status` type as previously this
was formatted as a String instead a pod.
Add a new mix-in class for classes that are serialized to PODs, the aim
of this class is to provide a way to ensure that both the original data
version and the current serialization version are known. When attempting
to de-serialize a POD the serialization version will be compared to the
latest version in WA if not matching will call the appropriate method to
upgrade the pod to a known structure state populating any missing fields
with a sensible default or converting the existing data to the new
format.
On some devices the score string obtained can contain extra characters.
Only use the numerical values from the score when converting, otherwise
if not found set the result to 'NaN'.
By clearing the application data each time the workload is run this
forces the required assets to be re-installed each time. As the
workload is not affected by persistent state do not perform the
clearing.
Add format_version class attribute to TargetInfo to track format
changes. This is checked when deserializing from POD to catch format
changes between cached and obtained TargetInfo's.
Previously run level artifacts would be added with a particular job_id,
and updated artifacts would be stored as new objects each time. Refactor
to remove unnecessary instance variables, only provide a job_id when
required and add an update capability for largeobjects to ensure this
does not happen.
In Android Q a popup will be displayed warning if the application has
not been designed for the latest version of android. This has currently been
dealt with on a per workload basis however this is a common popup so
add a method to dismiss the popup if present to the base class.
While attempting to run vellamo on Android Q, a popup warning with
the message, "This app was built for an older version of Android and may not
work properly. Try checking for updates, or contact the developer." would
appear, causing the workload to halt.
Close the popup warning before dismissing EULA and executing the remaining
steps to run vellamo.
Tested with vellamo apk version 3.2.4.
Signed-off-by: Lisa Nguyen <lisa.nguyen@linaro.org>
- force cast start/end timestamps to uint64_t to correct recording format issue on 32bit devices (i.e. 4 bytes timespec tv_sec written on 8 bytes memory slot)
Previously the LogcatRunMonitor left the logcat process running in the
background causing issues with concurrent accesses. Now ensure the thread
terminates correctly.
The RunInfo object in the run output is initally created before the
config has been fully parsed therefore attributes for the project and
run name are never updated, once the config has been finalized make sure
to update the relavant information.
- change magic string literal to a b'' string so that the comparison
works in python 3
- expand timestamp tuples (struct.unpack always returns a tuple) before
attempting to cast to float.
For workloads that support Linux targets do not use
`uninstall_executable` as this is not available, instead use `uninstall` as
other targets should be able to determine the appropriate uninstallation
method.
Previously the workload assumed that all the scores were visible on a
single screen however for devices with smaller displays the results need
to scrolled.
A new popup appears when running pcmark on android Q that complains
about the app being built for an older version of android.
Since this popup will be temporary, the fix has to make sure not to
break in the future when this popup disappears or when the test is ran
on a compatible version of android.
To achieve this, we attempt to dismiss the popup and if we timeout we
silently carry on with the test assuming no popup will appear.
Signed-off-by: Qais Yousef <qais.yousef@arm.com>
Creating a new workload to execute the following tests on GFXBench.
* Car Chase
* Car Chase Offscreen
* Manhattan 3.1
* 1080p Manhattan 3.1 Offscreen
* 1440p Manhattan 3.1 Offscreen
* Tessellation
* Tessellation Offscreen
Now allows for specifying a `group` value for each section which will
cross product the sections within that group with the sections in each
other group. Additionally classifiers will automatically be added to
each job spec with the relevant group information.
In the case of a failure in the initialization of one output_processor the
remaining `initialize` methods may not get called causing variables to
not be initialized correctly.
The 'jsonB' datatype was only added in v9.4 so ensure that the Postgres
server to is running this or later and inform the user if this is not
the case.
The Output processor which is used to upload the results found in the
wa_output folder to a Postgres database, whose schema is defined by the
WA Create Database command.
Add a command to create a PostgreSQL database with supplied parameters
which draws its structure from the supplied schema (Version 1.1). This
database is of a format intended to be used with the forthcoming WA
Postgres output processor.
Leftover code from WA2 meant that logcat was cleared on the device by
the workload directly instead of using devlib, this caused issues if logcat was
still being cleared from other areas of the code.
Expose the context to the initialize and finalize functions for Output
Processors. This was found to be necessary for the upcoming PostgreSQL
Output Processor.
Previously when using the create command for adding
EnergyInstruemntBackends they were treated like any other plugin and
generated incorrect configuration. Now automatically add the
`energy_measurement` instrument and populate it's configuration with the
relevant defaults for the specified Backend.
In commit d2ece we are now tracking augmentations which are used during a
run in the run_config via the context when installing augmentations.
Update the Process command and its ProcessContext with a dummy method
to relect this change.
Ensure that a plug-ins config entry is converted to an identifier before being
stored in the PluginCache so that the relevant configuration can be
retrieved appropriately. For example this allows for both 'trace-cmd' and
'trace_cmd' to be used as config entries to provide configuration for the
'trace-cmd' plugin.
In commit 8da911 the initialization of the target was split into a
separate method of the TargetManger. Ensure that we call the relevant
method after creating the manager.
Add Context.write_config() to write the combined config into run output
__meta. Use it after instruments and result processors get installed to
make sure their configuration gets serialized in the output.
- Rename "instrument_name" to "instrument" inside do_execute(), as
ConfigManger.get_instrument() returns a list of Instrument objects,
not names.
- To avoid name clash, rename the imported instrument module to
"instrumentation".
- Change how "source" and "dest" are handled inside merge() to be more
sane and less confusing, ensuring that disabling toggles are merged
correctly.
- Do not drop disabling values during merge, to ensure that merging
is a transitive operation.
- Add unit tests for the above fixes.
On some devices running Android 8.1 the start benchmark button was
failing to be clicked, this is a workaround to click on the coordinates
of the button instead of the UiObject iteslf.
- Add copyright headers to files that did not already have them
- Update the year to the last year the file was modified in files with
existing headers.
Remove unused ``platform`` attributes and add ``supported_platforms``
along with check so that workloads can fail early when attempting to run
on an incorrect platform.
Mongoperf currently has a non-ASCII character in the file that causes a failure.
This commit rewords the description to remove the incriminating character.
Add a workload that runs mongoperf benchmark that measures I/O
performance on a MongoDB server. This workload assumes that mongoperf
is already installed.
Consider headers when calculating column widths. This will ensure proper
alignment in cases where the column header is wider than any of the
column values.
When installing from source WA attempts to include the commit ID in the
version of the installed pacakge however this caused issues with package
managers like pip. PEP440 specifies that local identifiers must be in the
form `<public version identifier>[+<localversion label>]` so update the
version to conform.
https://www.python.org/dev/peps/pep-0440/#local-version-identifiers
Lengthen timeout for gemm and conv tests to be the same as for sparse
test. While the former two usually take a lot less time, their execution
time will vary significantly depending on the target and the runtime
environment (e.g. cpu frequencies might be forced to lowest values).
urllib2 does not exist in Python 3, and its methods have been moved into
urllib.request. Use future library to create aliases that work across
both 2 and 3.
Add a workload for lat_mem_rd and bw_mem tests of LMbench benchmark, a suite of
portable ANSI/C microbenchmarks for UNIX/POSIX.
This is a port of corresponding workload from WA2.
Ensure the serialized state gets written out in case of early abort
(before the run has been fully initialized and job execution has
started), and that the appropriate final output is generated.
This requires that ExecutionContext is created much earlier (in order
for postamble to be possible), and that KeyboardInteruptException is
caught and handled properly by the Executor.
Add a convenience method for invoking write_info(), write_state(), and
write_result() of run_output. This will make it easier to call it from
multiple places.
TargetManger.get_target_info() gets invoked twice: first on context
creation, and then a second lime shortly afterwards to populate
RunOutput. Change target_info attribute inside the context to be a
property that redirects to run_output. This will allow the context to be
created earlier, before a target connection is available.
Add run.log as an Artifact to the output, once it has been created.
Every file outsize of __meta (excepting result.json) should have an
associated artifact to describe it.
