- cross-compiled revent binaries to match latest version (with recording timestamp fix f64aaf6 on 12 Oct 2018)
toolchains used:
gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabi
gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu
- fixes error in utils/revent.py when reading timestamps from recordings made with previous wa revent binaries
- Move the slide editing test into the main runWorkload instead of
setup.
- On some devices the folder picker has changed layout so add support for
navigating.
- Add support for differently capitalized splash buttons.
- Add workaround for adding a new slide if click the button doesn't work
the first time.
- Fix formatting
- Skip teardown automation if elements are not present on some devices
instead of failing the workload.
- Give extra time for start button to appear as some devices can be slow
to load.
Add additional check for text based directions button as id can be
missing on some devices and allow for skipping the view steps stage for
large screen devices which do not require this step.
In commit bb282eb19c devlibs
`escape_double_quotes` method was retired in favour of the `pipes.quote`
method however this does not format correctly for this purpose therefore
revert back to the original escaping method.
Previously the `Status` Enum was converted to a string as part of
serialization however now use the Enum `to_pod` method and make the
respective changes for de-serialization.
Update the postgres database schema:
- Rename "resourcegetters" schema to "resource_getters" for
consistency
- Rename "retreies" colum to "retry" to better relflect it purpose
- Store additional information including:
- POD serialization data
- Missing target information
- JSON formatted runstate
Ensure that Run structures now have serialization versions.
Also fix serialization/de-serialization of `Status` type as previously this
was formatted as a String instead a pod.
Add a new mix-in class for classes that are serialized to PODs, the aim
of this class is to provide a way to ensure that both the original data
version and the current serialization version are known. When attempting
to de-serialize a POD the serialization version will be compared to the
latest version in WA if not matching will call the appropriate method to
upgrade the pod to a known structure state populating any missing fields
with a sensible default or converting the existing data to the new
format.
On some devices the score string obtained can contain extra characters.
Only use the numerical values from the score when converting, otherwise
if not found set the result to 'NaN'.
By clearing the application data each time the workload is run this
forces the required assets to be re-installed each time. As the
workload is not affected by persistent state do not perform the
clearing.
Add format_version class attribute to TargetInfo to track format
changes. This is checked when deserializing from POD to catch format
changes between cached and obtained TargetInfo's.
Previously run level artifacts would be added with a particular job_id,
and updated artifacts would be stored as new objects each time. Refactor
to remove unnecessary instance variables, only provide a job_id when
required and add an update capability for largeobjects to ensure this
does not happen.
In Android Q a popup will be displayed warning if the application has
not been designed for the latest version of android. This has currently been
dealt with on a per workload basis however this is a common popup so
add a method to dismiss the popup if present to the base class.
While attempting to run vellamo on Android Q, a popup warning with
the message, "This app was built for an older version of Android and may not
work properly. Try checking for updates, or contact the developer." would
appear, causing the workload to halt.
Close the popup warning before dismissing EULA and executing the remaining
steps to run vellamo.
Tested with vellamo apk version 3.2.4.
Signed-off-by: Lisa Nguyen <lisa.nguyen@linaro.org>
- force cast start/end timestamps to uint64_t to correct recording format issue on 32bit devices (i.e. 4 bytes timespec tv_sec written on 8 bytes memory slot)
Previously the LogcatRunMonitor left the logcat process running in the
background causing issues with concurrent accesses. Now ensure the thread
terminates correctly.
The RunInfo object in the run output is initally created before the
config has been fully parsed therefore attributes for the project and
run name are never updated, once the config has been finalized make sure
to update the relavant information.
- change magic string literal to a b'' string so that the comparison
works in python 3
- expand timestamp tuples (struct.unpack always returns a tuple) before
attempting to cast to float.
For workloads that support Linux targets do not use
`uninstall_executable` as this is not available, instead use `uninstall` as
other targets should be able to determine the appropriate uninstallation
method.
Previously the workload assumed that all the scores were visible on a
single screen however for devices with smaller displays the results need
to scrolled.
A new popup appears when running pcmark on android Q that complains
about the app being built for an older version of android.
Since this popup will be temporary, the fix has to make sure not to
break in the future when this popup disappears or when the test is ran
on a compatible version of android.
To achieve this, we attempt to dismiss the popup and if we timeout we
silently carry on with the test assuming no popup will appear.
Signed-off-by: Qais Yousef <qais.yousef@arm.com>
Creating a new workload to execute the following tests on GFXBench.
* Car Chase
* Car Chase Offscreen
* Manhattan 3.1
* 1080p Manhattan 3.1 Offscreen
* 1440p Manhattan 3.1 Offscreen
* Tessellation
* Tessellation Offscreen
Now allows for specifying a `group` value for each section which will
cross product the sections within that group with the sections in each
other group. Additionally classifiers will automatically be added to
each job spec with the relevant group information.
In the case of a failure in the initialization of one output_processor the
remaining `initialize` methods may not get called causing variables to
not be initialized correctly.
The 'jsonB' datatype was only added in v9.4 so ensure that the Postgres
server to is running this or later and inform the user if this is not
the case.
The Output processor which is used to upload the results found in the
wa_output folder to a Postgres database, whose schema is defined by the
WA Create Database command.
Add a command to create a PostgreSQL database with supplied parameters
which draws its structure from the supplied schema (Version 1.1). This
database is of a format intended to be used with the forthcoming WA
Postgres output processor.
Leftover code from WA2 meant that logcat was cleared on the device by
the workload directly instead of using devlib, this caused issues if logcat was
still being cleared from other areas of the code.
Expose the context to the initialize and finalize functions for Output
Processors. This was found to be necessary for the upcoming PostgreSQL
Output Processor.
Previously when using the create command for adding
EnergyInstruemntBackends they were treated like any other plugin and
generated incorrect configuration. Now automatically add the
`energy_measurement` instrument and populate it's configuration with the
relevant defaults for the specified Backend.
In commit d2ece we are now tracking augmentations which are used during a
run in the run_config via the context when installing augmentations.
Update the Process command and its ProcessContext with a dummy method
to relect this change.
Ensure that a plug-ins config entry is converted to an identifier before being
stored in the PluginCache so that the relevant configuration can be
retrieved appropriately. For example this allows for both 'trace-cmd' and
'trace_cmd' to be used as config entries to provide configuration for the
'trace-cmd' plugin.
In commit 8da911 the initialization of the target was split into a
separate method of the TargetManger. Ensure that we call the relevant
method after creating the manager.
Add Context.write_config() to write the combined config into run output
__meta. Use it after instruments and result processors get installed to
make sure their configuration gets serialized in the output.
- Rename "instrument_name" to "instrument" inside do_execute(), as
ConfigManger.get_instrument() returns a list of Instrument objects,
not names.
- To avoid name clash, rename the imported instrument module to
"instrumentation".
- Change how "source" and "dest" are handled inside merge() to be more
sane and less confusing, ensuring that disabling toggles are merged
correctly.
- Do not drop disabling values during merge, to ensure that merging
is a transitive operation.
- Add unit tests for the above fixes.
On some devices running Android 8.1 the start benchmark button was
failing to be clicked, this is a workaround to click on the coordinates
of the button instead of the UiObject iteslf.
- Add copyright headers to files that did not already have them
- Update the year to the last year the file was modified in files with
existing headers.
Remove unused ``platform`` attributes and add ``supported_platforms``
along with check so that workloads can fail early when attempting to run
on an incorrect platform.
Mongoperf currently has a non-ASCII character in the file that causes a failure.
This commit rewords the description to remove the incriminating character.
Add a workload that runs mongoperf benchmark that measures I/O
performance on a MongoDB server. This workload assumes that mongoperf
is already installed.
Consider headers when calculating column widths. This will ensure proper
alignment in cases where the column header is wider than any of the
column values.
When installing from source WA attempts to include the commit ID in the
version of the installed pacakge however this caused issues with package
managers like pip. PEP440 specifies that local identifiers must be in the
form `<public version identifier>[+<localversion label>]` so update the
version to conform.
https://www.python.org/dev/peps/pep-0440/#local-version-identifiers
Lengthen timeout for gemm and conv tests to be the same as for sparse
test. While the former two usually take a lot less time, their execution
time will vary significantly depending on the target and the runtime
environment (e.g. cpu frequencies might be forced to lowest values).
urllib2 does not exist in Python 3, and its methods have been moved into
urllib.request. Use future library to create aliases that work across
both 2 and 3.
Add a workload for lat_mem_rd and bw_mem tests of LMbench benchmark, a suite of
portable ANSI/C microbenchmarks for UNIX/POSIX.
This is a port of corresponding workload from WA2.
Ensure the serialized state gets written out in case of early abort
(before the run has been fully initialized and job execution has
started), and that the appropriate final output is generated.
This requires that ExecutionContext is created much earlier (in order
for postamble to be possible), and that KeyboardInteruptException is
caught and handled properly by the Executor.
Add a convenience method for invoking write_info(), write_state(), and
write_result() of run_output. This will make it easier to call it from
multiple places.
TargetManger.get_target_info() gets invoked twice: first on context
creation, and then a second lime shortly afterwards to populate
RunOutput. Change target_info attribute inside the context to be a
property that redirects to run_output. This will allow the context to be
created earlier, before a target connection is available.
Add run.log as an Artifact to the output, once it has been created.
Every file outsize of __meta (excepting result.json) should have an
associated artifact to describe it.
Add wa and devlib versions to RunOutput's metadata on creation. They
already get logged, but this will make them readily accessible to
scripts and output processors.
When iterating over PLATFORMS tuples, account for the
platform_target_defaults introduced by 7470fb5ee.
Also, correct the tuple description in the comments.
The ValueError does not given enough information in the case of an empty file
extension. Add a special case to handle files with no extension and
prompt the user.
Call resolver's get() method, rather than get_resource(), incorrectly
introduced in the previous commit. Also add get() alias to the context
to allow it to be used as a resolver.
Since 2ff06af63, resources are obtained via context.get_resource, rather
than context.resolve.get_resource. LightContext used by the record
command was not updated to implement that method. This fixes the issue
Allow specifying "augmentations" and "all" as the plugin kind to lost.
In the case of of the former, instruments and output processors get
listed. In the case of the latter, every plugin kind gets listed.
This should have been handled by the @total_ordering decorator, but
isn't due to
https://bugs.python.org/issue25732
(briefly, total_ordering is back-ported from Python 3, where the base
object provides the default implementation of __ne__ based on __eq__, so
total_ordering did not override it; this, however does not happen in
Python 2).
Also update unit tests to catch this edge case.
During Python 3 migration, the izip call inside diff_interrupt_files has
been replaced with a zip call (zip returning an iterator in Python 3).
Import zip from builtins to ensure that it also produces an iterator in
Python 2.
Also fix the associated unit test by importing the correct function.
This instrument will log serial traffic to a file. The idea is to
assist with debugging development boards, as otherwise there is a
chance that crash reports, for example, are lost when the device is
reset by WA.
It logs the pre-run and post-run traffic to preamble_serial.log and
postamble_serial.log , respectively, in the top-level output
directory. Serial logs for each workload are stored in the appropriate
workload directory as serial.log.
As we are tying up the serial port, we stop the logging when the
device is reset to allow board such as the Arm Juno to be reset via
the serial connection.
Python 3 has changed the behavior of subparsers so that they are no
longer mandatory by default. As a result, executing just "wa" under
Python 3 results in a random error, rather than a help message.
Fix this by making the subparsers mandatory.
Certain devices present a night time reading popup alert when opening the books workload.
This can throw off the workload so I've added a function to check for this and dismiss if neccessary.
Add a decorator to indicate that a callback runs entirely host-side and
does not rely on a connection to the target. This means it will be
invoked even if the target was detected to be unresponsive.
- Add "hard" parameter to TargetManager's reboot().
- Pass in context to TargetManager's reboot() and
verify_target_responsive() -- this is needed for signal dispatch.
- Call self.reboot() instead of self.target.reboot() when attempting a
reset for unresponsive targets inside TargetManager, in order to
ensure that appropriate signals are dispatched.
Fix a regression introduced with Python 3 port -- JSON deserializer
should check for basestring rather than str when deciding whether to try
to decode a custom type.
In a previous commit we filtered parameters so that '-h' is never passed
to 'parse_known_commands' to ensure our own custom help message is displayed,
now also filter for '--help' for the same reason.
Previously incorrect method calls were used to try and convert 'max' and
'min' into their corresponding frequencies, now call the correct methods
and refactor the conversion into a separate method.
Modify execution so that output processors' process_job_output() (but
not export_job_output()) is wrapped by the JOB_OUTPUT_PROCESSED signals.
This makes sense conceptually, and is more useful in practice, as there
are already WORKLOAD_RESULT_EXTRACTION and WORKLOAD_OUTPUT_UPDATE
signals sent by the job, if it's desirable to run before output
processors, but after the job results have been collected.
Since daq-server has been split out into a separate package, daq setup
is no longer part of WA documentation. This resulted in a warning when
building the docs.
As part of resolving a resource, record its MD5 hash in the output
metadata. In order to enable this, resource resolution is now done via
the context, rather than directly via the ResourceResolver (in order not
to introduce a context dependency in the Gui object, context now
emulates the resolver interface).
Metadata is a key-value mapping for arbitrary data, similar to
classifiers. Unlike classifiers, metadata does not directly relate to
the results of the execution, but to the execution itself, and typically
would not be processed by Output Processors in the same way as
classifiers. Metadata can also be a lot more free-form in it's value
structure; while classifier values are simple scalars, metadata values
can be arbitrary POD structures.
If the device is to perform an initial reboot, allow up to `max_retries`
from the run configuration for the reboot to succeed and add logging to
inform the user.
Treat the optional reboot of the device as part of the job itself so that
if the reboot fails it will be retried in accordance to the job
configuration.
Instead of starting and stopping the target from inside the Context,
move the calls to the Runner. This allows for the any initilizaton
errors to be dealt with as part of the job and retried as
appropriate.
