Use the context manager to simplify the file transfer cache management.
Also introduce devlib.utils.misc.nullcontext() mirroring the behavior of
contextlib.nullcontext().
When globbing=True, the source is interpreted as a globbing pattern on
the target and is expanded before pulling the files or folders.
This also aligns the behaviour of all targets:
* adb connection was supported a limited form of globbing by default
(only on the last component of the path)
* SCP was supporting a limited form of globbing
* GEM5 was not supporting globbing at all
* paramiko was not supporting globbing at all
Also fix a race condition on push/pull as root, where pushing/pulling
the same file from multiple threads would have ended up using the same
temporary file.
Create a new temporary directory to use when extracting apk methods,
as when running multiple processes in parallel the extracted files could
be overwritten by each other.
As the exoplayer workload did not specify a pre-existing logfile, it is
created for it by default in LogcatMonitor. This default method opens
the logfile in 'byte' mode rather than the expected 'string' mode.
Regex operations that depend on the logfile for event triggering expect it to
be in 'string' mode, which was not the case.
aapt is now depreciated in favour of aapt2 therefore prefer using the
newer binary if it is found on the system. If not present fallback to
the old implementation.
Currently all invocations of aapt within devlib are compatible with
aapt2 however expose the `aapt_version` attribute to indicate which
version has been selected to allow for allow maintaining future
compatibility.
No longer recursively attempt to push a file to the target. If the
second attempt goes wrong we assume there is something else wrong and
therefore let the error propagate.
Also log the original error in case it is not the error we were
expecting.
logcat -v lets you specify the format for the logcat output. Add a
parameter to the LogcatCollector to allow us to pass that information
down to the logcat invocation.
Paramiko uses sftp for file transfer rather then scp as with the
previous implementation however not all targets support this.
Expose a parameter to the SSHConnection to allow falling back to
the scp implementation.
In the case of connecting to a system without a password the password
parameter needs to be set to an empty string which will currently still
cause ssh keys to be loaded. Only check for ssh keys when the password
is explicitly set to `None`.
Allow for configuring the number of connection attempts that will be
made to the device before failing to connect. This allows for waiting longer
periods of time for the device to come online.
Unless we tell pexpect to expect something it will not read from the
process' buffer, or write anything to the logfile. If we follow the
collector instructions from devlib's documentation:
In [1]: from devlib import AndroidTarget, LogcatCollector
In [2]: t = AndroidTarget()
# Set up the collector on the Target.
In [3]: collector = LogcatCollector(t)
# Configure the output file path for the collector to use.
In [4]: collector.set_output('adb_log.txt')
# Reset the Collector to preform any required configuration or
# preparation.
In [5]: collector.reset()
# Start Collecting
In [6]: collector.start()
# Wait for some output to be generated
In [7]: sleep(10)
# Stop Collecting
In [8]: collector.stop()
# Retrieved the collected data
In [9]: output = collector.get_data()
adb_log.txt will be empty because between collector.start() and
collector.stop() there were no expect() calls to
LogcatMonitor._logcat. As the get_log() function already has code to
flush the log, abstract it to a function and call it in stop() before
terminating the pexpect.spawn().
By default, pexpect.spawn() is a bytes interface: its read methods
return bytes and its write/send and expect method expect bytes. Logcat
uses utf-8 strings, so put pexpect.spawn() in str mode.
This code can't have worked in python3 before, since the logcat file
is not opened in "b" mode.
If an error occurs while executing a command the `output` and `error`
variables may not get initialised, only attempted to decode their
contents if this is not the case.
check_output(combined_output=True) does not guarantee that stdout will
come before stderr, but the ordering is needed in case check_exit_code
is True, as we are expecting the exit code at the end of stdout.
Furthermore, the exceptions can't report what is stdout and what is
stderr as they are combined.
Partially revert 77a6de9453 ("utils/android: include stderr in adb_shell
output") and parse output and err independently. Return them combined
from adb_shell() to keep the functionality that 77a6de9453 was
implementing.
d4b0dedc2a ("utils/misc: add combined output option to
check_output") adds an option that combines stdout and stderr, but
their order is arbitrary (stdout may appear before or after
stderr). This leads to problems in adb_shell() when it tries to check
the error code. Now that adb_shell() doesn't use combined_output,
remove the option as there are no more users in devlib.
squash! utils/misc: Make the return of check_output consistent
By default parmaiko attempts to search for SSH keys even when connecting
with a password. Ensure this is disabled to prevent issues where
non-valid keys are found on the host when connecting using password
authentication.
* Unify the behavior of background commands in connections.BackgroundCommand().
This implements a subset of subprocess.Popen class, with a unified behavior
across all connection types
* Implement the SSH connection using paramiko rather than pxssh.
Update a command line to redirect standard streams as specified using the
parameters. This helper allows honoring streams specified in the same way as
subprocess.Popen, by doing it as much using shell redirections as possible.
Similar to a regular property(), with the following differences:
* Values are memoized and are threadlocal
* The value returned by the property needs to be called (like a weakref) to get
the actual value. This level of indirection is needed to allow methods to be
implemented in the proxy object without clashing with the value's methods.
* If the above is too annoying, a "sub property" can be created with the regular
property() behavior (and therefore without the additional methods) using
tls_property.basic_property .
Use the timeout parameter added in Python 3.3, which removes the need for the
timer thread and avoids some weird issues in preexec_fn, as it's now documented
to sometimes not work when threads are involved.
The `AndroidTarget` class should not depend on ADB specific commands as
is is possible to use this target with other connection types e.g. ssh.
Therefore move the adb specific commands into the `AdbConnection`.
- `wait_for_device` and `reboot_bootloader` are now exposed in AndroidTarget
as generic methods and call through to the connection method.
- `adb_kill_server` is now a standalone function of the AdbConnection.
The `AndroidTarget` would ensure that when connecting to a IP target
that it disconnected first to prevent the connection getting stuck if
the connection was not closed correctly. Move this code into the
`AdbConnection` instead as this is more relevant.
The user can pass a dictionary containg the key and value
pairs with the extra ssh configuration options. Multiple
options will be passed as '-o key1=value1 -o key2=value2'
Signed-off-by: Michalis Spyrou <michalis.spyrou@arm.com>
On newer devices dumpsys output is more explanatory and does not only
contain numerical data. Update the parser to ignore non numerical
data for example the arguments that were passed and section headers.
To allow for connecting to an `AndroidTarget` as root before the target
has been initialised, allow for passing `adb_as_root` as a connection
parameter to the `AdbConnection`. This will restart `adbd` as root
before attempting to connect to the target and will restart as unrooted
once all connections to that target have been closed.
Add a method to `AdbConnection` to control whether whether adb is
connected as root. This allows for the connection to track whether it is
connected as root for a particular device across all instances of a
connection.
Improve the detection of being `connected_as_root` from comparing the
username to checking the actual id of the user and export this as a
property for the connection.
The fix in commit 964fde2 caused issues with certain command structures,
for example running in the background. To prevent this run the original
command as a subshell.
When executing a command using `su`, the `echo` command was returning the
error code of the invocation of `su` rather than the command itself.
Usually `su` should mimic the return code of the command it is executing
however this is not always the case which can cause issues.
Commit 89c40fb switched from using `echo CMD | su` to `su -c CMD`
however not all modern versions of `su` support this format.
Automatically try and detect if the new style is supported when
connecting and if not, fall back to the old implementation.
