1
0
mirror of https://github.com/ARM-software/devlib.git synced 2025-01-31 02:00:45 +00:00

1155 Commits

Author SHA1 Message Date
Marc Bonnici
77f0b1f06d docs: Add readthedocs config and requirements.txt
The default versions used for sphinx and docuilts on
readthedocs are no longer compatible. Explicitly list
the package versions that should be used when building
the documentation.
2021-10-28 11:03:08 +01:00
Douglas Raillard
b4c76007c8 ftrace: Do not memoize FtraceCollector.available_events
ftrace events can be added and removed dynamically by kernel modules, so
they cannot be memoized.
2021-10-27 11:56:23 +01:00
Douglas Raillard
3f92d92a3f ssh: Reduce number of opened channels
SSH servers seem to have a maximum number of opened channels, after
which paramiko will raise an exception:

    Could not open an SSH channel: ChannelException(2, 'Connect failed')

Memoizing the SFTPClient object based on the timeout setting leads to
many opened sessions, since the timeout is typically adjusted, e.g. to
match the size of the file when pulling an ftrace trace.dat file.

Solve that by memoizing the SFTPClient based only on the connection
object, with a maximum number of 1 cached object, and update its timeout
setting inplace.
2021-10-15 16:10:02 +01:00
Douglas Raillard
0e0417c6b3 target: Fix undefined function
Remove references to non-existent dst_path_exists() function in
target.py
2021-10-11 14:42:25 +01:00
Douglas Raillard
e979bafb50 target: Speedup Target.write_value()
Avoid an execute() by doing the check in the same command. This also
allows to return early if the write is fast, and to extend for longer if
the write is slow. The speed at which you can observe a write in sysfs
depends on the backing kernel handlers, so there is a wide variety of
situations.

Also, make a more fine grained error detection by allowing the write
itself to fail, which can happen when writing invalid values to sysfs.
2021-10-08 18:22:04 +01:00
Douglas Raillard
0c1878786b Target.execute(): Add .returncode and .output exception attributes
Add Target{Stable,Transient}CalledProcessError exceptions, with an
.returncode and .output attributes, raised by Target.execute(),
mirroring subprocess.CalledProcessError.

This is very useful in client code that uses "exit N" to signal an
abnormal condition, and then inspects the output to find out more.
2021-10-08 18:22:04 +01:00
Douglas Raillard
ff57e785f8 exception: Deal with missing _message attribute
Allow subclasses of DevlibError to not have a _message attribute, in
which case it falls back on str(self), just as when _message is None.
2021-10-08 18:22:04 +01:00
Douglas Raillard
f1c8ca1a66 utils.android: Separate stdout and stderr by newline
Following what the other connections are doing, make the android
connection separate stdout and stderr by a newline.
2021-10-08 18:22:04 +01:00
Douglas Raillard
b719808ef2 target: Add Target.pull(via_temp=False) parameter
Allow pulling a file via a temporary location on the target, to
side-step performance issues when pulling big files from sysfs.
2021-10-08 13:14:42 +01:00
Douglas Raillard
477e82c444 target: Open the possiblity for optimizing multi-source push/pulls 2021-10-08 13:14:42 +01:00
Douglas Raillard
173df18f29 target: Factorize push/pull path resolution
Handle in one place the decision of what is the real destination of each
file in push/pull operations.

The following can now be assumed by the connection:

    * The destination does not exist.
    * The folder containing the destination does exist.

This ensures consistent errors and behaviors across all connection
types, at the cost of:

    * At least an extra execute() per source (up to 2 if the destination
      is a file that needs to be removed to make room).
    * For now, globbing will lead to a separate request for each file,
      rather than a merged one. This is because the destination of each
      source is prepared so that the connection will not have any
      interpretation work to do.
2021-10-08 13:14:42 +01:00
Douglas Raillard
79be8bc5ad ssh: Memoize the SshConnection._get_sftp()
Since we have the guarantee to have a different SshConnection per
thread, we can memoize paramiko's SFTPClient. This provides a great
performance boost.
2021-10-08 13:14:42 +01:00
Douglas Raillard
55d914bf93 target: Fix typo in Target._prepare_xfer 2021-10-08 13:14:42 +01:00
Douglas Raillard
528d3d4e0f target: Fix fallback path for kernel config loading
The main path is reading /proc/config.gz. If it does not exists, the
following paths are tested:
'/boot/config', '/boot/config-$(uname -r)'

Since the 2nd path contains a command to be executed, remove quoting of
the path when using "cat".
2021-09-30 13:20:33 +01:00
Peter Collingbourne
d6a2ed8247 Teach the script to understand "dumpsys power" output from newer Android versions.
Newer Android versions do not include any of the attributes that we're
currently looking for in the "dumpsys power" output. Instead they have
"mWakefulness" which can either be "Asleep" or "Awake". Adjust the regex to
look for that attribute as well.
2021-09-28 09:34:16 +01:00
Douglas Raillard
917800ffa6 collector: ftrace: Use trace-cmd start --cmdlines-size
Use --cmdlines-size option rather than setting the file manually.
2021-09-01 17:38:49 +01:00
Douglas Raillard
5671b49c2f collector: ftrace: Use trace-cmd start -C
Use -C to set the clock rather than writing manually to the file.
2021-09-01 17:38:49 +01:00
Douglas Raillard
27616813ea collector: ftrace: Use trace-cmd start -b
Avoid manually setting the buffer size since trace-cmd start can do it
directly.
2021-09-01 17:38:49 +01:00
Douglas Raillard
cdceba59ba bin: Update trace-cmd to 2.9.1
Build of commit 2191498dc35d629003591f727b604120fabbe02d, which is a few
commits after 2.9.1 release. This version has been in use in LISA for
months now so it should work well.
2021-09-01 17:38:49 +01:00
Douglas Raillard
301d43d140 module/cgroups: Fix exclude pattern quoting
Use shlex.quote() rather than "manual" quoting.
2021-08-19 18:32:46 +01:00
Douglas Raillard
8b92f5530a connection: Add BackgroundCommand.communicate()
Add a communicate() method in the style of Popen.communicate().

Unlike Popen.communicate, it will raise a CalledProcessError if the
command exit with non-zero code.
2021-08-19 18:32:33 +01:00
Douglas Raillard
ad5a97afcc connection: Add ParamikoBackgroundCommand.cmd attr for diagnostic
Add a "cmd" attribute for better exception messages.
2021-08-19 18:32:33 +01:00
Douglas Raillard
e231cb0849 host: Fix PopenBackgroundCommand stdin
The Popen object created for background command currently has no stdin
pipe, preventing the user from writing to it (it will be None instead of
being a file-like object).

Fix that by passing stdin=subprocess.PIPE to Popen constructor.
2021-08-18 16:52:48 +01:00
Douglas Raillard
47280f63da connection: Fix race in ParamikoBackgroundCommand API
When using an ssh background command, the data is read from paramiko as
it comes and stored in a buffering pipe by a thread. Currently, the
ParamikoBackgroundCommand API will report the command as having
completed as soon as paramiko reports it. This however does not
necessarily mean that the pipe-filling thread is finished copying the
streams, and the client will end up assuming there is no more data to
read even though it's not the case.

Fix that by ensuring that when poll() returns non-None, the output
streams are ready to be drained.
2021-08-18 16:52:18 +01:00
Douglas Raillard
a6dd4ddbce ssh: Flush writing end of pipes before closing
When running a background command, ensure the redirection thread flushes
the writing end of the pipe before closing it.
2021-08-18 16:52:18 +01:00
Douglas Raillard
9c8624833e target: Make Target.push/pull work with pathlib
Convert paths to str() so that passing a pathlib.Path works.
2021-08-12 18:49:29 +01:00
Douglas Raillard
dd7860d477 ssh: Move legacy scp out of SshConnectionBase
Move methods related to the scp command into TelnetConnection, since
SshConnection do not use it at all anymore (it has been replaced by the
"scp" python package, piggy backing on paramiko).
2021-08-11 18:40:39 +01:00
Douglas Raillard
914a93355a host: Fix typo s/src/source
Fix typo in LocalConnection._copy_path(): src => source
2021-08-10 11:28:16 +01:00
Marc Bonnici
676336a72a target: Use busybox implementation of kill
For greater portability ensure that we use the busybox
implementation for all kill commands.
2021-07-26 14:37:24 +01:00
Marc Bonnici
b5cd5358ab version: Bump dev version 2021-07-23 15:42:13 +01:00
Marc Bonnici
e6c52c49ff version: Bump revision number v1.3.2 2021-07-23 15:42:13 +01:00
Marc Bonnici
6825130e48 connection: Use busybox implementation of kill
Some target implementations of kill do not support killing
process groups so use the busybox implementation for greater
portability.
2021-07-23 12:35:51 +01:00
Douglas Raillard
80c0e37d11 utils/misc: Use an RLock in tls_property
Allow reentrancy of the lock to fix a deadlock that can occur if
self._get_tls() is called while holding the lock.
2021-07-21 16:44:49 +01:00
Douglas Raillard
f523afda95 target: Fix deadlock in Target.clear_logcat()
Ensure that only once clear_logcat() call is active at once, and just
ignore reentrant calls.
2021-07-21 16:44:49 +01:00
Douglas Raillard
b64ec714a0 utils/misc: Use RLock for check_output_lock
Using a threading.Lock leads to a deadlock in some circumstances.
2021-07-21 16:44:49 +01:00
Valentin Schneider
6249c06b44 modules/sched: Add awareness of new debug directory root
Scheduler debug information is being unified under /sys/kernel/debug/sched
for Linux v5.13. Plug in awareness for the new path while still trying the
old one(s) for backwards compatibility.
2021-07-12 15:16:59 +01:00
Marc Bonnici
3af3463c3c utils/ssh: Fix paramiko streams
Ensure that we use the input stream for reading.
2021-06-29 13:44:14 +01:00
Marc Bonnici
7065847f77 utils/ssh: Fix paramiko stdin
Ensure that we open the stdin stream for writing instead
of read only.
2021-06-29 13:44:14 +01:00
douglas-raillard-arm
79783fa09a target: Create new connection for reentrant calls
When Target.conn property is required while the current connection is
already in use, provide a fresh connection to avoid deadlocks. This is
enabled by the @call_conn decorator that is used on all Target methods
that use self.conn directly.
2021-06-03 17:24:50 +01:00
douglas-raillard-arm
796536d67d hotplug: Verify hotplug.online_all()
Check that all CPUs are effectively online after a call to
target.hotplug.online_all(), as hotplug issues are common and failure to
bring back up a CPU can be quite problematic.
2021-06-03 17:24:43 +01:00
douglas-raillard-arm
b9374d530e ssh: Raise explicit exception when SFTP is not available
When SFTP is not available on OpenSSH, paramiko will raise a generic
exception:

    paramiko.ssh_exception.SSHException: EOF during negotiation

In order to make it easier to debug, raise a TargetStableError telling
the user to enable SFTP on their server. On OpenSSH, this means
installing the sftp subsystem and enabling it in sshd_config.
2021-05-11 09:39:53 +01:00
Javi Merino
34e51e7230 collector/perf: raise an error if report_options or report_sample_options are specified when not using perf/simpleperf record 2021-04-27 10:40:06 +01:00
Marc Bonnici
fa595e1a3d version: Dev version bump 2021-04-19 11:02:53 +01:00
Marc Bonnici
78938cf243 version: Bump revision number v1.3.1 2021-04-19 11:02:53 +01:00
Marc Bonnici
4941a7183a setup.py: Update project description 2021-04-19 10:58:59 +01:00
Marc Bonnici
3ab9d23a4a setup.py: Add long description to package
Use the readme as the long description, this will be
displayed on the PyPi page.
2021-04-19 10:58:59 +01:00
Marc Bonnici
5cf18a7b3c docs: Add note to hostid about PowerPc64 devices
Add caveat for the hostid on PowerPC64 devices due to the library
used when linking the included binary.
2021-04-19 10:46:46 +01:00
Benjamin Crawford
5bfeae08f4 bin: update aarch64 and x86_64 busybox binaries
The original BusyBox binaries were statically linked
against glibc, which caused segmentation faults to occur
on __nss_* calls. This switches the libc implementation
to uClibc in both cases.

busybox ver bump to 1.32.1 for arm64 and x86_64

update x86_64-linux-uclibc busybox

update aarch64-linux-uclibc busybox
2021-04-16 18:25:06 +01:00
Marc Bonnici
a87a1df0fb bin/busybox: Update ppc64le busybox
Update the ppc64le busybox implementation to v1.32.1
to match other architecture binaries.

Note: This binary is linked with musl as glibc has been
reported to cause issues however uclibc does not appear
to be a support build configuration.
2021-04-16 18:24:12 +01:00
Marc Bonnici
3bf763688e bin/busybox: Update 32 bit busybox binaries
Update the arm32 and add i368 v1.32.1 busybox binaries statically linked
with uclibc rather than the previous version with glibc which
caused segfaults on some devices.
2021-04-16 18:24:12 +01:00