Support sending any signal to background commands, instead of only
supporting properly SIGKILL/SIGTERM/SIGQUIT.
The main issue is figuring out what PID to send the signal to, as the
devlib API allows running a whole snippet of shell script that typically
is wrapped under many layers of sh -c and sudo calls. In order to lift
the ambiguity, the user has access to a "devlib-signal-target" command
that points devlib at what process should be the target of signals:
# Run a "setup" command, then the main command that will receive the
# signals
cmd = 'echo setup; devlib-signal-target echo hello world'
with target.background(cmd) as bg:
bg.communicate()
The devlib-signal-target script can only be used once per background
command, so that it is never ambiguous what process is targeted, and so
that the Python code can cache the target PID. Subsequent invocations
of devlib-signal-target will fail.
Make cpu_frequency_devlib dependent on whether the "cpu_frequency" event
has been selected rather than dependent on the cpufreq devlib module
being loaded on the target.
The old behavior became particularly problematic with the lazy loading
of modules. However, it was never a reliable way of knowing if the user
was interested in the frequency or not.
Apply a similar mechanism for the extra idle state transitions only done
if "cpu_idle" event is selected.
Emiting the current frequency of all CPUs in the stop() hook is not
useful as the current frequency should already be known from the trace.
Either the event is emitted every time the frequency changes and the
up-to-date information is available, or the frequency never changes
(e.g. userspace governor) and the frequencies will be known from
emitting cpu_frequency_devlib in start().
_prepare_xfer() deals with all the paths resulting from glob expansion,
so it can benefit from async capabilities in order to process multiple
files concurrently.
Convert the internals to async/await to enable useful map_concurrently()
When pulling a file from the target, copy all paths as files and follow
symlinks if necessary. That fixes issues related to chmod not working on
symlinks and generally allows getting any path.
If we want to one day preserve symlinks in some capacities, we can
always add an option to Target.pull() to do so.
Paramiko seems to have had a slight change in behavior that broke
devlib: to save a remote command execution, we attempt to pull any path
as file first, and then as a folder if the former failed.
This is now broken as paramiko will create an empty destination file
when trying to pull as a file. When we attempt again to pull as folder,
the destination exists already (empty file) and we raise an exception.
To fix that, make sure we cleanup any attempt after pulling as a file
before trying again.
The logic in Target.disconnect() appears to have been duplicated by
error. While _probably_ harmless, this is at least confusing, and since
this happens outside of the lock, this may actually be a real problem.
The current behavior is to issue "adb unroot" if the device needed to be
rooted upon connection. This breaks use of nested Targets, which LISA
requires as some target interaction needs to happen in a subprocess.
Fix that by restoring the same adb root state that there was when
creating the connection, rather than blindly unrooting the device upon
disconnection.
Check that no element in the chain adds any unwanted content to stdout
or stderr when running a command. This is especially important as PAM
modules can just write arbitrary messages to stdout when using sudo,
such as password expiry notification. There unfortunately seems to be no
way of silencing it, but we can at least catch it before it triggers
errors down the line.
Since the prompt is added to stdout, remove the one-space-prompt that
currently corrupts stdout when a command is ran with sudo.
That non-empty prompt was added as Windows Subsystem for Linux version 1
(WSL1) has a broken sudo implementation that chokes on an empty prompt.
Considering this is not a platform that is normally suported by devlib,
we re-introduce that empty prompt.
Some drivers emit broken multiline dmesg output (with some Windows-style
newline ...) . In order to parse the rest of the content, allow not
raising on such input.
FtraceCollector.reset() executes 'trace-cmd reset ..' command which
clears all kprobes. This breaks tracing existing kprobe events (if any).
Thus, save kprobe events before trace-cmd reset and restore them after
the reset operation.
For the context, I want to trace an ordinary function in kernel (e.g.,
"echo 'p do_sys_open' > /sys/kernel/tracing/kprobe_events"). However,
FtraceCollector.reset() destroys kprobes, too. Preserving existing
kprobes allows me to use FtraceCollector class as is.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
inspect.iscoroutinefunction() currently detects
_AsyncPolymorphicFunction() as being a coroutine function since it
inspects x.__code__.co_flags to determine so. Since we delegate
attribute access to the async function, it makes
_AsyncPolymorphicFunction() appear as being a coroutine function even
though it is not.
Fix that by directing __code__ attribute access to __call__'s code.
Provide an implementation of re-entrant asyncio.run() that is less
brittle than what greenback provides (e.g. no use of ctypes to poke
extension types).
The general idea of the implementation consists in treating the executed
coroutine as a generator, then turning that generator into a generator
implemented using greenlet. This allows a nested function to make the
top-level parent yield values on its behalf, as if every call was
annotated with "yield from".
Once a thread exits, the connection instance it was using can be
returned to the pool so it can be reused by another thread.
Since there is no per-thread equivalent to atexit, this is achieved by
returning the connection to the pool after every top-level method call
that uses it directly, so the object the user can get by accessing
Target.conn can change after each call to Target method.
cpu-checker was planned to detect availability of KVM acceleration in
QEMU by running kvm-ok command. However, the implementation diverged
from plan and made cpu-checker redundant. Thus, remove it from apt
package list.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Some ADB servers may use non-standard port number. Hence, add 'adb_port'
property to AndroidTarget class and pass port number down to
adb_command().
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Apparently commit
492d42dddb63 ("target: tests: Address review comments on PR#667")
erroneously renamed target_configs.yaml to target_configs.yml.
Rename it to test_config.yml.
Also address 2 Docker warnings related to environment variables while we
are here.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
install_base.sh is left-over from LISA/install_base.sh. Scope of the
script in question is different (and potentially can diverge more) than
its root in LISA. Hence, give it a more descriptive (hopefully) name.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Both devlib and LISA utilizes install_base.sh script, but they install
different packages and support different input arguments. Also support
custom ANDROID_HOME environment variable in order to let LISA (or just
let users install Android SDK/tools wherever they want) choose install
location.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Apparently skins are just nice to have. Also devlib uses emulated
devices in command line (no GUI), so skins are unnecessary. Removing
skins will also reduce the disparity in install_base.sh scripts of LISA
and devlib.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Make sure devlib/install_base.sh has complete Android SDK support. This
will be the first step of removing duplicate Android SDK installation
functions from LISA/install_base.sh.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Just a house-keeping patch to do some trivial improvements:
- Move global variables to the beginning of the script
- Eliminate redundant echo commands
- Tidy up the system package list
- Drop superfluous ';'
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Raise an exception allowing diagnosis when a dmesg line does not match
the regex it is supposed to, rather than the cryptic groups()
AttributeError on None.
Use atexit handler to run Target.disconnect() when the process is about
to exit. This avoids running it with a half torn down namespace, with
ensuing exceptions and non-clean disconnect.
Allow cleanly disconnecting the Target object, so that we don't get
garbage output from __del__ later on when half of the namespace has
already disappeared.
On some systems this seems to have no effect, leaving the executed shell in the root cgroup. Before, this function would still execute and the end user would think the desired process was run in the cgroup when infact it had not.
Log any error happening in adb command ran by _ping() so it can be diagnosed.
Also fix possible deadlock by not using subprocess.PIPE along
subprocess.call(), as the documentation recommends against it.
PR#668: https://github.com/ARM-software/devlib/pull/668
- Fix mixed tab-space white-spacing issues
- s/CMDLINE_VERSION/ANDROID_CMDLINE_VERSION/ to be more precise
- s/set_host_arch/get_android_sdk_host_arch/ because the global variable
for Android host architecture is removed now
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
PR#667: https://github.com/ARM-software/devlib/pull/667
- Implement a test module initializer with a tear down method in
test/test_target.py
- Make various cleanups in test/test_target.py
- Improve structure of test/test_config.yml (previously
target_configs.yaml)
- Make docstrings Sphinx compatible
- Make ``TargetRunner`` and its subclasses private
- Cleanup tests/test_target.py
- Replace print()'s with appropriate logging calls
- Implement ``NOPTargetRunner`` class for simplifying tests
- Improve Python v3.7 compatibility
- Relax host machine type checking
- Escape user input strings
and more..
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Default modules are a recurrent source of errors as they fail to
initialize (cgroups particularly) on any recent target. This leads to
error in basically any workload-automation setup on Android 12 and
above targets.
Since modules can now be lazily loaded upon Target attribute access,
there is no reason to preload those anymore.
Cleanup the module loading code and enable lazy initialization of modules:
* Target.modules is now a read-only property, that is a list of strings
(always, not either strings or dict with mod name as key and params dict as
value).
Target._modules dict stores parameters for each module that was asked
for.
* Target.__init__() now makes thorough validation of the modules list it
is given:
* Specifying the same module mulitple time is only allowed if they
are specified with the same parameters. If a module is specified
both with and without parameters, the parameters take precedence
and the conflict is resolved.
* Only one module of each "kind" can be present in the list.
* Module subclasses gained a class attribute "attr_name" that computes
their "attribute name", i.e. the name under which they are expected to
be lookedup on a Target instance.
* Modules are now automatically registered by simple virtue of
inheriting from Module and defining a name, wherever the source
resides. They do not have to be located in devlib.modules anymore.
This allows 3rd party module providers to very easily add new ones.
* Modules are accessible as Target attribute as:
* Their "kind" if they specified one
* Their "name" (always)
This allows the consumer to either rely on a generic API (via the
"kind") or to expect a specific module (via the "name").
* Accessing a module on Target will lazily load it even if was not
selected using Target(modules=...):
* If the module parameters were specified in Target(modules=...) or
via platform modules, they will be applied automatically.
* Otherwise, no parameter is passed.
* If no module can be found with that name, the list of
Target.modules will be searched for a module matching the given
kind. The first one to be found will be used.
* Modules specified in Target(modules=...) are still loaded eagerly when
their stage is reached just like it used to. We could easily make
those lazily loaded though if we wanted.
* Specifying Target(modules={'foo': None}) will make the "foo" module
unloadable. This can be used to prevent lazy loading a specific
module.
Introduce a Dockerfile in order to create Docker image for devlib and
``run_tests.sh`` script to test Android, Linux, LocalLinux, and QEMU
targets on the Docker image.
The Dockerfile forks from ``Ubuntu-22.04``, installs required system
packages, checks out ``master`` branch of devlib, installs devlib,
creates Android virtual devices via ``tools/android/install_base.sh``,
and QEMU images for aarch64 and x86_84 architectures.
Note that Android command line tools version, buildroot and devlib
branches can be customized via environment variables.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Default connection timeout (30 secs) may be insufficient for some test
setups or in some conditions. Thus, support specifying timeout parameter
in target configuration file.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
