Check that no element in the chain adds any unwanted content to stdout
or stderr when running a command. This is especially important as PAM
modules can just write arbitrary messages to stdout when using sudo,
such as password expiry notification. There unfortunately seems to be no
way of silencing it, but we can at least catch it before it triggers
errors down the line.
Since the prompt is added to stdout, remove the one-space-prompt that
currently corrupts stdout when a command is ran with sudo.
That non-empty prompt was added as Windows Subsystem for Linux version 1
(WSL1) has a broken sudo implementation that chokes on an empty prompt.
Considering this is not a platform that is normally suported by devlib,
we re-introduce that empty prompt.
Some drivers emit broken multiline dmesg output (with some Windows-style
newline ...) . In order to parse the rest of the content, allow not
raising on such input.
FtraceCollector.reset() executes 'trace-cmd reset ..' command which
clears all kprobes. This breaks tracing existing kprobe events (if any).
Thus, save kprobe events before trace-cmd reset and restore them after
the reset operation.
For the context, I want to trace an ordinary function in kernel (e.g.,
"echo 'p do_sys_open' > /sys/kernel/tracing/kprobe_events"). However,
FtraceCollector.reset() destroys kprobes, too. Preserving existing
kprobes allows me to use FtraceCollector class as is.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
inspect.iscoroutinefunction() currently detects
_AsyncPolymorphicFunction() as being a coroutine function since it
inspects x.__code__.co_flags to determine so. Since we delegate
attribute access to the async function, it makes
_AsyncPolymorphicFunction() appear as being a coroutine function even
though it is not.
Fix that by directing __code__ attribute access to __call__'s code.
Provide an implementation of re-entrant asyncio.run() that is less
brittle than what greenback provides (e.g. no use of ctypes to poke
extension types).
The general idea of the implementation consists in treating the executed
coroutine as a generator, then turning that generator into a generator
implemented using greenlet. This allows a nested function to make the
top-level parent yield values on its behalf, as if every call was
annotated with "yield from".
Once a thread exits, the connection instance it was using can be
returned to the pool so it can be reused by another thread.
Since there is no per-thread equivalent to atexit, this is achieved by
returning the connection to the pool after every top-level method call
that uses it directly, so the object the user can get by accessing
Target.conn can change after each call to Target method.
cpu-checker was planned to detect availability of KVM acceleration in
QEMU by running kvm-ok command. However, the implementation diverged
from plan and made cpu-checker redundant. Thus, remove it from apt
package list.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Some ADB servers may use non-standard port number. Hence, add 'adb_port'
property to AndroidTarget class and pass port number down to
adb_command().
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Apparently commit
492d42dddb63 ("target: tests: Address review comments on PR#667")
erroneously renamed target_configs.yaml to target_configs.yml.
Rename it to test_config.yml.
Also address 2 Docker warnings related to environment variables while we
are here.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
install_base.sh is left-over from LISA/install_base.sh. Scope of the
script in question is different (and potentially can diverge more) than
its root in LISA. Hence, give it a more descriptive (hopefully) name.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Both devlib and LISA utilizes install_base.sh script, but they install
different packages and support different input arguments. Also support
custom ANDROID_HOME environment variable in order to let LISA (or just
let users install Android SDK/tools wherever they want) choose install
location.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Apparently skins are just nice to have. Also devlib uses emulated
devices in command line (no GUI), so skins are unnecessary. Removing
skins will also reduce the disparity in install_base.sh scripts of LISA
and devlib.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Make sure devlib/install_base.sh has complete Android SDK support. This
will be the first step of removing duplicate Android SDK installation
functions from LISA/install_base.sh.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Just a house-keeping patch to do some trivial improvements:
- Move global variables to the beginning of the script
- Eliminate redundant echo commands
- Tidy up the system package list
- Drop superfluous ';'
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Raise an exception allowing diagnosis when a dmesg line does not match
the regex it is supposed to, rather than the cryptic groups()
AttributeError on None.
Use atexit handler to run Target.disconnect() when the process is about
to exit. This avoids running it with a half torn down namespace, with
ensuing exceptions and non-clean disconnect.
Allow cleanly disconnecting the Target object, so that we don't get
garbage output from __del__ later on when half of the namespace has
already disappeared.
On some systems this seems to have no effect, leaving the executed shell in the root cgroup. Before, this function would still execute and the end user would think the desired process was run in the cgroup when infact it had not.
Log any error happening in adb command ran by _ping() so it can be diagnosed.
Also fix possible deadlock by not using subprocess.PIPE along
subprocess.call(), as the documentation recommends against it.
PR#668: https://github.com/ARM-software/devlib/pull/668
- Fix mixed tab-space white-spacing issues
- s/CMDLINE_VERSION/ANDROID_CMDLINE_VERSION/ to be more precise
- s/set_host_arch/get_android_sdk_host_arch/ because the global variable
for Android host architecture is removed now
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
PR#667: https://github.com/ARM-software/devlib/pull/667
- Implement a test module initializer with a tear down method in
test/test_target.py
- Make various cleanups in test/test_target.py
- Improve structure of test/test_config.yml (previously
target_configs.yaml)
- Make docstrings Sphinx compatible
- Make ``TargetRunner`` and its subclasses private
- Cleanup tests/test_target.py
- Replace print()'s with appropriate logging calls
- Implement ``NOPTargetRunner`` class for simplifying tests
- Improve Python v3.7 compatibility
- Relax host machine type checking
- Escape user input strings
and more..
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Default modules are a recurrent source of errors as they fail to
initialize (cgroups particularly) on any recent target. This leads to
error in basically any workload-automation setup on Android 12 and
above targets.
Since modules can now be lazily loaded upon Target attribute access,
there is no reason to preload those anymore.
Cleanup the module loading code and enable lazy initialization of modules:
* Target.modules is now a read-only property, that is a list of strings
(always, not either strings or dict with mod name as key and params dict as
value).
Target._modules dict stores parameters for each module that was asked
for.
* Target.__init__() now makes thorough validation of the modules list it
is given:
* Specifying the same module mulitple time is only allowed if they
are specified with the same parameters. If a module is specified
both with and without parameters, the parameters take precedence
and the conflict is resolved.
* Only one module of each "kind" can be present in the list.
* Module subclasses gained a class attribute "attr_name" that computes
their "attribute name", i.e. the name under which they are expected to
be lookedup on a Target instance.
* Modules are now automatically registered by simple virtue of
inheriting from Module and defining a name, wherever the source
resides. They do not have to be located in devlib.modules anymore.
This allows 3rd party module providers to very easily add new ones.
* Modules are accessible as Target attribute as:
* Their "kind" if they specified one
* Their "name" (always)
This allows the consumer to either rely on a generic API (via the
"kind") or to expect a specific module (via the "name").
* Accessing a module on Target will lazily load it even if was not
selected using Target(modules=...):
* If the module parameters were specified in Target(modules=...) or
via platform modules, they will be applied automatically.
* Otherwise, no parameter is passed.
* If no module can be found with that name, the list of
Target.modules will be searched for a module matching the given
kind. The first one to be found will be used.
* Modules specified in Target(modules=...) are still loaded eagerly when
their stage is reached just like it used to. We could easily make
those lazily loaded though if we wanted.
* Specifying Target(modules={'foo': None}) will make the "foo" module
unloadable. This can be used to prevent lazy loading a specific
module.
Introduce a Dockerfile in order to create Docker image for devlib and
``run_tests.sh`` script to test Android, Linux, LocalLinux, and QEMU
targets on the Docker image.
The Dockerfile forks from ``Ubuntu-22.04``, installs required system
packages, checks out ``master`` branch of devlib, installs devlib,
creates Android virtual devices via ``tools/android/install_base.sh``,
and QEMU images for aarch64 and x86_84 architectures.
Note that Android command line tools version, buildroot and devlib
branches can be customized via environment variables.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Default connection timeout (30 secs) may be insufficient for some test
setups or in some conditions. Thus, support specifying timeout parameter
in target configuration file.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
We can mimic ChromeOS target by combining a QEMU guest (for Linux
bindings of ``ChromeOsTarget`` class) with a Android virtual desktop
(for Android bits of ``ChromeOsTarget``).
Note that Android bindings of ``ChromeOsTarget`` class also requires
existence of ``/opt/google/containers/android`` folder on the Linux
guest.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
SshConnection registers an atexit handler so the connection is closed
upon exiting the process if it has not been done before. However, the
handler keeps a reference on the connection, which means it _will_ stay
alive. If lots of short-lived connections are created (which can happen
when using e.g. ThreadPoolExecutor), they will simply stay around and
leak.
Fix that by using a weak reference (WeakMethod) to register in the
atexit handler, with a callback to unregister it when the object is
deallocated.
Add support for launching emulated targets on QEMU. The base class
``TargetRunner`` has groundwork for target runners like
``QEMUTargetRunner``.
``TargetRunner`` is a contextmanager which starts runner process (e.g.,
QEMU), makes sure the target is accessible over SSH (if
``connect=True``), and terminates the runner process once it's done.
The other newly introduced ``QEMUTargetRunner`` class:
- performs sanity checks to ensure QEMU executable, kernel, and initrd
images exist,
- builds QEMU parameters properly,
- creates ``Target`` object,
- and lets ``TargetRunner`` manage the QEMU instance.
Also add a new test case in ``tests/test_target.py`` to ensure devlib
can run a QEMU target and execute some basic commands on it.
While we are in neighborhood, fix a typo in ``Target.setup()``.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Integrate buildroot into devlib in order to ease building kernel and
root filesystem images via 'generate-kernel-initrd.sh' helper script.
As its name suggests, the script builds kernel image which also includes
an initial RAM disk per default config files located under
configs/<arch>/.
Provide config files for buildroot and Linux kernel as well as a
post-build.sh script which tweaks (e.g., allowing root login on SSH)
target's root filesystem.
doc/tools.rst talks about details of kernel and rootfs configuration.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Test Android and Linux targets as well in addition to LocalLinux target.
In order to keep basic verification easy, list complete list of test
targets in tests/target_configs.yaml.example and keep the default
configuration file for targets simple.
Also:
- Create a test folder on target's working directory.
- Remove all devlib artefacts after execution of the test.
- Add logs to show progress of operations.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
``Target.make_temp()`` employs ``mktemp`` command to create a temporary
file or folder.
This method will be used in unit tests.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
SshConnection._make_client() may throw exceptions for several reasons
(e.g., target is not ready yet). The client should be closed if that is
the case. Otherwise Python unittest like tools report resource warning
for 'unclosed socket', etc.
Signed-off-by: Douglas Raillard <douglas.raillard@arm.com>
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
Introduce ``tools/android/install_base.sh`` [1] script to install
Android command line tools including necessary platforms and
system-images for Linux and create Android Virtual Devices (AVD) for
Pixel 6 with Android v12 & v14 as well as an Android virtual *desktop*
device (v13) for ChromeOS tests.
[1] Forked from https://github.com/ARM-software/lisa/blob/main/install_base.sh
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
This will be useful in automating CI tests without modifying the source
code.
Replace unittest with pytest in order to make parameter passing to test
functions easier.
Move target configuration reading and generating target object outside
of the test function. Because we will run the test function for new
targets and may want to add new test functions.
While we are here, also fix pylint issues.
Signed-off-by: Metin Kaya <metin.kaya@arm.com>
