c4patino/nixpkgs
1
0
Fork 0
mirror of https://github.com/NixOS/nixpkgs.git synced 2026-07-06 17:13:24 -05:00
Code Issues Projects Releases Packages Wiki Activity

nixos/test-driver: use vhost-device-vsock for SSH backdoor

Browse source 
`vhost-device-vsock`[1] is a custom implementation of AF_VSOCK, but the
application on the host-side uses a UNIX domain-socket. This gives us
the following nice properties:

* We don't need to do `--arg sandbox-paths /dev/vhost-vsock` anymore for
  debugging builds within the sandbox. That means, untrusted users can
  also debug these kinds of tests now.

* This prevents CID conflicts on the host-side, i.e. there's no need for
  using `sshBackdoor.vsockOffset` for tests anymore.

A big shout-out goes to Allison Karlitskaya, the developer of test.thing[2]
who talked about this approach to do AF_VSOCK on All Systems Go 2025.

This patch requires systemd 258[3] because this contains `vhost-mux` in
its SSH config which is needed to connect to the VMs from now on.

To not blow up the patches even more, this only uses AF_VSOCK for the
debugger. A potential follow-up for the future would be a removal of the
current `backdoor.service` and replace it entirely by this
functionality.

The internal implementation tries to be consistent with how VLANs and
machines are handled, i.e. the processes are started when the Driver's
context is entered and cleaned up in __exit__().

I decided to push the process management and creation of sockets for
vhost-device-vsock into its own class, that's an implementation detail
and not a concern for the test-driver. In fact, `vhost-device-vsock` is
something we can drop once QEMU implements native support for using
AF_UNIX on the host-side[4]. `VsockPair` is its own class since
returning e.g. a triple of `(Path, Path, Int)` would be ambiguous in
what is the guest and what the host path (and frankly, I found it hard
to distinguish the two when reading the docs of `vhost-device-vsock`
initially).

Finally, now that we can do the SSH backdoor without adding additional
devices to the sandbox, I figured, it's time to write a test-case for
it.

[1] https://github.com/rust-vmm/vhost-device/blob/main/vhost-device-vsock/README.md
[2] https://codeberg.org/lis/test.thing
[3] https://github.com/NixOS/nixpkgs/pull/427968
[4] https://gitlab.com/qemu-project/qemu/-/issues/2095
This commit is contained in:
Maximilian Bosch 2025-10-22 11:17:46 +02:00
commit 1987c483d8
No known key found for this signature in database
12 changed files with 213 additions and 96 deletions
Download patch file Download diff file Expand all files Collapse all files

48
nixos/doc/manual/development/running-nixos-tests-interactively.section.md
View file

@ -88,52 +88,34 @@ An SSH-based backdoor to log into machines can be enabled with
}
```
::: {.warning}
Make sure to only enable the backdoor for interactive tests
(i.e. by using `interactive.sshBackdoor.enable`)! This is the only
supported configuration.
Running a test in a sandbox with this will fail because `/dev/vhost-vsock` isn't available
in the sandbox.
:::
This creates a [vsock socket](https://man7.org/linux/man-pages/man7/vsock.7.html)
for each VM to log in with SSH. This configures root login with an empty password.
When the VMs get started interactively with the test-driver, it's possible to
connect to `machine` with
On the host-side a UNIX domain-socket is used with
[vhost-device-vsock](https://github.com/rust-vmm/vhost-device/blob/main/vhost-device-vsock/README.md).
That way, it's not necessary to assign system-wide unique vsock numbers.
```
$ ssh vsock/3 -o User=root
$ ssh vsock-mux//tmp/path/to/host -o User=root
```
The socket numbers correspond to the node number of the test VM, but start
at three instead of one because that's the lowest possible
vsock number. The exact SSH commands are also printed out when starting
`nixos-test-driver`.
The socket paths are printed when starting the test driver:
```
Note: this requires systemd-ssh-proxy(1) to be enabled (default on NixOS 25.05 and newer).
machine: ssh -o User=root vsock-mux//tmp/tmpg1rp9nti/machine_host.socket
```
On non-NixOS systems you'll probably need to enable
the SSH config from {manpage}`systemd-ssh-proxy(1)` yourself.
If starting VM fails with an error like
During a test-run, it's possible to print the SSH commands again by running
```
qemu-system-x86_64: -device vhost-vsock-pci,guest-cid=3: vhost-vsock: unable to set guest cid: Address already in use
```
it means that the vsock numbers for the VMs are already in use. This can happen
if another interactive test with SSH backdoor enabled is running on the machine.
In that case, you need to assign another range of vsock numbers. You can pick another
offset with
```nix
{
sshBackdoor = {
enable = true;
vsockOffset = 23542;
};
}
In [2]: dump_machine_ssh()
SSH backdoor enabled, the machines can be accessed like this:
Note: this requires systemd-ssh-proxy(1) to be enabled (default on NixOS 25.05 and newer).
machine: ssh -o User=root vsock-mux//tmp/tmpg1rp9nti/machine_host.socket
```
## Port forwarding to NixOS test VMs {#sec-nixos-test-port-forwarding}

14
nixos/doc/manual/development/writing-nixos-tests.section.md
View file

@ -512,19 +512,11 @@ Once you are in the sandbox shell, you can access the VMs (for example, `machine
with SSH over vsock:
```
bash# ssh -F ./ssh_config vsock/3
bash# ssh -F ./ssh_config -o User=root vsock-mux//tmp/.../machine_host.socket
```
For the AF_VSOCK feature to work, `/dev/vhost-vsock` is needed in the sandbox
which can be done with e.g.
```
nix-build -A nixosTests.foo --option sandbox-paths /dev/vhost-vsock
```
As described in [](#sec-nixos-test-ssh-access), the numbers for vsock start at
`3` instead of `1`. So the first VM in the network (sorted alphabetically) can
be accessed with `vsock/3`.
The socket paths are printed at the beginning of the test. See
[](#sec-nixos-test-ssh-access) for more context.
### SSH access to test containers {#sec-test-container-ssh-access}

3
nixos/doc/manual/redirects.json
View file

@ -2174,9 +2174,6 @@
"test-opt-sshBackdoor.enable": [
"index.html#test-opt-sshBackdoor.enable"
],
"test-opt-sshBackdoor.vsockOffset": [
"index.html#test-opt-sshBackdoor.vsockOffset"
],
"test-opt-enableDebugHook": [
"index.html#test-opt-enableDebugHook"
],

2
nixos/lib/test-driver/default.nix
View file

@ -14,6 +14,7 @@
remote-pdb,
netpbm,
vhost-device-vsock,
nixosTests,
qemu_pkg ? qemu_test,
qemu_test,
@ -56,6 +57,7 @@ buildPythonApplication {
socat
util-linux
vde2
vhost-device-vsock
]
++ lib.optionals enableNspawn [
systemd

9
nixos/lib/test-driver/src/test_driver/__init__.py
View file

@ -147,9 +147,9 @@ def main() -> None:
type=Path,
)
arg_parser.add_argument(
"--dump-vsocks",
"--enable-ssh-backdoor",
help="indicates that the interactive SSH backdoor is active and dumps information about it on start",
type=int,
action="store_true",
)
args = arg_parser.parse_args()
@ -197,9 +197,10 @@ def main() -> None:
keep_machine_state=args.keep_machine_state,
global_timeout=args.global_timeout,
debug=debugger,
enable_ssh_backdoor=args.enable_ssh_backdoor,
) as driver:
if offset := args.dump_vsocks:
driver.dump_machine_ssh(offset)
if args.enable_ssh_backdoor:
driver.dump_machine_ssh()
if args.interactive:
history_dir = os.getcwd()
history_path = os.path.join(history_dir, ".nixos-test-history")

103
nixos/lib/test-driver/src/test_driver/driver.py
View file

@ -8,6 +8,7 @@ import threading
import traceback
from collections.abc import Callable, Iterator
from contextlib import AbstractContextManager, contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import Any
from unittest import TestCase
@ -63,6 +64,45 @@ def pythonize_name(name: str) -> str:
return re.sub(r"^[^A-Za-z_]|[^A-Za-z0-9_]", "_", name)
@dataclass
class VsockPair:
guest: Path
host: Path
cid: int
class VHostDeviceVsock:
def __init__(self, tmp_dir: Path, machines: list[str]):
self.temp_dir_handle = tempfile.TemporaryDirectory(dir=tmp_dir)
self.temp_dir = Path(self.temp_dir_handle.name)
self.sockets = {
machine: VsockPair(
self.temp_dir / f"{machine}_guest.socket",
self.temp_dir / f"{machine}_host.socket",
cid,
)
for cid, machine in enumerate(machines, start=3)
}
self.vhost_proc = subprocess.Popen(
[
"vhost-device-vsock",
*(
arg
for vsock_pair in self.sockets.values()
for arg in (
"--vm",
f"guest-cid={vsock_pair.cid},socket={vsock_pair.guest},uds-path={vsock_pair.host}",
)
),
]
)
def __del__(self) -> None:
self.vhost_proc.kill()
self.temp_dir_handle.cleanup()
class Driver:
"""A handle to the driver that sets up the environment
and runs the tests"""
@ -80,6 +120,8 @@ class Driver:
keep_machine_state: bool
logger: AbstractLogger
debug: DebugAbstract
vhost_vsock: VHostDeviceVsock | None = None
enable_ssh_backdoor: bool
def __init__(
self,
@ -94,6 +136,7 @@ class Driver:
keep_machine_state: bool = False,
global_timeout: int = 24 * 60 * 60 * 7,
debug: DebugAbstract = DebugNop(),
enable_ssh_backdoor: bool = False,
):
self.tests = tests
self.out_dir = out_dir
@ -108,6 +151,7 @@ class Driver:
self.container_start_scripts = dict(
zip(container_names, container_start_scripts)
)
self.enable_ssh_backdoor = enable_ssh_backdoor
def __enter__(self) -> "Driver":
self.race_timer = threading.Timer(self.global_timeout, self.terminate_test)
@ -118,6 +162,12 @@ class Driver:
self.polling_conditions = []
if self.enable_ssh_backdoor and self.vm_start_scripts:
with self.logger.nested("start vhost-device-vsock"):
self.vhost_vsock = VHostDeviceVsock(
tmp_dir, list(self.vm_start_scripts.keys())
)
self.machines_qemu = [
QemuMachine(
name=name,
@ -127,6 +177,16 @@ class Driver:
callbacks=[self.check_polling_conditions],
out_dir=self.out_dir,
logger=self.logger,
vsock_host=(
self.vhost_vsock.sockets[name].host
if self.vhost_vsock is not None
else None
),
vsock_guest=(
self.vhost_vsock.sockets[name].guest
if self.vhost_vsock is not None
else None
),
)
for name, vm_start_script in self.vm_start_scripts.items()
]
@ -219,6 +279,14 @@ class Driver:
except Exception as e:
self.logger.error(f"Error during cleanup of vlan{vlan.nr}: {e}")
if self.enable_ssh_backdoor:
try:
del self.vhost_vsock
except Exception as e:
self.logger.error(
f"Error during cleanup of vhost-device-vsock process: {e}"
)
def subtest(self, name: str) -> Iterator[None]:
"""Group logs under a given test name"""
with self.logger.subtest(name):
@ -256,6 +324,7 @@ class Driver:
NspawnMachine=NspawnMachine, # for typing
t=AssertionTester(),
debug=self.debug,
dump_machine_ssh=self.dump_machine_ssh,
)
machine_symbols = {pythonize_name(m.name): m for m in self.machines}
# If there's exactly one machine, make it available under the name
@ -275,18 +344,28 @@ class Driver:
)
return {**general_symbols, **machine_symbols, **vlan_symbols}
def dump_machine_ssh(self, offset: int) -> None:
print("SSH backdoor enabled, the machines can be accessed like this:")
print(
f"{Style.BRIGHT}Note:{Style.RESET_ALL} vsocks require {Style.BRIGHT}systemd-ssh-proxy(1){Style.RESET_ALL} to be enabled (default on NixOS 25.05 and newer)."
)
longest_name = len(max((machine.name for machine in self.machines), key=len))
for index, machine in enumerate(self.machines, start=offset + 1):
name = machine.name
spaces = " " * (longest_name - len(name) + 2)
def dump_machine_ssh(self) -> None:
if not self.enable_ssh_backdoor:
return
assert self.vhost_vsock is not None
if self.machines:
print("SSH backdoor enabled, the machines can be accessed like this:")
print(
f" {name}:{spaces}{Style.BRIGHT}{machine.ssh_backdoor_command(index)}{Style.RESET_ALL}"
f"{Style.BRIGHT}Note:{Style.RESET_ALL} this requires {Style.BRIGHT}systemd-ssh-proxy(1){Style.RESET_ALL} to be enabled (default on NixOS 25.05 and newer)."
)
longest_name = len(
max((machine.name for machine in self.machines), key=len)
)
for index, machine in enumerate(self.machines):
name = machine.name
spaces = " " * (longest_name - len(name) + 2)
print(
f" {name}:{spaces}{Style.BRIGHT}{machine.ssh_backdoor_command()}{Style.RESET_ALL}"
)
else:
print("SSH backdoor enabled, but no machines defined")
def test_script(self) -> None:
"""Run the test script"""
@ -386,6 +465,10 @@ class Driver:
"""
tmp_dir = get_tmp_dir()
if self.enable_ssh_backdoor:
self.logger.warning(
f"create_machine({name}): not enabling SSH backdoor, this is not supported for VMs created with create_machine!"
)
return QemuMachine(
tmp_dir=tmp_dir,
out_dir=self.out_dir,

29
nixos/lib/test-driver/src/test_driver/machine/__init__.py
View file

@ -147,6 +147,7 @@ class QemuStartCommand:
qmp_socket_path: Path,
shell_socket_path: Path,
allow_reboot: bool = False,
vsock_guest: Path | None = None,
) -> str:
display_opts = ""
@ -170,6 +171,12 @@ class QemuStartCommand:
if not allow_reboot:
qemu_opts += " -no-reboot"
if vsock_guest is not None:
qemu_opts += (
f" -chardev socket,id=vsock_ssh,path={vsock_guest} "
f"-device vhost-user-vsock-pci,chardev=vsock_ssh "
)
return (
f"{self._cmd}"
f" -qmp unix:{qmp_socket_path},server=on,wait=off"
@ -203,10 +210,15 @@ class QemuStartCommand:
qmp_socket_path: Path,
shell_socket_path: Path,
allow_reboot: bool,
vsock_guest: Path | None = None,
) -> subprocess.Popen:
return subprocess.Popen(
self.cmd(
monitor_socket_path, qmp_socket_path, shell_socket_path, allow_reboot
monitor_socket_path,
qmp_socket_path,
shell_socket_path,
allow_reboot,
vsock_guest,
),
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
@ -724,6 +736,9 @@ class QemuMachine(BaseMachine):
shell: socket.socket | None
serial_thread: threading.Thread | None
vsock_guest: Path | None
vsock_host: Path | None
booted: bool
connected: bool
# Store last serial console lines for use
@ -741,6 +756,8 @@ class QemuMachine(BaseMachine):
name: str | None = None,
keep_machine_state: bool = False,
callbacks: list[Callable] | None = None,
vsock_guest: Path | None = None,
vsock_host: Path | None = None,
) -> None:
self.start_command = QemuStartCommand(start_command)
super().__init__(
@ -753,6 +770,8 @@ class QemuMachine(BaseMachine):
)
self.full_console_log = []
self.vsock_guest = vsock_guest
self.vsock_host = vsock_host
# set up directories
self.monitor_path = self.state_dir / "monitor"
@ -769,8 +788,9 @@ class QemuMachine(BaseMachine):
self.booted = False
self.connected = False
def ssh_backdoor_command(self, index: int) -> str:
return f"ssh -o User=root vsock/{index}"
def ssh_backdoor_command(self) -> str:
assert self.vsock_host is not None
return f"ssh -o User=root vsock-mux/{self.vsock_host}"
def is_up(self) -> bool:
return self.booted and self.connected
@ -1224,6 +1244,7 @@ class QemuMachine(BaseMachine):
self.qmp_path,
self.shell_path,
allow_reboot,
self.vsock_guest,
)
self.monitor, _ = monitor_socket.accept()
self.shell, _ = shell_socket.accept()
@ -1434,7 +1455,7 @@ class NspawnMachine(BaseMachine):
self.machine_sock_path = self.tmp_dir / f"{self.name}-nspawn.sock"
def ssh_backdoor_command(self, index: int) -> str:
def ssh_backdoor_command(self) -> str:
# documented in systemd-ssh-generator(8) and https://systemd.io/CONTAINER_INTERFACE/
socket_path = f"/run/systemd/nspawn/unix-export/{self.name}/ssh"
proxy_cmd = f"socat - UNIX-CLIENT:{socket_path}"

1
nixos/lib/test-script-prepend.py
View file

@ -58,3 +58,4 @@ serial_stdout_on: Callable[[], None]
polling_condition: PollingConditionProtocol
debug: DebugAbstract
t: TestCase
dump_machine_ssh: Callable[[], None]

24
nixos/lib/testing/driver.nix
View file

@ -7,6 +7,8 @@
let
inherit (lib) mkOption types literalMD;
inherit (config) sshBackdoor;
# Reifies and correctly wraps the python test driver for
# the respective qemu version and with or without ocr support
testDriver = config.pythonTestDriverPackage.override {
@ -232,5 +234,27 @@ in
# make available on the test runner
passthru.driver = config.driver;
nodeDefaults =
{ config, ... }:
{
# This is needed for the SSH backdoor to function.
# Set this to `true` by default to not change essential QEMU flags
# depending on whether debugging is enabled.
#
# If needed, this can still be turned off.
virtualisation.qemu.enableSharedMemory = lib.mkDefault true;
assertions = [
{
assertion = sshBackdoor.enable -> config.virtualisation.qemu.enableSharedMemory;
message = ''
When turning on the SSH backdoor of the NixOS test-framework,
`virtualisation.qemu.enableSharedMemory` MUST be `true`
(affected: ${config.networking.hostName}).
'';
}
];
};
};
}

39
nixos/lib/testing/nodes.nix
View file

@ -13,6 +13,7 @@ let
mapAttrs
mkIf
mkMerge
mkRemovedOptionModule
mkOption
optionalAttrs
types
@ -128,6 +129,12 @@ let
in
{
imports = [
(mkRemovedOptionModule [ "sshBackdoor" "vsockOffset" ] ''
The option `sshBackdoor.vsockOffset` has been removed from the testing framework.
The functionality provided by it is not needed anymore.
'')
];
options = {
sshBackdoor = {
@ -137,22 +144,6 @@ in
type = types.bool;
description = "Whether to turn on the VSOCK-based access to all VMs. This provides an unauthenticated access intended for debugging.";
};
vsockOffset = mkOption {
default = 2;
type = types.ints.between 2 4294967296;
description = ''
This field is only relevant when multiple users run the (interactive)
driver outside the sandbox and with the SSH backdoor activated.
The typical symptom for this being a problem are error messages like this:
`vhost-vsock: unable to set guest cid: Address already in use`
This option allows to assign an offset to each vsock number to
resolve this.
This is a 32bit number. The lowest possible vsock number is `3`
(i.e. with the lowest node number being `1`, this is 2+1).
'';
};
};
node.type = mkOption {
@ -318,7 +309,7 @@ in
passthru.containers = config.containers;
extraDriverArgs = mkIf config.sshBackdoor.enable [
"--dump-vsocks=${toString config.sshBackdoor.vsockOffset}"
"--enable-ssh-backdoor"
];
defaults = mkMerge [
@ -341,20 +332,6 @@ in
})
];
nodeDefaults = mkIf config.sshBackdoor.enable (
let
inherit (config.sshBackdoor) vsockOffset;
in
{ config, ... }:
{
virtualisation.qemu.options = [
"-device vhost-vsock-pci,guest-cid=${
toString (config.virtualisation.test.nodeNumber + vsockOffset)
}"
];
}
);
# Docs: nixos/doc/manual/development/writing-nixos-tests.section.md
/**
See https://nixos.org/manual/nixos/unstable#sec-override-nixos-test

1
nixos/tests/all-tests.nix
View file

@ -149,6 +149,7 @@ in
lib-extend = handleTestOn [ "x86_64-linux" "aarch64-linux" ] ./nixos-test-driver/lib-extend.nix { };
node-name = runTest ./nixos-test-driver/node-name.nix;
busybox = runTest ./nixos-test-driver/busybox.nix;
ssh-backdoor = runTestOn [ "x86_64-linux" ] ./nixos-test-driver/ssh-backdoor.nix;
console-log = runTest ./nixos-test-driver/console-log.nix;
containers = runTest ./nixos-test-driver/containers.nix;
driver-timeout =

36
nixos/tests/nixos-test-driver/ssh-backdoor.nix Normal file
View file

@ -0,0 +1,36 @@
{ pkgs, lib, ... }:
{
name = "ssh-backdoor";
sshBackdoor.enable = true;
nodes.machine = { };
testScript = ''
import subprocess
start_all()
machine.wait_for_unit("multi-user.target")
assert driver.vhost_vsock is not None
host_socket = driver.vhost_vsock.sockets["machine"].host
with subtest("ssh from the host via systemd-ssh-proxy"):
subprocess.run(
[
"${lib.getExe pkgs.openssh}",
"-vvv",
f"vsock-mux/{host_socket}",
"-o",
"User=root",
"-F",
# The backdoor feature of the driver copies this into NIX_BUILD_TOP.
# We can't do this here since `enableDebugHook=true;` would halt
# instead of terminating the test execution if it fails.
"${pkgs.systemd}/lib/systemd/ssh_config.d/20-systemd-ssh-proxy.conf",
"--",
"true"
],
check=True
)
'';
}