b7749c7671
Bash's standard behavior of not propagating non-zero exit codes through a pipeline is unexpected and almost universally unwanted. Default to setting `pipefail` for the command being run; it can still be turned off by prefixing the pipeline with `set +o pipefail` if needed. Also, set `errexit` and `nonunset` options to make the first command of consecutive commands separated by `;` fail, and disallow dereferencing unset variables respectively.
517 lines
15 KiB
XML
517 lines
15 KiB
XML
<section xmlns="http://docbook.org/ns/docbook"
|
|
xmlns:xlink="http://www.w3.org/1999/xlink"
|
|
xmlns:xi="http://www.w3.org/2001/XInclude"
|
|
version="5.0"
|
|
xml:id="sec-writing-nixos-tests">
|
|
<title>Writing Tests</title>
|
|
|
|
<para>
|
|
A NixOS test is a Nix expression that has the following structure:
|
|
<programlisting>
|
|
import ./make-test-python.nix {
|
|
|
|
# Either the configuration of a single machine:
|
|
machine =
|
|
{ config, pkgs, ... }:
|
|
{ <replaceable>configuration…</replaceable>
|
|
};
|
|
|
|
# Or a set of machines:
|
|
nodes =
|
|
{ <replaceable>machine1</replaceable> =
|
|
{ config, pkgs, ... }: { <replaceable>…</replaceable> };
|
|
<replaceable>machine2</replaceable> =
|
|
{ config, pkgs, ... }: { <replaceable>…</replaceable> };
|
|
…
|
|
};
|
|
|
|
testScript =
|
|
''
|
|
<replaceable>Python code…</replaceable>
|
|
'';
|
|
}
|
|
</programlisting>
|
|
The attribute <literal>testScript</literal> is a bit of Python code that
|
|
executes the test (described below). During the test, it will start one or
|
|
more virtual machines, the configuration of which is described by the
|
|
attribute <literal>machine</literal> (if you need only one machine in your
|
|
test) or by the attribute <literal>nodes</literal> (if you need multiple
|
|
machines). For instance,
|
|
<filename
|
|
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix">login.nix</filename>
|
|
only needs a single machine to test whether users can log in on the virtual
|
|
console, whether device ownership is correctly maintained when switching
|
|
between consoles, and so on. On the other hand,
|
|
<filename
|
|
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs/simple.nix">nfs/simple.nix</filename>,
|
|
which tests NFS client and server functionality in the Linux kernel
|
|
(including whether locks are maintained across server crashes), requires
|
|
three machines: a server and two clients.
|
|
</para>
|
|
|
|
<para>
|
|
There are a few special NixOS configuration options for test VMs:
|
|
<!-- FIXME: would be nice to generate this automatically. -->
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>
|
|
<option>virtualisation.memorySize</option>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
The memory of the VM in megabytes.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<option>virtualisation.vlans</option>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
The virtual networks to which the VM is connected. See
|
|
<filename
|
|
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix">nat.nix</filename>
|
|
for an example.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<option>virtualisation.writableStore</option>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
By default, the Nix store in the VM is not writable. If you enable this
|
|
option, a writable union file system is mounted on top of the Nix store
|
|
to make it appear writable. This is necessary for tests that run Nix
|
|
operations that modify the store.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
For more options, see the module
|
|
<filename
|
|
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix">qemu-vm.nix</filename>.
|
|
</para>
|
|
|
|
<para>
|
|
The test script is a sequence of Python statements that perform various
|
|
actions, such as starting VMs, executing commands in the VMs, and so on. Each
|
|
virtual machine is represented as an object stored in the variable
|
|
<literal><replaceable>name</replaceable></literal> if this is also the
|
|
identifier of the machine in the declarative config.
|
|
If you didn't specify multiple machines using the <literal>nodes</literal>
|
|
attribute, it is just <literal>machine</literal>.
|
|
The following example starts the machine, waits until it has finished booting,
|
|
then executes a command and checks that the output is more-or-less correct:
|
|
<programlisting>
|
|
machine.start()
|
|
machine.wait_for_unit("default.target")
|
|
if not "Linux" in machine.succeed("uname"):
|
|
raise Exception("Wrong OS")
|
|
</programlisting>
|
|
The first line is actually unnecessary; machines are implicitly started when
|
|
you first execute an action on them (such as <literal>wait_for_unit</literal>
|
|
or <literal>succeed</literal>). If you have multiple machines, you can speed
|
|
up the test by starting them in parallel:
|
|
<programlisting>
|
|
start_all()
|
|
</programlisting>
|
|
</para>
|
|
|
|
<para>
|
|
The following methods are available on machine objects:
|
|
<variablelist>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>start</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Start the virtual machine. This method is asynchronous — it does not
|
|
wait for the machine to finish booting.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>shutdown</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Shut down the machine, waiting for the VM to exit.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>crash</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Simulate a sudden power failure, by telling the VM to exit immediately.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>block</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Simulate unplugging the Ethernet cable that connects the machine to the
|
|
other machines.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>unblock</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Undo the effect of <methodname>block</methodname>.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>screenshot</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Take a picture of the display of the virtual machine, in PNG format. The
|
|
screenshot is linked from the HTML log.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>get_screen_text_variants</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Return a list of different interpretations of what is currently visible
|
|
on the machine's screen using optical character recognition. The number
|
|
and order of the interpretations is not specified and is subject to
|
|
change, but if no exception is raised at least one will be returned.
|
|
</para>
|
|
<note>
|
|
<para>
|
|
This requires passing <option>enableOCR</option> to the test attribute
|
|
set.
|
|
</para>
|
|
</note>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>get_screen_text</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Return a textual representation of what is currently visible on the
|
|
machine's screen using optical character recognition.
|
|
</para>
|
|
<note>
|
|
<para>
|
|
This requires passing <option>enableOCR</option> to the test attribute
|
|
set.
|
|
</para>
|
|
</note>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>send_monitor_command</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Send a command to the QEMU monitor. This is rarely used, but allows doing
|
|
stuff such as attaching virtual USB disks to a running machine.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>send_key</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Simulate pressing keys on the virtual keyboard, e.g.,
|
|
<literal>send_key("ctrl-alt-delete")</literal>.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>send_chars</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Simulate typing a sequence of characters on the virtual keyboard, e.g.,
|
|
<literal>send_chars("foobar\n")</literal> will type the string
|
|
<literal>foobar</literal> followed by the Enter key.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>execute</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Execute a shell command, returning a list
|
|
<literal>(<replaceable>status</replaceable>,
|
|
<replaceable>stdout</replaceable>)</literal>.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>succeed</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Execute a shell command, raising an exception if the exit status
|
|
is not zero, otherwise returning the standard output. Commands
|
|
are run with <literal>set -euo pipefail</literal> set:
|
|
<itemizedlist>
|
|
<listitem>
|
|
<para>
|
|
If several commands are separated by <literal>;</literal>
|
|
and one fails, the command as a whole will fail.
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
For pipelines, the last non-zero exit status will be
|
|
returned (if there is one, zero will be returned
|
|
otherwise).
|
|
</para>
|
|
</listitem>
|
|
<listitem>
|
|
<para>
|
|
Dereferencing unset variables fail the command.
|
|
</para>
|
|
</listitem>
|
|
</itemizedlist>
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>fail</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Like <methodname>succeed</methodname>, but raising an exception if the
|
|
command returns a zero status.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>wait_until_succeeds</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Repeat a shell command with 1-second intervals until it succeeds.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>wait_until_fails</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Repeat a shell command with 1-second intervals until it fails.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>wait_for_unit</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Wait until the specified systemd unit has reached the “active” state.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>wait_for_file</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Wait until the specified file exists.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>wait_for_open_port</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Wait until a process is listening on the given TCP port (on
|
|
<literal>localhost</literal>, at least).
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>wait_for_closed_port</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Wait until nobody is listening on the given TCP port.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>wait_for_x</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Wait until the X11 server is accepting connections.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>wait_for_text</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Wait until the supplied regular expressions matches the textual contents
|
|
of the screen by using optical character recognition (see
|
|
<methodname>get_screen_text</methodname> and
|
|
<methodname>get_screen_text_variants</methodname>).
|
|
</para>
|
|
<note>
|
|
<para>
|
|
This requires passing <option>enableOCR</option> to the test attribute
|
|
set.
|
|
</para>
|
|
</note>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>wait_for_console_text</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Wait until the supplied regular expressions match a line of the serial
|
|
console output. This method is useful when OCR is not possibile or
|
|
accurate enough.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>wait_for_window</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Wait until an X11 window has appeared whose name matches the given
|
|
regular expression, e.g., <literal>wait_for_window("Terminal")</literal>.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>copy_from_host</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Copies a file from host to machine, e.g.,
|
|
<literal>copy_from_host("myfile", "/etc/my/important/file")</literal>.
|
|
</para>
|
|
<para>
|
|
The first argument is the file on the host. The file needs to be
|
|
accessible while building the nix derivation. The second argument is the
|
|
location of the file on the machine.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>systemctl</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Runs <literal>systemctl</literal> commands with optional support for
|
|
<literal>systemctl --user</literal>
|
|
</para>
|
|
<para>
|
|
<programlisting>
|
|
machine.systemctl("list-jobs --no-pager") # runs `systemctl list-jobs --no-pager`
|
|
machine.systemctl("list-jobs --no-pager", "any-user") # spawns a shell for `any-user` and runs `systemctl --user list-jobs --no-pager`
|
|
</programlisting>
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
<varlistentry>
|
|
<term>
|
|
<methodname>shell_interact</methodname>
|
|
</term>
|
|
<listitem>
|
|
<para>
|
|
Allows you to directly interact with the guest shell.
|
|
This should only be used during test development, not in production tests.
|
|
Killing the interactive session with <literal>Ctrl-d</literal> or <literal>Ctrl-c</literal> also ends the guest session.
|
|
</para>
|
|
</listitem>
|
|
</varlistentry>
|
|
</variablelist>
|
|
</para>
|
|
|
|
<para>
|
|
To test user units declared by <literal>systemd.user.services</literal> the
|
|
optional <literal>user</literal> argument can be used:
|
|
<programlisting>
|
|
machine.start()
|
|
machine.wait_for_x()
|
|
machine.wait_for_unit("xautolock.service", "x-session-user")
|
|
</programlisting>
|
|
This applies to <literal>systemctl</literal>, <literal>get_unit_info</literal>,
|
|
<literal>wait_for_unit</literal>, <literal>start_job</literal> and
|
|
<literal>stop_job</literal>.
|
|
</para>
|
|
|
|
<para>
|
|
For faster dev cycles it's also possible to disable the code-linters (this shouldn't
|
|
be commited though):
|
|
<programlisting>
|
|
import ./make-test-python.nix {
|
|
skipLint = true;
|
|
machine =
|
|
{ config, pkgs, ... }:
|
|
{ <replaceable>configuration…</replaceable>
|
|
};
|
|
|
|
testScript =
|
|
''
|
|
<replaceable>Python code…</replaceable>
|
|
'';
|
|
}
|
|
</programlisting>
|
|
This will produce a Nix warning at evaluation time. To fully disable the
|
|
linter, wrap the test script in comment directives to disable the Black linter
|
|
directly (again, don't commit this within the Nixpkgs repository):
|
|
<programlisting>
|
|
testScript =
|
|
''
|
|
# fmt: off
|
|
<replaceable>Python code…</replaceable>
|
|
# fmt: on
|
|
'';
|
|
</programlisting>
|
|
</para>
|
|
</section>
|