b0956b0ae3
False statements don't cause NixOS tests to fail, we need to throw an exception or terminate.
421 lines
12 KiB
XML
421 lines
12 KiB
XML
<section xmlns="http://docbook.org/ns/docbook"
|
||
xmlns:xlink="http://www.w3.org/1999/xlink"
|
||
xmlns:xi="http://www.w3.org/2001/XInclude"
|
||
version="5.0"
|
||
xml:id="sec-writing-nixos-tests">
|
||
<title>Writing Tests</title>
|
||
|
||
<para>
|
||
A NixOS test is a Nix expression that has the following structure:
|
||
<programlisting>
|
||
import ./make-test.nix {
|
||
|
||
# Either the configuration of a single machine:
|
||
machine =
|
||
{ config, pkgs, ... }:
|
||
{ <replaceable>configuration…</replaceable>
|
||
};
|
||
|
||
# Or a set of machines:
|
||
nodes =
|
||
{ <replaceable>machine1</replaceable> =
|
||
{ config, pkgs, ... }: { <replaceable>…</replaceable> };
|
||
<replaceable>machine2</replaceable> =
|
||
{ config, pkgs, ... }: { <replaceable>…</replaceable> };
|
||
…
|
||
};
|
||
|
||
testScript =
|
||
''
|
||
<replaceable>Perl code…</replaceable>
|
||
'';
|
||
}
|
||
</programlisting>
|
||
The attribute <literal>testScript</literal> is a bit of Perl code that
|
||
executes the test (described below). During the test, it will start one or
|
||
more virtual machines, the configuration of which is described by the
|
||
attribute <literal>machine</literal> (if you need only one machine in your
|
||
test) or by the attribute <literal>nodes</literal> (if you need multiple
|
||
machines). For instance,
|
||
<filename
|
||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/login.nix">login.nix</filename>
|
||
only needs a single machine to test whether users can log in on the virtual
|
||
console, whether device ownership is correctly maintained when switching
|
||
between consoles, and so on. On the other hand,
|
||
<filename
|
||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nfs.nix">nfs.nix</filename>,
|
||
which tests NFS client and server functionality in the Linux kernel
|
||
(including whether locks are maintained across server crashes), requires
|
||
three machines: a server and two clients.
|
||
</para>
|
||
|
||
<para>
|
||
There are a few special NixOS configuration options for test VMs:
|
||
<!-- FIXME: would be nice to generate this automatically. -->
|
||
<variablelist>
|
||
<varlistentry>
|
||
<term>
|
||
<option>virtualisation.memorySize</option>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
The memory of the VM in megabytes.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<option>virtualisation.vlans</option>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
The virtual networks to which the VM is connected. See
|
||
<filename
|
||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/tests/nat.nix">nat.nix</filename>
|
||
for an example.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<option>virtualisation.writableStore</option>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
By default, the Nix store in the VM is not writable. If you enable this
|
||
option, a writable union file system is mounted on top of the Nix store
|
||
to make it appear writable. This is necessary for tests that run Nix
|
||
operations that modify the store.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
</variablelist>
|
||
For more options, see the module
|
||
<filename
|
||
xlink:href="https://github.com/NixOS/nixpkgs/blob/master/nixos/modules/virtualisation/qemu-vm.nix">qemu-vm.nix</filename>.
|
||
</para>
|
||
|
||
<para>
|
||
The test script is a sequence of Perl statements that perform various
|
||
actions, such as starting VMs, executing commands in the VMs, and so on. Each
|
||
virtual machine is represented as an object stored in the variable
|
||
<literal>$<replaceable>name</replaceable></literal>, where
|
||
<replaceable>name</replaceable> is the identifier of the machine (which is
|
||
just <literal>machine</literal> if you didn’t specify multiple machines
|
||
using the <literal>nodes</literal> attribute). For instance, the following
|
||
starts the machine, waits until it has finished booting, then executes a
|
||
command and checks that the output is more-or-less correct:
|
||
<programlisting>
|
||
$machine->start;
|
||
$machine->waitForUnit("default.target");
|
||
die unless $machine->succeed("uname") =~ /Linux/;
|
||
</programlisting>
|
||
The first line is actually unnecessary; machines are implicitly started when
|
||
you first execute an action on them (such as <literal>waitForUnit</literal>
|
||
or <literal>succeed</literal>). If you have multiple machines, you can speed
|
||
up the test by starting them in parallel:
|
||
<programlisting>
|
||
startAll;
|
||
</programlisting>
|
||
</para>
|
||
|
||
<para>
|
||
The following methods are available on machine objects:
|
||
<variablelist>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>start</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Start the virtual machine. This method is asynchronous — it does not
|
||
wait for the machine to finish booting.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>shutdown</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Shut down the machine, waiting for the VM to exit.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>crash</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Simulate a sudden power failure, by telling the VM to exit immediately.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>block</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Simulate unplugging the Ethernet cable that connects the machine to the
|
||
other machines.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>unblock</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Undo the effect of <methodname>block</methodname>.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>screenshot</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Take a picture of the display of the virtual machine, in PNG format. The
|
||
screenshot is linked from the HTML log.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>getScreenText</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Return a textual representation of what is currently visible on the
|
||
machine's screen using optical character recognition.
|
||
</para>
|
||
<note>
|
||
<para>
|
||
This requires passing <option>enableOCR</option> to the test attribute
|
||
set.
|
||
</para>
|
||
</note>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>sendMonitorCommand</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Send a command to the QEMU monitor. This is rarely used, but allows doing
|
||
stuff such as attaching virtual USB disks to a running machine.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>sendKeys</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Simulate pressing keys on the virtual keyboard, e.g.,
|
||
<literal>sendKeys("ctrl-alt-delete")</literal>.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>sendChars</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Simulate typing a sequence of characters on the virtual keyboard, e.g.,
|
||
<literal>sendKeys("foobar\n")</literal> will type the string
|
||
<literal>foobar</literal> followed by the Enter key.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>execute</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Execute a shell command, returning a list
|
||
<literal>(<replaceable>status</replaceable>,
|
||
<replaceable>stdout</replaceable>)</literal>.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>succeed</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Execute a shell command, raising an exception if the exit status is not
|
||
zero, otherwise returning the standard output.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>fail</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Like <methodname>succeed</methodname>, but raising an exception if the
|
||
command returns a zero status.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>waitUntilSucceeds</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Repeat a shell command with 1-second intervals until it succeeds.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>waitUntilFails</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Repeat a shell command with 1-second intervals until it fails.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>waitForUnit</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Wait until the specified systemd unit has reached the “active” state.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>waitForFile</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Wait until the specified file exists.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>waitForOpenPort</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Wait until a process is listening on the given TCP port (on
|
||
<literal>localhost</literal>, at least).
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>waitForClosedPort</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Wait until nobody is listening on the given TCP port.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>waitForX</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Wait until the X11 server is accepting connections.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>waitForText</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Wait until the supplied regular expressions matches the textual contents
|
||
of the screen by using optical character recognition (see
|
||
<methodname>getScreenText</methodname>).
|
||
</para>
|
||
<note>
|
||
<para>
|
||
This requires passing <option>enableOCR</option> to the test attribute
|
||
set.
|
||
</para>
|
||
</note>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>waitForWindow</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Wait until an X11 window has appeared whose name matches the given
|
||
regular expression, e.g., <literal>waitForWindow(qr/Terminal/)</literal>.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>copyFileFromHost</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Copies a file from host to machine, e.g.,
|
||
<literal>copyFileFromHost("myfile", "/etc/my/important/file")</literal>.
|
||
</para>
|
||
<para>
|
||
The first argument is the file on the host. The file needs to be
|
||
accessible while building the nix derivation. The second argument is the
|
||
location of the file on the machine.
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
<varlistentry>
|
||
<term>
|
||
<methodname>systemctl</methodname>
|
||
</term>
|
||
<listitem>
|
||
<para>
|
||
Runs <literal>systemctl</literal> commands with optional support for
|
||
<literal>systemctl --user</literal>
|
||
</para>
|
||
<para>
|
||
<programlisting>
|
||
$machine->systemctl("list-jobs --no-pager"); // runs `systemctl list-jobs --no-pager`
|
||
$machine->systemctl("list-jobs --no-pager", "any-user"); // spawns a shell for `any-user` and runs `systemctl --user list-jobs --no-pager`
|
||
</programlisting>
|
||
</para>
|
||
</listitem>
|
||
</varlistentry>
|
||
</variablelist>
|
||
</para>
|
||
|
||
<para>
|
||
To test user units declared by <literal>systemd.user.services</literal> the
|
||
optional <literal>$user</literal> argument can be used:
|
||
<programlisting>
|
||
$machine->start;
|
||
$machine->waitForX;
|
||
$machine->waitForUnit("xautolock.service", "x-session-user");
|
||
</programlisting>
|
||
This applies to <literal>systemctl</literal>, <literal>getUnitInfo</literal>,
|
||
<literal>waitForUnit</literal>, <literal>startJob</literal> and
|
||
<literal>stopJob</literal>.
|
||
</para>
|
||
</section>
|