This splits a few NixOS tests (namely Chromium, VirtualBox and the
networking tests) into several subtests that are exposed via attributes.
The networking tests were already split up but they didn't expose an
attribute set of available tests but used a function attribute to
specify the resulting test instead.
A new function callSubTests in nixos/release.nix is now responsible for
gathering subtests, which is also used for the installer and boot tests.
The latter is now placed in a tests.boot.* namespace rather than
"polluting" the tests attribute set with its subtest.
This makes it easier to test just a specific channel rather than to
force testing all builds down the users/testers throat. Especially this
makes it easier to test NixOS channel upgrades only against the Chromium
stable channel instead of just removing the beta/dev channels from the
tests entirely (as done in 69ec09f38a).
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Should clean up a lot of these redundant lines for various sub-tests.
Note that the tests.boot* are now called tests.boot.boot*, but otherwise
all the test attribute names should stay the same.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cc: @edolstra
Cc: @wkennington
Cc: @bobvanderlinden
This should de-clutter the various redundant lines of callTest's on
subtests so that every main test file should have only one line with a
callSubTests instead.
Overrides work the same way as callTest, except that if the system
attribute is explicitly specified we do not generate attributes for all
available systems.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Cc: @edolstra
Now subtests are separate derivations, because the individual tests do
not depend on state from previous test runs.
This has the advantage that it's easier to run individiual tests and
it's also easier to pinpoint individual tests that randomly fail.
I ran all of these tests locally and they still succeed.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
- Enforce that an option declaration has a "defaultText" if and only if the
type of the option derives from "package", "packageSet" or "nixpkgsConfig"
and if a "default" attribute is defined.
- Enforce that the value of the "example" attribute is wrapped with "literalExample"
if the type of the option derives from "package", "packageSet" or "nixpkgsConfig".
- Warn if a "defaultText" is defined in an option declaration if the type of
the option does not derive from "package", "packageSet" or "nixpkgsConfig".
- Warn if no "type" is defined in an option declaration.
The manuals are now evaluated with each derivation in `pkgs` (recursively)
replaced by a fake with path "\${pkgs.path.to.the.attribute}".
It isn't perfect, but it seems to cover a vast majority of use cases.
Caveat: even if the package is reached by a different means,
the path above will be shown and not e.g. `${config.services.foo.package}`.
As before, defaults created by `mkDefault` aren't displayed,
but documentation shouldn't (mostly) be a reason to use that anymore.
Note: t wouldn't be enough to just use `lib.mapAttrsRecursive`,
because derivations are also (special) attribute sets.
It serves as a regression test, because right now if you enable
networking.useNetworkd the default loopback interface doesn't get
assigned any IP addresses.
To be sure, I have bisected this and it has been introduced with the
update to systemd 228 in 1da87d4.
Only the "scripted" networking tests have to succeed in order to trigger
a channel update of nixos-unstable, so I'm leaving this test as broken
and we have to figure out next what's the *exact* reason for the
breakage.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
We already have separate tests for checking whether the ISO boots
correctly, so it's not necessary to do that here. So now
tests/installer.nix just tests nixos-install, from a regular NixOS VM
that uses the host's Nix store. This makes running the tests more
convenient because we don't have to build a new ISO after every
change.
We want to avoid getting broken LUKS systems into the latest channel, so
let's ensure that the channel update won't happen if LUKS support is
broken again.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
The i686-linux test has never worked and I wrote the VM test only on
x86_64-linux to verify whether hardening mode works. I don't know why it
fails on i686-linux, but that might be because the inner VirtualBox VM
we're starting during the VM test doesn't use hardware virtualization.
Closes#5708.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
It has been removed by 71a197bc6e.
I'm reintroducing the test mainly because it actually *is* useful,
because right now, machines with mdraid will not boot. In order to
prevent such things from happening in the future, we should *not* remove
this VM test again.
If it really goes back to failing randomly, we should really try to fix
it instead of removing it again.
Of course it fails right now because of the mdraid bump in 7719f7f.
Also, if you want to have additional info about the reasons, have a look
at the commit message of 666cf992f0.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
It now strictly evaluates all remaining attributes, preventing
unevaluated thunks that cannot be garbage-collected. It's also applied
to all jobs in Nixpkgs' release.nix.
This reduces hydra-eval-jobs' memory consumption on the 14.12
release-combined jobset from 5.1 GB to 2.0 GB.
Currently just makes sure that by default it's possible to open a
terminal.
And exactly this should be the main point that might confuse users of i3
in NixOS, because i3 doesn't print a warning/error if it is unable to
start the terminal emulator.
Thanks to @waaaaargh for reporting this issue.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Currently it pretty much tests starting up virtual machines and just
shutting down afterwards, but for both VBoxManage and the VirtualBox
GUI.
This helps catching errors in hardened mode, however we still need to
test whether networking works the way intended (and I fear that this is
broken at the moment).
The VirtualBox VM is _not_ using hardware virtualization support (thus
we use system = "i686-linux", because x86_64 has no emulation support),
because we're already within a qemu VM, which means it's going to be
slow as hell (that's why I've written own subs just for testing
startup/shutdown/whatnot with respective timeouts).
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
- Create container nixos profile
- Create lxc-container nixos config using container nixos profile
- Docker nixos image, use nixos profile for its base config
Currently, the test is only for testing the user namespace sandbox and
even that isn't very representative, because we're running the tests as
root.
But apart from that, we should have functionality for opening/closing
windows and the main goal here is to get them as deterministic as
possible, because Chromium usually isn't very nice to chained xdotool
keystrokes.
And of course, the most important "test" we have here: We know at least
whether Chromium works _at_all_.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
I'm really not sure whether these tests are actually run upstream,
because there are quite a few oddities which either are my fault by just
missing something important or upstream really doesn't bother to run
those tests.
One example of this are testDiskChunk1 and testDiskChunk2, which create
two non-existing partitions and tries to allocate them. Now, in
allocatePartitions(), the partedPartition attributes are reset to None
and shortly afterwards a for loop is expecting it to be NOT None.
So, for now I'm disabling these tests and will see if we stumble on them
during work on nixpart 1.0, so we're really sure whether it's my fault
or a real bug in blivet.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This causes some cruft to be uploaded (such as unit files) but it
ensures that every package used by the base system ends up in the
channel, not just environment.systemPackages.
(cherry picked from commit 4dfca8e14a5d1c2e62bb5ad267163a7a32ef872f)
Currently only tests basic resource record lookup against IPv4 and IPv6.
Nothing special yet, but probably enough for most setups.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
This reverts commit 469f22d717, reversing
changes made to 0078bc5d8f.
Conflicts:
nixos/modules/installer/tools/nixos-generate-config.pl
nixos/modules/system/boot/loader/grub/install-grub.pl
nixos/release.nix
nixos/tests/installer.nix
I tried to keep apparently-safe code in conflicts.
release.nix and release-combined.nix current hardcode the systems which
they are built for. This change introduces an argument to the
expressions called supportedSystems, which allows the builder to choose
which architectures he wants to build. By default, this uses the same
linux x86_64 and i686 architectures.
Now you can just say:
$ nix-build '<nixos/tests/login.nix>'
You can still get the driver script for interactive testing:
$ nix-build '<nixos/tests/login.nix>' -A driver
$ ./result/bin/nixos-test-driver
You can now run a test in the nixos/tests directory directly using
nix-build, e.g.
$ nix-build '<nixos/tests/login.nix>' -A test
This gets rid of having to add the test to nixos/tests/default.nix.
(Of course, you still need to add it to nixos/release.nix if you want
Hydra to run the test.)
It's now "<yy>.<mm>pre<commit-count>.<short-hash>"
(e.g. "13.10pre5678.gfedcba") for the unstable branch, and
"<yy>.<mm>.<commit-count>.<short-hash>" (e.g. "13.10.5678.gfedcba")
for stable branches.
