Commit graph

548 commits

Author SHA1 Message Date
pennae
40a35299fa nixos: add functions and documentation for escaping systemd Exec* directives
it's really easy to accidentally write the wrong systemd Exec* directive, ones
that works most of the time but fails when users include systemd metacharacters
in arguments that are interpolated into an Exec* directive. add a few functions
analogous to escapeShellArg{,s} and some documentation on how and when to use them.
2022-03-12 00:48:52 +01:00
Julien Moutinho
8b842173d0 nixos/make-options-doc: fix invalid ':' in XML NCName (non-colonized name) 2021-12-28 22:18:16 -05:00
pennae
a70b1eb630 nixos/lib/make-options-doc: fix with nix 2.3 2021-12-14 03:41:09 +01:00
Jacek Galowicz
b6bf1ca717
Merge pull request #149329 from marijanp/test-driver-restructuring
nixos/test-driver: make the test-driver a python package
2021-12-10 18:41:49 +00:00
Marijan Petričević
ab693de868 nixos/test-driver: make the test-driver a python package 2021-12-10 12:27:45 +00:00
Silvan Mosberger
2f2b60dd87 lib/nixos/eval-config.nix: Fix extraArgs
Fixes a mistake in https://github.com/NixOS/nixpkgs/pull/148315 that
caused https://github.com/NixOS/nixpkgs/issues/148343#issuecomment-990881216
2021-12-10 12:55:30 +01:00
Silvan Mosberger
15c41e1d54
Merge pull request #147265 from pennae/option-docs-build
put all option docs build flavors on equal footing
2021-12-08 21:27:02 +01:00
Artturi
3fe92b45cb
Merge pull request #149665 from Artturin/squashfscores 2021-12-08 20:20:32 +02:00
Artturin
eea6baad50 make-squashfs: use $NIX_BUILD_CORES or 48 cores if above 48
by default all cores are used

hoping this will fix the hydra i686 squashfs build issues as all the
failures were using 64 cores

Parallel mksquashfs: Using 64 processors
Creating 4.0 filesystem on ..., block size 1048576.
FATAL ERROR: mangle2:: xz compress failed with error code 5
2021-12-08 19:23:31 +02:00
ajs124
eee45bb295
Merge pull request #146815 from ElvishJerricco/systemd-utils-expressions
Move systemd-lib.nix and systemd-unit-options.nix into utils
2021-12-08 15:07:28 +00:00
Silvan Mosberger
490d46f044
Merge pull request #148315 from hercules-ci/nixos-evalModules-legacy-cleanup
NixOS/evalModules legacy cleanup
2021-12-07 18:30:52 +01:00
pennae
027f7e1b7f nixos/lib/make-options-doc: generate options.xml from options.json
to do this we must replace derivations with attrsets in make-options-doc, since
xml can represent derivations differently from attrset but json cannot. this
also given asciidoc and mddoc the ability to handle derivation differently,
which they previously didn't have.
2021-12-06 16:12:32 +01:00
pennae
9b97a2ea88 nix/lib/make-options-doc: remove nix-level sorting
there are no remaining users of sorted option lists except the docbook build,
which sorts its input separately.
2021-12-06 16:12:32 +01:00
pennae
4670400309 nixos/lib/make-options-doc: generate asciidoc/md in derivations
use the json file derivation we already have to also generate the asciidoc and
md options docs instead of formatting the options in nix. docbook docs are
already produced in derivations.

the new script produce the exact same output as the old in-nix generation.
2021-12-06 16:12:30 +01:00
Artturin
60422ba2ea nixos/test-driver: add 10ms delay to send_key
attempt to fix https://github.com/NixOS/nixpkgs/issues/147294
2021-12-03 20:04:56 +02:00
Robert Hensing
78ada83361 nixos/eval-config: Deprecate extraArgs and check parameters 2021-12-02 18:23:43 +00:00
Robert Hensing
0b5aea2b27 nixos/eval-config: Remove a rec 2021-12-02 18:23:43 +00:00
Robert Hensing
59c4a35aab nixos/eval-config: Avoid evalModules args and check parameters 2021-12-02 18:23:43 +00:00
Jacek Galowicz
58371472fe
Merge pull request #146512 from DeterminateSystems/better-visibility
nixos/test-driver: more context when step finishes, give more functions nested labels
2021-12-02 09:26:21 +00:00
Jacek Galowicz
57dc1085dc
Merge pull request #146271 from DeterminateSystems/better-timeouts
nixos/test-driver: add execute_with_timeout, add (functional) timeouts to more functions
2021-12-01 19:01:05 +00:00
Cole Helbling
af765f3abd nixos/test-driver: give more functions nested labels
This will make it easier to trace through the test execution without
having to scroll through the entire kernel output.
2021-12-01 10:29:56 -08:00
Cole Helbling
c6ee63259a nixos/test-driver: more context when step finishes
When displaying the amount of time some step took, with no other
context, it becomes nigh impossible (especially in longer tests) to see
when specific steps finished.
2021-12-01 10:29:56 -08:00
Robert Hensing
8a129f8cf0
Merge pull request #144094 from hercules-ci/nixos-specialisations-use-extendModules
nixos/specialisation: Rephrase in terms of extendModules, noUserModules
2021-12-01 11:03:36 +01:00
Martin Schwaighofer
af180d554b qemu, runInLinuxVM: change default cpu to qemu64
The flag -cpu max leaves QEMU 6.1.0 stuck on some systems,
for example when /dev/kvm is not read-writable.
This does not happen with -cpu qemu64.

Getting stuck like that is a regression in 6.1.0 not yet present in 6.0.0
and should be fixed with 6.2.0 according to early testing with rc1.

We should consider reverting this change when we merge QEMU 6.2.0.
See #146526.

fixes #141596
2021-11-30 13:06:22 +00:00
talyz
e8cc900eae
make-disk-image: Make additionalPaths work with Nix 2.4
The `nix` command is marked as experimental since 2.4, so an extra
flag is required to unlock it.
2021-11-29 21:11:15 +01:00
Ryan Burns
2175b157ac treewide: refactor isi686 && isx86_64 -> isx86 2021-11-20 17:50:41 -08:00
Will Fancher
851495a752 Move systemd-lib.nix and systemd-unit-options.nix into utils 2021-11-20 17:52:29 -05:00
Patrick Hilhorst
1334a62539
test-driver.py: directly import pathlib.Path 2021-11-20 01:37:08 +01:00
Patrick Hilhorst
9e71014ede
test-driver.py: always export single machine as 'machine' 2021-11-19 23:02:30 +01:00
Patrick Hilhorst
a8f693ed48
test-driver.py: fix weird non-pythonism 2021-11-19 22:28:32 +01:00
Cole Helbling
e62b8020f3 nixos/test-driver: add (functional) timeouts to more functions
A retry timeout doesn't really help if the thing it's retrying may block
forever.
2021-11-17 16:11:10 -08:00
Cole Helbling
363d7f3ae8 nixos/test-driver: add timeout parameter to execute 2021-11-17 16:11:03 -08:00
Artturin
83f892c511 nixos/lib: add /usr to pathsNeededForBoot
if /usr is split from root then it has to be mounted at boot
See https://github.com/NixOS/nixpkgs/issues/146115
2021-11-15 16:28:22 +02:00
Jacek Galowicz
2ba0732988
Merge pull request #142747 from helsinki-systems/fix/test-runner-execute
nixos/test-runner: Fix execute() flakiness
2021-11-02 14:51:50 +01:00
Robert Hensing
b5681a7a40 nixos/specialisation: Rephrase in terms of extendModules, noUserModules
By using the new extendModules function to produce the specialisations,
we avoid reimplementing the eval-config.nix logic in reverse and fix
cross compilation support for specialisations in the process.
2021-11-01 10:59:47 +01:00
Robert Hensing
0b0d2637fc Revert "nixos/specialisation: Rephrase in terms of extendModules, noUserModules"
This reverts commit e2bea4427b.

While this commit was probably fine, I want to be conservative
with changes right before the release branch-off.
So far the extendModules commits have been adding and refactoring
expressions that did not affect derivation hashes, whereas this
commit changes the module ordering. I will open a separate PR for
it.
2021-11-01 09:41:55 +01:00
Robert Hensing
e2bea4427b nixos/specialisation: Rephrase in terms of extendModules, noUserModules 2021-11-01 09:40:16 +01:00
Robert Hensing
22584ce667 nixos/eval-config.nix: Expose type
Also works for (pkgs.nixos {}).type.
2021-11-01 09:38:49 +01:00
Robert Hensing
56c4f9d005 nixosTest: Fix infinite recursion involving hasContext testScript when useNixStoreImage = true
The involved test was nixosTests.nextcloud.basic21.

It has a testScript that is strict in nodes.nextcloud.config.system.build.vm,
in assertions about imagemagick being in the system closure.

The recursion was introduced in 329a4461a7 from
https://github.com/NixOS/nixpkgs/pull/140792
2021-10-31 18:25:19 +01:00
talyz
46f7521144 make-disk-image: Add an argument to only build a Nix store image
Add the `onlyNixStore` argument which enables building images
containing the contents of the Nix store at the root, instead of a
full system.
2021-10-28 12:55:01 +02:00
talyz
329a4461a7 nixos/testing-python: Copy test script derivations to nodes
Make sure the all derivations referenced by the test script are
available on the nodes. Accessing these derivations works just fine
without this change when using 9p to mount the host's store, but when
an image is built (virtualisation.buildRootImage), the dependencies
need to be copied to the image. We don't want to copy the script
itself, though, since that would trigger unnecessary image rebuilds.
2021-10-28 12:55:01 +02:00
talyz
87d8eec069 make-disk-image: Add additionalPaths argument
Add an argument which makes it possible to add additional paths to the
image's nix store.
2021-10-28 12:55:01 +02:00
talyz
398a73ac98 make-disk-image: Add copyChannel argument
Add a copyChannel argument which controls whether the current source
tree will be made available as a nix channel in the image or
not. Previously, it always was. Making it available is useful for
interactive use of nix utils, but changes the hash of the image when
the sources are updated.
2021-10-28 12:55:01 +02:00
talyz
56c5efa25b make-disk-image: Reintroduce the installBootLoader argument
installBootLoader was removed in
f1708a9d7d79e2bf2961fc648625578b23b3460f; probably by mistake, since
only the logic was removed, not the argument.
2021-10-28 12:55:01 +02:00
Janne Heß
1640359f33
nixos/test-runner: Fix execute() flakiness
Instead of using the magic string, we now just base64-encode everything
and check for a newline.
2021-10-28 11:51:20 +02:00
Janne Heß
1ed5606d8a
nixos/test-runner: Fix thread cleanup 2021-10-24 15:11:46 +02:00
K900
9611041ef5 nixos/lib/test-driver: set defaults in __init__ instead of class level
Just a minor code style nit to avoid potential footguns
2021-10-19 16:29:19 +03:00
K900
a874235dff nixos/lib/test-driver: clean up threads correctly
The current implementation just forks off a thread to read
QEMU's stdout and lets it exist forever. This, however,
makes the interpreter shutdown racy, as the thread could
still be running and writing out buffered stdout when the
main thread exits (and since it's using the low level API,
the worker thread does not get cleaned up by the atexit hooks
installed by `threading`, either). So, instead of doing that,
let's create a real `threading.Thread` object, and also
explicitly `join` it along with the other stuff when cleaning up.
2021-10-19 16:23:51 +03:00
Domen Kožar
8fdc8fe5ce
Merge pull request #142040 from pennae/optimize-module-docs
make-options-docs: don't sort the options XML file
2021-10-18 16:38:32 -05:00
pennae
24eb353907 make-options-docs: don't sort the options XML file
we need the file itself as a dependency for the docbook build, but we don't need
it to be properly sorted at the nix level. push the sort out to a python script
instead to save eval time. on the machine used to write this `nix-instantiate
<nixos/nixos> -A system` went down from 7.1s to 5.4s and GC heap size decreased
by 50MB (or 70MB max RSS).
2021-10-18 03:45:33 +02:00