I made a mistake merge. Reverting it in c778945806 undid the state
on master, but now I realize it crippled the git merge mechanism.
As the merge contained a mix of commits from `master..staging-next`
and other commits from `staging-next..staging`, it got the
`staging-next` branch into a state that was difficult to recover.
I reconstructed the "desired" state of staging-next tree by:
- checking out the last commit of the problematic range: 4effe769e2
- `git rebase -i --preserve-merges a8a018ddc0` - dropping the mistaken
merge commit and its revert from that range (while keeping
reapplication from 4effe769e2)
- merging the last unaffected staging-next commit (803ca85c20)
- fortunately no other commits have been pushed to staging-next yet
- applying a diff on staging-next to get it into that state
defaultHardeningFlags is set to enable pie for Musl, but is not
actually used because the default is never put into
NIX_HARDENING_ENABLE. That still works for cases other than Musl
only because NIX_HARDENING_ENABLE is defaulted in the binutils and
cc-wrapper setup-hook.sh scripts.
This hook moves systemd user service file from `lib/systemd/user` to
`share/systemd/user`. This is to allow systemd to find the user
services when installed into a user profile. The `lib/systemd/user`
path does not work since `lib` is not in `XDG_DATA_DIRS`.
The Rust `cc` crate started running `xcrun` when SDKROOT is defined:
a970b0ab0b
Consequently, building crates that use newer versions of the `cc`
crate fail, because xcrun is not available in pure build environments.
This provides consistency with the pure stdenv, which provides
patchelf this way. Native stdenv can always just manually install
patchelf on their system, but like xz, it’s unlikely to be provided in
/usr/bin/. In addition, it’s not even in the RHEL7 repos.
This introduces the .inputDerivation attribute on all derivations
created with mkDerivation. This is another derivation that can always
build successfully and whose runtime dependencies are the build time
dependencies of the original derivation.
This allows easy building and distributing of all derivations needed to
enter a nix-shell with
nix-build shell.nix -A inputDerivation
Update gnu-config (config.sub, config.guess) to suport the Genode
platform and apply the updateAutotoolsGnuConfigScriptsHook to Genode
cross-compilation.
I hate the thing too even though I made it, and rather just get rid of
it. But we can't do that yet. In the meantime, this brings us more
inline with autoconf and will make it slightly easier for me to write a
pkg-config wrapper, which we need.
The cross file is added in the `mkDerivation`. It isn't nice putting
build tool-specific stuff here, but our current architecture gives us
little alternative.
Currently it's not possible to determine the reason why a package is
unavailable without evaluating nixpkgs multiple times with different
settings. eg.
nix-repl> :p android-studio.meta
{ available = false; broken = false; unfree = true; unsupported = true; ... }
The following snippet is an example that uses this information to query
the availability information of all packages in nixpkgs, giving an
overview of all the packages currently marked as broken, etc.
{ pkgs }:
with import <nixpkgs/lib>;
let
mapPkgs =
let mapPkgs' = path: f: mapAttrs (n: v:
let result = builtins.tryEval (v ? meta); in
if !result.success then {} else
if isDerivation v then f (path ++ [n]) v else
if isAttrs v && v.recurseForDerivations or false then mapPkgs' (path ++ [n]) f v else
{}
);
in mapPkgs' [];
getMeta = path: drv:
if drv.meta ? available then
let meta = {
pkg = concatStringsSep "." path;
inherit (drv.meta) broken unfree unsupported insecure;
};
in builtins.trace meta.pkg meta
else {};
metaToList = attrs: flatten (map (v: if v ? pkg then v else metaToList v) (attrValues attrs));
in metaToList (mapPkgs getMeta pkgs)
These files never existed, so best to not leave the reference. If
someone want to step up to maintain this, that would be fine. I don’t
have the hardware to test these out. In addition, someone tried to use
the bootstrap-tools currently built by Hydra and found that they were
broken in some unclear way.
The linker scripts no longer contain store paths, so this does nothing. More
importantly, libpthread.so is not longer a linker script on ARM, so the patching
would corrupt it.
There's a generated header that got comment about the source header
from glibc.dev, which added unwanted runtime dependency. Tested:
nix build -f pkgs/top-level/release.nix stdenvBootstrapTools.{aarch64,i686,x86_64}-linux.test
Fixes#21629
Passing these extra linker flags removes both the semi-random uuid
included in most binaries as well as making the sdk version consistent
instead of based on the current os version.
Load command 8
cmd LC_UUID
cmdsize 24
uuid 70FAF921-5DC8-371C-B814-4F121FADFDF4
Load command 9
cmd LC_VERSION_MIN_MACOSX
cmdsize 16
version 10.12
sdk 10.13
The -macosx_version_min flag isn't strictly necessary since that's
already handled by MACOSX_DEPLOYMENT_TARGET.
While looking at the graph of all the outputs in my personal binary
cache it became obvious that we have a lot of self references within the
package set. That isn't an isuse by itself. However it increases the
size of the binary cache for every (reproducible) build of a package
that carries references to itself. You can no longer deduplicate the
outputs since they are all unique. One of the ways to get rid of (a few)
references is to rewrite all the symlinks that are currently used to be
relative symlinks. Two build of something that didn't really change but
carries a self-reference can the be store as the same NAR file again.
I quickly hacked together this change to see if that would yield and
success. My bash scripting skills are probably not great but so far it
seem to somewhat work.
- Replaced python override from the final stdenv, instead we
propagate our bootstrap python to stage4 and override both
CF and xnu to use it.
- Removed CF argument from python interpreters, this is redundant
since it's not overidden anymore.
- Inherit CF from stage4, making it the same as the stdenv.
Before, we'd always use `cc = null`, and check for that. The problem is
this breaks for cross compilation to platforms that don't support a C
compiler.
It's a very subtle issue. One might think there is no problem because we
have `stdenvNoCC`, and presumably one would only build derivations that
use that. The problem is that one still wants to use tools at build-time
that are themselves built with a C compiler, and those are gotten via
"splicing". The runtime version of those deps will explode, but the
build time / `buildPackages` versions of those deps will be fine, and
splicing attempts to work this by using `builtins.tryEval` to filter out
any broken "higher priority" packages (runtime is the default and
highest priority) so that both `foo` and `foo.nativeDrv` works.
However, `tryEval` only catches certain evaluation failures (e.g.
exceptions), and not arbitrary failures (such as `cc.attr` when `cc` is
null). This means `tryEval` fails to let us use our build time deps, and
everything comes apart.
The right solution is, as usually, to get rid of splicing. Or, baring
that, to make it so `foo` never works and one has to explicitly do
`foo.*`. But that is a much larger change, and certaily one unsuitable
to be backported to stable.
Given that, we instead make an exception-throwing `cc` attribute, and
create a `hasCC` attribute for those derivations which wish to
condtionally use a C compiler: instead of doing `stdenv.cc or null ==
null` or something similar, one does `stdenv.hasCC`. This allows quering
without "tripping" the exception, while also allowing `tryEval` to work.
No platform without a C compiler is yet wired up by default. That will
be done in a following commit.
Rewrite the `stripHash` helper function with 2 differences:
* Paths starting with `--` will no longer produce an error.
* Use Bash string manipulation instead of shelling out to `grep` and
`cut`. This should be faster.
A bunch of stdenv-internal variables were deleted in
1601a7fcce, but these are needed in the
fixup phase, whereas the rest are just needed for the initial work
(findInputs, etc) before the user phases.
CC @matthewbauer
There were two issues:
* builtins.getEnv was called deep into the nixpkgs tree making it hard
to discover. This is solved by moving the call into
pkgs/top-level/impure.nix
* when the config was explicitly set by the user to false, it would
still try and load the environment variable. This meant that it was
not possible to guarantee the same outcome on two different systems.
Apparently this option trades compression time for size,
and explicitly does so without increasing resources needed in decomp.
Doesn't make tarball creation unbearable, so add it to options!
Before, we very carefully unapplied and reapplied `set -u` so the rest
of Nixpkgs could continue to not fail on undefined variables. Let's rip
off the band-aid.
Go beyond the obvious setup hooks now, with a bit of sed, with a skipped case:
- cc-wrapper's `dontlink`, because it already is handled.
Also, in nix files escaping was manually added.
EMP
A package's meta.license can either be a single license or a list. The
code to check config.whitelistedLicenses and config.blackListedLicenses
wasn't handling this, nor was the showLicense function.
This logic should be in the pkgs/top-level/static.nix. We don’t want
to pollute Nixpkgs with =if stdenv.static=. Also, "static" is not
descriptive. We have two types of static stdenvs, ‘makeStaticLibaries’
and ‘makeStaticBinaries’. We shouldn’t rely on a static boolean like
this.
These can be used to determine whether a ELF file with ELF header is an
executable or shared library.
We can't implement it in pure bash, as bash has problems with null
bytes.
That's very much consistent with the spirit of nix-shell --pure
BTW, nix 1.x shells will be always treated as pure;
in that version detection isn't possible.
https://github.com/NixOS/nix/commit/1bffd83e1a9c
Some SSL libs don't react to $SSL_CERT_FILE.
That actually makes sense to me, as we add this behavior
as nixpkgs-specific, so it seems "safer" to use $NIX_*.
We want initialPath to have lowest precedence.
In addition, unset _PATH and _HOST_PATH as they shouldn’t be needed
after final PATH and HOST_PATH are set.
Adds pkgsCross.wasm32 and pkgsCross.wasm64. Use it to build Nixpkgs
with a WebAssembly toolchain.
stdenv/cross: use static overlay on isWasm
isWasm doesn’t make sense dynamically linked.
This puts patches in all derivations even if it unspecified by the
derivation. By default it will be an empty list. This simplifies
overrides, as we can now assume that patches is a valid name so that
this works:
self: super: {
mypkg = super.pkg.overrideAttrs (o: {
patches = o.patches ++ [ ./my-very-own.patch ];
});
}
That is, you don’t need to provide a default "or []", make-derivation
provides one for you.
Unfortunately, this is a mass rebuild.
this adds libc++ to the LLVM cross, giving us access to the full
Nixpkgs set. This requires 4 stages of wrapped compilers:
- Clang with no libraries
- Clang with just compiler-rt
- Clang with Libc, and compiler-rt
- Clang with Libc++, Libc, and compiler-rt
New android ndk (18) now uses clang. We were going through the wrapper
that are provided. This lead to surprising errors when building.
Ideally we could use the llvm linker as well, but this leads to errors
as many packages don’t support the llvm linker.
This is needed to avoid confusing and repeated boilerplate for
`fooForTarget`. The vast majority of use-cases can still use
`buildPackages or `targetPackages`, which are now defined in terms of
these.
You can build (partially) with LLVM toolchain using the useLLVM flag.
This works like so:
nix-build -A hello --arg crossSystem '{ system =
"aarch64-unknown-linux-musl"; useLLVM = true }'
also don’t separate debug info in lldClang
It doesn’t work currently with that setup hook. Missing build-id?
Comments on conflicts:
- llvm: d6f401e1 vs. 469ecc70 - docs for 6 and 7 say the default is
to build all targets, so we should be fine
- some pypi hashes: they were equivalent, just base16 vs. base32
For a long time now, tracing has been broken in Nixpkgs. So when you
have an eval error you would get something like this:
error: while evaluating the attribute 'buildInputs' of the derivation 'hello-2.10' at /home/mbauer/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:185:11:
while evaluating 'chooseDevOutputs' at /home/mbauer/nixpkgs/lib/attrsets.nix:474:22, called from undefined position:
while evaluating 'optionals' at /home/mbauer/nixpkgs/lib/lists.nix:257:5, called from /home/mbauer/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:132:17:
This is coming from how Nix handles string context and how
make-derivation messes with the "name" attribute. This commit should
restore the old behavior so you get a nice line number like:
error: while evaluating the attribute 'buildInputs' of the derivation 'hello-2.10' at /home/mbauer/nixpkgs/pkgs/applications/misc/hello/default.nix:4:3:
while evaluating 'chooseDevOutputs' at /home/mbauer/nixpkgs/lib/attrsets.nix:474:22, called from undefined position:
while evaluating 'optionals' at /home/mbauer/nixpkgs/lib/lists.nix:257:5, called from /home/mbauer/nixpkgs/pkgs/stdenv/generic/make-derivation.nix:132:17:
NOTE: This will still be broken for cross compilation due to the
prefixes we are adding to name.
This behavior ended up breaking the handleEvalIssue functionality by hiding those packages. So something like this:
$ nix-env -iA nixpkgs.zoom-us
would silently fail, without telling the user how to fix it! Regardless, this "bug" should be handled in Nix - not Nixpkgs.
Fixes#38952.
We can't run the checkPhase when build != host, so we may as well make
the checkInputs native.
This signicantly improves the situation of Python packages when enabling
strictDeps.
* add generic x86_32 support
- Add support for i386-i586.
- Add `isx86_32` predicate that can replace most uses of `isi686`.
- `isi686` is reinterpreted to mean "exactly i686 arch, and not say i585 or i386".
- This branch was used to build working i586 kernel running on i586 hardware.
* revert `isi[345]86`, remove dead code
- Remove changes to dead code in `doubles.nix` and `for-meta.nix`.
- Remove `isi[345]86` predicates since other cpu families don't have specific model predicates.
* remove i386-linux since linux not supported on that cpu
Hydra's page showing evaluation errors is about a mile long, showing
buckets of user-friendly errors, like this:
in job ‘seyren.aarch64-linux’:
Package ‘oraclejre-8u191’ in /nix/store/fa9zzkbljkvdavwzirkrr5irg25ymbjl-source/pkgs/development/compilers/oraclejdk/jdk-linux-base.nix:71 has an unfree license (‘unfree’), refusing to evaluate.
a) For `nixos-rebuild` you can set
{ nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.
b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
{ allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.
in job ‘jetbrains.webstorm.x86_64-linux’:
Package ‘webstorm-2018.3.1’ in /nix/store/fa9zzkbljkvdavwzirkrr5irg25ymbjl-source/pkgs/applications/editors/jetbrains/default.nix:230 has an unfree license (‘unfree’), refusing to evaluate.
a) For `nixos-rebuild` you can set
{ nixpkgs.config.allowUnfree = true; }
in configuration.nix to override this.
b) For `nix-env`, `nix-build`, `nix-shell` or any other Nix command you can add
{ allowUnfree = true; }
to ~/.config/nixpkgs/config.nix.
This makes it extremely hard to find actual issues in the output. This
patch set makes the output much more condensed in Hydra:
Failed to evaluate nifticlib-2.0.0: «unsupported»: is not supported on ‘x86_64-apple-darwin’
Failed to evaluate dmd-2.081.2: «unsupported»: is not supported on ‘aarch64-unknown-linux-gnu’
Failed to evaluate dmdBuild-2.081.2: «unsupported»: is not supported on ‘aarch64-unknown-linux-gnu’
Failed to evaluate ldc-1.11.0: «unsupported»: is not supported on ‘aarch64-unknown-linux-gnu’
Failed to evaluate ldcBuild-1.11.0: «unsupported»: is not supported on ‘aarch64-unknown-linux-gnu’
Failed to evaluate ldc-0.17.5: «unsupported»: is not supported on ‘aarch64-unknown-linux-gnu’
Failed to evaluate ldcBuild-0.17.5: «unsupported»: is not supported on ‘aarch64-unknown-linux-gnu’
Adds the static overlay that can be used to build Nixpkgs statically.
Can be used like:
nix build pkgsStatic.hello
Not all packages build, as some rely on dynamic linking.
crossOverlays only apply to the packages being built, not the build
packages. It is useful when you don’t care what is used to build your
packages, just what is being built. The idea relies heavily on the
cross compiling infrastructure. Using this implies that we need to
create a cross stdenv.
Some trivial builders use the name attr to choose the exec name
produced. For example nixos-install,
nixos-install = makeProg {
name = "nixos-install";
src = ./nixos-install.sh;
nix = config.nix.package.out;
path = makeBinPath [ nixos-enter ];
};
When cross compiling, this puts the prog in,
/bin/nixos-install-powerpc64le-unknown-linux-gnu
When separateDebugInfo = true & !stdenv.hostPlatform.isLinux, we
always gave an error. There is no reason to do this. Instead, just
don’t add separate debug info when we aren’t on Linux.
propagateNativeBuildInputs will end up going in the output derivation.
This case is allowed to end up in references because of that. Sorry
for the disruption!
Fixes#50865
Completely breaks darwin. Every package in the stdenv that has shebangs
in the output will end up with references to bootstrap-tools.
This reverts commit eb7c50a993.
Completely breaks darwin. Every package in the stdenv that has shebangs
in the output will end up with references to bootstrap-tools.
This reverts commit 2f2e635dd5.
This will make the list much easier to re-use, eg. for `nixosTests`
The drawback is that this approaches makes the
```
nix-build release.nix -A tests.opensmtpd.x86_64-linux
```
command about twice as slow (3s to 6s): it now has to evaluate `nixpkgs`
once for each architecture, instead of just having the hardcoded list of
tests that allowed to say “ok just evaluate for x86_64-linux”.
On the other hand, complete evaluation of `release.nix` should be much
faster because we no longer import `nixpkgs` for each test: testing with
the following command went from 30s to 18s, and that's just for a few
tests.
```
time nix-instantiate --eval --strict nixos/release.nix -A tests.nat
```
I initially wanted to test on the whole `release.nix`, but there are too
many broken tests and it takes too long to eval them all, especially
compared to the fact that the current implementation breaks some setup.
Given developers can just `nix-build nixos/tests/my-test.nix`, it sounds
like an overall win.
Fixes#49071
On ld.gold, we produce broken executables when linking with the Musl
libc. This appears to be a known bug when using ld.gold and Musl. This
thread describes the workaround as enabling PIE when using ld.gold and
Musl:
https://www.openwall.com/lists/musl/2015/05/01/5
By default we don’t enable PIE to avoid breaking things. But in the
Musl case we are breaking things by not enabling PIE. So this adds a
special case for defaultHardeningFlags which keeps the pie hardening
for everything. Any packages that break with PIE can add the pie flag
to disableHardeningFlags array (a no-op for now on anything but Musl).
When strictDeps = true, we don’t want native build inputs to end up in
the output. For instance gcc is a builtin native build input and
should only show up in an output if it is also listed in buildInputs.
/cc @ericson2314
In strictDeps=false, autoPatchshebangs should use
--build (corresponding to PATH) to lookup commands. This restores the
previous behavior of patchshebangs so that we don’t break stuff that
isn’t careful in the buildInputs vs. nativeBuildInputs distinction.
Unfortunately this won’t work under cross compilation.
Rationale
---------
Currently, tests are hard to discover. For instance, someone updating
`dovecot` might not notice that the interaction of `dovecot` with
`opensmtpd` is handled in the `opensmtpd.nix` test.
And even for someone updating `opensmtpd`, it requires manual work to go
check in `nixos/tests` whether there is actually a test, especially
given not so many packages in `nixpkgs` have tests and this is thus most
of the time useless.
Finally, for the reviewer, it is much easier to check that the “Tested
via one or more NixOS test(s)” has been checked if the file modified
already includes the list of relevant tests.
Implementation
--------------
Currently, this commit only adds the metadata in the package. Each
element of the `meta.tests` attribute is a derivation that, when it
builds successfully, means the test has passed (ie. following the same
convention as NixOS tests).
Future Work
-----------
In the future, the tools could be made aware of this `meta.tests`
attribute, and for instance a `--with-tests` could be added to
`nix-build` so that it also builds all the tests. Or a `--without-tests`
to build without all the tests. @Profpatsch described in his NixCon talk
such systems.
Another thing that would help in the future would be the possibility to
reasonably easily have cross-derivation nix tests without the whole
NixOS VM stack. @7c6f434c already proposed such a system.
This RFC currently handles none of these concerns. Only the addition of
`meta.tests` as metadata to be used by maintainers to remember to run
relevant tests.
