Commit graph

2375 commits

Author SHA1 Message Date
Sandro
b04d4bad27
Merge pull request #216992 from SuperSandro2000/stdenvNative-fix-eval
{bintools,cc}-wrapper: don't fallback to version = null
2023-04-14 11:22:20 +02:00
Sandro Jäckel
b0ee79f9fd
mkDerivation: show meaningful error when version is set to null
instead of `Cannot coerce null to string`
2023-04-12 22:08:36 +02:00
Artturi
c88044fe0b
Merge pull request #225929 from Artturin/fixadapters2 2023-04-12 23:04:31 +03:00
Artturin
67a84d1428 stdenvAdapters: fix preservation of env.NIX_CFLAGS_COMPILE
we have managed to migrate to NIX_CFLAGS_COMPILE to the env attrset well
enough that we don't need to support having it toplevel. mkDerivation
will throw if there's a attr in both env and toplevel so no need to
worry about that
2023-04-12 21:27:34 +03:00
github-actions[bot]
47157337bf
Merge master into staging-next 2023-04-12 00:02:18 +00:00
Artturin
9adf83737f stdenvAdapters: fix preservation of env 2023-04-11 21:25:52 +03:00
Adam Joseph
3778e07994 stdenv: fix pkgsMusl
I broke `pkgsMusl` with #209870.

Something odd is happening with `xgcc` (the temporary compiler that
should be used only to compile `gcc`, although we are using it to
compile a temporary `patchelf` too) and `libstdc++`.

The temporary fix in this commit is to use `-static-libstdc++` for
the ephemeral `patchelf` built by `xgcc`.  It isn't pretty, but it
appears to work.

Incorporates:

- https://github.com/NixOS/nixpkgs/pull/224945
2023-04-06 03:16:31 -07:00
Sergei Trofimovich
c2e14a73e6 xgcc: avoid libc.so mix between gcc and binutils-ld linker plugin
Without the change when I attempt to built `nixpkgs` with weekly
`gcc-13` (it pulls in `flex` as a build input`) I am getting build
failure related to glibc mix caused by glibc loading:

    ...-binutils-patchelfed-ld-2.40/bin/ld: ...-xgcc-13.0.0/libexec/gcc/x86_64-unknown-linux-gnu/13.0.1/liblto_plugin.so:
    error loading plugin: ...-bootstrap-tools/lib/libpthread.so.0: undefined symbol: __libc_vfork, version GLIBC_PRIVATE

The change disables LTO plugin entirely to avoid loading of `glibc` mix.
2023-04-05 07:43:22 +01:00
Bernardo Meurer
f1f6ca8bcd
Merge pull request #209870 from amjoseph-nixpkgs/pr/stdenv/external-gcc-bootstrap 2023-04-03 08:19:03 -07:00
Adam Joseph
96588eb3de gcc: add common/checksum.nix
This commit adds `gcc/common/checksum.nix`, which contains code
common to both gcc11 and gcc12, implementing the `enableChecksum`
feature.

When gcc's built-in bootstrap (`--enable-bootstrap`) is used, gcc
compiles itself three times and compares a hash of the unlinked `.o`
files from the second and third compilation.  The
`enableChecksum=true` parameter performs the same comparison as part
of the `postInstall` phase.

Notably, `enableChecksum=true` can be used with `enableBootstrap=false`.

Co-authored-by: Sandro <sandro.jaeckel@gmail.com>
2023-04-02 13:49:53 -07:00
Adam Joseph
fed2300bea unpack-bootstrap-tools.sh: patchelf libgcc_s.so.1 2023-04-02 13:49:53 -07:00
Adam Joseph
d7fe0a5548 make-bootstrap-tools.nix: use a patchelf built with -static-{libgcc,libstdc++}
Our bootstrap-files unpacker has always relied on a lot of unstated
assumptions, one of them being that every library has a DT_NEEDED
for librt.so, so patchelf'ing something into the RUNPATH into
librt.so means that it will be searched for every library load in
all of the bootstrap-files.

Unfortunately that assumption is not true for libgcc.

This causes problems, because patchelf links against libgcc (and
against libstdc++, which links against libgcc).  So we can't use
patchelf on libgcc, because it needs libgcc, so patchelf doesn't
work until libgcc is patchelfed.

The robust solution here is to use static linking for the copy of
patchelf that is shipped with the bootstrap-files.  We don't have to
go all the way to a statically linked libc; just -static-libgcc and
-static-libstdc++ are enough to break the circular dependency.
2023-04-02 13:49:53 -07:00
Adam Joseph
86ca0faff7 make-bootstrap-tools.nix: cp libgcc_s without -d
We do not want to preserve the symlinks from libgcc_s, since they
point to another outpath.  We want to copy from that outpath rather
than link to it.
2023-04-02 13:49:53 -07:00
Adam Joseph
c6bd37a691 make-bootstrap-tools.nix: ship libisl.so
Now that we've dropped the
gcc-links-statically-to-lib{isl,mpfr,mpc,gmp} hack, our gcc needs
libisl.so.  Let's add it to the bootstrap-files.
2023-04-02 13:49:53 -07:00
Adam Joseph
7553d0fe29 stdenv: Nix-driven bootstrap of gcc
#### Summary

By default, when you type `make`, GCC will compile itself three
times.  This PR inhibits that behavior by configuring GCC with
`--disable-bootstrap`, and reimplements the triple-rebuild using
Nix rather than `make`/`sh`.

 #### Immediate Benefits

- Allow `gcc11` and `gcc12` on `aarch64` (without needing new
  `bootstrapFiles`)
- Faster stdenv rebuilds: the third compilation of gcc
  (i.e. stageCompare) is no longer a `drvInput` of the final stdenv.
  This allows Nix to build stageCompare in parallel with the rest of
  nixpkgs instead of in series.
- No more copying `libgcc_s` out of the bootstrap-files or other
  derivations
- No more Frankenstein compiler: the final gcc and the libraries it
  links against (mpfr, mpc, isl, glibc) are all built by the same
  compiler (xgcc) instead of a mixture of the bootstrapFiles'
  compiler and xgcc.
- No more [static lib{mpfr,mpc,gmp,isl}.a hack]
- Many other small `stdenv` hacks eliminated
- `gcc` and `clang` share the same codepath for more of `cc-wrapper`.

 #### Future Benefits

- This should allow using a [foreign] `bootstrap-files` so long as
  `hostPlatform.canExecute bootstrapFiles`.
- This should allow each of the libraries that ship with `gcc`
  (lib{backtrace, atomic, cc1, decnumber, ffi, gomp, iberty,
  offloadatomic, quadmath, sanitizer, ssp, stdc++-v3, vtv}) to be
  built in separate (one-liner) derivations which `inherit src;`
  from `gcc`, much like https://github.com/NixOS/nixpkgs/pull/132343

 #### Incorporates

- https://github.com/NixOS/nixpkgs/pull/210004
- https://github.com/NixOS/nixpkgs/pull/36948 (unreverted)
- https://github.com/NixOS/nixpkgs/pull/210325
- https://github.com/NixOS/nixpkgs/pull/210118
- https://github.com/NixOS/nixpkgs/pull/210132
- https://github.com/NixOS/nixpkgs/pull/210109
- https://github.com/NixOS/nixpkgs/pull/213909
- https://github.com/NixOS/nixpkgs/pull/216136
- https://github.com/NixOS/nixpkgs/pull/216237
- https://github.com/NixOS/nixpkgs/pull/210019
- https://github.com/NixOS/nixpkgs/pull/216232
- https://github.com/NixOS/nixpkgs/pull/216016
- https://github.com/NixOS/nixpkgs/pull/217977
- https://github.com/NixOS/nixpkgs/pull/217995

 #### Closes

- Closes #108305
- Closes #108111
- Closes #201254
- Closes #208412

 #### Credits

This project was made possible by three important insights, none of
which were mine:

1. @ericson2314 was the first to advocate for this change, and
   probably the first to appreciate its advantages.  Nix-driven
   (external) bootstrap is "cross by default".

2. @trofi has figured out a lot about how to get gcc to not mix up
   the copy of `libstdc++` that it depends on with the copy that it
   builds, by moving the `bootstrapFiles`' `libstdc++` into a
   [versioned directory].  This allows a Nix-driven bootstrap of gcc
   without the final gcc would still having references to the
   `bootstrapFiles`.

3. Using the undocumented variable [`user-defined-trusted-dirs`]
   when building glibc.  When glibc `dlopen()`s `libgcc_s.so`, it
   uses a completely different and totally special set of rules for
   finding `libgcc_s.so`.  This trick is the only way we can put
   `libgcc_s.so` in its own separate outpath without creating
   circular dependencies or dependencies on the bootstrapFiles.  I
   would never have guessed to use this (or that it existed!) if it
   were not for a [comment in guix] which @Mic92 [mentioned].

My own role in this PR was basically: being available to go on a
coding binge at an opportune moment, so we wouldn't waste a
[crisis].

[aarch64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662822938
[amd64-compare-ofborg]: https://github.com/NixOS/nixpkgs/pull/209870/checks?check_run_id=10662825857
[nonexistent sysroot]: https://github.com/NixOS/nixpkgs/pull/210004
[versioned directory]: https://github.com/NixOS/nixpkgs/pull/209054
[`user-defined-trusted-dirs`]: https://sourceware.org/legacy-ml/libc-help/2013-11/msg00026.html
[comment in guix]: 5e4ec82181/gnu/packages/gcc.scm (L253)
[mentioned]: https://github.com/NixOS/nixpkgs/pull/210112#issuecomment-1379608483
[crisis]: https://github.com/NixOS/nixpkgs/issues/108305
[foreign]: https://github.com/NixOS/nixpkgs/pull/170857#issuecomment-1170558348
[static lib{mpfr,mpc,gmp,isl}.a hack]: 2f1948af9c/pkgs/stdenv/linux/default.nix (L380)
2023-04-02 13:49:41 -07:00
Adam Joseph
70d34589bd stdenv/linux: factor out commonGccOverrides
This commit has no effect on eval.  It simply factors out a common
subexpression.
2023-04-02 03:04:20 -07:00
Adam Joseph
e8b10284f3 linux/default.nix: use mipsel.nix instead of longson.nix for mips32
Right now our bootstrapFiles-selecting algorithm uses the
`loongson2f.nix` bootstrapFiles (which were not built by Hydra).

These bootstrapFiles don't work anymore.  They were added in 2010 by
40405d03ac.

This commit causes mipsel-linux native builds to use the Hydra-built
bootstrap files from this PR instead:

  https://github.com/NixOS/nixpkgs/pull/183487
2023-04-01 23:25:25 -07:00
Vladimír Čunát
12dd95fbb1
Merge branch 'master' into staging-next 2023-03-24 09:07:41 +01:00
Adam Joseph
13507da345 check-meta.nix: fix self-contradictory error messages
See https://github.com/NixOS/nixpkgs/pull/222792#pullrequestreview-1356114111

You can't just `lib.filter _ lib.systems.all` -- that throws away
important information, leading to nixpkgs disagreeing with itself
like this:

```
$ NIXPKGS_ALLOW_BROKEN=1 nix-instantiate . -A pkgsStatic.systemd
error: Package ‘systemd-252.5’ in ... is only supported on ... x86_64-linux but not on requested x86_64-linux, refusing to evaluate.
```

After:

```
$ NIXPKGS_ALLOW_BROKEN=1 nix-instantiate . -A pkgsStatic.systemd
error: Package ‘systemd-252.5’ in ... is not available on the requested hostPlatform:
         hostPlatform.config = "x86_64-unknown-linux-musl"
         package.meta.platforms = [
           "aarch64-linux"
           "armv5tel-linux"
           "armv6l-linux"
           "armv7a-linux"
           "armv7l-linux"
           "i686-linux"
           "m68k-linux"
           "microblaze-linux"
           "microblazeel-linux"
           "mipsel-linux"
           "mips64el-linux"
           "powerpc64-linux"
           "powerpc64le-linux"
           "riscv32-linux"
           "riscv64-linux"
           "s390-linux"
           "s390x-linux"
           "x86_64-linux"
         ]
         package.meta.badPlatforms = [
           {
             isStatic = true;
             parsed = { };
           }
         ]
       , refusing to evaluate.
```
2023-03-24 00:15:57 -07:00
Weijia Wang
b46c97e1dc stdenv: generalise showPlatforms 2023-03-23 23:47:11 +02:00
github-actions[bot]
307b719414
Merge master into staging-next 2023-03-23 18:01:20 +00:00
Weijia Wang
2a7e1e0228 stdenv: fix error with patterned platforms 2023-03-23 17:27:49 +02:00
Sandro
979c6c1fe4
Merge pull request #213780 from SuperSandro2000/check-meta-platform
stdenv: show supported and requested platforms when check meta fails
2023-03-23 15:01:34 +01:00
github-actions[bot]
455127ad5e
Merge master into staging-next 2023-03-16 18:01:20 +00:00
Bernardo Meurer
6e55733359
Merge pull request #219747 from Stunkymonkey/deprecate-isNull 2023-03-16 11:10:22 -03:00
Martin Weinelt
19680e9902
Merge pull request #217568 from trofi/stdenv-parallel-install
stdenv/generic/setup.sh: enable parallel installs for parallel builds
2023-03-15 17:10:19 +00:00
Martin Weinelt
ef91384e6f Merge remote-tracking branch 'origin/master' into staging-next 2023-03-10 13:09:25 +01:00
Bernardo Meurer
e9cf2d1a41
Merge pull request #188334 from amjoseph-nixpkgs/pr/bootstrapFiles/mips64n32 2023-03-09 21:21:15 -05:00
Adam Joseph
49878856e6 https://github.com/NixOS/nixpkgs/pull/188334#issuecomment-1445425412 2023-03-07 02:11:30 -08:00
Felix Buehler
d10e69c86b treewide: deprecate isNull
https://nixos.org/manual/nix/stable/language/builtins.html#builtins-isNull
2023-03-06 22:40:04 +01:00
Martin Weinelt
5aeab34845
Merge pull request #218301 from rrbutani/fix/separate-debuginfo-with-lld
Fix `separate-debug-info` with lld
2023-03-04 00:43:26 +00:00
Weijia Wang
5028ec96c9
Merge pull request #202347 from stephank/feat/bootstrap-aarch64-darwin
stdenvBootstrapTools: native aarch64-darwin build
2023-03-03 20:05:28 +02:00
Vladimír Čunát
763470bbb0
Merge #207135: stdenv: aarch64-linux: gcc9 -> gcc12
...into staging
Also merge the commit that was referenced as the base for the build
of the new bootstrap tools (although others would give the same).
2023-03-01 12:54:52 +01:00
Sergei Trofimovich
69cf5181c3 stdenv/generic/setup.sh: enable parallel installs by default
The primary motivating example is openssl:

Before the change full package build took 1m54s minutes.
After the change full package build takes 59s.

About a 2x speedup.

The difference is visible because openssl builds hundreds of manpages
spawning a perl process per manual in `install` phase. Such a workload
is very easy to parallelize.

Another example would be `autotools`+`libtool` based build system where
install step requires relinking. The more binaries there are to relink
the more gain it will be to do it in parallel.

The change enables parallel installs by default only for buiilds that
already have parallel builds enabled. There is a high chance those build
systems already handle parallelism well but some packages will fail.

Consistently propagated the enableParallelBuilding to:
- cmake (enabled by default, similar to builds)
- ninja (set parallelism explicitly, don't rely on default)
- bmake (enable when requested)
- scons (enable when requested)
- meson (set parallelism explicitly, don't rely on default)
- waf (set parallelism explicitly, don't rely on default)
- qmake-4/5/6 (enable by default, similar to builds)
- xorg (always enable, similar to builds)
2023-02-26 22:02:09 +00:00
Weijia Wang
f05b5d4054 stdenv: aarch64-linux: gcc9 -> gcc12
Hydra job building them: https://hydra.nixos.org/build/208909151

The bootstrap files can be reproduced on the commit 21ec906463, e.g. by:
  cat $(nix-build pkgs/top-level/release.nix -QA stdenvBootstrapTools.aarch64-linux.dist)/nix-support/hydra-build-products
    file tarball /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    file busybox /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
and the hashes as well:
  nix hash file /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/bootstrap-tools.tar.xz
    sha256-aJvtsWeuQHbb14BGZ2EiOKzjQn46h3x3duuPEawG0eE=
  nix hash path /nix/store/kdpbw0plmjqlafjnpbz31ja51m4bd2dk-stdenv-bootstrap-tools/on-server/busybox
    sha256-0MuIeQlBUaeisqoFSu8y+8oB6K4ZG5Lhq8RcS9JqkFQ=

You can check this on any machine, as the builds are on cache.nixos.org
but also you can reproduce the hashes when rebuilt on aarch64-linux HW.
2023-02-26 23:26:30 +02:00
Rahul Butani
d364ee8d13
mkDerivation: do not disable separateDebugInfo on LLVM stdenvs
This was disabled here: b86e62d30d (diff-282a02cc3871874f16401347d8fadc90d59d7ab11f6a99eaa5173c3867e1a160)

h/t to @teh: b86e62d30d (commitcomment-77916294)
for pointing out that the failure that @matthewbauer was
seeing was caused by the `separate-debug-info.sh` `build-id` length
requirement that #146275 will relax

`lld` has had `--build-id` support dating back to LLVM4: https://reviews.llvm.org/D18091

This predates every `llvmPackages_` version currently in nixpkgs (and
certainly every version actually still used in `useLLVM` stdenvs) so
with the previous commit (asking `ld` for sufficiently long SHA1 hashes)
I think we can safely enable `separateDebugInfo` when using LLVM
bintools.
2023-02-25 12:49:40 -06:00
github-actions[bot]
3cdd771820
Merge staging-next into staging 2023-02-23 18:01:49 +00:00
Alyssa Ross
52c286ee5b
Merge remote-tracking branch 'origin/master' into staging-next
Conflicts:
	pkgs/development/libraries/pmdk/default.nix
2023-02-23 13:51:34 +00:00
Artturin
f9fdf2d402 treewide: move NIX_CFLAGS_COMPILE to the env attrset
with structuredAttrs lists will be bash arrays which cannot be exported
which will be a issue with some patches and some wrappers like cc-wrapper

this makes it clearer that NIX_CFLAGS_COMPILE must be a string as lists
in env cause a eval failure
2023-02-22 21:23:04 +02:00
Sergei Trofimovich
3057968396
Merge pull request #210019 from amjoseph-nixpkgs/pr/stdenv/verify-comments
express #208478 as assertions
2023-02-21 19:38:58 +00:00
Adam Joseph
d7aad24531 express #208478 as assertions
PR #208478 added a lot of documentation about which packages were
rebuilt in each stage of the stdenv bootstrap.  However nothing
checks that these comments agree with reality; they can bitrot over
time.  This PR rewrites those comments as assertions, so they cannot
bitrot.

This conversion did expose some ambiguity in our scheme for naming
the stages.   Suppose that `pkgs.stdenv.name=="stdenv-stage4", then
which of these is "the stage4 coreutils"?

```
pkgs.coreutils
pkgs.stdenv.__bootPackages.coreutils
```

The choice is arbitrary, and both choices have confusing corner
cases.  We should revisit this at some point.
2023-02-20 23:20:30 -08:00
Artturi
ee54eb7d21
Merge pull request #216383 from Artturin/bintoolswrappermold 2023-02-17 19:32:06 +02:00
Artturin
299a7bd35e stdenvAdapters: add useMoldLinker 2023-02-17 06:35:24 +02:00
Robert Scott
0eedcfc3f4
Merge pull request #212498 from risicle/ris-fortify3
hardening flags: add `FORTIFY_SOURCE=3` support
2023-02-16 21:19:30 +00:00
github-actions[bot]
c4fe2133de
Merge staging-next into staging 2023-02-15 06:01:44 +00:00
github-actions[bot]
58cfebde3e
Merge staging-next into staging 2023-02-15 00:02:50 +00:00
Felix Buehler
bc3d5934d7 treewide: use lib.optionals 2023-02-14 19:11:59 +01:00
Felix Buehler
cdb39a86e0 treewide: use optionalString 2023-02-13 21:52:34 +01:00
github-actions[bot]
29c2637dda
Merge staging-next into staging 2023-02-13 12:02:05 +00:00
github-actions[bot]
fd8daee7c2
Merge master into staging-next 2023-02-13 12:01:30 +00:00