Source of this change goes back to 2009 and original version of
fetchgit at 205fb0c87e.
The nondeterminism is really caused by changing .git so leave other
files alone as they might be interesting.
Note: this causes a hash mismatch with Hydra's version of Git Plugin
which we should fix to comply.
Move all the dependencies to their own derivation, so that we don't publish all
of them if the wrapper is installed in a profile.
The previous solution just moved them to a custom directory to avoid conflicts,
this refactors that and completely hides them, while preserving the desired
improvement of adding only one directory to each of the emacs search paths
The chroot caps restriction disallows chroot'ed processes from running
any command that requires `CAP_SYS_ADMIN`, breaking `nixos-rebuild`. See
e.g., https://github.com/NixOS/nixpkgs/issues/15293
This significantly weakens chroot protections, but to break
nixos-rebuild out of the box is too severe.
There are no users of it in main tree and recent merge
of multiple outputs branch makes it obsolete for private trees
too.
At the time hook was created, recently merged multiple output
branch was relying on passing flags to autotools to split
outputs, which obviously wasn't working for other build systems
Scatter output was taking different approach where files were
moved out from a build tree based on known paths, which is more
or less what current multiple-outputs.sh hook is able to do too.
A nix specific set of tools for converting icon files
that are not in a freedesktop ready format.
I plan on using these tools for both `keepass` and
`retroarch` packages. It may benifit many other packages.
For every *.{exe,dll} in $output/bin/ we try to find all (potential)
transitive dependencies and symlink those DLLs into $output/bin
so they are found on invocation.
(DLLs are first searched in the directory of the running exe file.)
The links are relative, so relocating whole /nix/store won't break them.
The hook is activated on cygwin and when cross-compiling to mingw.
- the default --docdir is typically DATAROOTDIR/doc/pkgName
- I saw no other way than to employ some magic to guess this `pkgName`
- user can override it by setting $shareDocName
I supplied meta.outputsToInstall automatically in all
mkDerivation products, but some packages still don't use it.
The reported case: jekyll -> bundlerEnv -> buildEnv -> runCommand.
The /nix path in 4d200538 of the layer tar didn't exist for some
packages, such as cacert. This is because cacert just creates an /etc
directory and doesn't depend on any other /nix paths. If we tried
putting this directory in the tar and using overlayfs with it, we'd get
"Invalid argument" when trying to remove the directory.
We now check whether the closure is non-empty before telling tar to
store the /nix directory.
Fixes#14710.
There were two sources of non-determinisim coming into the images. The
first was tar mtimes, the second was pigz/gzip times.
An example image now passes with the --check flag.
I think the intention of this functionality was to provide a simple
alternative to the "runAsRoot" and "contents" attributes.
The implementation caused very slow builds of Docker images. Almost all
of the build time was spent in IO for tar, due to tarballs being
created, immediately extracted, then recreated. I had 30 minute builds
on some of my images which are now down to less than 2 minutes. A couple
of other users on #nix IRC have observed similar improvements.
The implementation also mutated the produced Docker layers without
changing their hashes. Using non-empty tarballs would produce images
which got cached incorrectly in Docker.
I have a commit which just fixes the performance problem but I opted to
completely remove the tarball feature after I found out that it didn't
correctly implement the Docker Image Specification due to the broken
hashing.
This is to avoid unwanted side effects when installing a wrapped emacs in the environment:
* All executables in the dependencies become available in the user environment
* All site-lisp binaries in the dependencies become accessible to unwrapped emacs
Also, both bin and site-lisp would generate conflicts so installing a wrapped emacs becomes really cumbersome
1. Update bower2nix version and add new/updated dependencies into
node-packages-generated.nix. This was done manually, with npm2nix
generating the initial set of derivations. In future, it would be
nice to have an automatic process (see #10358, #9332).
2. Add an override to nodePackages.bower2nix wrapping the commands so
that git is on the PATH.
3. Update fetchbower to support new command-line options of bower2nix,
and to allow github URL tag versions.
Previously, nix-prefetch-git would report the same JSON whether submodules were being fetched or not; with this change, the --fetch-submodules option will cause the JSON output to include "fetchSubmodules": true, so that fetchgit (builtins.fromJSON (builtins.readFile ./path/to/output.json)) will work.
The following parameters are now available:
* hardeningDisable
To disable specific hardening flags
* hardeningEnable
To enable specific hardening flags
Only the cc-wrapper supports this right now, but these may be reused by
other wrappers, builders or setup hooks.
cc-wrapper supports the following flags:
* fortify
* stackprotector
* pie (disabled by default)
* pic
* strictoverflow
* format
* relro
* bindnow
Some recent perl version introduced "keys" to return the keys
in random order. As some of the packages are solved by "provides" and
based on the order, this randomness affects what packages get into the
closure.
This problem may be in other nix perl scripts.
The importance of glibc makes it worthwhile to provide debug
symbols. However, this revealed an issue with separateDebugInfo: it
was indiscriminately adding --build-id to all ld invocations, while in
fact it should only do that for final links. Glibc also uses non-final
("relocatable") links, leading to subsequent failure to apply a build
ID ("Cannot create .note.gnu.build-id section, --build-id
ignored"). So now ld-wrapper.sh only passes --build-id for final
links.
Otherwise, when building glibc and other packages, the "strip" from
bootstrapTools is used, which doesn't recognise some tags produced by
the newer "ld" from binutils.
The two lines I removed technically assert the exact same thing, since `!a -> b`
is equivalent to `a || b`. So, I replaced the two lines with the more symmetric
form to make it clearer.
This commit fixes#6651.
Before this change the `nix-prefetch-git` script would use a different store
name than nix's `fetchgit` function. Because of that it was not possible to
use `nix-prefetch-git` as a way to pre-populate the store (for example when
the user it using private git dependencies that needs access to the ssh agent)
In some cases the $sourceRoot is missing. Skip the hook instead
of showing the following cryptic error:
find: cannot search `': No such file or directory
/nix/store/0p1afvl8jcpi6dvsq2n58i90w9c59vz1-set-source-date-epoch-to-latest.sh: line 12: [: : integer expression expected
vcunat removed the warning; the hook will just skip silently in these cases.
Perhaps someone can improve on it some time.
Regression introduced by 4529ed1259.
I've missed this in #5096, not because of a messed up rebase as I have
guessed from a comment on #12635 but missed this in the first place.
The testing I did while working on the pull request weren't exhaustive
enough to cover this, because I haven't tested with packages that use
the propagatedUserEnvPkgs attribute.
In order to make the test a bit more exhaustive this time, let's test it
using:
nix-build -E 'with import ./. {}; buildEnv {
name = "testenv";
paths = [
pkgs.hello pkgs.binutils pkgs.libsoup pkgs.gnome3.yelp
pkgs.gnome3.totem
];
}'
And with this commit the errors no longer show up and the environment is
built correctly.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Fixes: #12635
- Now `pkg.outputUnspecified = true` but this attribute is missing in
every output, so we can recognize whether the user chose or not.
If (s)he didn't choose, we put `pkg.bin or pkg.out or pkg` into
`systemPackages`.
- `outputsToLink` is replaced by `extraOutputsToLink`.
We add extra outputs *regardless* of whether the user chose anything.
It's mainly meant for outputs with docs and debug symbols.
- Note that as a result, some libraries will disappear from system path.
* authorization token is optional
* registry url is taken from X-Docker-Endpoints header
* pull.sh correctly resumes partial layer downloads
* detjson.py does not fail on missing keys
The comment related to the `deepClone` and `no-deepClone` options was
misleading as these options have no relation with submodules, but on the
the depth in `git clone --depth n`.
Login mode can cause hidden problems, e.g. #12406. Generally we don't want
to read user's .bash_profile when we don't start an interactive shell inside
a chroot.
These environment variables allow using fetchgit with git:// URLs using
the SOCKS proxy technique described in 'Using Git with a SOCKS proxy':
http://www.patthoyts.tk/blog/using-git-with-socks-proxy.html
Briefly, GIT_PROXY_COMMAND is set to a script which invokes connect[1],
which reads SOCKS_PROXY, which might be pointing to a local instance of
'ssh -D'.
[1] pkgs/tools/networking/connect
The ld-wrapper.sh script calls `readlink` in some circumstances. We need
to ensure that this is the `readlink` from the `coreutils` package so
that flag support is as expected.
This is accomplished by explicitly setting PATH at the top of each shell
script.
Without doing this, the following happens with a trivial `main.c`:
```
nix-env -f "<nixpkgs>" -iA pkgs.clang
$ clang main.c -L /nix/../nix/store/2ankvagznq062x1gifpxwkk7fp3xwy63-xnu-2422.115.4/Library -o a.out
readlink: illegal option -- f
usage: readlink [-n] [file ...]
```
The key element is the `..` in the path supplied to the linker via a
`-L` flag. With this patch, the above invocation works correctly on
darwin, whose native `/usr/bin/readlink` does not support the `-f` flag.
The explicit path also ensures that the `grep` called by `cc-wrapper.sh`
is the one from Nix.
Fixes#6447
Building packages requires package-build.el from Melpa, but installing
packages only requires package.el. Packages from ELPA are already built,
so there is no need to involve package-build.el.
When building a package from a Melpa recipe file, get the Emacs package
name from the recipe. Nix is more restrictive about packages names than
Emacs, so the Nix name for a package is sometimes different.
Checking file contents is redundant in this case, because we will go
ahead anyway, regardless of whether the content is the same.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Originally wanted to include ignoreCollisions in cups-progs, but I think
it's better if we use ignoreCollisions only if there are _real_
collisions between files with different contents.
Of course, we also check whether the file permissions match, so you get
a collision if contents are the same but the permissions are different.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
For instance, a binary like libfoo.so will cause a symlink
lib/debug/libfoo.so.debug -> .build-id/<build-ID>.debug to be
created. This is primarily useful for use with eu-addr2line, if you
know the name of a binary and the relative address, but not the build
ID.
- fix in silencing some moveToOutput messages
- allow removing (developer) documentation even without defining outputs
(note: some paths are auto-removed by default, e.g. gtk-doc and man3)
Unfortunately, yesterday Nix got reverted to a version with broken
passAsFile implementation on some Hydra machines, so we have corrupted
files again. (E.g. http://hydra.nixos.org/build/29777678.) Forcing
another gratuitous rebuild to get rid of them.
(cherry picked from commit 75974d9220b8397c736ada76fb24eb934fa62f6c)
Also fix the hash in goPackages.inflect, the only user of the fetcher ATM.
Closes#12002 (different `inflect` fix), fixes#12012.
Using fetchzip-derived functions is likely more efficient than fetchhg,
and it's lighter on dependencies (hash is the same as with fetchhg in this case).
fetchbzr always uses the derivation name `bzr-export`. nix-prefetch-bzr
should use the same name for its output. This avoids duplicate downloads
and problems with forbidden characters in bazaar repository names.
Fixes#10819. emacsWithPackages will know its own package set. This
requires it to be in a package set, rather than at the top level, so it
lives in emacsPackagesNg.
This seems to be the root cause of the random page allocation failures
and @wizeman did a very good job on not only finding the root problem
but also giving a detailed explanation of it in #10828.
Here is an excerpt:
The problem here is that the kernel is trying to allocate a contiguous
section of 2^7=128 pages, which is 512 KB. This is way too much:
kernel pages tend to get fragmented over time and kernel developers
often go to great lengths to try allocating at most only 1 contiguous
page at a time whenever they can.
From the error message, it looks like the culprit is unionfs, but this
is misleading: unionfs is the name of the userspace process that was
running when the system ran out of memory, but it wasn't unionfs who
was allocating the memory: it was the kernel; specifically it was the
v9fs_dir_readdir_dotl() function, which is the code for handling the
readdir() function in the 9p filesystem (the filesystem that is used
to share a directory structure between a qemu host and its VM).
If you look at the code, here's what it's doing at the moment it tries
to allocate memory:
buflen = fid->clnt->msize - P9_IOHDRSZ;
rdir = v9fs_alloc_rdir_buf(file, buflen);
If you look into v9fs_alloc_rdir_buf(), you will see that it will try
to allocate a contiguous buffer of memory (using kzalloc(), which is a
wrapper around kmalloc()) of size buflen + 8 bytes or so.
So in reality, this code actually allocates a buffer of size
proportional to fid->clnt->msize. What is this msize? If you follow
the definition of the structures, you will see that it's the
negotiated buffer transfer size between 9p client and 9p server. On
the client side, it can be controlled with the msize mount option.
What this all means is that, the reason for running out of memory is
that the code (which we can't easily change) tries to allocate a
contiguous buffer of size more or less equal to "negotiated 9p
protocol buffer size", which seems to be way too big (in our NixOS
tests, at least).
After that initial finding, @lethalman tested the gnome3 gdm test
without setting the msize parameter at all and it seems to have resolved
the problem.
The reason why I'm committing this without testing against all of the
NixOS VM test is basically that I think we can only go better but not
worse than the current state.
Signed-off-by: aszlig <aszlig@redmoonstudios.org>
Previously is was assumed that bash was in the path when calling the
environment setup script. This changes all of the references of bash to
be absolute paths so that the user doesn't have to worry about the
environment they call it with.
