Commit graph

51 commits

Author SHA1 Message Date
Peter Kolloch
8a6638daa9 build-support/rust/buildRustCrate: Search for matching Cargo.toml in sub directories
This is what cargo does for git repositories.

See related issues:

* https://github.com/kolloch/crate2nix/issues/53
* https://github.com/kolloch/crate2nix/issues/33
2020-03-09 15:11:50 +01:00
Peter Kolloch
04e7462ee6 buildRustCrate: refactor colored logging
* Make errors include the crate name and make them much more prominent.
* Move more code into lib.sh
* Already source generated logging code and lib.sh in configure
2020-03-09 14:26:28 +01:00
Andreas Rammhold
453589696b
Merge pull request #79816 from andir/buildRustCrate-no-override-dep
buildRustCrate: remove superfluous dependency overrides
2020-02-18 15:18:44 +01:00
Andreas Rammhold
be5597fc9d
buildRustCrate: remove superfluous dependency overrides
By overriding each dependency on every level of the dependency tree we
are creating a lot of unnecessary instances of the same derivation

Looking at the output size of `nix-instantiate --trace-function-calls
-vvvv …` and the execution time I got about a 10x improvement after
applying this change.

It was probably good intentions that lead to these overrides but in
practice no tooling (that I know of) really needs this. `carnix` and
`crate2nix` are fine without those overrides. Furthermore I believe that
it is the job of the tooling around `buildRustCrate` to provide a
coherent set of overrides. By not enforcing all of the overrides, debug
flags, verbosity, … to be the same throughout the closure we also allow
consumers to override specific aspects of the crates. Some (older?)
crates might need different `crateOverrides` then newer crates with the
same name. Currently such situations can not (easily) be implemented
with the override in-place.
2020-02-11 11:48:45 +01:00
Andreas Rammhold
56e11bc8df
buildRustCrate: remap the current build dir to / for (more) reproducible builds 2020-02-06 01:18:59 +01:00
Andreas Rammhold
a57d0fe0bb
buildRustCrate: fix #78412
`build.rs` files might create files. Those files are supposed to go into
`OUT_DIR` (envirionment variable) and not be overlayed onto the source
tree.
2020-01-28 14:07:58 +01:00
Andreas Rammhold
19698d15ce
buildRustCrateTests: add regression test for #74071 2020-01-28 14:07:58 +01:00
Andreas Rammhold
78faab1be0 buildRustCrateTests: add test case for rlib linking 2020-01-21 17:46:32 +01:00
Andreas Rammhold
406e0c9d51
buildRustCrateTests: fix some formatting issues 2020-01-21 17:32:48 +01:00
Andreas Rammhold
d6a8b55fb0
buildRustCrate: treat rlib crates just like lib crates
Both version provide `rlib` files to link against. Previously we would
try to find a matching shared library in the `lib` output.
2020-01-21 17:22:59 +01:00
Andreas Rammhold
69c96adc53
buildRustCrateTests: use releaseTools.aggregate
Previously I did use `runCommand` to do the same. Using
releaseTools.aggregate seems a lot saner and we might get nicer hydra
output of the tests that are failing.
2020-01-16 13:24:15 +01:00
Andreas Rammhold
29a8575e3d
buildRustCrate: remove one of the odd library filename cases
It used to be the case (ref missing) that cargo did treat
`src/$libName.rs` as an alternative to `src/lib.rs` when the latter
wasn't present. Recently I failed to reproduce that with vanilla cargo
and it started to cause pain with some crates of the form:

some_crate/
 `- src
   `- main.rs
   `- some_crate.rs

We would build `src/some_crate.rs` and thing it is a library while that
might not be the actual case. This crate is a valid `bin` crate not a
`lib` crate as far as I can tell from the samples I took.

I removed support for the previously required heuristic and commented
out the test cases in case we will need them again. We could crawl in
the Git history but chances are that the next person looking into this
doesn't know about the history.
2020-01-16 13:24:13 +01:00
Andreas Rammhold
3e61906e1c
buildRustCrate: slight "rewording" and reformatting
There is no point in reinventinb builtins through `filterAttrs` or the
like. Lets just stick to what we already have in our toolbelt.
2020-01-07 11:57:34 +01:00
Andreas Rammhold
a3a51763f9
buildRustCrate: add buildTests flag to tell rustc to build tests instead of binaries
This helps us instruct rustc to build tests instead of binaries. The
actual build will then ONLY produce test executables. This is a first
step towards having rust crate tests within nixpkgs.

We default back to only a single output in test cases since that is the
only reasonable thing to do here.

Producing libraries or binaries in addition to tests would theoretically
be feasible but usually generates different dependency trees. It is very
common to have some libraries in `[dev-depdendencies]` within Cargo.toml
just for your tests. To not start mixing things up going with a
dedicated derivation for the test build sounds like the best choice for
now.

To use this you must provide a proper test dependency chain to
`buildRustCrate` (as you would usually do with your non-test inputs).
And then set the `buildTests` attribute to `true`. The derivation will
then contain all tests that were built in `$out/tests`. All common test
patterns and directories should be supported and tested by this change.

Below is an example how you would run a single test from the derivation.
This commit contains some more examples in the `buildRustCrateTests`
attribute set that might be helpful.

```
let
  drv = buildRustCrate {
     …
     buildTests true;
  };
in runCommand "test-my-crate" {} ''
  touch $out
  exec ${drv}/tests/my-test
''
```
2020-01-07 11:57:34 +01:00
Andreas Rammhold
6383b42dae
buildRustCrate: fixup usage of builtins.filterSource
While unifying most of the lib function calls I accidentially changed
the filterSource functions as well. Since there were no tests I ended
up forgetting about this case (even thought I ran into it…).
2020-01-07 00:49:48 +01:00
Andreas Rammhold
2eaaf7aafd
buildRustCrate: move common build functions to a dedicated file
This means we aren't rebuilding hat file for each crate we are building
and the buildPhase expression is a lot easier to comprehent.
2019-12-12 13:55:04 +01:00
Andreas Rammhold
3f49d7a3ea
buildRustCrate: deduplicate dependency override code
The previous lines were only different in the kind of dependencies but
otherwise exactly the same. It makes the entire thing a bit more
readable by moving this into a function that takes care of this.
2019-12-12 01:03:41 +01:00
Andreas Rammhold
6ad22f5b4d
buildRustCrate: use less bash for the build script
We can get rid of a bunch of workarounds that were in the build script
before by just passing on the `crateBin` attribute.

Before we converted the list of attributes to a string only to convert
it back in bash during the build phase. We can do the entire looping
through builds in Nix and thus need no conversion and parsing of
attributes over and over again.

The big part that still remains bash is the heuristic that cargo
introduced and that we can't do at eval time.
2019-12-12 01:03:11 +01:00
Andreas Rammhold
5ad83267ed
buildRustCrate: reflow the way extraRustcOpts is constructed
This should make it more obvious that we have three parts to it and not
just one long gibberish string that makes up all of it.
2019-12-11 23:27:58 +01:00
Andreas Rammhold
d37f001164
buildRustCrate: rename makeDeps function to mkRustcDepArgs
This should carry the function better then `makeDeps` as it isn't
producing deps but the rustc arguments required to link against those.
2019-12-11 23:23:55 +01:00
Andreas Rammhold
f4aeabd04a
buildRustCrate: document and cleanup the symbol seeding
That code had been in the derivation for a while but no explanation was
given why that is needed. It might be helpful to our future selfs to
document why things are done the way they are.
2019-12-11 23:23:55 +01:00
Andreas Rammhold
db55d1f89d
buildRustCrate: use tr instead of sed (it reads a bit nicer)
I already have a few changes in here that will trigger rebuilds so I
might as well do that substitution now.
2019-12-11 22:40:19 +01:00
Andreas Rammhold
50b2ef28f7
buildRustCrate: move the color loggign & remove some runtime checks
The expression is already long and confusing enough without the color
stuff sprinkled in. Moving it to a dedicated file makes sense.

I switched a bit of the color support code to pure Nix since there
wasn't much point in doing that in bash while we can just do it in Nix.
2019-12-11 22:35:44 +01:00
Andreas Rammhold
0aac0e8d2c
buildRustCrate: builtins -> lib where possible
We can just use `lib` instead of `builtins` in all cases but the
`hashString` case. Also changed a few lines to make use of some optional
helpers from lib.
2019-12-11 22:01:36 +01:00
Jörg Thalheim
435c3ecde7
rust: add support for armv6l-linux and armv7l-linux (#73472)
rust: add support for armv6l-linux and armv7l-linux
2019-11-29 12:47:15 +00:00
Andreas Rammhold
1b748554d5
buildRustCrate: add lib output
This cuts down the dependency tree on some rust builds where a crate not
just exposes a binary but also a library. `$out/lib` contained a bunch
of extra support files that among other information carry linker flags
(including the full path to link-time dependencies). Worst case this led
to some binary outputs depending on the full build closure of rust
crates.

Moving all the `$out/lib` files to `$lib/lib` solves this nicely.

`lib` might be a bit weird here as they are most of the time just rlib
files (rust libraries). Those are essential only required during
compilation but they can also be shared objects (like with traditional
C-style packages). Which is why I went with `lib` for the new output.

One of the caveats we are running into here is that we do not (always)
know ahead of time of a crate produces just a library or just a binary.
Cargo allows for some ambiguity regarding whether or not a crate
provides one, two, … binaries and libraries as it's outputs. Ideally we
would be able to rely on the `crateType` entirely but so far that isn't
the case. More work on that area might show how difficult that actually
is.
2019-11-26 15:05:01 +01:00
Ben Wolsieffer
83ac9c07e4 rust: add support for armv6l-linux and armv7l-linux 2019-11-23 19:19:31 -05:00
Joachim Fasting
bad07dfac5
tree-wide: replace uses of splitString "." with lib.versions
Quoting from the splitString docstring:

   NOTE: this function is not performant and should never be used.

This replaces trivial uses of splitString for splitting version
strings with the (potentially builtin) splitVersion.
2019-09-26 17:42:49 +02:00
Andreas Rammhold
a69a6c1117
Merge pull request #68296 from danieldk/crateRenames
buildRustCrate: add support for renaming crates
2019-09-10 10:57:54 +02:00
Daniël de Kok
85c6d72011 buildRustCrate: add support for renaming crates
Before this change, buildRustCrate always called rustc with

--extern libName=[...]libName[...]

However, Cargo permits using a different name under which a dependency
is known to a crate. For example, rand 0.7.0 uses:

[dependencies]
getrandom_package = { version = "0.1.1", package = "getrandom", optional = true }

Which introduces the getrandom dependency such that it is known as
getrandom_package to the rand crate. In this case, the correct extern
flag is of the form

--extern getrandom_package=[...]getrandom[...]

which is currently not supported. In order to support such cases, this
change introduces a crateRenames argument to buildRustCrate. This
argument is an attribute set of dependencies that should be renamed. In
this case, crateRenames would be:

{
  "getrandom" = "getrandom_package";
}

The extern options are then built such that if the libName occurs as
an attribute in this set, it value will be used as the local
name. Otherwise libName will be used as before.
2019-09-08 19:17:52 +02:00
volth
08f68313a4 treewide: remove redundant rec 2019-08-28 11:07:32 +00:00
volth
35d68ef143 treewide: remove redundant quotes 2019-08-26 21:40:19 +00:00
volth
f3282c8d1e treewide: remove unused variables (#63177)
* treewide: remove unused variables

* making ofborg happy
2019-06-16 19:59:05 +00:00
Peter Kolloch
61ac550082 Fix #60125 - buildRustCrate: Always set CARGO_PKG_VERSION_PRE and CARGO_PKG_HOMEPAGE
(as cargo does)
2019-04-23 23:41:57 +02:00
Andreas Rammhold
e0b4356c0d
Merge pull request #57936 from andir/build-rust-crate-nix-build-cores
buildRustCrate: use $NIX_BUILD_CORES for each of the crates
2019-03-29 12:29:43 +01:00
Teo Klestrup Röijezon
5eec83eb83 buildRustCrate: Fix include filter
buildRustCrate has a handy `include` helper, that only imports those whitelisted
files and folders to the store.

However, the function's matching logic is broken and includes all files,
regardless of whether or not they're whitelisted, as long as the whitelist
contains at least one name (regardless of whether that name exists). This is
because it doesn't take into account that
`lib.strings.removePrefix "foo" "bar" == "bar"` (that is, paths that don't match
the prefix are passed straight through).
2019-03-25 15:24:42 +01:00
Andreas Rammhold
4c89619152
buildRustCrate: use $NIX_BUILD_CORES for each of the crates 2019-03-20 02:19:50 +01:00
Peter Kolloch
baa3d6f43b
buildRustCrate: Make CARGO_MANIFEST_DIR absolute
While it is not obvious from the source, cargo sets CARGO_MANIFEST_DIR to an absolute directory. This let to a build problem with the popular "tera" crate using the "pest" crate.

## Cargo details

The variable is set here:

f7c91ba622/src/cargo/core/compiler/compilation.rs (L229)

and computed from the `manifest_path`:

f7c91ba622/src/cargo/core/package.rs (L163)

The manifest path is also exported via `cargo metadata` where you can see that it is absolute.
2019-03-03 12:02:26 +01:00
Pierre-Étienne Meunier
32e94c2674 buildRustCrate: adding the description field 2019-02-25 15:22:03 +00:00
Andreas Rammhold
6a3b144d02
buildRustCrate: support rust editions
In combination with carnix we can now build crates that require a
specific edition of rust features. A few crates started requiring that
already and having this in nixpkgs is just logical.
2019-02-18 01:09:54 +01:00
Andreas Rammhold
044a8a24ed
buildRustCrate: pass extraRustcOpts to configure crate
Previously build flags would not be available during the configure phase
while they might be required to build the `build.rs` file.
2019-02-18 00:10:36 +01:00
Darius Jahandarie
da84602f61 buildRustCrate: fix extraRustcOpts so it is not discarded 2018-12-12 01:21:46 -06:00
Austin Seipp
3aa9091162 buildRustCreate: export RUSTDOC during cargo config
See https://github.com/NixOS/nixpkgs/pull/50452#issuecomment-443455411

Signed-off-by: Austin Seipp <aseipp@pobox.com>
2018-12-04 19:51:51 -06:00
Pierre-Étienne Meunier
0e8332ca2b Fixing "include" 2018-11-22 11:40:03 +00:00
Pierre-Étienne Meunier
f1de24feb8 Rust build-support: fixing a compilation error in some crates (such as proc-macro2) 2018-11-16 12:12:59 +00:00
Pierre-Etienne Meunier
ae3b4655a4 Carnix: 0.7.2 -> 0.8.10 (#40587)
Carnix: splits input into two parts: creates from creates.io and local ones
2018-10-28 00:06:29 +01:00
Andreas Rammhold
fc5e595003
buildRustCrate: added some edge cases with binaries
This commit adds test based on real-world crates (brotli).
There were a few more edge cases that were missing beforehand. Also it
turned out that we can get rid of the `finalBins` list since that will
now be handled during runtime.
2018-09-13 22:00:29 +02:00
Andreas Rammhold
0f95d05548 buildRustCrate: add test cases 2018-09-13 20:28:39 +02:00
Andreas Rammhold
fdc2017f1c buildRustCrate: binary heuristic should be able to treat spaces 2018-09-13 20:28:39 +02:00
Andreas Rammhold
1371815060 buildRustCrate: extracted builder scripts into dedicated files
The build expression got quiet large over time and to make it a bit
easier to grasp the different scripts involved in the build are now
separated from the nix file.
2018-09-13 20:28:39 +02:00