Commit graph

42 commits

Author SHA1 Message Date
Matthew Bauer
0046802ab6 fetchurl: only allow empty hash when cacert is available
We can use cacert to validate that the data passes SSL certificates.
Normally, this doesn’t happen because we already have the hash, but in
the hash = "" case we don’t.
2020-06-09 12:56:27 -05:00
Matthew Bauer
6367a55312 fetchurl: don’t run ‘postFetch’ on hashed-mirrors
hashed-mirrors are content addressed. So if $outputHash is in the
hashed-mirror, changes from ‘postFetch’ would already be made. So,
running postFetch will end up applying the change /again/, which we
don’t want.
2020-04-16 16:36:56 -05:00
Yegor Timoshenko
6fad2ae94b
fetchurl: warn on rev archives, resolves #32999 2018-10-16 13:46:19 +00:00
Yegor Timoshenko
a26d9e3ba8
Readd --retry 3 2018-06-11 01:48:00 +00:00
Dan Peebles
0cb623c3d9
fetchurl: add user agent
It would be nice to be able to track Nix requests. It's not trustworthy,
but can be helpful for stats and routing in HTTP logs.

Since `fetchurl` is used so widely, we should "magically" get a UA on
`fetchzip`, `fetchFromGitHub`, and other related fetchers.

Since `fetchurl` is only used for fixed-output derivations, this should
cause no mass rebuild.

User-Agent example: curl/7.57.0 Nixpkgs/18.03
2017-12-23 22:20:56 +00:00
Thomas Tuegel
c25199f697
fetchurl: remove unpaired call to `stopNest'
Fixes #27406.

Commit 5d4efb2c81 added an assertion to `stopNest'
which requires it be correctly paired with `startNest'. `fetchurl' calls
`stopNest', but never calls `startNest'; the former calls are removed.
2017-07-21 15:45:29 -05:00
Domen Kožar
c7383cb34b fetchurl: support executables 2016-01-19 11:17:49 +01:00
Eelco Dolstra
bb672805cd fetchurl: Always use tarballs.nixos.org
Otherwise, if the upstream mirror changes (rather than deletes) a
file, then tarballs.nixos.org won't be used even if it has a copy of
the original file, and so we'll get a hash mismatch.
2015-12-10 16:06:12 +01:00
ts468
db25ece3c8 fetchurl/builder.sh: style fix, close #10260 2015-10-07 09:48:58 +02:00
Aristid Breitkreuz
5c0816567d fix fetchurl 2014-10-05 19:23:27 +02:00
Domen Kožar
d6b7778753 fetchurl: fix mirroring after ecf5a08831 2014-10-05 09:53:18 +02:00
Domen Kožar
fd48e801d2 fetchurl: resume download of curl exits with status code 18 2014-10-01 13:58:42 +02:00
Eelco Dolstra
c8df888858 Add a function "fetchzip"
This function downloads and unpacks a file in one fixed-output
derivation. This is primarily useful for dynamically generated zip
files, such as GitHub's /archive URLs, where the unpacked content of
the zip file doesn't change, but the zip file itself may (e.g. due to
minor changes in the compression algorithm, or changes in timestamps).

Fetchzip is implemented by extending fetchurl with a "postFetch" hook
that is executed after the file has been downloaded. This hook can
thus perform arbitrary checks or transformations on the downloaded
file.
2014-05-08 15:30:17 +02:00
Gergely Risko
07b26ce493 fetchUrl: add curlOpts parameter (merge #535) 2013-05-16 10:45:16 +02:00
Eelco Dolstra
04765da09a fetchurl: Don't wait too long for the hashed mirrors
If the hashed mirror (nixos.org/tarballs) doesn't respond in 15
seconds, give up and try the original URL.
2013-02-06 15:15:28 +01:00
Eelco Dolstra
f337ab4430 fetchurl: Remove some compatibility hacks 2013-02-06 15:00:33 +01:00
Eelco Dolstra
4b50191fd4 * Remove a not very useful message that didn't get printed anyway
because the test condition was wrong.

svn path=/nixpkgs/trunk/; revision=26147
2011-03-03 14:59:58 +00:00
Yury G. Kudryashov
c74d23dbdf Fix typo, thanks to Rob Vermaas
svn path=/nixpkgs/trunk/; revision=22031
2010-05-28 06:49:32 +00:00
Yury G. Kudryashov
952bc80b4d Let fetchurl retry failed downloads. Add NIX_CURL_FLAGS impure var
svn path=/nixpkgs/trunk/; revision=22006
2010-05-27 18:59:19 +00:00
Marc Weber
ec0e356b9a Revert "adding NIX_CONTINUE_DOWNLOADS feature, see mkdir comment"
This reverts commit -r16401

svn path=/nixpkgs/trunk/; revision=16405
2009-07-16 17:17:23 +00:00
Marc Weber
10f6c2592c adding NIX_CONTINUE_DOWNLOADS feature, see mkdir comment
svn path=/nixpkgs/trunk/; revision=16401
2009-07-16 15:18:26 +00:00
Michael Raskin
f067828cdc Allow question marks in URL...
svn path=/nixpkgs/trunk/; revision=15507
2009-05-08 17:00:33 +00:00
Eelco Dolstra
aa84bce64d * Get rid of all references to nix.cs.uu.nl.
svn path=/nixpkgs/trunk/; revision=13299
2008-11-14 16:57:19 +00:00
Eelco Dolstra
eebb241519 * Write the list of mirrors to a file that we can reuse between
fetchurl instantiations, instead of passing the mirrors to fetchurl
  instantiations via environment variables.  This makes the resulting
  store derivations (.drv files) much smaller, which in turn makes
  nix-env/nix-instantiate faster (4.8 -> 4.2 seconds on nix-env -qa
  --out-path).

svn path=/nixpkgs/trunk/; revision=12695
2008-08-22 15:53:21 +00:00
Eelco Dolstra
a7c209e474 * Interface to support mirror:// in nix-prefetch-url. The function
resolveMirrorURLs produces a file containing the URLs to which a
  mirror:// URL resolves.

svn path=/nixpkgs/trunk/; revision=12410
2008-07-23 16:04:10 +00:00
Eelco Dolstra
76a8d120a4 * Allow the user to override the list of mirrors for specific
mirror:// sites through environment variables, e.g.

  NIX_MIRRORS_gnu="ftp://ftp.nluug.nl/pub/gnu/ ftp://ftp.gnu.org/pub/gnu/"

  or

  NIX_MIRRORS_sourceforge="http://surfnet.dl.sourceforge.net/sourceforge/"

svn path=/nixpkgs/trunk/; revision=9302
2007-09-11 15:00:49 +00:00
Eelco Dolstra
6c4fd2e3df * Allow the user to override the list of content-addressable mirrors
from the command-line by setting the NIX_HASHED_MIRRORS environment
  variable.

svn path=/nixpkgs/trunk/; revision=9301
2007-09-11 13:48:53 +00:00
Eelco Dolstra
45a2c87402 * Support for mirror:// URLs a la Gentoo (NIXPKGS-70). Instead of
fetchurl {
      url = http://heanet.dl.sourceforge.net/sourceforge/zapping/zapping-0.9.6.tar.bz2;
      md5 = "8306775c6a11de4d72345b5eee970ea6";
    };

  you can write

    fetchurl {
      url = mirror://sourceforge/zapping/zapping-0.9.6.tar.bz2;
      md5 = "8306775c6a11de4d72345b5eee970ea6";
    };

  which causes fetchurl to try the SourceForge mirrors listed in the
  `sourceforge' attribute in build-support/fetchurl/mirrors.nix.
  (They're currently tried in sequence, and the lists of mirrors are
  not configurable yet.)

  The syntax for mirror URLs is mirror://site/path/to/file, where
  `site' is currently one of `sourceforge', `gnu' (mirrors of
  ftp://ftp.gnu.org/pub/gnu) and `kernel' (mirrors of
  http://www.all.kernel.org/pub/).

svn path=/nixpkgs/trunk/; revision=9197
2007-08-27 12:44:01 +00:00
Eelco Dolstra
50321b735b * Basic transparent mirror support in fetchurl (NIXPKGS-70). When
fetching a file with hash HASH of type TYPE, we first try to
  download <base-url>/<type>/<hash>, where <base-url> is one of a list
  of mirrors.  For instance, given

    src = fetchurl {
      url = http://releases.mozilla.org/pub/mozilla.org/firefox/releases/2.0.0.6/source/firefox-2.0.0.6-source.tar.bz2;
      sha1 = "eb72f55e4a8bf08e8c6ef227c0ade3d068ba1082";
    };

  and the mirror list [http://nix.cs.uu.nl/dist/tarballs], we first
  try to download

    eb72f55e4a

  and if that fails, we use the original URL.

  The list of mirrors is not yet user-configurable.

* `fetchurl' now also accepts an argument `urls' instead of `url' for
  a list of alternative download locations, which fetchurl will try in
  sequence.

svn path=/nixpkgs/trunk/; revision=9190
2007-08-23 15:22:30 +00:00
Eelco Dolstra
f59f5d7d96 * Handle weird cases when the server redirects us while setting a
cookie.

svn path=/nixpkgs/trunk/; revision=7352
2006-12-15 21:28:59 +00:00
Eelco Dolstra
f95a93197a * Don't use EPSV.
svn path=/nixpkgs/trunk/; revision=6894
2006-10-28 22:07:25 +00:00
Eelco Dolstra
82e678362f * "." -> "source".
svn path=/nixpkgs/trunk/; revision=4335
2005-12-05 14:11:09 +00:00
Eelco Dolstra
44a1f60787 * md5' -> id' in compatibility mode.
svn path=/nixpkgs/trunk/; revision=2273
2005-02-22 15:34:58 +00:00
Eelco Dolstra
4eba4a211e * Check the hash, but only on Nix <= 0.7.
svn path=/nixpkgs/trunk/; revision=2272
2005-02-22 15:23:56 +00:00
Eelco Dolstra
cdaecff49f * Use Nix 0.8-style fixed outputs.
svn path=/nixpkgs/trunk/; revision=2257
2005-02-21 15:52:37 +00:00
Eelco Dolstra
7bf54b734e * Follow redirects.
svn path=/nixpkgs/trunk/; revision=899
2004-04-01 19:10:37 +00:00
Eelco Dolstra
ab7af4709a * Add `--fail' to curl invocation to catch errors.
svn path=/nixpkgs/trunk/; revision=889
2004-04-01 12:23:02 +00:00
Eelco Dolstra
12ae5363ea * Remove trivial builders.
* Make builders unexecutable by removing the hash-bang line and
  execute permission.
* Convert calls to `derivation' to `mkDerivation'.
* Remove `system' and `stdenv' attributes from calls to
  `mkDerivation'.  These transformations were all done automatically,
  so it is quite possible I broke stuff.
* Put the `mkDerivation' function in stdenv/generic.

svn path=/nixpkgs/trunk/; revision=874
2004-03-29 17:23:01 +00:00
Eelco Dolstra
4df6ec860c * Typo fix.
svn path=/nixpkgs/trunk/; revision=850
2004-03-27 15:44:05 +00:00
Eelco Dolstra
5941f66f0e * The stdenv setup script now defines a generic builder that allows
builders for typical Autoconf-style to be much shorten, e.g.,

    . $stdenv/setup
    genericBuild

  The generic builder does lots of stuff automatically:

  - Unpacks source archives specified by $src or $srcs (it knows about
    gzip, bzip2, tar, zip, and unpacked source trees).
  - Determines the source tree.
  - Applies patches specified by $patches.
  - Fixes libtool not to search for libraries in /lib etc.
  - Runs `configure'.
  - Runs `make'.
  - Runs `make install'.
  - Strips debug information from static libraries.
  - Writes nested log information (in the format accepted by
    `log2xml').

  There are also lots of hooks and variables to customise the generic
  builder.  See `stdenv/generic/docs.txt'.

* Adapted the base packages (i.e., the ones used by stdenv) to use the
  generic builder.

* We now use `curl' instead of `wget' to download files in `fetchurl'.

* Neither `curl' nor `wget' are part of stdenv.  We shouldn't
  encourage people to download stuff in builders (impure!).

* Updated some packages.

* `buildinputs' is now `buildInputs' (but the old name also works).

* `findInputs' in the setup script now prevents inputs from being
  processed multiple times (which could happen, e.g., if an input was
  a propagated input of several other inputs; this caused the size
  variables like $PATH to blow up exponentially in the worst case).

* Patched GNU Make to write nested log information in the format
  accepted by `log2xml'.  Also, prior to writing the build command,
  Make now writes a line `building X' to indicate what is being
  built.  This is unfortunately often obscured by the gigantic tool
  invocations in many Makefiles.  The actual build commands are marked
  `unimportant' so that they don't clutter pages generated by
  `log2html'.


svn path=/nixpkgs/trunk/; revision=845
2004-03-19 16:53:04 +00:00
Eelco Dolstra
38bc744a0b * Use md5sum instead of nix-hash.
svn path=/nixpkgs/trunk/; revision=584
2003-12-21 16:00:17 +00:00
Eelco Dolstra
48c3faca51 * Renamed pkgs-ng to pkgs.
svn path=/nixpkgs/trunk/; revision=502
2003-11-14 09:59:13 +00:00
Renamed from pkgs-ng/build-support/fetchurl/builder.sh (Browse further)