2018-07-02 13:02:23 +02:00
|
|
|
{ stdenv49
|
foundationdb: rework python bindings, build system
FoundationDB uses Python at build time for some code generation.
However, it also has the official python bindings inside the source code
too, and the code for the Python bindings has some of it auto-generated
at compile time.
This made building python packages unattractive: we want to use the
source code generated from the FoundationDB build, but we don't want to
rebuild it. Previously we would override the 'python' input to the
FoundationDB module, but this meant we would do a complete rebuild, as
it was a necessary build time dependency, even though the resulting
generated code itself would not change. Furthermore, FoundationDB
versions < 6.0 don't properly support Python 3 *for the build system*,
though the bindings supported it, so that caused build failures. But the
first effect is the worst: it meant building separate python2 and
python3 packages implied two complete rebuilds of a single FoundationDB
version. This meant rather than 3 FDB builds, we'd do 3*N where N = the
number of major Python versions we support.
Finally, because we did not use pip to generate a wheel that we install
with metadata recorded for the installation, the FoundationDB python
package couldn't be used as an input to other setup.py-based packages:
there would be no recorded metadata in the dist-info folder which would
say this is the foundationdb package. This greatly limits its utility.
To fix all this, we do a few things:
- Apply some patches to fix the build system with Python 3.x for
older FoundationDB versions. (This is nice if end-users have
overridden the global Python version for some reason.)
- Move python directly into nativeBuildInputs, so it is only a
build time dependency.
- Take the python source code from the ./bindings directory and
tar it up use later after the build is done, so we get to keep
the generated code. This is the new 'pythonsrc' output from the
build. This code doesn't change based on whether or not the input
or resulting package is using Python 2 or 3, it's totally
deterministic.
- The build system also patches up the python source code a little,
so it can be installed directly with setup.py (it needs a little
stuff that it normally expects the build system to do.)
- Rework the python package to a separate file that uses
buildPythonPackage directly. Because the source code is already
prepared, it needs almost nothing else. Furthermore, this kills
the override itself for the foundationdb package, meaning rebuilds
are no longer needed.
- This package is very simple and just uses foundationdb.pythonsrc
as its source input. It also ensures a link to libfdb_c.so can
be found by ctypes (using substituteInPlace)
- python-packages.nix now just uses callPackage directly.
The net effect of this is, most importantly, that python packages do not
imply a full rebuild of the server source code: building python2 and
python3 packages from a version of FoundationDB now does not need to
override the foundationdb python input, reducing the number of needless
builds. They instead just run setup.py with the given version as input.
The second biggest effect is that wheel metadata is recorded correctly,
meaning dependent-python-packages that want to use the FoundationDB
bindings e.g. from PyPi should now work fine with buildPythonPackage.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2018-11-16 00:03:31 +01:00
|
|
|
, lib, fetchurl, fetchpatch, fetchFromGitHub
|
2018-07-02 13:02:23 +02:00
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
, which, findutils, m4, gawk
|
2018-12-23 00:38:37 +01:00
|
|
|
, python, openjdk, mono, libressl
|
2018-04-20 13:35:35 +02:00
|
|
|
}:
|
|
|
|
|
|
|
|
let
|
2018-07-02 13:02:23 +02:00
|
|
|
# hysterical raisins dictate a version of boost this old. however,
|
|
|
|
# we luckily do not need to build anything, we just need the header
|
|
|
|
# files.
|
|
|
|
boost152 = stdenv49.mkDerivation rec {
|
|
|
|
name = "boost-headers-1.52.0";
|
|
|
|
|
|
|
|
src = fetchurl {
|
|
|
|
url = "mirror://sourceforge/boost/boost_1_52_0.tar.bz2";
|
|
|
|
sha256 = "14mc7gsnnahdjaxbbslzk79rc0d12h1i681cd3srdwr3fzynlar2";
|
|
|
|
};
|
|
|
|
|
|
|
|
configurePhase = ":";
|
|
|
|
buildPhase = ":";
|
|
|
|
installPhase = "mkdir -p $out/include && cp -R boost $out/include/";
|
|
|
|
};
|
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
makeFdb =
|
|
|
|
{ version
|
|
|
|
, branch
|
2018-07-02 13:02:23 +02:00
|
|
|
, sha256
|
|
|
|
|
|
|
|
# the revision can be inferred from the fdb tagging policy
|
|
|
|
, rev ? "refs/tags/${version}"
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-07-02 13:02:23 +02:00
|
|
|
# in theory newer versions of fdb support newer compilers, but they
|
|
|
|
# don't :( maybe one day
|
|
|
|
, stdenv ? stdenv49
|
|
|
|
|
|
|
|
# in theory newer versions of fdb support newer boost versions, but they
|
|
|
|
# don't :( maybe one day
|
|
|
|
, boost ? boost152
|
2018-11-18 05:10:12 +01:00
|
|
|
|
|
|
|
# if an release is unofficial/a prerelease, then make sure this is set
|
|
|
|
, officialRelease ? true
|
2018-05-01 09:17:46 +02:00
|
|
|
}: stdenv.mkDerivation rec {
|
|
|
|
name = "foundationdb-${version}";
|
|
|
|
inherit version;
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
src = fetchFromGitHub {
|
|
|
|
owner = "apple";
|
|
|
|
repo = "foundationdb";
|
|
|
|
inherit rev sha256;
|
|
|
|
};
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-12-23 00:38:37 +01:00
|
|
|
nativeBuildInputs = [ python openjdk gawk which m4 findutils mono ];
|
foundationdb: rework python bindings, build system
FoundationDB uses Python at build time for some code generation.
However, it also has the official python bindings inside the source code
too, and the code for the Python bindings has some of it auto-generated
at compile time.
This made building python packages unattractive: we want to use the
source code generated from the FoundationDB build, but we don't want to
rebuild it. Previously we would override the 'python' input to the
FoundationDB module, but this meant we would do a complete rebuild, as
it was a necessary build time dependency, even though the resulting
generated code itself would not change. Furthermore, FoundationDB
versions < 6.0 don't properly support Python 3 *for the build system*,
though the bindings supported it, so that caused build failures. But the
first effect is the worst: it meant building separate python2 and
python3 packages implied two complete rebuilds of a single FoundationDB
version. This meant rather than 3 FDB builds, we'd do 3*N where N = the
number of major Python versions we support.
Finally, because we did not use pip to generate a wheel that we install
with metadata recorded for the installation, the FoundationDB python
package couldn't be used as an input to other setup.py-based packages:
there would be no recorded metadata in the dist-info folder which would
say this is the foundationdb package. This greatly limits its utility.
To fix all this, we do a few things:
- Apply some patches to fix the build system with Python 3.x for
older FoundationDB versions. (This is nice if end-users have
overridden the global Python version for some reason.)
- Move python directly into nativeBuildInputs, so it is only a
build time dependency.
- Take the python source code from the ./bindings directory and
tar it up use later after the build is done, so we get to keep
the generated code. This is the new 'pythonsrc' output from the
build. This code doesn't change based on whether or not the input
or resulting package is using Python 2 or 3, it's totally
deterministic.
- The build system also patches up the python source code a little,
so it can be installed directly with setup.py (it needs a little
stuff that it normally expects the build system to do.)
- Rework the python package to a separate file that uses
buildPythonPackage directly. Because the source code is already
prepared, it needs almost nothing else. Furthermore, this kills
the override itself for the foundationdb package, meaning rebuilds
are no longer needed.
- This package is very simple and just uses foundationdb.pythonsrc
as its source input. It also ensures a link to libfdb_c.so can
be found by ctypes (using substituteInPlace)
- python-packages.nix now just uses callPackage directly.
The net effect of this is, most importantly, that python packages do not
imply a full rebuild of the server source code: building python2 and
python3 packages from a version of FoundationDB now does not need to
override the foundationdb python input, reducing the number of needless
builds. They instead just run setup.py with the given version as input.
The second biggest effect is that wheel metadata is recorded correctly,
meaning dependent-python-packages that want to use the FoundationDB
bindings e.g. from PyPi should now work fine with buildPythonPackage.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2018-11-16 00:03:31 +01:00
|
|
|
buildInputs = [ libressl boost ];
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
patches =
|
2018-07-02 13:02:23 +02:00
|
|
|
[ # For 5.2+, we need a slightly adjusted patch to fix all the ldflags
|
|
|
|
(if lib.versionAtLeast version "5.2"
|
2018-07-21 00:50:08 +02:00
|
|
|
then (if lib.versionAtLeast version "6.0"
|
|
|
|
then ./ldflags-6.0.patch
|
|
|
|
else ./ldflags-5.2.patch)
|
2018-07-02 13:02:23 +02:00
|
|
|
else ./ldflags-5.1.patch)
|
foundationdb: rework python bindings, build system
FoundationDB uses Python at build time for some code generation.
However, it also has the official python bindings inside the source code
too, and the code for the Python bindings has some of it auto-generated
at compile time.
This made building python packages unattractive: we want to use the
source code generated from the FoundationDB build, but we don't want to
rebuild it. Previously we would override the 'python' input to the
FoundationDB module, but this meant we would do a complete rebuild, as
it was a necessary build time dependency, even though the resulting
generated code itself would not change. Furthermore, FoundationDB
versions < 6.0 don't properly support Python 3 *for the build system*,
though the bindings supported it, so that caused build failures. But the
first effect is the worst: it meant building separate python2 and
python3 packages implied two complete rebuilds of a single FoundationDB
version. This meant rather than 3 FDB builds, we'd do 3*N where N = the
number of major Python versions we support.
Finally, because we did not use pip to generate a wheel that we install
with metadata recorded for the installation, the FoundationDB python
package couldn't be used as an input to other setup.py-based packages:
there would be no recorded metadata in the dist-info folder which would
say this is the foundationdb package. This greatly limits its utility.
To fix all this, we do a few things:
- Apply some patches to fix the build system with Python 3.x for
older FoundationDB versions. (This is nice if end-users have
overridden the global Python version for some reason.)
- Move python directly into nativeBuildInputs, so it is only a
build time dependency.
- Take the python source code from the ./bindings directory and
tar it up use later after the build is done, so we get to keep
the generated code. This is the new 'pythonsrc' output from the
build. This code doesn't change based on whether or not the input
or resulting package is using Python 2 or 3, it's totally
deterministic.
- The build system also patches up the python source code a little,
so it can be installed directly with setup.py (it needs a little
stuff that it normally expects the build system to do.)
- Rework the python package to a separate file that uses
buildPythonPackage directly. Because the source code is already
prepared, it needs almost nothing else. Furthermore, this kills
the override itself for the foundationdb package, meaning rebuilds
are no longer needed.
- This package is very simple and just uses foundationdb.pythonsrc
as its source input. It also ensures a link to libfdb_c.so can
be found by ctypes (using substituteInPlace)
- python-packages.nix now just uses callPackage directly.
The net effect of this is, most importantly, that python packages do not
imply a full rebuild of the server source code: building python2 and
python3 packages from a version of FoundationDB now does not need to
override the foundationdb python input, reducing the number of needless
builds. They instead just run setup.py with the given version as input.
The second biggest effect is that wheel metadata is recorded correctly,
meaning dependent-python-packages that want to use the FoundationDB
bindings e.g. from PyPi should now work fine with buildPythonPackage.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2018-11-16 00:03:31 +01:00
|
|
|
]
|
2018-07-02 13:02:23 +02:00
|
|
|
# for 6.0+, we do NOT need to apply this version fix, since we can specify
|
|
|
|
# it ourselves. see configurePhase
|
foundationdb: rework python bindings, build system
FoundationDB uses Python at build time for some code generation.
However, it also has the official python bindings inside the source code
too, and the code for the Python bindings has some of it auto-generated
at compile time.
This made building python packages unattractive: we want to use the
source code generated from the FoundationDB build, but we don't want to
rebuild it. Previously we would override the 'python' input to the
FoundationDB module, but this meant we would do a complete rebuild, as
it was a necessary build time dependency, even though the resulting
generated code itself would not change. Furthermore, FoundationDB
versions < 6.0 don't properly support Python 3 *for the build system*,
though the bindings supported it, so that caused build failures. But the
first effect is the worst: it meant building separate python2 and
python3 packages implied two complete rebuilds of a single FoundationDB
version. This meant rather than 3 FDB builds, we'd do 3*N where N = the
number of major Python versions we support.
Finally, because we did not use pip to generate a wheel that we install
with metadata recorded for the installation, the FoundationDB python
package couldn't be used as an input to other setup.py-based packages:
there would be no recorded metadata in the dist-info folder which would
say this is the foundationdb package. This greatly limits its utility.
To fix all this, we do a few things:
- Apply some patches to fix the build system with Python 3.x for
older FoundationDB versions. (This is nice if end-users have
overridden the global Python version for some reason.)
- Move python directly into nativeBuildInputs, so it is only a
build time dependency.
- Take the python source code from the ./bindings directory and
tar it up use later after the build is done, so we get to keep
the generated code. This is the new 'pythonsrc' output from the
build. This code doesn't change based on whether or not the input
or resulting package is using Python 2 or 3, it's totally
deterministic.
- The build system also patches up the python source code a little,
so it can be installed directly with setup.py (it needs a little
stuff that it normally expects the build system to do.)
- Rework the python package to a separate file that uses
buildPythonPackage directly. Because the source code is already
prepared, it needs almost nothing else. Furthermore, this kills
the override itself for the foundationdb package, meaning rebuilds
are no longer needed.
- This package is very simple and just uses foundationdb.pythonsrc
as its source input. It also ensures a link to libfdb_c.so can
be found by ctypes (using substituteInPlace)
- python-packages.nix now just uses callPackage directly.
The net effect of this is, most importantly, that python packages do not
imply a full rebuild of the server source code: building python2 and
python3 packages from a version of FoundationDB now does not need to
override the foundationdb python input, reducing the number of needless
builds. They instead just run setup.py with the given version as input.
The second biggest effect is that wheel metadata is recorded correctly,
meaning dependent-python-packages that want to use the FoundationDB
bindings e.g. from PyPi should now work fine with buildPythonPackage.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2018-11-16 00:03:31 +01:00
|
|
|
++ (lib.optional (!lib.versionAtLeast version "6.0") ./fix-scm-version.patch)
|
|
|
|
# Versions less than 6.0 have a busted Python 3 build due to an outdated
|
|
|
|
# use of 'print'. Also apply an update to the six module with many bugfixes,
|
|
|
|
# which is in 6.0+ as well
|
|
|
|
++ (lib.optional (!lib.versionAtLeast version "6.0") (fetchpatch {
|
|
|
|
name = "update-python-six.patch";
|
|
|
|
url = "https://github.com/apple/foundationdb/commit/4bd9efc4fc74917bc04b07a84eb065070ea7edb2.patch";
|
|
|
|
sha256 = "030679lmc86f1wzqqyvxnwjyfrhh54pdql20ab3iifqpp9i5mi85";
|
|
|
|
}))
|
|
|
|
++ (lib.optional (!lib.versionAtLeast version "6.0") (fetchpatch {
|
|
|
|
name = "import-for-python-print.patch";
|
|
|
|
url = "https://github.com/apple/foundationdb/commit/ded17c6cd667f39699cf663c0e87fe01e996c153.patch";
|
|
|
|
sha256 = "11y434w68cpk7shs2r22hyrpcrqi8vx02cw7v5x79qxvnmdxv2an";
|
|
|
|
}))
|
|
|
|
;
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
postPatch = ''
|
2018-07-02 13:02:23 +02:00
|
|
|
# note: this does not do anything for 6.0+
|
2018-05-01 09:17:46 +02:00
|
|
|
substituteInPlace ./build/scver.mk \
|
|
|
|
--subst-var-by NIXOS_FDB_VERSION_ID "${rev}" \
|
|
|
|
--subst-var-by NIXOS_FDB_SCBRANCH "${branch}"
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
substituteInPlace ./Makefile \
|
|
|
|
--replace 'shell which ccache' 'shell true' \
|
|
|
|
--replace -Werror ""
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
substituteInPlace ./Makefile \
|
|
|
|
--replace libstdc++_pic libstdc++
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
substituteInPlace ./build/link-validate.sh \
|
|
|
|
--replace 'exit 1' '#exit 1'
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
patchShebangs .
|
2018-07-21 00:50:08 +02:00
|
|
|
'' + lib.optionalString (lib.versionAtLeast version "6.0") ''
|
|
|
|
substituteInPlace ./Makefile \
|
|
|
|
--replace 'TLS_LIBS +=' '#TLS_LIBS +=' \
|
|
|
|
--replace 'LDFLAGS :=' 'LDFLAGS := -ltls -lssl -lcrypto'
|
2018-05-01 09:17:46 +02:00
|
|
|
'';
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-07-27 01:43:45 +02:00
|
|
|
separateDebugInfo = true;
|
2018-05-01 09:17:46 +02:00
|
|
|
enableParallelBuilding = true;
|
2018-07-21 00:50:08 +02:00
|
|
|
|
2018-08-04 23:15:20 +02:00
|
|
|
makeFlags = [ "all" "fdb_java" "fdb_python" ]
|
2018-07-21 00:50:08 +02:00
|
|
|
# Don't compile FDBLibTLS if we don't need it in 6.0 or later;
|
|
|
|
# it gets statically linked in
|
|
|
|
++ lib.optional (!lib.versionAtLeast version "6.0") [ "fdb_c" ]
|
|
|
|
# Needed environment overrides
|
2018-07-27 01:43:45 +02:00
|
|
|
++ [ "KVRELEASE=1"
|
|
|
|
"NOSTRIP=1"
|
2018-11-18 05:10:12 +01:00
|
|
|
] ++ lib.optional officialRelease [ "RELEASE=true" ];
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-07-02 13:02:23 +02:00
|
|
|
# on 6.0 and later, we can specify all this information manually
|
|
|
|
configurePhase = lib.optionalString (lib.versionAtLeast version "6.0") ''
|
|
|
|
export SOURCE_CONTROL=GIT
|
|
|
|
export SCBRANCH="${branch}"
|
|
|
|
export VERSION_ID="${rev}"
|
|
|
|
'';
|
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
installPhase = ''
|
|
|
|
mkdir -vp $out/{bin,libexec/plugins} $lib/{lib,share/java} $dev/include/foundationdb
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-07-21 00:50:08 +02:00
|
|
|
'' + lib.optionalString (!lib.versionAtLeast version "6.0") ''
|
2018-08-04 23:15:20 +02:00
|
|
|
# we only copy the TLS library on < 6.0, since it's compiled-in otherwise
|
2018-05-01 09:17:46 +02:00
|
|
|
cp -v ./lib/libFDBLibTLS.so $out/libexec/plugins/FDBLibTLS.so
|
2018-07-21 00:50:08 +02:00
|
|
|
'' + ''
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-08-04 23:15:20 +02:00
|
|
|
# C API
|
|
|
|
cp -v ./lib/libfdb_c.so $lib/lib
|
2018-05-01 09:17:46 +02:00
|
|
|
cp -v ./bindings/c/foundationdb/fdb_c.h $dev/include/foundationdb
|
|
|
|
cp -v ./bindings/c/foundationdb/fdb_c_options.g.h $dev/include/foundationdb
|
2018-11-16 01:03:17 +01:00
|
|
|
cp -v ./fdbclient/vexillographer/fdb.options $dev/include/foundationdb
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-08-04 23:15:20 +02:00
|
|
|
# java
|
2018-05-05 08:29:39 +02:00
|
|
|
cp -v ./bindings/java/foundationdb-client.jar $lib/share/java/fdb-java.jar
|
2018-05-01 06:35:12 +02:00
|
|
|
|
2018-08-04 23:15:20 +02:00
|
|
|
# python
|
foundationdb: rework python bindings, build system
FoundationDB uses Python at build time for some code generation.
However, it also has the official python bindings inside the source code
too, and the code for the Python bindings has some of it auto-generated
at compile time.
This made building python packages unattractive: we want to use the
source code generated from the FoundationDB build, but we don't want to
rebuild it. Previously we would override the 'python' input to the
FoundationDB module, but this meant we would do a complete rebuild, as
it was a necessary build time dependency, even though the resulting
generated code itself would not change. Furthermore, FoundationDB
versions < 6.0 don't properly support Python 3 *for the build system*,
though the bindings supported it, so that caused build failures. But the
first effect is the worst: it meant building separate python2 and
python3 packages implied two complete rebuilds of a single FoundationDB
version. This meant rather than 3 FDB builds, we'd do 3*N where N = the
number of major Python versions we support.
Finally, because we did not use pip to generate a wheel that we install
with metadata recorded for the installation, the FoundationDB python
package couldn't be used as an input to other setup.py-based packages:
there would be no recorded metadata in the dist-info folder which would
say this is the foundationdb package. This greatly limits its utility.
To fix all this, we do a few things:
- Apply some patches to fix the build system with Python 3.x for
older FoundationDB versions. (This is nice if end-users have
overridden the global Python version for some reason.)
- Move python directly into nativeBuildInputs, so it is only a
build time dependency.
- Take the python source code from the ./bindings directory and
tar it up use later after the build is done, so we get to keep
the generated code. This is the new 'pythonsrc' output from the
build. This code doesn't change based on whether or not the input
or resulting package is using Python 2 or 3, it's totally
deterministic.
- The build system also patches up the python source code a little,
so it can be installed directly with setup.py (it needs a little
stuff that it normally expects the build system to do.)
- Rework the python package to a separate file that uses
buildPythonPackage directly. Because the source code is already
prepared, it needs almost nothing else. Furthermore, this kills
the override itself for the foundationdb package, meaning rebuilds
are no longer needed.
- This package is very simple and just uses foundationdb.pythonsrc
as its source input. It also ensures a link to libfdb_c.so can
be found by ctypes (using substituteInPlace)
- python-packages.nix now just uses callPackage directly.
The net effect of this is, most importantly, that python packages do not
imply a full rebuild of the server source code: building python2 and
python3 packages from a version of FoundationDB now does not need to
override the foundationdb python input, reducing the number of needless
builds. They instead just run setup.py with the given version as input.
The second biggest effect is that wheel metadata is recorded correctly,
meaning dependent-python-packages that want to use the FoundationDB
bindings e.g. from PyPi should now work fine with buildPythonPackage.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2018-11-16 00:03:31 +01:00
|
|
|
cp LICENSE ./bindings/python
|
|
|
|
substitute ./bindings/python/setup.py.in ./bindings/python/setup.py \
|
|
|
|
--replace 'VERSION' "${version}"
|
|
|
|
rm -f ./bindings/python/setup.py.in
|
2018-08-04 23:15:20 +02:00
|
|
|
rm -f ./bindings/python/fdb/*.pth # remove useless files
|
foundationdb: rework python bindings, build system
FoundationDB uses Python at build time for some code generation.
However, it also has the official python bindings inside the source code
too, and the code for the Python bindings has some of it auto-generated
at compile time.
This made building python packages unattractive: we want to use the
source code generated from the FoundationDB build, but we don't want to
rebuild it. Previously we would override the 'python' input to the
FoundationDB module, but this meant we would do a complete rebuild, as
it was a necessary build time dependency, even though the resulting
generated code itself would not change. Furthermore, FoundationDB
versions < 6.0 don't properly support Python 3 *for the build system*,
though the bindings supported it, so that caused build failures. But the
first effect is the worst: it meant building separate python2 and
python3 packages implied two complete rebuilds of a single FoundationDB
version. This meant rather than 3 FDB builds, we'd do 3*N where N = the
number of major Python versions we support.
Finally, because we did not use pip to generate a wheel that we install
with metadata recorded for the installation, the FoundationDB python
package couldn't be used as an input to other setup.py-based packages:
there would be no recorded metadata in the dist-info folder which would
say this is the foundationdb package. This greatly limits its utility.
To fix all this, we do a few things:
- Apply some patches to fix the build system with Python 3.x for
older FoundationDB versions. (This is nice if end-users have
overridden the global Python version for some reason.)
- Move python directly into nativeBuildInputs, so it is only a
build time dependency.
- Take the python source code from the ./bindings directory and
tar it up use later after the build is done, so we get to keep
the generated code. This is the new 'pythonsrc' output from the
build. This code doesn't change based on whether or not the input
or resulting package is using Python 2 or 3, it's totally
deterministic.
- The build system also patches up the python source code a little,
so it can be installed directly with setup.py (it needs a little
stuff that it normally expects the build system to do.)
- Rework the python package to a separate file that uses
buildPythonPackage directly. Because the source code is already
prepared, it needs almost nothing else. Furthermore, this kills
the override itself for the foundationdb package, meaning rebuilds
are no longer needed.
- This package is very simple and just uses foundationdb.pythonsrc
as its source input. It also ensures a link to libfdb_c.so can
be found by ctypes (using substituteInPlace)
- python-packages.nix now just uses callPackage directly.
The net effect of this is, most importantly, that python packages do not
imply a full rebuild of the server source code: building python2 and
python3 packages from a version of FoundationDB now does not need to
override the foundationdb python input, reducing the number of needless
builds. They instead just run setup.py with the given version as input.
The second biggest effect is that wheel metadata is recorded correctly,
meaning dependent-python-packages that want to use the FoundationDB
bindings e.g. from PyPi should now work fine with buildPythonPackage.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2018-11-16 00:03:31 +01:00
|
|
|
rm -f ./bindings/python/*.rst ./bindings/python/*.mk
|
|
|
|
|
|
|
|
cp -R ./bindings/python/ tmp-pythonsrc/
|
|
|
|
tar -zcf $pythonsrc --transform s/tmp-pythonsrc/python-foundationdb/ ./tmp-pythonsrc/
|
2018-08-04 23:15:20 +02:00
|
|
|
|
|
|
|
# binaries
|
2018-05-01 09:17:46 +02:00
|
|
|
for x in fdbbackup fdbcli fdbserver fdbmonitor; do
|
|
|
|
cp -v "./bin/$x" $out/bin;
|
|
|
|
done
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
ln -sfv $out/bin/fdbbackup $out/bin/dr_agent
|
|
|
|
ln -sfv $out/bin/fdbbackup $out/bin/fdbrestore
|
|
|
|
ln -sfv $out/bin/fdbbackup $out/bin/fdbdr
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
ln -sfv $out/bin/fdbbackup $out/libexec/backup_agent
|
|
|
|
'';
|
2018-05-01 06:35:12 +02:00
|
|
|
|
foundationdb: rework python bindings, build system
FoundationDB uses Python at build time for some code generation.
However, it also has the official python bindings inside the source code
too, and the code for the Python bindings has some of it auto-generated
at compile time.
This made building python packages unattractive: we want to use the
source code generated from the FoundationDB build, but we don't want to
rebuild it. Previously we would override the 'python' input to the
FoundationDB module, but this meant we would do a complete rebuild, as
it was a necessary build time dependency, even though the resulting
generated code itself would not change. Furthermore, FoundationDB
versions < 6.0 don't properly support Python 3 *for the build system*,
though the bindings supported it, so that caused build failures. But the
first effect is the worst: it meant building separate python2 and
python3 packages implied two complete rebuilds of a single FoundationDB
version. This meant rather than 3 FDB builds, we'd do 3*N where N = the
number of major Python versions we support.
Finally, because we did not use pip to generate a wheel that we install
with metadata recorded for the installation, the FoundationDB python
package couldn't be used as an input to other setup.py-based packages:
there would be no recorded metadata in the dist-info folder which would
say this is the foundationdb package. This greatly limits its utility.
To fix all this, we do a few things:
- Apply some patches to fix the build system with Python 3.x for
older FoundationDB versions. (This is nice if end-users have
overridden the global Python version for some reason.)
- Move python directly into nativeBuildInputs, so it is only a
build time dependency.
- Take the python source code from the ./bindings directory and
tar it up use later after the build is done, so we get to keep
the generated code. This is the new 'pythonsrc' output from the
build. This code doesn't change based on whether or not the input
or resulting package is using Python 2 or 3, it's totally
deterministic.
- The build system also patches up the python source code a little,
so it can be installed directly with setup.py (it needs a little
stuff that it normally expects the build system to do.)
- Rework the python package to a separate file that uses
buildPythonPackage directly. Because the source code is already
prepared, it needs almost nothing else. Furthermore, this kills
the override itself for the foundationdb package, meaning rebuilds
are no longer needed.
- This package is very simple and just uses foundationdb.pythonsrc
as its source input. It also ensures a link to libfdb_c.so can
be found by ctypes (using substituteInPlace)
- python-packages.nix now just uses callPackage directly.
The net effect of this is, most importantly, that python packages do not
imply a full rebuild of the server source code: building python2 and
python3 packages from a version of FoundationDB now does not need to
override the foundationdb python input, reducing the number of needless
builds. They instead just run setup.py with the given version as input.
The second biggest effect is that wheel metadata is recorded correctly,
meaning dependent-python-packages that want to use the FoundationDB
bindings e.g. from PyPi should now work fine with buildPythonPackage.
Signed-off-by: Austin Seipp <aseipp@pobox.com>
2018-11-16 00:03:31 +01:00
|
|
|
outputs = [ "out" "lib" "dev" "pythonsrc" ];
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
meta = with stdenv.lib; {
|
|
|
|
description = "Open source, distributed, transactional key-value store";
|
|
|
|
homepage = https://www.foundationdb.org;
|
|
|
|
license = licenses.asl20;
|
2018-12-05 02:57:18 +01:00
|
|
|
platforms = [ "x86_64-linux" ];
|
2018-05-01 09:17:46 +02:00
|
|
|
maintainers = with maintainers; [ thoughtpolice ];
|
|
|
|
};
|
|
|
|
};
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
in with builtins; {
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-07-02 13:02:23 +02:00
|
|
|
foundationdb51 = makeFdb rec {
|
2018-05-01 09:17:46 +02:00
|
|
|
version = "5.1.7";
|
|
|
|
branch = "release-5.1";
|
|
|
|
sha256 = "1rc472ih24f9s5g3xmnlp3v62w206ny0pvvw02bzpix2sdrpbp06";
|
|
|
|
};
|
|
|
|
|
|
|
|
foundationdb52 = makeFdb rec {
|
2018-08-04 16:42:44 +02:00
|
|
|
version = "5.2.8";
|
2018-07-02 13:02:23 +02:00
|
|
|
branch = "release-5.2";
|
2018-08-04 16:42:44 +02:00
|
|
|
sha256 = "1kbmmhk2m9486r4kyjlc7bb3wd50204i0p6dxcmvl6pbp1bs0wlb";
|
2018-05-01 09:17:46 +02:00
|
|
|
};
|
2018-04-20 13:35:35 +02:00
|
|
|
|
2018-05-01 09:17:46 +02:00
|
|
|
foundationdb60 = makeFdb rec {
|
2018-12-05 02:41:50 +01:00
|
|
|
version = "6.0.17";
|
2018-07-21 00:50:08 +02:00
|
|
|
branch = "release-6.0";
|
2018-12-05 02:41:50 +01:00
|
|
|
sha256 = "00m6dkv2nm51zhiq049fiivnz8hpc8w21y024lykhn16kyjdnfhs";
|
2018-04-20 13:35:35 +02:00
|
|
|
};
|
|
|
|
}
|