Commit graph

119 commits

Author SHA1 Message Date
pennae
54f4992e80 nixos/manual: render html with nixos-render-doc if !allowDocBook
this reproduces the docbook-generated html manual exactly enough to
appease the compare workflows while we still support both toolchains.
it's also a lot faster than the docbook toolchain, rendering the entire
html manual in about two seconds on this machine (while docbook needs
about 20).
2023-02-21 18:26:41 +01:00
pennae
81cf173256 nixos-render-docs: use multiprocessing for options
options processing is pretty slow right now, mostly because the
markdown-it-py parser is pure python (and with performance
pessimizations at that). options parsing *is* embarassingly parallel
though, so we can just fork out all the work to worker processes and
collect the results.

multiprocessing probably has a greater benefit on linux than on darwin
since the worker spawning method darwin uses is less efficient than
fork() on linux. this hasn't been tested on darwin, only on linux, but
if anything darwin will be faster with its preferred method.
2023-02-18 13:17:57 +01:00
pennae
1229e735ac nixos-render-docs: add structural includes, use for manual
this adds support for structural includes to nixos-render-docs.
structural includes provide a way to denote the (sub)structure of the
nixos manual in the markdown source files, very similar to how we used
literal docbook blocks before, and are processed by nixos-render-docs
without involvement of xml tooling. this will ultimately allow us to
emit the nixos manual in other formats as well, e.g. html, without going
through docbook at all.

alternatives to this source layout were also considered:

a parallel structure using e.g. toml files that describe the document
tree and links to each part is possible, but much more complicated to
implement than the solution chosen here and makes it harder to follow
which files have what substructure. it also makes it much harder to
include a substructure in the middle of a file.

much the same goes for command-line arguments to the converter, only
that command-lined arguments are even harder to specify correctly and
cannot be reasonably pulled together from many places without involving
another layer of tooling. cli arguments would also mean that the manual
structure would be fixed in default.nix, which is also not ideal.
2023-02-12 13:02:42 +01:00
pennae
8b7f0e559a nixos/manual: clean up default.nix a bit
- inline copySources into single user
- remove `inherit sources` where it's not necessary
- inline generatedSources. this will go away completely soon so we may
  as well.
- inline modulesDoc into manual-combined. this too will go away soon.
- inline sources into manual-combined. this too will go away soon.
2023-02-12 11:58:20 +01:00
pennae
ef413e3eac nixos/manual: split manpages-combined from manual-combined
once we generate the entire manual-combined.xml with a single
nixos-render-docs invocation we will no longer need any options xml
files. likewise we do not need the test options xml in the manpage
build. splitting manpages-combined from manual-combined also allows
these two to run in parallel, slightly improving build times.
2023-02-12 11:56:26 +01:00
pennae
d041641b1a nixos/manual: remove md-to-db
with manual chapters no longer needing pandoc for their conversion to
xml we can get rid of this source of confusion, and its huge cache of
xml files.
2023-02-10 06:40:15 +01:00
pennae
652a283e51 nixos-render-docs: render manual chapters during manual build
render all manual chapters to docbook from scratch every time the manual
is built. nixos-render-docs is quick enough at this to not worry about
the cost (needing only about a second), and it means we can remove
md-to-db.sh in the next commit.

no changes to the rendered html manual except for replacements and smartquotes.
2023-02-10 06:40:02 +01:00
pennae
67917ac102 nixos-render-docs: rename manual docbook converter to docbook-section
we'll soon add another docbook converter that does not emit a section as
a collection of chapters, but sections or chapters on their own. this
should clarify naming a bit before there can be any confusion.
2023-02-10 06:40:02 +01:00
pennae
10f2c3cacf nixos/manual: remove sources input from manpages drv
it's not used. holdover from when manpages were written in docbook.
2023-02-10 06:40:00 +01:00
pennae
10a4f0daca nixos-render-docs: add options manpage converter
mdoc is just too slow to render on groff, and semantic markup doesn't
help us any for generated pages.

this produces a lot of changes to configuration.nix.5, but only few
rendering changes. most of those seem to be place losing a space where
docbook emitted roff code that did not faithfully represent the input
text, though a few places also gained space where docbook dropped them.
notably we also don't need the compatibility code docbook-xsl emitted
because that problem was fixed over a decade ago.

this will handle block quotes, which the docbook stylesheets turned into
a mess of roff requests that ended up showing up in the output instead
of being processed.
2023-02-08 15:23:34 +01:00
figsoda
42d1d60a92
Merge pull request from SuperSandro2000/cleanup-unused-bindings
treewide: cleanup some unused bindings
2023-02-06 20:07:50 -05:00
Sandro Jäckel
50e0012f9d
treewide: cleanup some unused bindings 2023-02-07 01:36:15 +01:00
pennae
0c601b12bf nixos/manual: translate manpages to mdoc
since we want to move away from docbook and having docbook manpages
around is going to block further progress we have to translate the
current nixos manpages from docbook it *something* else. mdoc seems the
most appropriate choice since markdown can't represent all the things
manpages can differentiate with even more extensions. if we ever need
html manpages we can render them using troff, like many of the online
manpage viewers, or rewrite them again. since we haven't had html
manpages for any of these in many years that seems unlikely to happen.

unlike most of the recent markdown conversions this comes with a lot of
minor rendering changes, mostly in spacing. docbook-xslt creates manual
pages in a very different dialect than mdoc (which is used here).
2023-02-03 02:23:07 +01:00
pennae
0a6e6cf7e6 nixos/manual: render module chapters with nixos-render-docs
this converts meta.doc into an md pointer, not an xml pointer. since we
no longer need xml for manual chapters we can also remove support for
manual chapters from md-to-db.sh

since pandoc converts smart quotes to docbook quote elements and our
nixos-render-docs does not we lose this distinction in the rendered
output. that's probably not that bad, our stylesheet didn't make use of
this anyway (and pre-23.05 versions of the chapters didn't use quote
elements either).

also updates the nixpkgs manual to clarify that option docs support all
extensions (although it doesn't support headings at all, so heading
anchors don't work by extension).
2023-01-27 20:07:34 +01:00
pennae
617822c95b nixos/manual: remove olinkdb
as far as we can tell nixos has only ever had a total of one olink, and
currently has no olinks at all. we can't currently represent olinks in
markdown docs, and if we re-add support for cross-document links they
will take a different form (and not use docbook, which will have to be
phased out before we re-add anything).

the olinkdb is thus unused and takes 10 seconds on our machine to build,
holding up the rest of the manual for no benefit.
2023-01-26 00:32:56 +01:00
pennae
df09c21fb2 nixos/documentation: deprecate docbook option docs
following the plan in https://github.com/NixOS/nixpkgs/pull/189318#discussion_r961764451

also adds an activation script to print the warning during activation
instead of during build, otherwise folks using the new CLI that hides
build logs by default might never see the warning.
2023-01-22 17:08:40 +01:00
Robert Hensing
c23666473c doc: Remove all section numbers
Simplicity.

We use anchors nowadays. Nobody should be printing this.
2022-12-12 13:04:39 +01:00
Robert Hensing
216c5dc10d nixos/doc: Disambiguate test option ids
Changing this later on would break external links into the manual.
2022-09-29 12:41:59 +02:00
Robert Hensing
ac03757eb2 nixos/doc: Wire up the test options reference 2022-09-24 17:38:10 +01:00
Robert Hensing
6205d37747 nixos/testing: Improve option docs 2022-09-24 17:38:10 +01:00
pennae
767485a0de lib/options: deprecate docbook text and literalDocBook
deprecate literalDocBook by adding a warning (that will not fire yet) to
its uses and other docbook literal strings by adding optional warning
message to mergeJSON.
2022-09-10 18:23:13 +02:00
Robert Hensing
49237015d6 nixos/doc: Make nixos-manual-combined fail easy to inspect
By writing the unchecked outputs before checking them, they will
be written to a store path, which appears in the log, and which
sticks around even if the build fails. Eventually it is GCed, but
until then, you can open the file.

If you run it in a terminal+editor combination like VSCode, the
failure location is just one Ctrl+click away.
2022-06-27 17:41:32 +02:00
Jan Tojnar
84d6565d52 nixos/man: prevent duplication of options
libxslt 1.1.35 fixed conflict resolution for templates to match the specification.
This uncovered a bug in docbook-xsl (https://github.com/docbook/xslt10-stylesheets/issues/240),
which causes option names to be duplicated into the option descriptions.

Let’s resolve the conflict by patching the stylesheets.

Fixes: https://github.com/NixOS/nixpkgs/issues/166304
2022-04-03 21:22:47 +02:00
pennae
9d3ba92d63 nixos/documentation: fix docs cross build
a few things should've used buildPackages/nativeBuildInputs to not not require
the host architecture for building docs. tested by building aarch64-linux docs
on x86_64-linux, and the result looks good.
2022-01-08 16:03:23 +01:00
pennae
50954ad1c5 nixos/make-options-doc: treat missing descriptions as errors by default
this partially solves the problem of "missing description" warnings of the
options doc build being lost by nix build, at the cost of failing builds that
previously ran. an option to disable this behaviour is provided.
2022-01-02 19:46:13 +01:00
pennae
fc614c37c6 nixos/documentation: split options doc build
most modules can be evaluated for their documentation in a very
restricted environment that doesn't include all of nixpkgs. this
evaluation can then be cached and reused for subsequent builds, merging
only documentation that has changed into the cached set. since nixos
ships with a large number of modules of which only a few are used in any
given config this can save evaluation a huge percentage of nixos
options available in any given config.

in tests of this caching, despite having to copy most of nixos/, saves
about 80% of the time needed to build the system manual, or about two
second on the machine used for testing. build time for a full system
config shrank from 9.4s to 7.4s, while turning documentation off
entirely shortened the build to 7.1s.
2022-01-02 19:46:13 +01:00
pennae
c533b01863 nixos/doc/manual: remove non-matching optionsXML inherit 2021-12-06 16:12:32 +01:00
Ben Siraphob
1c2a2b0a08 treewide: fold -> foldr 2021-01-26 10:57:07 +07:00
zimbatm
7d834eff6c
nixos/manual: make reproducible () 2020-10-31 21:18:16 +01:00
Frederik Rietdijk
70bb2e4151 nixos manual: have a toc for each part and chapter
to be inline with the Nixpkgs manual. This makes navigating to
particular sections easier.
2019-10-30 10:25:09 +01:00
Frederik Rietdijk
227dff4e70 nixos manual: reduce toc depth
to reduce the amount of information presented.
2019-10-30 10:25:09 +01:00
Domen Kožar
5cfd034af0
Extract NixOS options documentation generation to a function
Motivation is to support other repositories containing nixos
modules that would like to generate options documentation:

- nix-darwin
- private repos
- arion
- ??
2019-08-08 16:18:09 +02:00
Graham Christensen
777e94d903
Merge pull request from layus/warn-manual-ids
Nixos manual: error out on missing IDs
2019-03-09 08:21:07 -05:00
Jan Malakhovski
ca496b194f nixos: doc: increase maxdepth for xsltproc
See https://github.com/NixOS/nixpkgs/issues/37903#issuecomment-376618117
for details. With the previous patch and some custom modules included in
`configuration.nix` the above bug is very easy to trigger.

This is a simplest workaround I have. A proper solution would look like
https://github.com/NixOS/nixpkgs/issues/37903#issuecomment-376980838.
2019-03-05 09:41:40 +00:00
Guillaume Maudoux
a65974c604 nixos/manual: error out on missing IDs 2019-02-08 10:59:46 +01:00
Guillaume Maudoux
4c0230eb56 nixos/manual: warn on missing xml:id 2019-02-07 16:35:24 +01:00
Jan Malakhovski
fe744d3fb1 nixos: doc: rename manual to manualHTML, cleanup references
Because when I see "config.system.build.manual.manual" after I forgot
what it means I ask "Why do I need that second `.manual` there again?".
Doesn't happen with `config.system.build.manual.manualHTML`.
2018-09-23 20:47:58 +00:00
aszlig
f865d0feab
nixos: Split paras by \n\n in option descriptions
What annoyed me for a long time was the fact, that in order to break
into a new paragraph, you need to insert </para><para> in the
description attribute of an option.

Now we will automatically create <para/> elements for every block that
is separated by two consecutive newlines.

I first tried to do this within options-to-docbook.xsl, but it turns
out[1] that this isn't directly possible with XSLT 1.0, so I added
another XSLT file that postprocesses the option descriptions that are
now enclosed in <nixos:option-description/> by options-to-docbook.xsl.

The splitting itself is a bit more involved, because we can't simply
split on every \n\n because we'd also split text nodes of elements, for
example:

  <screen><![CDATA[

    one line

    another one

  ]]></screen>

This would create one <para/> element for "one line" and another for
"another line", which we obviously don't want because <screen/> is used
to display verbatim contents of what a user is seeing on the screen.

So what we do instead is splitting *only* the top-level text nodes
within the outermost <para/> and leave all elements as-is. If there are
more than one <para/> elements at the top-level, we simply don't process
it at all, because the description then already contains </para><para>.

https://www.mhonarc.org/archive/html/xsl-list/2012-09/msg00319.html

Signed-off-by: aszlig <aszlig@nix.build>
Cc: @edolstra, @domenkozar
2018-09-02 08:10:37 +02:00
Matthew Bauer
76999cc40e treewide: remove aliases in nixpkgs
This makes the command ‘nix-env -qa -f. --arg config '{skipAliases =
true;}'’ work in Nixpkgs.

Misc...

- qtikz: use libsForQt5.callPackage

  This ensures we get the right poppler.

- rewrites:

  docbook5_xsl -> docbook_xsl_ns
  docbook_xml_xslt -> docbook_xsl

diffpdf: fixup
2018-07-18 23:25:20 -04:00
Jan Malakhovski
2a5688574c nixos: doc: make relatedPackages a bit smarter 2018-06-13 16:25:10 +00:00
Graham Christensen
0ff0d138e4
nixos docs: Add a makefile for hacking on the nixos docs 2018-04-28 04:00:55 -04:00
Graham Christensen
59f8b1e844
nixos docs: Move generated XML in to a specific subdirectory to allow easier hacking 2018-04-27 22:44:51 -04:00
Graham Christensen
8f33464ca7
nixos docs: syntax highlight
- Rectifies diverging CSS by combining
   nixos/nixpkgs docs CSS
 - Moves our custom Highlight.js loader in to
   the hljs package
 - Switches the nixos docs to use SVG
   callouts too
2018-04-05 07:54:01 -04:00
Tuomas Tynkkynen
3918182cb5 nixos/manual: Don't depend on non-bin outputs of libxml2 and libxslt
Might fix https://hydra.nixos.org/build/71580290.
2018-03-21 22:18:49 +02:00
Shea Levy
20072d733b
nixos: manual: Fix cross-compilation. 2018-03-01 10:41:50 -05:00
Jan Malakhovski
810c4702cf stdenv.mkDerivation: rename meta.evaluates -> meta.available
A much better name.
2018-02-18 13:33:25 +00:00
Jan Malakhovski
b468f98b02 nixos: doc: trivial cleanup and docstring fix 2018-02-11 22:41:06 +00:00
Jan Malakhovski
252ec7da0a nixos: doc: make option sorting somewhat more efficient 2018-02-11 22:07:11 +00:00
Jan Malakhovski
660806066a nixos, lib: implement relatedPackages option
This allows one to specify "related packages" in NixOS that get rendered into
the configuration.nix(5) man page. The interface philosophy is pretty much
stolen from TeX bibliography.

See the next several commits for examples.
2018-02-09 19:51:06 +00:00
Jan Malakhovski
a7d75ab648 nixos/doc: push all the enable*' and package*` options to the top of their option group
Why? Because this way configuration.nix(5) can be read linearly.

Before:

> virtualisation.xen.bootParams
> ...
> virtualisation.xen.enable
> ...
> virtualisation.xen.package
> ...

After:

> virtualisation.xen.enable
> virtualisation.xen.package
> virtualisation.xen.bootParams
> ...
2018-02-09 19:51:05 +00:00