codegen/vulkan/BUILD.adoc

// Copyright 2014-2021 The Khronos Group Inc.
// SPDX-License-Identifier: CC-BY-4.0

= Vulkan^(R)^ Specification Build Instructions and Notes
:toc2:
:toclevels: 2

ifdef::env-github[]
:note-caption: :information_source:
endif::[]

[[intro]]
== Introduction

This README describes how to build the Vulkan API specification, reference
pages, and/or other related targets.

It documents how to set up your build environment, build steps and targets,
and contains some troubleshooting advice.

[[building]]
== Building The Spec

First, clone the Khronos Github repository containing the Vulkan
specification to your local Linux, Windows, or Mac PC.
The repository is located at https://github.com/KhronosGroup/Vulkan-Docs/.

Next, install all the necessary build tools (see <<depends,Software
Dependencies>> below).
If you are using the <<depends-docker, Khronos-Provided Docker Image>>,
which we strongly recommend, then one way to build using the image (assuming
a Linux docker host) is:

    $ docker run --user `id -u`:`id -g` -it --rm \
      -v <vulkan-path>:/vulkan \
      khronosgroup/docker-images:asciidoctor-spec /bin/bash

where <vulkan-path> is the path to the cloned repository. This runs the
image with the cloned repository under /vulkan and accesses it as a
specified user (set to your own user and group ID above), so that it doesn't
get filled with files owned by another user.

Then in the running image,

    $ cd /vulkan
    $ ./makeSpec -spec core html

which builds an HTML5 specification output for the core Vulkan 1.2
specification, with no extensions included, or

    $ ./makeSpec -spec all all

which builds the spec targets `html`, `pdf`, `styleguide`, `registry`,
`manhtmlpages`, and `allchecks`, with *all* registered extensions included.

There are many other ways of using the image, including inside a Continuous
Integration pipeline; locally with persistent Docker volume storage of the
repository; and so on.

If you are not using our Docker image to build with, and you have a
<<depends-nondocker, Non-Docker Build Environment>> with the entire
toolchain installed, you can go to <vulkan-path> and invoke the same `make`
commands there.

[NOTE]
.Note
====
  * While it's possible to invoke `make` directly, this is rarely
    appropriate or useful.
    Usually dozens to hundreds of build options must be set to specify the
    desired set of extensions to include in the specification.
    The `makeSpec` python script, which is discussed in more detail
    <<building-extensions, below>>, simplifies this process for common
    cases.
  * The `all` target takes a long time to run, and generates outputs that
    are irrelevant for most users.
    The `html` target just generates the HTML output, which is often all
    that's needed for spec bugfixes not involving extensions.
  * The default `make` options build a Vulkan 1.2 specification with no
    optional extensions.
  * The `validusage` target is not built as part of the `all` target, due to
    it needing to be built with all extensions enabled (`-spec all`).
    Building the `validusage` target will fail otherwise.
====

These targets generate a variety of output documents in the directory
specified by the Makefile variable `$(OUTDIR)` (by default, `out/`).
The checked-in file `out/index.html` links to all these
targets, or they can individually be found as follows:

Vulkan Specification::
  * `html` -- Single-file HTML5 in `$(OUTDIR)/html/vkspec.html`, and KaTeX
    dependency in $(OUTDIR)/katex
  * `chunked` -- Chunked HTML5 in `$(OUTDIR)/html/chap?.html`
  * `pdf` -- PDF in `$(OUTDIR)/pdf/vkspec.pdf`
"`styleguide`" (Vulkan Documentation and Extensions: Procedures and Conventions)::
  * `styleguide` -- Single-file HTML5 in `$(OUTDIR)/styleguide.html`
XML Registry schema document::
  * `registry` -- Single-file HTML5 in `$(OUTDIR)/registry.html`
<<building-diff,Diff spec>>::
  * `diff_html` -- Single-file HTML5 in `$(OUTDIR)/html/diff.html`
<<refpages,Reference pages>>::
  * `manhtmlpages` -- File-per-entry-point HTML in `$(OUTDIR)/man/html/*`.
    Must be built with all extensions enabled (using `makeSpec -spec all`).
<<validation-scripts,Validator output>>::
  * None at present. The `allchecks` target writes to standard output unless
    the underlying script is given additional options.
Valid usage database::
  * `validusage` - json database of all valid usage statements in the
     specification. Must be built with `./makeAllExts` (for now).
     Output in `$(OUTDIR)/validation/validusage.json`.
     A validated schema for the output of this is stored in
     `$(CURDIR)/config/vu-to-json/vu_schema.json`

Once you have the basic build working, an appropriate parallelization
option, such as `-j 8`, should significantly speed up the reference page
builds.

If you encounter problems refer to the <<troubleshooting>> section.


[[building-versions]]
=== Building Specifications For Different API Versions

The `Makefile` defaults to building a Vulkan 1.2 specification.
This is controlled by Asciidoctor attributes passed in the Makefile variable
`$(VERSIONS)`
To instead build a Vulkan 1.1 specification, pass

----
VERSIONS="VK_VERSION_1_0 VK_VERSION_1_1"
----

on the `makeSpec` command line.


[[building-extensions]]
=== Building With Extensions Included

Extensions are defined in the same source as the core Specification, but
are only conditionally included in the output.
http://asciidoctor.org/docs/user-manual/#attributes[Asciidoctor attributes]
of the same name as the extension are used to define whether the extension
is included or not -- defining such an attribute will cause the output to
include the text for that extension.

When building the specification, the extensions included are those specified
as a space-separated list of extension names (e.g. `VK_KHR_surface`) in the
Makefile variable `$(EXTENSIONS)`, usually set on the make command line.
When changing the list of extensions, it is critical to remove all generated
files using the `clean_generated` Makefile target, as the contents of
generated files depends on `$(EXTENSIONS)`.

The `makeSpec` wrapper script can clean generated files and then build one
or more specification targets for a set of explicitly specified extensions,
including all implicit extension dependencies of that set.
It accepts these options:

  * -clean - remove generated targets before building
  * -v - print actions as well as executing them
  * -n - print actions without executing them
  * -genpath *path* - specify path to generated files (default `gen`)
  * -spec *type* - build with sepcified sets of extensions.
    *type* may be
  ** *core* - no extensions added (default if not specified)
  ** *khr* - all KHR extensions added
  ** *all* - all registered extensions added
  * -extension *extname* - build with specified extension included,
    as well as the set specified by `-spec`.
    Can be given multiple times.
  * All remaining targets are arbitrary `make` options or
    targets in the Makefile.

The `target(s)` passed to these scripts are arbitrary `make` options, and
can be used to set Makefile variables and options discussed above, as well
as specify actual build targets.
For example, to build the HTML specification with all KHR extensions
included as well as a single vendor extension:

----
$ ./makeSpec -clean -spec khr -extension VK_EXT_debug_report html
----

The scripts `makeAllExts`, `makeKHR`, and `makeExt` set appropriate options
and invoke `makeSpec`, for backwards compatibility, but are no longer used
by Khronos.

The Makefile variable `$(APITITLE)` defines an additional string which is
appended to the specification title.
When building with extensions enabled, this should be set to something like
`(with extension VK_extension_name)`.
The `makeSpec` script already does this.

The reference pages (the `manhtmlpages` target) must be built using the
`-spec all` option; there are markup and scripting issues which will
probably cause any more restricted set of refpages to fail to build.


[[building-diff]]
==== Building A Highlighted Extension Diff

The `diff_html` target in the Makefile can be used to generate a version of
the specification which highlights changes made to the specification by the
inclusion of a particular set of extensions.

Extensions in the Makefile variable `$(EXTENSIONS)` define the base
extensions to be enabled by the specification, and these will not be
highlighted in the output.
Extensions in the Makefile variable `$(DIFFEXTENSIONS)` define the set of
extensions whose changes to the text will be highlighted when they are
enabled.
Any extensions in both variables will be treated as if they were only
included in `$(DIFFEXTENSIONS)`.
`$(DIFFEXTENSIONS)` can be set when using the `makeSpec` script described
above.

In the resulting HTML document, content that has been added by one of the
extensions will be highlighted with a lime background, and content that was
removed will be highlighted with a pink background.
Each section has an anchor of `#differenceN`, with an arrow (=>) at the end
of each section which links to the next difference section.
The first diff section is `#difference1`.

[NOTE]
.Note
====
This output is not without errors.
It may instead result in visible `+++[.added]##content##+++` and
`+++[.removed]##content##+++`, and so also highlights not being rendered.
But such visible markup still correctly encapsulates the modified content.
====

[[building-test]]
=== Alternate and Test Builds

If you are just testing Asciidoctor formatting, macros, stylesheets, etc.,
you may want to edit `vkspec.txt` to just include your test code.
The asciidoctor HTML build is very fast, even for the whole Specification,
but PDF builds take several minutes.


=== Images Used In The Specification

All images used in the specification are in the `images/` directory in the
SVG format, and were created with Inkscape.
We recommend using Inkscape to modify or create new images, as we've had
problems using SVG files created by some other tools; especially in the PDF
builds.


[[validation-scripts]]
=== Validation Scripts

The `allchecks` Makefile target runs a Python script that looks for markup
errors, missing interfaces, macro misuse, and inconsistencies in the
specification text.
This script is necessarily heuristic, since it's dealing with lots of
hand-written material, but it identifies many problems and can suggest
solutions.
This script is also run as part of the CI tests in the internal Khronos
gitlab repository.


[[macros]]
== Our Asciidoctor Macros

We use many custom Ruby macros in the reference pages and API spec
Asciidoctor sources.
The validator scripts rely on these macros as part of their sanity checks,
and you should use the macros whenever referring to an API command, struct,
token, or enum name, so the documents are semantically tagged and more
easily verifiable.

The supported macros are defined in the `config/spec-macros/extension.rb`
asciidoctor extension script.

The tags used are described in the
link:https://www.khronos.org/registry/vulkan/specs/1.1/styleguide.html[style
guide] (generated from `styleguide.txt`).

We (may) eventually tool up the spec and reference pages to the point that
anywhere there's a type or token referred to, clicking on (or perhaps
hovering over) it in the HTML view will take reader to the definition of
that type/token.
That will take some more plumbing work to tag the stuff in the autogenerated
include files, and do something sensible in the spec (e.g. resolve links to
internal references).

Most of these macros deeply need more intuitive names.


[[refpages]]
== Reference Pages

The reference pages are extracted from the API Specification source, which
has been tagged to help identify boundaries of language talking about
different commands, structures, enumerants, and other types.
A set of Python scripts extract and lightly massage the relevant tagged
language into corresponding reference page sources.

To regenerate the reference page sources from scratch yourself, execute:

----
./makeSpec -spec all refpages
----

The `genRef.py` script will generate many warnings, but most are just
reminders that some pages are automatically generated.
If everything is working correctly, all the `$(GENERATED)/refpage/*.txt`
files will be regenerated, but their contents will not change.

If you add new API features to the Specification in a branch, make sure that
the commands have the required tagging and that reference pages are
generated for them, and build properly.

When executing the `manhtmlpages` target in the Makefile, after building
HTML versions of all reference pages extracted from the spec, symbolic links
from aliases to the reference page for the API they alias will also be
created.


[[styles]]
== Our stylesheets

We use an HTML stylesheet `config/khronos.css` derived from the
http://asciidoctor.org/docs/produce-custom-themes-using-asciidoctor-stylesheet-factory/[Asciidoctor
stylesheet factory] "`colony`" theme, with the default Arial font family
replaced by the sans-serif https://en.wikipedia.org/wiki/Noto_fonts[Noto
font family].


[[styleguide]]
== Vulkan Style Guide


If you're writing new spec language or modifying existing language, see the
link:https://www.khronos.org/registry/vulkan/specs/1.2/styleguide.html["`style
guide`"] (formally titled "`Vulkan Documentation and Extensions: Procedures
and Conventions`") document for details of our asciidoctor macros,
extensions, mathematical equation markup, writing style, etc.


[[depends]]
== Software Dependencies

This section describes the software components used by the Vulkan spec
toolchain.

In the past, we previously specified package versions and instructions for
installing the toolchain in multiple desktop environments including Linux,
MacOS X, and Microsoft Windows.
The underlying components evolve rapidly, and we have not kept those
instructions up to date.


[[depends-docker]]
=== Khronos-Provided Docker Image

Khronos has published a Docker image containing a Debian Linux distribution
with the entire toolchain preinstalled.

We will occasionally update this image if needed, and we recommend people
needing to build from this repository use the Docker image.

Docker installation is beyond the scope of this document.
Refer to link:https://docs.docker.com/get-docker/[the Docker website] for
information about installing Docker on Linux, Windows, and MacOS X.

The name of the build image is

    khronosgroup/docker-images:asciidoctor-spec

It can be pulled from the
link:https://hub.docker.com/repository/docker/khronosgroup/docker-images[Dockerhub
repository] with the command

    docker pull khronosgroup/docker-images:asciidoctor-spec

Once docker is installed and the image is available, it can be executed as
described above under <<building, Building the Spec>> to generate
Specification output documents or other Makefile targets.

[NOTE]
.Note
====
The old `vulkan-docs-base` and `vulkan-docs` images continue to be hosted,
but the new `asciidoctor-spec` image is preferred - the added functionality
of `vulkan-docs`, to set the user/group inside Docker based on environment
variables passed into Docker, is no longer needed with the Docker `--user`
option.
====


[[depends-nondocker]]
=== Non-Docker Build Environments

We do not actively support building outside of our Docker image, but it is
straightforward to reproduce our toolchain in a Debian (or similar APT-based
Linux) distribution by executing the same steps as the
link:https://github.com/KhronosGroup/DockerContainers/blob/master/asciidoctor-spec.dockerfile[Dockerfile]
used to build our Docker image.

It should be possible to apply the same steps in a Windows Subsystem for
Linux (WSL2) environment on Windows 10, as well.

For other native environments, such as MacOS X and older Unix-like
environments for Windows such as MinGW and Cygwin, we provided instructions
in older versions of this document.
While those instructions are out of date and have been removed from current
versions of this document, you may be able to make use of
link:https://github.com/KhronosGroup/Vulkan-Docs/blob/v1.2.135/BUILD.adoc#depends[the
version of BUILD.adoc in the v1.2.135 repository tag]

[NOTE]
.Note
====
While we have no intention of forcing people to use our Docker image, we
cannot support every possible environment.
The Docker image is a straightforward way to use the Vulkan-Docs repository
with almost all modern desktop environments.
====


[[history]]
== Revision History

  * 2021-03-12 - Use the new Docker image.
  * 2020-07-15 - Update to use `makeSpec` instead of `makeAllExts`.
  * 2020-03-23 - Document Khronos' published Docker image for building the
    spec, and remove all platform-specific instructions.
  * 2018-12-04 - Update Rbenv and ruby gem installation instructions and
    package dependencies for Linux and Ubuntu/Windows 10.
  * 2018-10-25 - Update Troubleshooting, and Windows and Linux build. Plus
    random editing.
  * 2018-03-13 - Rename to BUILD.adoc and update for new directory
    structure.
  * 2018-03-05 - Update README for Vulkan 1.1 release.
  * 2017-03-20 - Add description of prawn versioning problem and how to fix
    it.
  * 2017-03-06 - Add description of ruby-enum versioning problem and how to
    fix it.
  * 2017-02-13 - Move some comments here from ../../../README.md. Tweak
    asciidoctor markup to more clearly delineate shell command blocks.
  * 2017-02-10 - Add more Ruby installation guidelines and reflow the
    document in accordance with the style guide.
  * 2017-01-31 - Add rbenv instructions and update the README elsewhere.
  * 2017-01-16 - Modified dependencies for Asciidoctor
  * 2017-01-06 - Replace MathJax with KaTeX.
  * 2016-08-25 - Update for the single-branch model.
  * 2016-07-10 - Update for current state of spec and ref page generation.
  * 2015-11-11 - Add new can: etc.
    macros and DBLATEXPREFIX variable.
  * 2015-09-21 - Convert document to asciidoc and rename to README.md in the
    hope the gitlab browser will render it in some fashion.
  * 2015-09-21 - Add descriptions of LaTeX and MathJax math support for all
    output formats.
  * 2015-09-02 - Added Cygwin package info.
  * 2015-09-02 - Initial version documenting macros, required toolchain
    components and versions, etc.