Grpc.Tools MSBuild integration overview

This is an overview for maintainers of Grpc.Tools.

The Grpc.Tools NuGet package provides custom build targets to make it easier to specify .proto files in a project and for those files to be compiled and their generated files to be included in the project.

Files in the NuGet package

.props and .target files

MSBuild properties and targets included from the Grpc.Tools NuGet package are in:

build\Grpc.Tools.props, which imports
- build\_grpc\_Grpc.Tools.props
- build\_protobuf\Google.Protobuf.Tools.props
build\Grpc.Tools.targets, which imports
- build\_grpc\_Grpc.Tools.targets
- build\_protobuf\Google.Protobuf.Tools.targets

Details of how NuGet packages can add custom build targets and properties to a project is documented here: MSBuild .props and .targets in a package

Basically the .props and .targets files are automatically included in the projects - the .props at the top of the project and the .targets are added to the bottom of the project.

Visual Studio property pages

For Visual Studio integration - these files provide the properties pages:

build\_protobuf\Protobuf.CSharp.xml (included from Google.Protobuf.Tools.targets)
build\_grpc\Grpc.CSharp.xml (included from _Grpc.Tools.targets)

Custom tasks DLLs

DLLs containing the custom tasks are in:

build\_protobuf\netstandard1.3
build\_protobuf\net45

Protobuf compiler and C# gRPC plugin binaries

Native binary executables for the protobuf compiler (protoc) and C# gRPC plugin (grpc_csharp_plugin) are included in the NuGet package. Included are binaries for various OSes (Windows, Linux, macOS) and CPU architectures (x86, x64, arm64).

The build determines which executables to use for the particular machine that the it is being run on. These can be overridden by specifying MSBuild properties or environment variables to give the paths to custom executables:

Protobuf_ProtocFullPath property or PROTOBUF_PROTOC environment variable
Full path of protoc executable
gRPC_PluginFullPath property or GRPC_PROTOC_PLUGIN environment variable
Full path of gRPC C# plugin

Grpc.Tools custom build targets

Hooking the custom targets into the project build

The custom targets hook into various places in a normal MSBuild build by specifying before/after targets at the relevant places. See msbuild-targets for predefined targets.

Before PrepareForBuild
- we add Protobuf_SanityCheck that checks this is a supported project type, e.g. a C# project.
Before BeforeCompile
- we add all the targets that compile the .proto files and generate the expected .cs files. These files are added to those that get compiled by the C# compiler.
  - The target _Protobuf_Compile_BeforeCsCompile is the glue inserting the targets into the build. It may look like it isn't doing anything but by specifying BeforeTargets and DependsOnTargets it inserts Protobuf_Compile into the build - but only doing so if this is a C# project.
After CoreClean
- we add Protobuf_Clean that cleans the files generated by the protobuf compiler.
  - The target _Protobuf_Clean_AfterCsClean is the glue inserting Protobuf_Clean into the build - but only doing so if this is a C# project.

Custom tasks

There are a few custom tasks needed by the targets. These are implemented in C# in the Grpc.Tools project and packaged in the file Protobuf.MSBuild.dll in the NuGet package. See task writing for information about implementing custom tasks.

ProtoToolsPlatform
- Works out the operating system and CPU architecture
ProtoCompilerOutputs
- Tries to work out the names and paths of the files that would be generated by the protobuf compiler and returns these as a list of items
- Also returns list of items that is the same as the Protobuf items passed in with the output directory metadata updated
ProtoReadDependencies
- Read generated files from previously written dependencies file and return as items.
ProtoCompile
- Runs the protobuf compiler for a proto file. The executable to run is specified by the property Protobuf_ProtocFullPath
- To do this it:
  - first writes out a response file containing the parameters for protobuf compiler
  - runs the executable, which generates the .cs files and a .protodep dependencies file
  - reads the dependencies file to find the files that were generated and these are returned as a list of items that can then be used in the MSBuild targets

Build steps

The names of these items and properties are correct at the time of writing this document.

High level builds steps:

Prepare list of .proto files to compile
- Makes sure all needed metadata is set for the <Protobuf> item, defaulting some values
- Removes <Protobuf> items that no longer exist or are marked as don't compile
Handling incremental builds
- Work out files that need to be created or have changed
Compile the .proto files
Add generated files to the list of files for the C# compiler

Prepare the list of .proto files to compile

At various stages of the build copies of the original <Protobuf> items are created and/or updated to set metadata and to prune out unwanted items.

Firstly, the build makes sure ProtoRoot metadata is set for all Protobuf items. A new list of items - Protobuf_Rooted - is created from the Protobuf items with ProtoRoot metadata set:

If ProtoRoot already set in the <Protobuf> item in the project file then it is left as-is.
If the .proto file is under the project's directory then set ProtoRoot=".".
If the .proto file is outside of the project's directory then set ProtoRoot="<relative path to project directory>".

Now prune out from Protobuf_Rooted the items that the user doesn‘t want to compile - those don’t have ProtoCompile metadata as true. The pruned list is now called Protobuf_Compile.

Set the Source metadata on Protobuf_Compile items to be the name of the .proto file. The Source metadata is used later as a key to map generated files to .proto files.

Handling incremental builds

Gathering files to check for incremental builds

The target Protobuf_PrepareCompile tries to work out which files the protobuf compiler will generate without actually calling the protobuf compiler. This is a best-effort guess. The custom task ProtoCompilerOutputs is called to do this. The results are stored in the item list Protobuf_ExpectedOutputs.

The target Protobuf_PrepareCompile also reads previously written .protodep files to get any actual files previously generated. The custom task ProtoReadDependencies is called to do this. The results are stored in the item list Protobuf_Dependencies. This is in case the list of actual files is different from the previous best-effort guess from ProtoCompilerOutputs.

The expected outputs and previous outputs are needed so that the timestamps of those files can be checked later when handling an incremental build.

Understanding incremental builds

To avoid unnecessarily recompiling the .proto files during an incremental build the target _Protobuf_GatherStaleBatched tries to work out if any files have changed.

It checks for out of date files using MSBuilds incremental build feature that compares the timestamps on a target's Input files to its Output files. See How to: Build incrementally

The Inputs that are checked are:

Timestamps of the .proto files
Timestamps of previously generated files (list of these files read from .protodep files)
Timestamps of MSBuild project files

These are checked against the Outputs:

Timestamps of the expected generated files

MSBuild target batching is used to check each .proto file against its expected outputs. The batching is done by specifying the Source metadata in the Input. Items where Source metadata matches in both input and output are in each batch.

The target _Protobuf_GatherStaleBatched sets the metadata _Exec=true on _Protobuf_OutOfDateProto items that are out of date.

Later in the target _Protobuf_GatherStaleFiles, the items in _Protobuf_OutOfDateProto that don't have metadata _Exec==true are removed from the list of items, leaving only those that need compiling.

Compile the .proto files

The target _Protobuf_CoreCompile is run for each .proto file that needs compiling. These are in the item list _Protobuf_OutOfDateProto. The custom task ProtoCompile is called to run the protobuf compiler. The files that were generated are returned in the item list _Protobuf_GeneratedFiles.

If there are expected files that were not actually generated then the behaviour depends on whether the generated files should have been within the project (e.g. in the intermediate directories) or were specified to be outside of the project.

If within the project - empty files are created to prevent incremental builds doing unnecessary recompiles
If outside the project - by default empty files are not created and a warning is output (this behaviour is configurable)

TODO: why are files inside and outside the project treated differently?

Add generated .cs files to the list of files for the C# compiler

The target _Protobuf_AugmentLanguageCompile adds to the Compile item list (the list of files that CSC compiles) the expected generated files.

Note - this is the expected files not the actual generated files and this is done before the protobuf compiler is called.

TODO: why are the expected files not the actual generated files added?

Handling design time builds

Design-time builds are special builds that Visual Studio uses to gather information about the project. They are not user-initiated but may be triggered whenever files are added, removed or saved. See Design-Time Builds.

The Grpc.Tools build targets used to try and optimise design time builds by disabling calling the protobuf compiler during a design time build. However this optimisation can lead to errors in Visual Studio because the generated files may not exist or be out of date and any code that relies on them will then have errors.

Now design time builds behave exactly the same as a normal build. The old behaviour can be enabled by setting the setting DisableProtobufDesignTimeBuild property to true in the project file if it is a design time build, e.g. by adding

<PropertyGroup Condition="'$(DesignTimeBuild)' == 'true' ">
    <DisableProtobufDesignTimeBuild>true</DisableProtobufDesignTimeBuild>
</PropertyGroup>

Automatically including .proto files

For SDK projects it is possible to automatically include .proto files found in the project directory or sub-directories, without having to specify them with a <Protobuf> item. To do this the property EnableDefaultProtobufItems has be set to true in the project file.

By default it is not set and <Protobuf> items must be included in the project for the .proto files to be compiled.