src/csharp/Grpc.Tools/implementation_notes.md

# Grpc.Tools MSBuild integration overview

This is an overview for maintainers of Grpc.Tools.

The Grpc.Tools NuGet package provides custom build targets to make it easier to specify `.proto` files
in a project and for those files to be compiled and their generated files to be included in the project.

# Files in the NuGet package

## .props and .target files

MSBuild properties and targets included from the Grpc.Tools NuGet package are in:

* `build\Grpc.Tools.props`, which imports
    * `build\_grpc\_Grpc.Tools.props`
    * `build\_protobuf\Google.Protobuf.Tools.props`
* `build\Grpc.Tools.targets`, which imports
    * `build\_grpc\_Grpc.Tools.targets`
    * `build\_protobuf\Google.Protobuf.Tools.targets`

Details of how NuGet packages can add custom build targets and properties to a project is documented
here: [MSBuild .props and .targets in a package](https://learn.microsoft.com/en-us/nuget/concepts/msbuild-props-and-targets)

Basically the `.props` and `.targets` files are automatically included in the projects - the `.props` at the top
of the project and the `.targets` are added to the bottom of the project.

## Visual Studio property pages

For Visual Studio integration - these files provide the properties pages:

* `build\_protobuf\Protobuf.CSharp.xml` (included from `Google.Protobuf.Tools.targets`)
* `build\_grpc\Grpc.CSharp.xml` (included from `_Grpc.Tools.targets`)

## Custom tasks DLLs

DLLs containing the custom tasks are in:

* `build\_protobuf\netstandard1.3`
* `build\_protobuf\net45`

## Protobuf compiler and C# gRPC plugin binaries

Native binary executables for the protobuf compiler (_protoc_) and C# gRPC plugin (_grpc_csharp_plugin_) are
included in the NuGet package. Included are binaries for various OSes (Windows, Linux, macOS) and
CPU architectures (x86, x64, arm64).

The build determines which executables to use for the particular machine that the it is being run on.
These can be overridden by specifying MSBuild properties or environment variables to give the paths to custom executables:

* `Protobuf_ProtocFullPath` property or `PROTOBUF_PROTOC` environment variable \
Full path of protoc executable
* `gRPC_PluginFullPath` property or `GRPC_PROTOC_PLUGIN` environment variable \
Full path of gRPC C# plugin

# Grpc.Tools custom build targets

## Hooking the custom targets into the project build

The custom targets hook into various places in a normal MSBuild build by specifying
before/after targets at the relevant places. See
[msbuild-targets](https://learn.microsoft.com/en-us/visualstudio/msbuild/msbuild-targets)
for predefined targets.

* Before `PrepareForBuild`
    * we add `Protobuf_SanityCheck` that checks this is a supported project type, e.g. a C# project.
* Before `BeforeCompile`
    * we add all the targets that compile the `.proto` files and generate the expected `.cs` files. These files are added to those that get compiled by the C# compiler.
        * The target `_Protobuf_Compile_BeforeCsCompile` is the _glue_ inserting the targets into the build.
        It may look like it isn't doing anything but by specifying `BeforeTargets` and `DependsOnTargets` it inserts `Protobuf_Compile` into the build - but only doing so if this is a C# project.
* After `CoreClean`
    * we add `Protobuf_Clean` that cleans the files generated by the protobuf compiler.
        * The target `_Protobuf_Clean_AfterCsClean` is the _glue_ inserting `Protobuf_Clean` into the build - but only doing so if this is a C# project.

## Custom tasks

There are a few custom tasks needed by the targets. These are implemented in C# in the Grpc.Tools project
and packaged in the file `Protobuf.MSBuild.dll` in the NuGet package. See [task writing](https://learn.microsoft.com/en-us/visualstudio/msbuild/task-writing) for information about implementing custom tasks.

* ProtoToolsPlatform
    * Works out the operating system and CPU architecture
* ProtoCompilerOutputs
    * Tries to work out the names and paths of the files that would be generated by the protobuf compiler and returns these as a list of items
    * Also returns list of items that is the same as the `Protobuf` items passed in with the output directory metadata updated
* ProtoReadDependencies
    * Read generated files from previously written dependencies file and return as items.
* ProtoCompile
    * Runs the protobuf compiler for a proto file. The executable to run is specified by the property `Protobuf_ProtocFullPath`
    * To do this it:
        * first writes out a response file containing the parameters for protobuf compiler 
        * runs the executable, which generates the `.cs` files and a `.protodep` dependencies file
        * reads the dependencies file to find the files that were generated and these are returned as a list of _items_ that can then be used in the MSBuild targets

## Build steps

The names of these items and properties are correct at the time of writing this document.

High level builds steps:

* Prepare list of `.proto` files to compile
    * Makes sure all needed metadata is set for the `<Protobuf>` item, defaulting some values
    * Removes `<Protobuf>` items that no longer exist or are marked as don't compile
* Handling incremental builds
    * Work out files that need to be created or have changed
* Compile the `.proto` files
* Add generated files to the list of files for the C# compiler

### Prepare the list of .proto files to compile

At various stages of the build copies of the original `<Protobuf>` items are created
and/or updated to set metadata and to prune out unwanted items.

Firstly, the build makes sure `ProtoRoot` metadata is set for all `Protobuf` items.
A new list of items - `Protobuf_Rooted` - is created from the `Protobuf` items with `ProtoRoot` metadata set:

* If `ProtoRoot` already set in the `<Protobuf>` item in the project file then it is left as-is.
* If the `.proto` file is under the project's directory then set `ProtoRoot="."`.
* If the `.proto` file is outside of the project's directory then set `ProtoRoot="<relative path to project directory>"`.

Now prune out from `Protobuf_Rooted` the items that the user doesn't want to compile - those don't have
`ProtoCompile` metadata as `true`.  The pruned list is now called `Protobuf_Compile`.

Set the `Source` metadata on `Protobuf_Compile` items to be the name of the `.proto` file.
The `Source` metadata is used later as a key to map generated files to `.proto` files.

### Handling incremental builds

#### Gathering files to check for incremental builds

The target `Protobuf_PrepareCompile` tries to work out which files the protobuf compiler will
generate without actually calling the protobuf compiler. This is a best-effort guess.
The custom task `ProtoCompilerOutputs` is called to do this. The results are stored in the
item list `Protobuf_ExpectedOutputs`.

The target `Protobuf_PrepareCompile` also reads previously written `.protodep` files to get
any actual files previously generated. The custom task `ProtoReadDependencies` is called to
do this. The results are stored in the item list `Protobuf_Dependencies`.
This is in case the list of actual files is different from the previous best-effort guess
from `ProtoCompilerOutputs`.

The expected outputs and previous outputs are needed so that the timestamps of those files
can be checked later when handling an incremental build.

#### Understanding incremental builds

To avoid unnecessarily recompiling the `.proto` files during an incremental build the
target `_Protobuf_GatherStaleBatched` tries to work out if any files have changed.

It checks for out of date files using MSBuilds incremental build feature that compares the 
timestamps on a target's _Input_ files to its _Output_ files.
See [How to: Build incrementally](https://learn.microsoft.com/en-us/visualstudio/msbuild/how-to-build-incrementally)

The _Inputs_ that are checked are:
* Timestamps of the `.proto` files
* Timestamps of previously generated files (list of these files read from `.protodep` files)
* Timestamps of MSBuild project files

These are checked against the _Outputs_:
* Timestamps of the expected generated files

[MSBuild target batching](https://learn.microsoft.com/en-us/visualstudio/msbuild/item-metadata-in-target-batching)
is used to check each `.proto` file against its expected outputs. The batching is done by specifying the `Source` 
metadata in the _Input_. Items where `Source` metadata matches in both input and output are in each batch.

The target `_Protobuf_GatherStaleBatched` sets the metadata `_Exec=true` on `_Protobuf_OutOfDateProto` 
items that are out of date.

Later in the target `_Protobuf_GatherStaleFiles`, the items in `_Protobuf_OutOfDateProto` that don't have
metadata `_Exec==true` are removed from the list of items, leaving only those that need compiling.

### Compile the .proto files

The target `_Protobuf_CoreCompile` is run for each `.proto` file that needs compiling.
These are in the item list `_Protobuf_OutOfDateProto`. The custom task `ProtoCompile` is called to run the
protobuf compiler. The files that were generated are returned in the item list `_Protobuf_GeneratedFiles`.

If there are expected files that were not actually generated then the behaviour depends on whether the
generated files should have been within the project (e.g. in the intermediate directories) or were
specified to be outside of the project.
* If within the project - empty files are created to prevent incremental builds doing unnecessary recompiles
* If outside the project - by default empty files are not created and a warning is output (this behaviour is configurable)

**TODO:** why are files inside and outside the project treated differently?

### Add generated .cs files to the list of files for the C# compiler

The target `_Protobuf_AugmentLanguageCompile` adds to the `Compile` item list
(the list of files that CSC compiles) the expected generated files.

**Note** - this is the _expected_ files not the _actual_ generated files and this is done
before the protobuf compiler is called.

**TODO:** why are the _expected_ files not the _actual_ generated files added?

## Handling design time builds

Design-time builds are special builds that Visual Studio uses to gather information about the project.
They are not user-initiated but may be triggered whenever files are added, removed or saved.
See [Design-Time Builds](https://github.com/dotnet/project-system/blob/main/docs/design-time-builds.md).

The Grpc.Tools build targets used to try and optimise design time builds by disabling calling the
protobuf compiler during a design time build.  However this optimisation can lead to errors in
Visual Studio because the generated files may not exist or be out of date and any code that relies
on them will then have errors.

Now design time builds behave exactly the same as a normal build.
The old behaviour can be enabled by setting the setting `DisableProtobufDesignTimeBuild` property
to `true` in the project file **_if_** it is a design time build, e.g. by adding

```xml
<PropertyGroup Condition="'$(DesignTimeBuild)' == 'true' ">
    <DisableProtobufDesignTimeBuild>true</DisableProtobufDesignTimeBuild>
</PropertyGroup>
```

## Automatically including .proto files

For SDK projects it is possible to automatically include `.proto` files found in the project
directory or sub-directories, without having to specify them with a `<Protobuf>` item.
To do this the property `EnableDefaultProtobufItems` has be set to `true` in the project file.

By default it is not set and `<Protobuf>` items must be included in the project for
the `.proto` files to be compiled.