SynthTool (for client libraries)

This tool helps to generate and layout cloud client libraries. Synthtool runs the GAPIC (Generated API Client) Generator via Google API Artifact Manager (artman).

Prerequisites

1. Linux This tool runs on Linux only. No other platforms are supported.

2. Python 3.6 Either install it from python.org or use pyenv to get 3.6.

3. Bazel can be downloaded from bazel.build.

4. Docker Some synth.py files require Docker to generate code.

5. Clone this repository and install this library with pip:

   cd synthtool
   python3 -m pip install -e .
   

Basic usage

To start the process of generation, clone the destination repository.

git clone [email protected]:googleapis/python-tasks.git
cd python-tasks/

Running synthtool

If a synth.py script is not present, create a new one.

You can create one from scratch or copy one from another library.

• e.g. the synth.py for the Cloud Tasks API for Python, Java, Node.js, PHP, or Ruby.

Run synthtool:

python3 -m synthtool

After synthtool runs successfully:

• Investigate the changes it made
• Run the library tests
• Commit and push the changes to a branch and open a Pull Request

Find examples below in different programming languages (Cloud Tasks API used as an example).

Python

• Clone the destination repository:

  git clone [email protected]:googleapis/python-tasks.git
  cd python-tasks/
  

• Run synthtool to generate using the existing synth.py file for the Python Client for Cloud Tasks API:

  python3 -m synthtool
  

• See the Python Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.

Java

• Clone the destination repository:

  git clone [email protected]:googleapis/java-tasks.git
  cd java-tasks/
  

• Run synthtool to generate using the existing synth.py file for the Google Cloud Java Client for Cloud Tasks:

  python3 -m synthtool
  

• See the Java Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.

Node.js

• Clone the destination repository:

  git clone [email protected]:googleapis/nodejs-tasks.git
  cd nodejs-tasks/
  

• Run synthtool to generate using the existing synth.py file for the Google Cloud Tasks Node.js Client:

  python3 -m synthtool
  

• See the Node.js Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.

PHP

• Clone the destination repository:

  git clone [email protected]:googleapis/google-cloud-php.git
  cd google-cloud-php/
  

• Navigate to the destination directory:

  cd Tasks/
  

• Run synthtool to generate using the existing synth.py file for the Google Cloud Tasks client for PHP:

  python3 -m synthtool
  

• See the PHP Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.

Ruby

• Clone the destination repository:

  git clone [email protected]:googleapis/google-cloud-ruby.git
  cd google-cloud-ruby/
  

• Navigate to the destination directory:

  cd google-cloud-tasks/
  

• Run synthtool to generate using the existing synth.py file for the Ruby Client for Cloud Tasks API:

  python3 -m synthtool
  

• See the Ruby Contributing Guide or instructions to install dependencies, run tests, and submit a contribution.

Features

Common transforms

Functions in synthtool make it easier to copy files, merge files, etc. See the pydocs for more details.

Templating

SynthTool supports template files using Jinja.

Templates are found in subdirectories of synthtool/gcp/templates/ for each language,

• e.g. the template directories for Python, Node.js, PHP, or Ruby.

You can generate and copy templates using gcp.CommonTemplates in your synth.py:

common_templates = gcp.CommonTemplates()

templates = common_templates.node_library()
s.copy(templates)

You can provide variables to templates as keyword arguments to the library generation method:

common_templates = gcp.CommonTemplates()

templates = common_templates.node_library(version=5, show_version=True, previous_versions=[1,2,3,4])

s.copy(templates)

Template files can access any values provided, e.g.

• README.md.j2

  {% if show_version %}
  The version is {{ version }}
  
  {% if previous versions is defined %}
  Previous versions:
    {% for ver in previous_versions %}
    - {{ ver }}
    {% endfor %}
  {% endif %}
  {% endif %}
  

For more information on how to use Synthtool templating for Python Samples, view /py_templating_instructions

You can learn more about Jinga templating in the Template Designer Documentation.

googleapis-private

SynthTool supports generation from googleapis/googleapis-private.

gapic = gcp.GAPICGenerator()

library = gapic.node_library('speech', 'v1', private=True)

2FA is required to clone a private repo.

• Using SSH: Before running Synthtool, set the environment variable AUTOSYNTH_USE_SSH to true.

The repo is cloned using SSH.

• Using HTTPS: Generate a GitHub Personal Access Token with scope repo. Run synthtool.

When GitHub prompts for your GitHub password, provide the access token instead.

synthtool > Cloning googleapis-private.
Username for 'https://github.com': busunkim96
Password for 'https://[email protected]':

Artman Version

SynthTool uses the latest version of the Artman Docker image. You can change this by setting the environment variable SYNTHTOOL_ARTMAN_VERSION to the desired version tag.

export SYNTHTOOL_ARTMAN_VERSION=0.16.2

GAPIC Generator Python Version

SynthTool uses the latest version of gcr.io/gapic-images/gapic-generator-python. You can change this by setting the environment variable SYNTHTOOL_GAPIC_GENERATOR_PYTHON_VERSION to the desired version tag.

export SYNTHTOOL_GAPIC_GENERATOR_PYTHON_VERSION=0.22.0

Alternatively you can set the generator version by passing it to gapic.py_library.

import synthtool as s
import synthtool.gcp as gcp

gapic = gcp.GAPICMicrogenerator()

library = gapic.py_library(
    "bigquery/connection", "v1beta1", generator_version="0.22.0"
)

Local Googleapis

SynthTool supports generation from a local copy of googleapis. Specify the path to googleapis in the environment variable SYNTHTOOL_GOOGLEAPIS.

export SYNTHTOOL_GOOGLEAPIS=path/to/local/googleapis

Local GAPIC Generator

SynthTool supports generation from a local copy of gapic-generator. Specify the path to gapic-generator in the environment variable SYNTHTOOL_GENERATOR.

export SYNTHTOOL_GENERATOR=path/to/local/gapic-generator

Don't forget to compile gapic-generator before running SynthTool.

cd path/to/local/gapic-generator
./gradlew fatJar

Local Template Files

SynthTool supports specifying a local directory of templates. Specify the path to the root template directory (not a SynthTool clone) in the environment variable SYNTHTOOL_TEMPLATES.

export SYNTHTOOL_TEMPLATES=path/to/local/templates

Include .proto files

SynthTool supports copying .proto API definition files from googleapis.

gapic = gcp.GAPICGenerator()

library = gapic.node_library('speech', 'v1', include_protos=True)

Context-Aware Commits

Autosynth runs synthtool on your synth.py nightly or more frequently. By default, it runs synthtool once, and if the generated code differs, creates a PR with the differences.

Autosynth can also find which changes in upstream repositories triggered changes in the generated code. To enable this behavior (context-aware commits), set one or both of the following flags in you synth.py file:

AUTOSYNTH_MULTIPLE_COMMITS
AUTOSYNTH_MULTIPLE_PRS

Example

Assume that since the library source code was last generated, A, B and X, Y were committed to googleapis and synthtool respectively, and they all triggered changes in the generated library code.

Here's what autosynth generates for each flag setting.

AUTOSYNTH_MULTIPLE_COMMITS = True

Autosynth creates one PR, with a single commit for each original commit: | PR | | - | | A | | B | | X | | Y |

AUTOSYNTH_MULTIPLE_COMMITS = True
AUTOSYNTH_MULTIPLE_PRS = True

Autosynth creates two PRs, with a single commit for each original commit: | PR1 | | - | | A | | B |

AUTOSYNTH_MULTIPLE_PRS = True

Autosynth creates two PRs, with a single commit combining all the original commits.

Helpful tips

Where does the generated code go?

SynthTool runs Artman which creates generated code that can be found at ~/.cache/synthtool/googleapis<-private>/artman_genfiles. This is useful for figuring out what it is you need to copy for your specific library.

Warning: Don't lint manually-written code in synth.py!

Ben had the misfortune to discover a corner case where autosynth deleted a file that Ben never intended or expected it to delete.

Here is what happened:

1. Autosynth cannot directly observe which files your synth.py generates, because synth.py could literally do anything, including launch the space shuttle. So, Autosynth figures out which files were generated by examining all the file system reads and writes that happened while synth.py was executing. Any file that is written to or copied to is deemed to have been generated by synth.py and recorded as a generatedFiles in synth.metadata.

2. NodeJS's synth.py ran the linter on manually-written sample files in the repo. A new version of the linter was pulled in, which modified a manually-written sample file. Autosynth observed the write and concluded that the manually-written file was a generated file, and listed it in generatedFiles in synth.metadata. The next time Autosynth ran, the linter made no changes, and the manually-written sample file was not written to. Autosynth concluded the manually-written sample file was no longer being generated, and deleted it.

Lesson Learned:

Make sure your synth.py does not touch any manually-written files in the repo.