blob: c1fb7159a68bb0d2f56087193416172c9e7b03ac [file] [log] [blame] [edit]
//! # Snapshot testing for a herd of CLI tests
//!
//! `trycmd` is a test harness that will enumerate test case files and run them to verify the
//! results, taking inspiration from
//! [trybuild](https://crates.io/crates/trybuild) and [cram](https://bitheap.org/cram/).
//!
//! Which tool is right:
//! - Hand-written test cases: for peculiar circumstances
//! - [assert_cmd](https://crates.io/crates/assert_cmd): Test cases follow a certain pattern but
//! special attention is needed in how to verify the results.
//! - `trycmd`: For running a lot of blunt tests (limited test predicates)
//! - Particular attention is given to allow the test data to be pulled into documentation, like
//! with [mdbook](https://rust-lang.github.io/mdBook/)
//! - [cram](https://bitheap.org/cram/): For cases agnostic of any programming language
//!
//! ## Getting Started
//!
//! To create a minimal setup, create a `tests/cli_tests.rs` with
//! ```rust,no_run
//! #[test]
//! fn cli_tests() {
//! trycmd::TestCases::new()
//! .case("tests/cmd/*.trycmd");
//! }
//! ```
//!
//! The test can be run with `cargo test`. This will enumerate all `.trycmd` files and run
//! them as test cases, failing if they do not pass.
//!
//! To temporarily override the results, you can do:
//! ```rust,no_run
//! #[test]
//! fn cli_tests() {
//! trycmd::TestCases::new()
//! .case("tests/cmd/*.trycmd")
//! // See Issue #314
//! .fail("tests/cmd/buggy-case.trycmd");
//! }
//! ```
//!
//! ## Workflow
//!
//! To generate snapshots, run
//! ```console
//! $ TRYCMD=dump cargo test --test cli_tests
//! ```
//! This will write all of the `.stdout` and `.stderr` files in a `dump/` directory.
//!
//! You can then copy over to `tests/cmd` the cases you want to test
//!
//! To update snapshots, run
//! ```console
//! $ TRYCMD=overwrite cargo test --test cli_tests
//! ```
//! This will overwrite any existing `.stdout` and `.stderr` file in `tests/cmd`
//!
//! To filter the tests to those with `name1`, `name2`, etc in their file names, you can run:
//! ```console
//! cargo test --test cli_tests -- cli_tests trycmd=name1 trycmd=name2...
//! ```
//!
//! To debug what `trycmd` is doing, add the feature flag `debug`.
//!
//! ## File Formats
//!
//! Say you have `tests/cmd/help.trycmd`, `trycmd` will look for:
//! - `tests/cmd/help.in/`
//! - `tests/cmd/help.out/`
//!
//! For `tests/cmd/help.toml`, `trycmd` will look for:
//! - `tests/cmd/help.stdin`
//! - `tests/cmd/help.stdout`
//! - `tests/cmd/help.stderr`
//! - `tests/cmd/help.in/`
//! - `tests/cmd/help.out/`
//!
//! ### `*.trycmd`
//!
//! `*.trycmd` files are literate test cases good for:
//! - Markdown-compatible syntax for directly rendering them
//! - Terminal-like appearance for extracting subsections into documentation
//! - Reducing the proliferation of files
//! - Running multiple commands within the same temp dir
//!
//! The syntax is:
//! - Test cases live inside of ` ``` ` fenced code blocks
//! - Everything out of them is ignored
//! - Blocks with info strings with an unsupported language (not `trycmd`, `console`) or the
//! `ignore` attribute are ignored
//! - "`$ `" line prefix starts a new command
//! - "`> `" line prefix appends to the prior command
//! - "`? <status>`" line indicates the exit code (like `echo "? $?"`) and `<status>` can be
//! - An exit code
//! - `success` *(default)*, `failed`, `interrupted`, `skipped`
//! - All following lines are treated as stdout + stderr
//!
//! The command is then split with [shlex](https://crates.io/crates/shlex), allowing quoted content
//! to allow spaces. The first argument is the program to run which maps to `bin.name` in the
//! `.toml` file.
//!
//! Example:
//! ~~~md
//! With the following code:
//! ```rust
//! println!("{}", message);
//! ```
//!
//! You get the following:
//! ```
//! $ my-cmd --print 'Hello World'
//! Hello
//! ```
//! ~~~
//!
//! ### `*.toml`
//!
//! As an alternative to `.trycmd`, the `toml` are good for:
//! - Precise control over current dir, stdin/stdout/stderr (including binary support)
//! - 1-to-1 with dumped results
//! - `TRYCMD=overwrite` support
//!
//! [schema](https://github.com/assert-rs/trycmd/blob/main/schema.json):
//! - `bin.name`: The name of the binary target from `Cargo.toml` to be used to find the file path
//!
//! #### `*.stdin`
//!
//! Data to pass to `stdin`.
//! - If not present, nothing will be written to `stdin`
//! - If `binary = false` in `*.toml` (the default), newlines and path separators will be normalized.
//!
//! #### `*.stdout` and `*.stderr`
//!
//! Expected results for `stdout` or `stderr`.
//! - If not present, we'll not verify the output
//! - If `binary = false` in `*.toml` (the default), newlines and path separators will be normalized before comparing
//!
//! **Eliding Content**
//!
//! Sometimes the output either includes:
//! - Content that changes from run-to-run (like time)
//! - Content out of scope of your tests and you want to exclude it to reduce brittleness
//!
//! To elide a section of content:
//! - `...` as its own line: match all lines until the next one. This is equivalent of
//! `\n(([^\n]*\n)*?`.
//! - `[..]` as part of a line: match any characters. This is equivalent of `[^\n]*?`.
//! - `[EXE]` as part of the line: On Windows, matches `.exe`, ignored otherwise
//! - `[ROOT]` as part of the line: The root directory for where the test is running
//! - `[CWD]` as part of the line: The current working directory within the root
//! - `[YOUR_NAME_HERE]` as part of the line: See [`TestCases::insert_var`]
//!
//! We will preserve these with `TRYCMD=dump` and will make a best-effort at preserving them with
//! `TRYCMD=overwrite`.
//!
//! ### `*.in/`
//!
//! When present, this will automatically be picked as the CWD for the command.
//!
//! `.keep` files will be ignored but their parent directories will be created.
//!
//! ### `*.out/`
//!
//! When present, each file in this directory will be compared to generated or modified files.
//!
//! See also "Eliding Content" for `.stdout`
//!
//! `.keep` files will be ignored.
// Doesn't distinguish between incidental sharing vs essential sharing
#![allow(clippy::branches_sharing_code)]
#[macro_use]
mod macros;
pub mod cargo;
pub mod schema;
#[cfg(feature = "diff")]
pub(crate) mod diff;
pub(crate) mod elide;
pub(crate) mod lines;
mod cases;
mod color;
mod command;
mod error;
mod filesystem;
mod registry;
mod runner;
mod spec;
pub use cases::TestCases;
pub use error::Error;
pub(crate) use color::Palette;
pub(crate) use command::wait_with_input_output;
pub(crate) use filesystem::{shallow_copy, File, FilesystemContext, Iterate as FsIterate};
pub(crate) use registry::BinRegistry;
pub(crate) use runner::{Case, Mode, Runner};
pub(crate) use spec::RunnerSpec;