cli-test/README.md

# cli-test

## What?

`cli-test` makes integration testing of command-line tools easier.

## Goals

* Readable syntax. Common cases should be concise, and pretty much anyone
  should be able to read tests even if they've never seen this tool before.

* Minimal issues with quoting. The toybox tests -- being shell scripts --
  quickly become a nightmare of quoting. Using a non ad hoc format (such as
  JSON) would have introduced similar but different quoting issues. A custom
  format, while annoying, side-steps this.

* Sensible defaults. We expect your exit status to be 0 unless you say
  otherwise. We expect nothing on stderr unless you say otherwise. And so on.

* Convention over configuration. Related to sensible defaults, we don't let you
  configure things that aren't absolutely necessary. So you can't keep your test
  data anywhere except in the `files/` subdirectory of the directory containing
  your test, for example.

## Non Goals

* Portability. Just being able to run on Linux (host and device) is sufficient
  for our needs. macOS is probably easy enough if we ever need it, but Windows
  probably doesn't make sense.

## Syntax

Any all-whitespace line, or line starting with `#` is ignored.

A test looks like this:
```
name: unzip -l
command: unzip -l $FILES/example.zip d1/d2/x.txt
after: [ ! -f d1/d2/x.txt ]
expected-stdout:
	Archive:  $FILES/example.zip
	  Length      Date    Time    Name
	---------  ---------- -----   ----
	     1024  2017-06-04 08:45   d1/d2/x.txt
	---------                     -------
	     1024                     1 file
---
```

The `name:` line names the test, and is only for human consumption.

The `command:` line is the command to be run. Additional commands can be
supplied as zero or more `before:` lines (run before `command:`) and zero or
more `after:` lines (run after `command:`). These are useful for both
setup/teardown but also for testing post conditions (as in the example above).

Any `command:`, `before:`, or `after:` line is expected to exit with status 0.
Anything else is considered a test failure.

The `expected-stdout:` line is followed by zero or more tab-prefixed lines that
are otherwise the exact output expected from the command. (There's magic behind
the scenes to rewrite the test files directory to `$FILES` because otherwise any
path in the output would depend on the temporary directory used to run the test.)

There is currently no `expected-stderr:` line. Standard error is implicitly
expected to be empty, and any output will cause a test failure. (The support is
there, but not wired up because we haven't needed it yet.)

The fields can appear in any order, but every test must contain at least a
`name:` line and a `command:` line.

## Output

The output is intended to resemble gtest.

## Future Directions

* It's often useful to be able to *match* against stdout/stderr/a file rather
  than give exact expected output. We might want to add explicit support for
  this. In the meantime, it's possible to use an `after:` with `grep -q` if
  you redirect in your `command:`.

* In addition to using a `before:` (which will fail a test), it can be useful
  to be able to specify tests that would cause us to *skip* a test. An example
  would be "am I running as root?".

* It might be useful to be able to make exit status assertions other than 0?

* There's currently no way (other than the `files/` directory) to share repeated
  setup between tests.