ignored and add ignore-if. The latter runs an arbitrary shell command which, if it returns zero, causes the test to be ignored. This allows much more flexibility than the overly simplistic “always ignore this test” of ignored. Tests with ignored: <reason> can be changed to ignore-if: true followed (or preceded) by a comment # <reason> (assuming comment_prefix is set: see below).comment_prefix("..."). For example LangTester::new().comment_prefix("#") causes lines in tests starting with # to be entirely ignored by lang_tester.test_file_filter is deprecated in favour of test_path_filter. The latter doesn't pre-filter non-files, making it more flexible. A simple way of moving from test_file_filter to test_path_filter is to change test_file_filter(|p| ...) to test_path_fiter(|p| p.is_file() & ...).lang_tests::a::b::c) rather than just the leaf (e.g c).FMBuilder throws an error.extra-args key has been renamed to exec-arg to reflect the fact that each key is a single argument.env-var key has been added. This allows environment variables to be set on a per-test basis e.g.:Compiler: env-var: DEBUG=1 stdout: xyz
Fix file descriptor race for tests that contain stdin data: files were closed twice, which could lead to an active (reused) file descriptor being closed incorrectly.
Documentation fixes.
test_extract) panics, lang_tester now considers that a test failure and reports it to the user. Because this uses catch_unwind underneath, the functions passed to lang_tester must now be RefUnwindSafe.The test_extract function signature has changed from:
Fn(&str) -> Option<String> + Send + Sync,
to:
Fn(&Path) -> String + Send + Sync,
In other words, users now have to both:
String rather than an Option<String>.In practise, most test_extract functions can be changed from (roughly):
test_extract(|s| { s.lines() ... })
to:
test_extract(|p| { std::fs::read_to_string(p).lines() })
fm_options function. See the fm changes for more information.Remove the built-in fuzzy matcher and use the fm library instead. This should be entirely backwards compatible in its default state. Users who want non-default fm options can use the new fm_options function in LangTester.
Add a stdin key to allow users to specify stdin input which should be passed to a sub-command.
Lines are no longer stripped of their leading or trailing whitespace allowing tests to be whitespace sensitive if required. Since matching in fm defaults to ignoring leading and trailing whitespace, the previous behaviour is preserved unless users explicitly tell fm to match whitespace.
a/x and b/x these are pretty printed as a::x and b::x respectively (whereas before they were pretty printed as simply x, meaning that you could not tell which had succeeded / failed).Add test_threads function which allows you to specify the number of test threads programatically.
Move from the deprecated tempdir to the maintained tempfile crate.
Add support for ignorable tests. A test command ignore: is interpreted as causing that entire test file to be ignored. As with cargo test, such tests can be run with the --ignored switch.
Fix a bug whereby the number of ignored tests was incorrectly reported.
status: signal to indicate that a test should exit due to a signal: on platforms which do not support this (e.g. Windows), such tests are ignored. Similarly, if a program was terminated due to a signal then, on Unix, the user is informed of that after test failure.Add support for --nocapture to better emulate cargo test. As with cargo test, if you're running more than one test then --nocapture is generally best paired with --test-threads=1 to avoid confusing, multiplexed output to the terminal.
Be clearer that tests can have defaults: notably commands default to status: success unless overridden.
Individual tests can now add extra arguments to an invoked command with the extra-args field.
Ensure that, if a command in a chain fails, the whole chain of commands fails. This means that if, for example, compilation of command C fails, we do not try and run C anyway (which can end up doing confusing things like running an old version of C).
Add support for running a defined number of parallel processes, using the cargo test-ish option --test-threads=n. For example, to run tests sequentially, specify --test-threads=1.
Warn users if a given test has run unexpectedly long (currently every multiple of 60 seconds). This is often a sign that a test has entered an infinite loop.
Use better terminology in the documentation. Previously “test” was used to mean a number of subtly different things which was rather confusing. Now test files contain test data. Test data contains test commands. Test commands contain sub-tests.
Stop testing a given test file on the first failed sub-test. Previously only a test command which exited unsuccessfully caused a test file to be considered as failed, causing the source of errors to sometimes be missed.
test_extract and test_cmds functions must now satisfy the Sync trait. This is a breaking change, albeit one that nearly all such functions already satisfied.Accept cargo-ish command-line parameters. In particular, this lets users run a subset of tests e.g. “ ab cd” only runs tests with “ab” or “cd” in their name. If you don't want lang_tester to look at your command-line arguments, set use_cmdline_args(false) (the default is true).
Run tests in parallel (one per CPU core). Depending on the size of your machine and the size of your test suite, this can be a significant performance improvement.
The status field can now take integer exit codes. i.e. if you specify status: 7 then the exit code of the binary being run will be checked to see if it is 7.
First stable release.