package tezt

Changelog

Version 4.2.0

New Features

• Added Log.clear_error_context_queue which clears the queue of messages that were skipped but that may still be printed in case of error.
• Added Test.current_test_seed_specification which returns the value of the ~seed argument of the current test.
• SIGTERM can now also be used to abort a run. Previously, only SIGINT had this effect. Note that while two consecutive SIGINT cause the process to exit immediately, two consecutive SIGTERM do not (the equivalent of "a second SIGTERM" is SIGKILL).
• Added new library tezt.scheduler. It is used internally to replace the scheduler which is used when running tests in parallel with --job-count. The library is general and flexible enough to be used outside of Tezt however.
• Added options to Runner.t. It allows to pass additional options to SSH.

Bug Fixes

• When running tests in parallel with --job-count N with N >= 2, the scheduler now aborts (with SIGTERM and then SIGINT) tests that reach their timeout. This means that even tests that are non-cooperative (do not yield with Lwt) can get killed when using --test-timeout and/or --global-timeout.

Version 4.1.0

Breaking Changes

• The addition of Check.result breaks the common pattern let result = ... in Check.(result = ...). Fix this by renaming the result variable.

New Features

• Added --on-empty-test-list. In particular, --on-empty-test-list ignore and --on-empty-test-list warn cause Tezt to exit with code 0 instead of 3 when the list of selected tests is empty.
• Added Check.result to make comparable result types.
• Added Check.get_pp, Check.get_equal and Check.get_compare to make it easier to make custom comparable types.
• Added Test.current_test_seed to get the random seed for the test that is currently running.
• Added Log.Style. It provides functions to set the style of timestamps and prefixes in log messages. In particular it allows to disable them.
• Added command-line options --log-timestamp, --no-log-timestamp, --log-prefix, --no-log-prefix, and their corresponding values Cli.Logs.timestamp and Cli.Logs.prefix.
• When running with --job-count 2 (-j 2) or more, Tezt now warns if a test has been running for more than 60 seconds. This delay can be modified or disabled with new command-line option --warn-after-timeout.
• The value of --job-count can now also be specified using the TEZT_JOB_COUNT environment variable.
• Process.kill now sends SIGKILL instead of SIGTERM on Win32.
• Process.program_path now uses where.exe instead of sh on Win32.
• The Process module now internally uses Lwt notifications instead of promises to handle process outputs. This makes it work with multiple threads.
• Added modules Process_common and Echo to the UNIX backend. Those modules are experimental; their interface may change in future releases and they are not documented in the generated ocamldoc. They are meant for users who want to implement custom versions of the Process module.

Minor Improvements

• Improved the documentation of the Test Selection Language (TSL) in --help.
• Improved the error message when passing invalid regular expressions in command-line arguments.

Bug Fixes

• Fixed Cli.Selecting_tests.tsl_expression which would not return the same result after the first time it was called.

Version 4.0.0

Breaking Changes

• The Cli module has changed a lot: instead of being all together in a record with mutable fields, command-line argument values are now separate values and are grouped in thematically-related submodules. Types such as log_level have also moved to those submodules. Those values are usually not used in user-code however, so this is only a breaking change for very niche cases. Functions Cli.get_* are unchanged, although there is now a better way to define custom command-line arguments.
• true and false are now invalid tags.

• The JSON module is now in its own library tezt.json. This allows one to use the JSON module of Tezt without linking with Tezt, and in particular without Tezt parsing the command-line.

  tezt.core depends on tezt.json but does not include the JSON module itself. If your code accessed Tezt_core.JSON, you must replace Tezt_core.JSON with JSON and you should add tezt.json as a dependency (especially if your dune-project contains (implicit_transitive_deps false)).

  tezt and tezt.js, however, still include the JSON module in their interface. If your code only accessed Tezt.JSON or Tezt_js.JSON, you have nothing to do.

• Using both --loop-count N and --loop now always limits the number of iterations to N. Before, --loop-count N --loop would result in an unbounded number of iterations, while --loop --loop-count N limited the number of iterations to N.

New Features

• --help is now much prettier.
• Tezt now uses Clap instead of Stdlib.Arg. This means that you can now define custom command-line arguments directly using Clap instead of using Cli.get_* functions. Such custom command-line arguments do not need to be prefixed with -a and will appear in --help.
• Tests can now be selected using generic predicates from the command-line. See SELECTING TESTS in --help (search for "TSL").
• Added new module Version with value Version.full which contains the version number of Tezt.
• Added --version to print the version number of Tezt on standard output and exit.
• Added --cleanup-timeout to be able to specify the delay between SIGTERM and SIGKILL when cleaning up external processes after a test.
• Added Test.current_test_file.
• Added Test.current_test_title.
• Added Test.current_test_tags and Test.current_test_has_tag.
• Added Test.declare_clean_up_function.

Bug Fixes

• Fixed a bug which caused tests to be started after the global timeout was reached if --keep-going was specified.
• Fixed a bug in Process where terminating a process running on a remote runner would not behave the same as terminating a local process. More precisely, the remote process used to only receive SIGKILL. Now it receives SIGTERM then SIGKILL after the timeout delay, which is the behavior for local processes.
• Fixed a bug which caused "bad file descriptor" (EBADF) errors when logging in at_exit handlers if those at_exit were declared at the toplevel of modules linked before the Log module. A rare occurrence since one does not have access to the Log module in such handlers without some reference hacks.

Version 3.1.1

Bug Fixes

• Fixed a bug causing records to not be taken into account when balancing jobs for --job.

Version 3.1.0

Breaking Changes

• On OCaml 4.12.1 or earlier, the new ?seed argument can break calls to Test.register which are written using the form Test.register ~__FILE__ ~title ~tags @@ fun () -> .... To fix this, either upgrade to OCaml 4.13.0 or later, add ?seed: None, or replace @@ with parentheses.
• Internal type Test.SCHEDULER.response changed. This change is very unlikely to impact you unless you wrote your own backend.
• --match now applies to both test title and test filename.
• Removed the --starting-port command-line argument. You can use -a starting-port= instead.
• Selecting a --job that is empty now results in an error. This means that if you have more jobs than tests in your CI, your CI will fail.
• Skipping all tests with --skip or --only now results in an error. This makes those command-line arguments behave like other filters.
• Tests are now registered with the full value passed as ~__FILE__ to Test.register, instead of just the basename.

New Features

• Added ?seed to Test.register and the --seed command-line parameter to help control the determinism of randomness.
• Added --log-worker-ids which adds worker ids to logs when --job-count is more than 1.
• Added --not-match to deselect tests with a title that matches a regular expression.
• Added --not-file to deselect tests with a given filename.
• --time can now be passed along with --from-record <record.json> --list to pretty-print timings from previous executions.
• --file FILE now selects all tests registered from a source file that ends with FILE.
• Added --not-file FILE to deselect tests registered from a source file that ends with FILE.
• In Cli: added _opt variants for functions that retrieve custom arguments.

Bug Fixes

• Fixed a bug where the log file does not contain logs from tests in the presence of --job-count. In the presence of --job-count N and --log-file FILE, test results are now written to FILE, where as the logs from tests are written in a separate file per worker named BASENAME-WORKER_ID.EXT, as detailed in --help.
• Fixed a bug where a warning is not emitted when the argument to --not-title does not correspond to any known files.
• Fixed a bug where the list of selectors printed with an empty test selection would not contain --not-title.

Version 3.0.0

Breaking Changes

• Argument ~output_file of Regression.register is now optional and has been renamed into ?file. Previously, ~output_file was automatically prefixed by the value given to --regression-dir. Now, files are put in a directory named expected next to the test itself, and the default filename is a sanitized version of the test title.
• Removed the --regression-dir command-line parameter.

New Features

• The tezt Dune library was split into tezt.core and tezt. Library tezt.core contains the parts that can run both on UNIX and using Node.js. It does not contain Test.run though so it cannot be used on its own. Library tezt is the UNIX backend. It also includes tezt.core, so this is not a breaking change.
• New library tezt.js is a partial backend for Node.js. Compared to tezt, it does not contain modules Process, Temp and Runner.
• Added --resume (short-hand: -r) and --resume-file, which allow to resume a previous run. For instance, if the previous run stopped after a failed test, and if that previous run was itself ran with --resume or --resume-file, resume from this run to avoid running the tests that already succeeded again.
• Added --match (short-hand: -m) to select tests with a title that match a regular expression.
• Added a ?timeout argument to Process.terminate and Process.clean_up. These functions send SIGTERM to processes. If this timeout is reached without the process actually terminating, Process.terminate also sends SIGKILL.
• Added module Diff which allows to compare two sequences. It is used internally by the Regression module to compare test outputs between runs.
• Added module Main with function Main.run, which is the same as Test.run. In practice you can still use Test.run, which now delegates to Main.run. But Test.run is not available in tezt.core, only in tezt and tezt.js. This is not a breaking change since existing applications would use tezt, not tezt.core directly.
• Added Base.project_root which is the path to the root of the current dune project according to environment variable DUNE_SOURCEROOT, falling back to the current directory if unavailable.
• Added Base.span to split a list in two sublists given a predicate.
• Added Base.rexf, a short-hand for rex @@ sf.
• Added Check.file_exists, Check.file_not_exists, Check.directory_exists and Check.directory_not_exists.
• Added Check.is_true and Check.is_false.
• Added Check.json and Check.json_u, type definitions for JSON.t and JSON.u respectively.
• Added support for int32 in the JSON module.
• Added JSON.merge_objects, JSON.filter_map_object and JSON.filter_object.
• Added JSON.equal and JSON.equal_u, equality predicates on JSON.t and JSON.u respectively.
• Added Temp.set_pid. The Temp module no longer calls Unix.getpid directly, this is done by Test.run instead, which calls Temp.set_pid. This allows backends where Unix.getpid cannot be used, like JS.
• Added Test.run_with_scheduler, and the Test.SCHEDULER signature. Added Test.get_test_by_title and Test.run_one. Those functions are used internally to provide different backends (UNIX / JS), but users usually would not have any use for them.
• Some fields in records obtained with --record became optional, with default values. New records may not be compatible with old versions of Tezt.
• Added module Process_hooks which contains the type definition for Process.hooks.

Bug Fixes

• Fixed some cases where regression tests would crash instead of failing gracefully for some system errors.
• Temp.clean_up no longer follows symbolic links, it just deletes the links themselves.
• Fixed --junit which no longer worked when using -j.
• Fixed --loop-count 0 which did not actually run zero loops.
• Fixed a bug where Tezt would exit prematurely without running tests when using --on-unknown-regression-files delete or --on-unknown-regression-files fail.
• Fixed a redundancy in the warning message when removing a non-empty temporary directories registered with the Temp module.
• Fixed some whitespace issues in --help.

Older Versions

Tezt 2.0.0 was the first announced release and as such it had no changelog. Tezt 1.0.0 was released in opam but was not announced.