package alcotest

  1. Overview
  2. Docs
Alcotest is a lightweight and colourful test framework

Install

Dune Dependency

github.com Readme Changelog ISC License Edit opam file Versions (40)

Authors

  1. T
    Thomas Gazagnaire

Maintainers

  1. T
    thomas@gazagnaire.org

Sources

alcotest-lwt-1.1.0.tbz
sha256=212827a49abf4008581c0da53e7ec78a9d639b415380dcb1fdeeb23f3ff083e2
sha512=c47d04b3c7100af703b470b93ff9fe9fe22790415370b6d5972736f46a5c83901717d67caf0c4115d01020d3078dc7f3063838578174921cab352546dad00148

Description

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Published: 03 Apr 2020

README

Alcotest is a lightweight and colourful test framework.

Alcotest exposes simple interface to perform unit tests. It exposes a simple TESTABLE module type, a check function to assert test predicates and a run function to perform a list of unit -> unit test callbacks.

Alcotest provides a quiet and colorful output where only faulty runs are fully displayed at the end of the run (with the full logs ready to inspect), with a simple (yet expressive) query language to select the tests to run.

Examples

A simple example (taken from examples/simple.ml):

(* Build with `ocamlbuild -pkg alcotest simple.byte` *)

(* A module with functions to test *)
module To_test = struct
  let lowercase = String.lowercase_ascii
  let capitalize = String.capitalize_ascii
  let str_concat = String.concat ""
  let list_concat = List.append
end

(* The tests *)
let test_lowercase () =
  Alcotest.(check string) "same string" "hello!" (To_test.lowercase "hELLO!")

let test_capitalize () =
  Alcotest.(check string) "same string" "World." (To_test.capitalize "world.")

let test_str_concat () =
  Alcotest.(check string) "same string" "foobar" (To_test.str_concat ["foo"; "bar"])

let test_list_concat () =
  Alcotest.(check (list int)) "same lists" [1; 2; 3] (To_test.list_concat [1] [2; 3])

(* Run it *)
let () =
  let open Alcotest in
  run "Utils" [
      "string-case", [
          test_case "Lower case"     `Quick test_lowercase;
          test_case "Capitalization" `Quick test_capitalize;
        ];
      "string-concat", [ test_case "String mashing" `Quick test_str_concat  ];
      "list-concat",   [ test_case "List mashing"   `Slow  test_list_concat ];
    ]

The result is a self-contained binary which displays the test results. Use ./simple.byte --help to see the runtime options.

$ ./simple.native
Testing Utils.
[OK]       string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
Test Successful in 0.001s. 4 tests run.

Selecting tests to execute

You can filter which tests to run by supplying a regular expression matching the names of the tests to execute, or by passing a regular expression and a comma-separated list of test numbers (or ranges of test numbers, e.g. 2,4..9):

$ ./simple.native test '.*concat*'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[SKIP]     string-case            1   Capitalization.
[OK]       string-concat          0   String mashing.
[OK]       list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 2 tests run.

$ ./simple.native test 'string-case' '1..3'
Testing Utils.
[SKIP]     string-case            0   Lower case.
[OK]       string-case            1   Capitalization.
[SKIP]     string-concat          0   String mashing.
[SKIP]     list-concat            0   List mashing.
The full test results are available in `_build/_tests`.
Test Successful in 0.000s. 1 test run.

Note that you cannot filter by test case name (i.e. Lower case or Capitalization), you must filter by test name & number instead. Test names may contain only alphanumeric characters, spaces, hyphens and underscores.

See the examples folder for more examples.

Quick and Slow tests

In general you should use `Quick tests: tests that are ran on any invocations of the test suite. You should only use `Slow tests for stress tests that are ran only on occasion (typically before a release or after a major change). These slow tests can be suppressed by passing the -q flag on the command line, e.g.:

$ ./test.exe -q # run only the quick tests
$ ./test.exe    # run quick and slow tests

Passing custom options to the tests

In most cases, the base tests are unit -> unit functions. However, it is also possible to pass an extra option to all the test functions by using 'a -> unit, where 'a is the type of the extra parameter.

In order to do this, you need to specify how this extra parameter is read on the command-line, by providing a Cmdliner term for command-line arguments which explains how to parse and serialize values of type 'a (note: do not use positional arguments, only optional arguments are supported).

For instance:

let test_nice i = Alcotest.(check int) "Is it a nice integer?" i 42

let int =
  let doc = "What is your prefered number?" in
  Cmdliner.Arg.(required & opt (some int) None & info ["n"] ~doc ~docv:"NUM")

let () =
  Alcotest.run_with_args "foo" int [
    "all", ["nice", `Quick, test_nice]
  ]

Will generate test.exe such that:

$ test.exe test
test.exe: required option -n is missing

$ test.exe test -n 42
Testing foo.
[OK]                all          0   int.

Lwt

Alcotest provides an Alcotest_lwt module that you could use to wrap Lwt test cases. The basic idea is that instead of providing a test function in the form unit -> unit, you provide one with the type unit -> unit Lwt.t and alcotest-lwt calls Lwt_main.run for you.

However, there are a couple of extra features:

  • If an async exception occurs, it will cancel your test case for you and fail it (rather than exiting the process).

  • You get given a switch, which will be turned off when the test case finishes (or fails). You can use that to free up any resources.

For instance:

let free () = print_endline "freeing all resources"; Lwt.return ()

let test_lwt switch () =
  Lwt_switch.add_hook (Some switch) free;
  Lwt.async (fun () -> failwith "All is broken");
  Lwt_unix.sleep 10.

let () =
  Lwt_main.run @@ Alcotest_lwt.run "foo" [
    "all", [
      Alcotest_lwt.test_case "one" `Quick test_lwt
    ]
  ]

Will generate:

$ test.exe
Testing foo.
[ERROR]             all          0   one.
-- all.000 [one.] Failed --
in _build/_tests/all.000.output:
freeing all resources
[failure] All is broken

Screenshots

The following screenshots demonstrate the HTML testing output from the odoc project.

All tests passed Some tests failed Failed test with custom diffing
ok err diff

Comparison with other testing frameworks

The README is pretty clear about that:

Alcotest is the only testing framework using colors!

More seriously, Alcotest is similar to ounit but it fixes a few of the problems found in that library:

  • Alcotest has a nicer output, it is easier to see what failed and what succeeded and to read the log outputs of the failed tests;

  • Alcotest uses combinators to define pretty-printers and comparators between the things to test.

Other nice tools doing different kind of testing also exist:

  • qcheck qcheck does random generation and property testing (e.g. Quick Check)

  • crowbar and bun are similar to qcheck, but use compiler-directed randomness, e.g. it takes advantage of the AFL support the OCaml compiler.

  • ppx_inline_tests allows to write tests in the same file as your source-code; they will be run only in a special mode of compilation.

Dependencies (8)

  1. stdlib-shims
  2. re >= "1.7.2"
  3. uuidm
  4. cmdliner >= "1.0.3" & < "1.1.0"
  5. astring
  6. fmt >= "0.8.6"
  7. ocaml >= "4.03.0"
  8. dune >= "2.0"

Dev Dependencies (1)

  1. odoc with-doc

  1. ahrocksdb
  2. albatross >= "1.5.0"
  3. alcotest-async < "1.0.0" | = "1.1.0"
  4. alcotest-lwt < "1.0.0" | = "1.1.0"
  5. alg_structs_qcheck
  6. ambient-context
  7. ambient-context-eio
  8. ambient-context-lwt
  9. angstrom >= "0.7.0"
  10. ansi >= "0.6.0"
  11. anycache >= "0.7.4"
  12. anycache-async
  13. anycache-lwt
  14. archetype >= "1.4.2"
  15. archi
  16. arp
  17. arp-mirage
  18. arrakis
  19. art
  20. asak >= "0.2"
  21. asli >= "0.2.0"
  22. asn1-combinators >= "0.2.2"
  23. atd >= "2.3.3"
  24. atdgen >= "2.10.0"
  25. atdpy
  26. atdts
  27. base32
  28. base64 >= "2.1.2" & < "3.2.0" | >= "3.4.0"
  29. bastet
  30. bastet_async
  31. bastet_lwt
  32. bech32
  33. bechamel >= "0.5.0"
  34. bigarray-overlap
  35. bigstring >= "0.3"
  36. bigstring-unix >= "0.3"
  37. bigstringaf
  38. bitlib
  39. blake2
  40. bloomf
  41. bls12-381 < "0.4.1" | >= "3.0.0" & < "18.0"
  42. bls12-381-hash
  43. bls12-381-js >= "0.4.2"
  44. bls12-381-js-gen >= "0.4.2"
  45. bls12-381-legacy
  46. bls12-381-signature
  47. bls12-381-unix
  48. blurhash
  49. builder-web
  50. bulletml
  51. bytebuffer
  52. ca-certs
  53. ca-certs-nss
  54. cactus
  55. caldav
  56. calendar >= "3.0.0"
  57. callipyge
  58. camlix
  59. camlkit
  60. camlkit-base
  61. capnp-rpc < "1.2.3"
  62. capnp-rpc-lwt < "0.3"
  63. capnp-rpc-mirage >= "0.9.0"
  64. capnp-rpc-unix >= "0.9.0" & < "1.2.3"
  65. carray
  66. carton
  67. cborl
  68. ccss >= "1.6"
  69. cf-lwt
  70. chacha
  71. channel
  72. charrua-client
  73. charrua-client-lwt
  74. charrua-client-mirage < "0.11.0"
  75. checked_oint < "0.1.1"
  76. checkseum >= "0.0.3"
  77. cid
  78. clarity-lang
  79. class_group_vdf
  80. cohttp >= "0.17.0"
  81. cohttp-curl-async
  82. cohttp-curl-lwt
  83. cohttp-eio >= "6.0.0~beta2"
  84. colombe >= "0.2.0"
  85. color
  86. conan
  87. conan-cli
  88. conan-database
  89. conan-lwt
  90. conan-unix
  91. conduit = "3.0.0"
  92. conex < "0.10.0"
  93. conex-mirage-crypto
  94. conex-nocrypto
  95. cookie
  96. cow >= "2.2.0"
  97. css
  98. css-parser
  99. cstruct >= "3.3.0"
  100. cstruct-sexp
  101. ctypes-zarith
  102. cuid
  103. curly
  104. current_incr
  105. cwe_checker
  106. data-encoding
  107. datakit >= "0.12.0"
  108. datakit-bridge-github >= "0.12.0"
  109. datakit-ci
  110. datakit-client-git >= "0.12.0"
  111. decompress >= "0.8" & < "1.5.3"
  112. depyt
  113. digestif >= "0.8.1"
  114. dispatch >= "0.4.1"
  115. dkim
  116. dkim-bin
  117. dkim-mirage
  118. dns >= "4.0.0"
  119. dns-cli
  120. dns-client >= "4.6.0"
  121. dns-forward < "0.9.0"
  122. dns-forward-lwt-unix
  123. dns-resolver
  124. dns-server
  125. dns-tsig
  126. dnssd
  127. dnssec
  128. docfd >= "2.2.0"
  129. dog < "0.2.1"
  130. domain-name
  131. dream
  132. dream-pure
  133. duff
  134. dune-release >= "1.0.0"
  135. duration >= "0.1.1"
  136. emile
  137. encore
  138. eqaf >= "0.5"
  139. equinoxe
  140. equinoxe-cohttp
  141. equinoxe-hlc
  142. eris
  143. eris-lwt
  144. ezgzip
  145. ezjsonm >= "0.4.2" & < "1.3.0"
  146. ezjsonm-lwt < "1.3.0"
  147. FPauth
  148. FPauth-core
  149. FPauth-responses
  150. FPauth-strategies
  151. faraday != "0.2.0"
  152. farfadet
  153. fat-filesystem >= "0.12.0"
  154. ff
  155. ff-pbt
  156. fiat-p256
  157. flex-array
  158. fsevents-lwt
  159. functoria >= "2.2.0"
  160. functoria-runtime >= "2.2.0" & != "3.0.1" & < "4.0.0~beta1"
  161. geojson
  162. geoml >= "0.1.1"
  163. git = "1.4.10" | = "1.5.0" | >= "1.5.2" & != "1.10.0"
  164. git-mirage < "3.0.0"
  165. git-unix >= "1.10.0" & != "2.1.0"
  166. gitlab-unix
  167. glicko2
  168. gmap >= "0.3.0"
  169. gobba
  170. gpt
  171. graphql
  172. graphql-async
  173. graphql-cohttp >= "0.13.0"
  174. graphql-lwt
  175. graphql_parser != "0.11.0"
  176. graphql_ppx >= "0.7.1"
  177. h1_parser
  178. h2
  179. hacl
  180. hacl-star >= "0.6.0"
  181. hacl_func
  182. hacl_x25519 >= "0.2.0"
  183. highlexer
  184. hkdf
  185. hockmd
  186. html_of_jsx
  187. http
  188. http-multipart-formdata < "2.0.0"
  189. httpaf >= "0.2.0"
  190. httpun
  191. httpun-ws
  192. hvsock
  193. icalendar >= "0.1.4"
  194. imagelib >= "20200929"
  195. index
  196. inferno >= "20220603"
  197. influxdb-async
  198. influxdb-lwt
  199. inquire < "0.2.0"
  200. interval-map
  201. iomux
  202. irmin < "0.8.0" | >= "0.9.6" & != "0.11.1" & < "1.0.0" | >= "2.0.0" & != "2.3.0"
  203. irmin-bench >= "2.7.0"
  204. irmin-chunk < "1.3.0" | >= "2.3.0"
  205. irmin-cli
  206. irmin-containers
  207. irmin-fs < "1.3.0" | >= "2.3.0"
  208. irmin-git < "2.0.0" | >= "2.3.0"
  209. irmin-http < "2.0.0"
  210. irmin-mem < "1.3.0"
  211. irmin-pack >= "2.4.0" & != "2.6.1"
  212. irmin-pack-tools
  213. irmin-test >= "2.2.0" & < "3.0.0"
  214. irmin-tezos
  215. irmin-tezos-utils
  216. irmin-unix >= "1.0.0" & < "1.3.3" | >= "2.4.0" & != "2.6.1"
  217. irmin-watcher
  218. jekyll-format
  219. jerboa
  220. jitsu
  221. jose
  222. json-data-encoding >= "0.9"
  223. json_decoder
  224. jsonxt
  225. junit_alcotest
  226. jwto
  227. ke >= "0.2"
  228. kkmarkdown
  229. lambda-runtime
  230. lambda_streams
  231. lambda_streams_async
  232. lambdapi >= "2.0.0"
  233. lambdoc >= "1.0-beta4"
  234. ledgerwallet-tezos >= "0.2.1" & < "0.4.0"
  235. letters
  236. lmdb >= "1.0"
  237. logical
  238. logtk >= "1.6"
  239. lp
  240. lp-glpk
  241. lp-glpk-js
  242. lp-gurobi
  243. lru
  244. lt-code
  245. luv
  246. mbr-format >= "1.0.0"
  247. mdx >= "1.6.0"
  248. mec
  249. mechaml >= "1.0.0"
  250. merge-queues >= "0.2.0"
  251. merge-ropes >= "0.2.0"
  252. metrics
  253. minicaml = "0.3.1" | >= "0.4"
  254. mirage >= "4.0.0~beta1"
  255. mirage-block-partition
  256. mirage-block-ramdisk >= "0.3"
  257. mirage-channel >= "4.0.0"
  258. mirage-channel-lwt
  259. mirage-crypto-ec
  260. mirage-flow >= "1.0.2" & < "1.2.0"
  261. mirage-flow-unix
  262. mirage-fs-mem
  263. mirage-fs-unix >= "1.2.0"
  264. mirage-kv >= "2.0.0"
  265. mirage-kv-mem
  266. mirage-kv-unix
  267. mirage-logs >= "0.3.0"
  268. mirage-nat
  269. mirage-net-unix >= "2.3.0"
  270. mirage-runtime >= "4.0.0~beta1" & < "4.5.0"
  271. mirage-tc
  272. mjson
  273. mmdb
  274. mnd
  275. monocypher
  276. mrmime >= "0.2.0"
  277. mrt-format
  278. msgpck >= "1.6"
  279. mssql >= "2.0.3"
  280. multibase
  281. multihash
  282. multihash-digestif
  283. multipart-form-data
  284. multipart_form
  285. multipart_form-eio
  286. multipart_form-lwt
  287. named-pipe
  288. nanoid
  289. nbd >= "4.0.3"
  290. nbd-tool
  291. nloge
  292. nocoiner
  293. non_empty_list
  294. OCADml >= "0.6.0"
  295. ocaml-r >= "0.5.0"
  296. ocaml-version >= "3.1.0"
  297. ocamlformat >= "0.13.0" & != "0.19.0~4.13preview" & < "0.25.1"
  298. ocamlformat-rpc < "removed"
  299. ocamline
  300. ocluster < "0.3.0"
  301. odoc >= "1.4.0" & < "2.1.0"
  302. ohex
  303. oidc
  304. opam-0install
  305. opam-file-format >= "2.1.1"
  306. opentelemetry >= "0.6"
  307. opentelemetry-client-cohttp-lwt >= "0.6"
  308. opentelemetry-client-ocurl >= "0.6"
  309. opentelemetry-cohttp-lwt >= "0.6"
  310. opentelemetry-lwt >= "0.6"
  311. opium >= "0.15.0"
  312. opium-graphql
  313. opium-testing
  314. opium_kernel
  315. orewa
  316. ortac-core
  317. osx-acl
  318. osx-attr
  319. osx-cf
  320. osx-fsevents
  321. osx-membership
  322. osx-mount
  323. osx-xattr
  324. otoggl
  325. owl >= "0.6.0" & != "0.9.0" & != "1.0.0"
  326. owl-base < "0.5.0"
  327. owl-ode >= "0.1.0" & != "0.2.0"
  328. owl-symbolic
  329. passmaker
  330. patch
  331. pbkdf
  332. pecu >= "0.2"
  333. pf-qubes
  334. pg_query >= "0.9.6"
  335. pgx >= "1.0"
  336. pgx_unix >= "1.0"
  337. pgx_value_core
  338. pgx_value_ptime < "2.2"
  339. phylogenetics
  340. piaf
  341. polyglot
  342. polynomial
  343. ppx_blob >= "0.3.0"
  344. ppx_deriving_cmdliner
  345. ppx_deriving_rpc
  346. ppx_deriving_yaml
  347. ppx_graphql >= "0.2.0"
  348. ppx_inline_alcotest
  349. ppx_parser
  350. ppx_protocol_conv >= "5.0.0"
  351. ppx_protocol_conv_json >= "5.0.0"
  352. ppx_protocol_conv_jsonm >= "5.0.0"
  353. ppx_protocol_conv_msgpack >= "5.0.0"
  354. ppx_protocol_conv_xml_light >= "5.0.0"
  355. ppx_protocol_conv_xmlm
  356. ppx_protocol_conv_yaml >= "5.0.0"
  357. ppx_repr < "0.4.0"
  358. ppx_subliner
  359. ppx_units
  360. ppx_yojson >= "1.1.0"
  361. pratter
  362. prc
  363. preface
  364. pretty_expressive
  365. prettym
  366. proc-smaps
  367. producer < "0.2.0"
  368. progress < "0.2.0"
  369. prom
  370. prometheus < "1.2"
  371. prometheus-app
  372. protocell
  373. protocol-9p >= "0.3" & < "0.11.0" | >= "0.11.2"
  374. protocol-9p-unix
  375. psq
  376. qcheck >= "0.18"
  377. qcheck-alcotest
  378. qcheck-core >= "0.18"
  379. quickjs
  380. radis
  381. randii
  382. reason-standard
  383. reparse >= "2.0.0" & < "3.0.0"
  384. reparse-unix < "2.1.0"
  385. resp
  386. resp-unix >= "0.10.0"
  387. rfc1951 < "1.0.0"
  388. routes < "2.0.0"
  389. rpc >= "7.1.0"
  390. rpclib >= "7.1.0"
  391. rpclib-async
  392. rpclib-lwt >= "7.1.0"
  393. rubytt
  394. SZXX >= "4.0.0"
  395. salsa20
  396. salsa20-core
  397. sanddb >= "0.2"
  398. scaml >= "1.5.0"
  399. scrypt-kdf
  400. secp256k1 >= "0.4.1"
  401. secp256k1-internal
  402. semver >= "0.2.1"
  403. sendmail
  404. sendmail-lwt
  405. sendmsg
  406. server-reason-react
  407. session-cookie
  408. session-cookie-async
  409. session-cookie-lwt
  410. sherlodoc
  411. slug
  412. sodium-fmt
  413. spin >= "0.6.0"
  414. squirrel
  415. ssh-agent
  416. ssl >= "0.6.0"
  417. stramon-lib
  418. styled-ppx
  419. syslog-rfc5424
  420. tcpip >= "2.4.2" & < "4.0.0" | >= "5.0.1" & < "7.0.0"
  421. tdigest < "2.1.0"
  422. terminal_size >= "0.1.1"
  423. terminus
  424. terminus-cohttp
  425. terminus-hlc
  426. terml
  427. textrazor
  428. tezos-base-test-helpers < "13.0"
  429. tezos-bls12-381-polynomial
  430. tezos-client-base < "12.0"
  431. tezos-crypto >= "8.0" & < "9.0"
  432. tezos-lmdb
  433. tezos-plompiler = "0.1.3"
  434. tezos-plonk = "0.1.3"
  435. tezos-signer-backends >= "8.0" & < "13.0"
  436. tezos-stdlib >= "8.0" & < "12.0"
  437. tezos-test-helpers < "12.0"
  438. tftp
  439. timedesc
  440. timere
  441. tls >= "0.12.0"
  442. toc
  443. topojson
  444. topojsone
  445. transept
  446. twostep
  447. type_eq
  448. type_id
  449. typebeat
  450. typeid >= "1.0.1"
  451. tyre >= "0.4"
  452. tyxml >= "4.0.0"
  453. tyxml-jsx
  454. tyxml-ppx >= "4.3.0"
  455. tyxml-syntax
  456. uecc
  457. ulid
  458. universal-portal
  459. unix-dirent
  460. unix-errno >= "0.3.0"
  461. unix-fcntl >= "0.3.0"
  462. unix-sys-resource
  463. unix-sys-stat
  464. unix-time
  465. unstrctrd
  466. user-agent-parser
  467. uspf
  468. uspf-lwt
  469. uspf-unix
  470. utop >= "2.13.0"
  471. validate
  472. validator
  473. vercel
  474. vpnkit
  475. wcwidth
  476. websocketaf
  477. x509 >= "0.7.0"
  478. xapi-rrd >= "1.8.2"
  479. xapi-stdext-date
  480. xapi-stdext-encodings
  481. xapi-stdext-std >= "4.16.0"
  482. yaml < "3.2.0"
  483. yaml-sexp
  484. yocaml
  485. yocaml_yaml
  486. yojson >= "1.6.0"
  487. yojson-five
  488. yuscii >= "0.3.0"
  489. zar
  490. zed >= "3.2.2"
  491. zlist < "0.4.0"

Conflicts

None

OCaml

Innovation. Community. Security.

GitHub Discord Twitter Peertube RSS
About
Changelog Releases Industrial Users Academic Users Why OCaml
Ecosystem
Packages Community Events OCaml Planet Jobs
Resources
Install OCaml Get Started Platform Tools Language Manual Standard Library API Books Exercises Papers OCaml Playground Logo
Policies
Carbon Footprint Governance Privacy Code of Conduct