package bisect_ppx
Install
Dune Dependency
Authors
Maintainers
Sources
sha256=6552116b8bce000dad7623c38fc8695d7ec6ec0b8ed67abee707031fcdb2cad0
md5=9551a49de6b80aa7caf20ec693387c86
Description
Bisect_ppx helps you test thoroughly. It is a small preprocessor that inserts instrumentation at places in your code, such as if-then-else and match expressions. After you run tests, Bisect_ppx gives a nice HTML report showing which places were visited and which were missed.
Usage is simple - add package bisect_ppx when building tests, run your tests, then run the Bisect_ppx report tool on the generated visitation files.
Published: 11 May 2020
README
Bisect_ppx
Bisect_ppx is a code coverage tool for OCaml and Reason. It helps you test thoroughly by showing what's not tested.
You can browse the report seen above online here. The details of how it is generated are in the worked example.
Table of contents
Usage
Dune
Refer to aantron/bisect-starter-dune, which produces this report.
Depend on Bisect_ppx in your
opam
file:depends: [ "bisect_ppx" {dev & >= "2.0.0"} ]
Mark the code under test for preprocessing by
bisect_ppx
in yourdune
file:(* -*- tuareg -*- *) let preprocess = match Sys.getenv "BISECT_ENABLE" with | "yes" -> "(preprocess (pps bisect_ppx))" | _ -> "" | exception Not_found -> "" let () = Jbuild_plugin.V1.send @@ {| (library (public_name my_lib) |} ^ preprocess ^ {|) |}
This uses Dune's OCaml syntax to completely take
bisect_ppx
out as a dependency, except when the environment variableBISECT_ENABLE
is set toyes
. This is so that you can release your project without it depending onbisect_ppx
for non-dev
builds.After ocaml/dune#57, Dune will have a lighter-weight built-in syntax for conditional preprocessing.
For now, the OCaml syntax can be understood as prepending a few lines of OCaml code to a regular
dune
file, and then replacing thepreprocess
stanza with|} ^ preprocess ^ {|
. See here for a neat summary of the patch.Build and run your test binary. In addition to testing your code, when exiting, it will write one or more files with names like
bisect0123456789.coverage
:BISECT_ENABLE=yes dune runtest --force
The
--force
flag forces all your tests to run, which is needed for an accurate coverage report.To run tests without coverage, do
dune runtest
Generate the coverage report in
_coverage/index.html
:bisect-ppx-report html
You can also generate a short summary in the terminal:
bisect-ppx-report summary
esy
Refer to aantron/bisect-starter-esy, which produces this report.
The instructions are the same as for regular Dune usage, but...
Depend on Bisect_ppx in
package.json
, instead of in anopam
file:"devDependencies": { "@opam/bisect_ppx": "^2.0.0", }
Use the
esy
command for the build and for running binaries:esy install BISECT_ENABLE=yes esy dune runtest --force esy dune exec bisect-ppx-report -- html
BuckleScript
Refer to aantron/bisect-starter-bsb, which produces this report.
Depend on Bisect_ppx in
package.json
, and install it:"devDependencies": { "bisect_ppx": "^2.0.0" }, "dependencies": { "bs-platform": "*" }
npm install
If you are using Yarn, you need to run an extra command because of yarnpkg/pkg#3421:
yarn add bisect_ppx yarn --check-files
If pre-built binaries aren't available for your system, the build will automatically fall back to building Bisect_ppx from source using esy, which will take a few minutes the first time. If this happens, you may need to install esy, if it is not already installed:
npm install -g esy npm install
Add Bisect_ppx to your
bsconfig.json
:"bs-dependencies": [ "bisect_ppx" ], "ppx-flags": [ "bisect_ppx/ppx" ]
If you are using Jest, add this to your
package.json
:"setupFilesAfterEnv": ["bisect_ppx/lib/js/src/runtime/bucklescript/jest.js"]
Or, if you have enabled the
package-specs.in-source
flag inbsconfig.json
,"setupFilesAfterEnv": ["bisect_ppx/src/runtime/bucklescript/jest.js"]
If the tests will be running in the browser, at the end of testing, call
Bisect.Runtime.get_coverage_data();
This returns binary coverage data in a
string option
, which you should upload or otherwise get out of the browser, and write into a.coverage
file.Build in development with
BISECT_ENABLE=yes
, run tests, and generate the coverage report in_coverage/index.html
:BISECT_ENABLE=yes npm run build npm run test npx bisect-ppx-report.exe html
To exclude your test files from the report, change your PPX flags like so:
"ppx-flags": [ ["bisect_ppx/ppx", "--exclude-files", ".*test\\.re"] ]
The last argument is a regular expression in the syntax of OCaml's
Str
module. Note that backslashes need to be escaped both inside the regular expression, and again because they are inside a JSON string.Multiple
--exclude-files
option can be specified if you want to provide multiple patterns.If your project uses both BuckleScript and native Dune, native Dune will start picking up OCaml files that are part of the BuckleScript
bisect_ppx
package. To prevent this, add adune
with the following contents to the root of your project:(data_only_dirs node_modules)
Js_of_ocaml
Refer to aantron/bisect-starter-jsoo, which produces this report.
Follow the Dune instructions above, except that the final test script must be linked with
bisect_ppx.runtime
(but not instrumented):(executable (name my_tester) (libraries bisect_ppx.runtime))
If the tests will run on Node, call this function at the end of testing to write
bisect0123456789.coverage
:Bisect.Runtime.write_coverage_data ()
If the tests will run in the browser, call
Bisect.Runtime.get_coverage_data ()
to get binary coverage data in a string option. Upload this string or otherwise extract it from the browser to create an
.coverage
file.Build the usual Js_of_ocaml target, including the instrumented code under test, then run the reporter to generate the coverage report in
_coverage/index.html
:BISECT_ENABLE=yes dune build my_tester.bc.js bisect-ppx-report html
Ocamlfind, Ocamlbuild, and OASIS
Ocamlbuild and OASIS instructions can be found at aantron/bisect_ppx-ocamlbuild.
With Ocamlfind, you must have your build script issue the right commands, to instrument the code under test, but not the tester:
ocamlfind opt -package bisect_ppx -c src/source.ml ocamlfind opt -c test/test.ml ocamlfind opt -linkpkg -package bisect_ppx src/source.cmx test/test.cmx
Running the tester will then produce
bisect0123456789.coverage
files, which you can process withbisect-ppx-report
.
Sending to Coveralls
bisect-ppx-report
can send reports to Coveralls and Codecov directly from Travis, CircleCI, and GitHub Actions. To do this, run
bisect-ppx-report send-to Coveralls
or
bisect-ppx-report send-to Codecov
When sending specifically from GitHub Actions to Coveralls, use
- run: bisect-ppx-report send-to Coveralls
env:
COVERALLS_REPO_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PULL_REQUEST_NUMBER: ${{ github.event.number }}
Put these commands in your CI script in the same place you would run bisect-ppx-report html
locally. See bisect-ci-integration-megatest for example CI scripts and current status of these integrations.
If you'd like Bisect_ppx to support other CI and/or coverage services, please open an issue or send a pull request!
As a workaround for missing CI/coverage integrations, and for development, bisect-ppx-report
can also generate a JSON report in Coveralls format, which can be uploaded to a service of your choice using a separate command. For example, to send manually from Travis to Coveralls:
bisect-ppx-report \
coveralls coverage.json \
--service-name travis-ci \
--service-job-id $TRAVIS_JOB_ID
curl -L -F json_file=@./coverage.json https://coveralls.io/api/v1/jobs
For other CI services, replace --service-name
and --service-job-id
as follows:
CI service | --service-name |
--service-job-id |
---|---|---|
Travis | travis-ci |
$TRAVIS_JOB_ID |
CircleCI | circleci |
$CIRCLE_BUILD_NUM |
Semaphore | semaphore |
$REVISION |
Jenkins | jenkins |
$BUILD_ID |
Codeship | codeship |
$CI_BUILD_NUMBER |
GitHub Actions | github |
$GITHUB_RUN_NUMBER |
Note that Coveralls-style reports are less precise than the HTML reports generated by Bisect_ppx, because Coveralls considers entire lines as visited or not visited. Bisect_ppx instead considers individual expressions. There can be many expressions on a single line, and the HTML report separately considers each expression as visited or not visited.
Controlling coverage with [@coverage off]
You can tag expressions with [@coverage off]
, and neither they, nor their subexpressions, will be instrumented by Bisect_ppx.
Likewise, you can tag module-level let
-declarations with [@@coverage off]
, and they won't be instrumented.
You can also turn off instrumentation for blocks of declarations inside a module with [@@@coverage off]
and [@@@coverage on]
.
Finally, you can exclude an entire file by putting [@@@coverage exclude_file]
into its top-level module. However, whenever possible, it is recommended to exclude files by not preprocessing with Bisect_ppx to begin with.
Real-world example examined
Refer to:
aantron/markup.ml, which produces this local report, and this report on Coveralls.
The Dune instructions and Coveralls instructions above.
The details:
The project depeds on package
bisect_ppx
, so that Bisect_ppx is installed byopam pin --dev-repo markup
andopam install .
There are three libraries in
src/
, each set to have its sources preprocessed by Bisect_ppx:Because of the
--conditional
flag, preprocessing is enabled only whenBISECT_ENABLE=yes
is set in the environment, so it is off by default.A coverage build is triggered by running
make coverage
. This target...Depends on
make clean
. This is a workaround until ocaml/dune#57 is solved. The problem is that doing a coverage build, after normal builds, should force all sources to be recompiled, so that they can be instrumented by the Bisect_ppx preprocessor. However, Dune doesn't know about this — it doesn't know that the behavior of the preprocessor depends on theBISECT_ENABLE
environment variable.Indeed, the preprocessor shouldn't read this environment variable. The preprocessor should just be turned off by Dune when not building for coverage. However, Dune does not currently have the ability to conditionally turn off a preprocessor.
In any case, to deal with this problem, the project always does a clean build when building for coverage.
Does a fresh build with
BISECT_ENABLE=yes
, causing the sources of the three libraries mentioned above to be instrumented.Runs the test suite.
bisect*.coverage
files with coverage data are produced as a side effect.Runs
bisect-ppx-report
to generate both the typical HTML report in_coverage/index.html
, and also a textual summary in the terminal for very fast iteration.
make coverage
is also used in Travis to submit coverage reports to Coveralls. At the end ofmake coverage
, thebisect*.coverage
files are still present, so.travis.yml
runsbisect-ppx-report
again to generate the Coveralls report. This follows the Coveralls instructions exactly.Coveralls can be configured to leave comments about changes in coverage. It is usually configured to at least add an additional check to branches and PRs — see the "3 checks passed" in the hidden Details of the linked PR.
During release,
(preprocess (pps bisect_ppx))
is removed from all libraries that are being released. This is typically in a one-commit release branch off master, which is what ends up being tagged.This won't be necessary after ocaml/dune#57 is addressed.
Other topics
See advanced usage for:
Exhaustiveness checking.
Excluding generated files from coverage.
Environment variables.
Bisect_ppx users
A small sample of projects using Bisect_ppx:
Core tools
Libraries
Applications
Contributing
Bug reports and pull requests are warmly welcome. Bisect_ppx is developed on GitHub, so please open an issue.
Bisect_ppx is developed mainly using opam. To get the latest development version, run
opam source --dev-repo --pin bisect_ppx
You will now have a bisect_ppx
subdirectory to work in. Try these Makefile
targets:
make test
for unit tests.make usage
for build system integration tests, except BuckleScript.make -C test/bucklescript full-test
for BuckleScript. This requires NPM and esy.
Dependencies (6)
-
ppx_tools_versioned
>= "5.4.0"
-
ocaml-migrate-parsetree
>= "1.7.0" & < "2.0.0"
-
ocaml
>= "4.02.0"
- dune
-
cmdliner
>= "1.0.0"
- base-unix
Used by (55)
- ambient-context
- ambient-context-eio
- ambient-context-lwt
- amf
-
arp
>= "1.0.0" & < "2.3.1"
-
azure-cosmos-db
>= "0.2.3"
-
base58
>= "0.1.2"
-
bastet
>= "1.2.0"
-
bio_io
< "0.5.1"
-
bisect_ppx-ocamlbuild
>= "1.0.1"
- checked_oint
- cll
-
cuid
>= "0.2"
- daypack-lib
- easy_xlsx
- exit
-
GT
>= "0.5.2"
- gobba
-
hc
< "0.2"
- jose
-
lambdasoup
>= "0.6.4" & < "0.7.3"
- little_logger
-
lwt
>= "4.2.0" & < "5.4.0"
-
markup
>= "0.8.1" & < "1.0.0-1"
- mazeppa
-
memo
< "0.2"
-
minicaml
= "0.3.1"
-
mirage-block-ccm
>= "1.1.0"
- mirage-btrees
- mnd
- mssql
- obeam
-
ocaml-protoc-plugin
>= "5.0.0"
-
ocamlformat
= "0.11.0"
-
odoc
>= "1.4.0" & < "2.0.0"
- omg
- opazl
- open_packaging
- partition_map
- pf-qubes
- pgx
-
ppx_make
>= "0.3.4"
-
ppx_subliner
>= "0.2.0"
-
pyml_bindgen
< "0.3.0"
- reed-solomon-erasure
-
rfc6287
>= "1.0.4"
-
routes
>= "0.7.2" & < "1.0.0"
- sentry
-
shared-block-ring
>= "2.3.0" & < "3.0.0"
- so
- spreadsheetml
- tezos-bls12-381-polynomial
-
tezos-plompiler
= "0.1.3"
-
tezos-plonk
= "0.1.3"
-
validate
>= "1.0.0"
Conflicts
None