OCaml Changelog


Read the latest releases and updates from the OCaml ecosystem.

About six months ago, opam-repo-ci (opam.ci.ocaml.org) suffered from a lack of system memory (issue 220) which caused it to be moved to the machine hosting ocaml-ci (ocaml.ci.dev).

Subsequently, that machine suffered from BTRFS volume corruption (issue 51). Therefore, we moved both services to a larger new server. The data was efficiently migrated using BTRFS tools: btrfs send | btrfs receive.

Since the move, we have seen issues with BTRFS metadata. Plus, we have suffered from a build-up of subvolumes, as reported by other users: Docker gradually exhausts disk space on BTRFS.

Unfortunately, both services went down on Friday evening (issue 85). Analysis showed over 500 BTRFS subvolumes, a shortage of metadata space, and insufficient space to perform a BTRFS rebalance.

Returning to the original configuration of splitting the ci.dev and OCaml.org services, they have been moved onto new and separate hardware. The underlying filesystem is now a RAID1-backed ext4, formatted with -i 8192 in order to ensure the availability of sufficient inodes. Docker uses Overlayfs. RSYNC was used to copy the databases and logs from the old server. This change should add resilience and has doubled the capacity for storing history logs.

Previously the infrastructure team had made FreeBSD available for opam-repo-ci. Now we can announce that the same support has been added to ocaml-ci, giving coverage for both OCaml 4.14 and the new OCaml 5.1 release. opam-repo-ci has also been upgraded to support OCaml 5.1. We aim to support both 4.14 as the Long Term Support release and the latest 5.* release.

Additionally an opam-health-check instance has been setup to provide continuous checking of opam repository packages against FreeBSD 13.2 x86_64 for both the 4.14 and 5.1 releases of OCaml. This will allow the community to check whether packages work on FreeBSD and provide fixes to opam-repository that will then get tested on FreeBSD. Closing the loop and giving the community the tools to support OCaml on FreeBSD effectively.

We hope the community finds the FreeBSD support useful.

The OCaml infrastructure team is going to move to Debian 12 as the main distribution from Debian 11. We will continue to provide Debian 11 and 10 images while they are supported, dropping Debian 10 when it reaches end of life in 2024-06-30. In addition to these changes we are deprecating Ubuntu 18.04, Alpine 3.16/17, OL7, OpenSuse 15.2 distributions as the have reached end of life. We strongly recommend updating to a newer version if you are still using them.

Please get in touch on https://github.com/ocaml/infrastructure/issues if you have questions or requests for additional support.

See full changelog
  • OCaml Debian images upgraded to Debian 12 (ocaml-dockerfile#172, @MisterDA)
  • Deprecation of Ubuntu 18.04, Alpine 3.16 and 3.17, OracleLinux 7, OpenSUSE 15.2 images (ocaml-dockerfile#176, @avsm)
  • Deprecate Ubuntu 18.04, Alpine 3.16/17, OL7, OpenSuse 15.2 for OCaml-CI (ocaml-ci#832, @tmcgilchrist)
  • Deprecate Ubuntu 18.04, Alpine 3.16/17, OL7, OpenSuse 15.2 for opam-repo-ci (opam-repo-ci#226, @tmcgilchrist)
  • Deprecate Ubuntu 18.04, Alpine 3.16/17, OL7, OpenSuse 15.2 for docker-base-images (docker-base-images#237, @tmcgilchrist)

The server toxis where Opam-Repo-CI and OCaml-CI were deployed suffered hardware difficulties yesterday, resulting in BTRFS filesystem corruption and memory issues. These issues are tracked on ocaml/infrastructure#51. Services were restored temporarily using a spare spinning disk, but we continued to see ECC memory issues.

All services have now been redeployed on new ARM64 hardware. We retained the databases for Prometheus, OCaml-CI and Opam-Repo-CI, but unfortunately, older job logs have been lost.

The external URLs for these services are unchanged.

New Ubuntu 23.04 and Fedora 38 distributions have been added to docker base image builder. Following their respective releases:

  • Ubuntu 23.04 was released 20 April 2023
  • Fedora 38 was released 18 April 2023 These images started building from Apr 25th 2023 and have been pushed to https://hub.docker.com/r/ocaml/opam with a full range of OCaml versions and variants. As always the status of the images can be viewed on images.ci.ocaml.org.

Alongside the Linux updates base images containing OCaml 5.1 have also been published for supported Linux platforms. Following the OCaml 5.1 Alpha release announcement on discuss.ocaml.org. Enjoy and please report any issues on ocurrent/docker-base-images/issues.

Due to maintenance operations on Tuesday 11th April between 7:30am and 11:00am (UTC+1), the build system will be running at approximately 50% capacity. You may experience some build delays in Opam Repo CI and OCaml CI.

Thank you for your patience and understanding during this time.

The intention is to retire the ocamllabs.io domain. Therefore any services using the domain will be redirected. From today, the Web UI for Opam Repo CI is available on both opam-ci.ci3.ocamllabs.io and opam.ci.ocaml.org with the service available at both opam-repo-ci.ci3.ocamllabs.io and opam-repo.ci.ocaml.org. In time, the ocamllabs.io sites will issue HTTP 301 permanent redirect messages.

Previously, opam.ci.ocaml.org targetted a web server which issued an HTTP 302 redirect to opam.ci3.ocamllabs.io. This redirection has been removed. opam.ci.ocaml.org points to the actual site.

As previously announced, "opam-repository-mingw" is no longer receiving updates.

We're actively working on getting the Windows compiler packages into ocaml/opam-repository. There are two issues which are taking (me) a little while to finish solving, but more on that further below.

In the gap - of hopefully only a month or so - for this being ready, there's is an issue that new releases are of course not available when opam-repository-mingw is being used with ocaml/setup-ocaml@v2 GitHub actions workflows. I'm hoping here to set out what's happening, and what steps you may need to take to keep your GitHub Actions Windows workflows running smoothly over the next few months.

What's happening right now?

We've updated setup-ocaml to use ocaml-opam/opam-repository-mingw instead of fdopen/opam-repository-mingw (see ocaml/setup-ocaml#651). This clone has been augmented with:

  • OCaml 4.14.1 packages, in the same style as the 4.14.0 forked packages (the "pre-compiled" package variants exist, but they're not pre-compiled)
  • Changes to the constraints for existing packages only

If you're using setup-ocaml in its default configuration, you should notice no change except that 4.14.x builds should now use 4.14.1 and the initial build will be a little slower as it builds from sources (GitHub Actions caching will then take over for subsequent runs).

For new releases of packages, it's necessary to add opam-repository to the repositories selections for the switches. It's important that opam-repository is at a lower priority than opam-repository-mingw for existing packages, so it's better to use these lines in your ocaml/setup-ocaml@v2 step than to issue opam repo add --rank=1000 later:

uses: ocaml/setup-ocaml@v2
  opam-repositories: |
    opam-repository-mingw: https://github.com/ocaml-opam/opam-repository-mingw.git#sunset
    default: https://github.com/ocaml/opam-repository.git

What do I do when things are broken?

There's an issue tracker on ocaml-opam/opam-repository-mingw, and this is a very good place to start.

If a version of a package isn't building, there are three possible remedies:

  • Previous versions of the package may have carried non-upstreamed patches in opam-repository-mingw. opam-repository's policy is not to carry such patches. In this case, the package actually doesn't work on Windows.
    • opam-repository should be updated to have os != "win32" added to the available field for the package
    • An issue on the package's upstream repo should be opened highlighting the need to upstream patches (or even a pull request with them!)
    • The patches in opam-repository-mingw make changes which may not necessarily be accepted/acceptable upstream in their current form, so the issue may be a better starting point than simply taking a patch and opening a pull request for it (for example, the utop package contains patches which may require further work and review)
  • The package relies on environment changes in "OCaml for Windows". For example, the Zarith package works in "OCaml for Windows" because the compiler packages unconditionally set the CC environment variable. This change is both not particularly desirable change to upstream (it is very confusing, for example, when working on the compiler itself) and also extremely difficult to upstream, so the fix here is instead to change the package's availability with (os != "win32" | os-distribution = "cygwinports") and constrain away OCaml 5 on Windows ("ocaml" {< "5.0" | os != "win32"})
  • Package constraints on existing packages need updating in ocaml-opam/opam-repository-mingw. For example, the release of ppxlib 0.29 required some existing packages to have upperbounds added.

What about OCaml 5.0.0?

OCaml 5.0.0 was released with support for the mingw-w64 port only, however, there's a quite major bug which wasn't caught by OCaml's testsuite, but is relatively easily triggered by opam packages. I've previously announced how to add OCaml 5 to a workflow. For the time being, the packages for OCaml 5 aren't automatically made available.

What's next?

The ultimate goal is to be using an upstream build of opam.exe with ocaml/opam-repository, just as on Unix. Once opam 2.2 is generally available (we're aiming for an alpha release at the end of March) and the compiler packages in opam-repository support the Windows builds, we will recommend stopping use of opam-repository-mingw completely. The default in setup-ocaml won't change straight away, since that risks breaking existing workflows.

With upstream compiler support, we'll be able to extend some of the existing bulk build support already being done for Linux to Windows and start to close the gap of patches in opam-repository-mingw.

Windows compiler packages

I mentioned earlier the problems with moving the compiler packages into opam-repository, and just for general interest this elaborates on them.

The first issue affects the use of the Visual Studio port ("MSVC") and is a consequence of the somewhat strange way that the C compiler is added to the environment when using the Visual Studio C compiler. "OCaml for Windows" (as well as Diskuv) use a wrapper command (it's ocaml-env in "OCaml for Windows" and with-dkml in Diskuv). Those commands are Windows-specific, which is an issue for upstream opam. There's an alternate way which sets the environment variables in a more opam-like way. Doing it that way, though, requires an improvement to opam's environment handling which is in opam 2.2, otherwise there's an easy risk of "blowing" the environment.

The second issue is selecting the C compiler. On Unix, this is easy with ocaml-base-compiler because there is only one "system" C compiler. Windows has two ports of OCaml, and the configuration requires it to be explicitly selected. That requires input from the user on switch creation for a Windows switch.

"OCaml for Windows" solves this by packaging the Windows compilers with the variant name appended, just as opam-repository used to, so ocaml-variants.4.14.1+mingw64 selects the the mingw-w64 port and ocaml-variants.4.14.1+msvc64 selects the MSVC64 port. The problem, as we already had in opam-repository, is that this adds 4 packages for each release of OCaml in ocaml-variants, and leads to a combinatorial explosion when we start considering flambda and other relevant compiler options.

opam-repository switched to using the ocaml-option- packages to solve the combinatorial explosion which was already present in opam-repository. The demonstration repo for OCaml 5 on Windows is already using an adapted version of this so that ocaml-option-mingw selects the mingw-w64 port (by default 64-bit, with ocaml-option-32bit then selecting the 32-bit port).

This work is all in progress and being tested alongside changes in opam 2.2 to support the depext experience on Windows. The only reason that's not being upstreamed piecemeal is that changes to the compiler packages in opam-repository trigger switch rebuilds all over the world, so we don't want to that until we're sure that the packages are correct. The intention is to do this around the time of the alpha release of opam 2.2, once the work in opam itself has settled down.

Thanks for getting to the end, and happy Windows CI testing!

Discuss this post further on the forums.

Following @avsm post on discuss.ocaml.org, we are pleased to announce that DNS names have now been switched over.

We are moving the opam.ocaml.org 2 servers between hosting providers, and wanted to give everyone clear notice that this happening. Over the next 24-48 hours, if you notice any unexpected changes in the way your opam archives work (for example, in your CI or packaging systems), then please do let us know immediately, either here or in ocaml/infrastructure#19 2.

The reason for the move is to take advantage of Scaleway’s generous sponsorship of ocaml.org, and to use their energy efficient renewable infrastructure 4 for our machines.

This also marks a move to building the opam website via the ocurrent 1 infrastructure, which leads to faster and more reliable updates to the hosted package archives (see here for the service graph and build logs 4). There are also now multiple machines behind the opam.ocaml.org 2 DNS (via round-robin DNS), and this makes it easier for us to publish the archives to a global CDN in the future.

But in the very short term, if something explodes unexpectedly, please do let us know.

watch.ocaml.org has been updated to run as a Docker service stack rather than via docker-compose. This change allowed an OCurrent pipeline to monitor the Docker repository and update the image via docker service update when a new version is available.

We have several other services updated via OCurrent: deploy.ci.ocaml.org

In OCurrent, we can create a schedule node that triggers every seven days and invokes a docker pull, yielding the current image SHA. If this has changed, run docker service update with the new image.

  let peertube =
     let weekly = Current_cache.Schedule.v ~valid_for:(Duration.of_day 7) () in
     let image = Cluster.Watch_docker.pull ~schedule:weekly "chocobozzz/peertube:production-bullseye" in
     Cluster.Watch_docker.service ~name:"infra_peertube" ~image ()

The deployment is achieved through an Ansible Playbook. Further details are available here.

The second part of the update was to improve the visibility of the backups for watch.ocaml.org. As noted previously, these use Tarsnap running monthly via CRON.

For this, a new plugin was added to OCurrent called ocurrent_ssh. This plugin allows arbitrary SSH commands to be executed as part of an OCurrent pipeline.

Again using a schedule node, the Current_ssh.run node will be triggered on a 30-day schedule, and the logs for each run will be available on deploy.ci.ocaml.org.

  let monthly = Current_cache.Schedule.v ~valid_for:(Duration.of_day 30) () in
   let tarsnap = Current_ssh.run ~schedule:monthly "watch.ocaml.org" ~key:"tarsnap" (Current.return ["./tarsnap-backup.sh"])

Some of our ocaml.org services such as <watch.ocaml.org> involve storing user uploaded content. We need a way to make sure these are backed up in case of information loss, and to date this has been adhoc (involving rsyncing to another machine).

We now have a Tarsnap account as tarsnap@ocaml.org, and it is first being used to store backups of the videos uploaded to the <watch.ocaml.org> service. We'll expand its use to other infrastructures that also have precious data.

Other suggestions for backup services are welcome. In general, we're looking for solutions that do not involve a lot of key management, and a reasonable amount of redundancy (but backing up across 2-3 other machines in different datacentres is probably sufficient).

If you want to contribute to a new release announcement, check out the Contributing Guide on GitHub.