Jekyll2026-02-02T06:43:11+00:00/feed.xmlOCaml InfrastructureWIki to hold the information about the machine resources available to OCaml.orgCurrent Bench Maintenance2024-02-19T00:00:00+00:002024-02-19T00:00:00+00:00/2024/02/19/current-bench-maintenanceDue to storage related maintenance operations on Monday 19th February ‘24 between 6:30am and 9:30am (UTC+1), the current-bench benchmarking service will be unavailable.

Thank you for your patience and understanding during this time.

]]>Electrial Work in the Datacentre2024-01-14T00:00:00+00:002024-01-14T00:00:00+00:00/2024/01/14/electrical-workThere is an electrical shutdown for essential maintenance this weekend, which will affect large parts of the cluster. All Power9, ARM64, RISC-V, Windows and FreeBSD workers and docs.ci.ocaml.org, docs-data.ocaml.org and the benchmarking servers will be unavailable. x86_64 workers will be substantially reduced. Service is expected to resume late on Sunday evening.

]]>Relocating opam.ci.ocaml.org and ocaml.ci.dev2023-12-04T00:00:00+00:002023-12-04T00:00:00+00:00/2023/12/04/services-movedAbout six months ago, opam-repo-ci (opam.ci.ocaml.org) suffered from a lack of system memory (issue 220) which caused it to be moved to the machine hosting ocaml-ci (ocaml.ci.dev).

Subsequently, that machine suffered from BTRFS volume corruption (issue 51). Therefore, we moved both services to a larger new server. The data was efficiently migrated using BTRFS tools: btrfs send | btrfs receive.

Since the move, we have seen issues with BTRFS metadata. Plus, we have suffered from a build-up of subvolumes, as reported by other users: Docker gradually exhausts disk space on BTRFS.

Unfortunately, both services went down on Friday evening (issue 85). Analysis showed over 500 BTRFS subvolumes, a shortage of metadata space, and insufficient space to perform a BTRFS rebalance.

Returning to the original configuration of splitting the ci.dev and OCaml.org services, they have been moved onto new and separate hardware. The underlying filesystem is now a RAID1-backed ext4, formatted with -i 8192 in order to ensure the availability of sufficient inodes. Docker uses Overlayfs. RSYNC was used to copy the databases and logs from the old server. This change should add resilience and has doubled the capacity for storing history logs.

]]>macOS Sonoma Update2023-11-09T00:00:00+00:002023-11-09T00:00:00+00:00/2023/11/09/macos-sonomaOBuilder workers for both Intel and Apple Silicon have been updated from macOS Ventura to macOS Sonoma 14.1.1.

From today, ocaml-ci and opam-repo-ci will test against Sonoma rather than Ventura.

]]>Current Bench Maintenance2023-11-06T00:00:00+00:002023-11-06T00:00:00+00:00/2023/11/06/current-bench-maintenanceDue to storage related maintenance operations on Wednesday 8th November ‘23 between 12:00pm and 3:00pm (UTC+1), the current-bench benchmarking service will be unavailable.

Thank you for your patience and understanding during this time.

]]>FreeBSD is Available in More Places2023-09-21T00:00:00+00:002023-09-21T00:00:00+00:00/2023/09/21/more-freebsd-newsPreviously the infrastructure team had made FreeBSD available for opam-repo-ci. Now we can announce that the same support has been added to ocaml-ci, giving coverage for both OCaml 4.14 and the new OCaml 5.1 release. opam-repo-ci has also been upgraded to support OCaml 5.1. We aim to support both 4.14 as the Long Term Support release and the latest 5.* release.

Additionally an opam-health-check instance has been setup to provide continuous checking of opam repository packages against FreeBSD 13.2 x86_64 for both the 4.14 and 5.1 releases of OCaml. This will allow the community to check whether packages work on FreeBSD and provide fixes to opam-repository that will then get tested on FreeBSD. Closing the loop and giving the community the tools to support OCaml on FreeBSD effectively.

We hope the community finds the FreeBSD support useful.

]]>FreeBSD is now tested in opam-repo-ci2023-08-08T00:00:00+00:002023-08-08T00:00:00+00:00/2023/08/08/freebsd-testingOBuilder now supports FreeBSD which has allowed opam-repo-ci to be extended to test against it. Currently, we are testing Opam 2.1.5 with OCaml 4.14.1 and OCaml 5.0 with FreeBSD 13.2 (AMD64).

]]>Upgrading to Debian 12 for ocaml docker images2023-06-28T00:00:00+00:002023-06-28T00:00:00+00:00/2023/06/28/upgrading-linux-distrosThe OCaml infrastructure team is going to move to Debian 12 as the main distribution from Debian 11. We will continue to provide Debian 11 and 10 images while they are supported, dropping Debian 10 when it reaches end of life in 2024-06-30. In addition to these changes we are deprecating Ubuntu 18.04, Alpine 3.16/17, OL7, OpenSuse 15.2 distributions as the have reached end of life. We strongly recommend updating to a newer version if you are still using them.

Please get in touch on https://github.com/ocaml/infrastructure/issues if you have questions or requests for additional support.

]]>Relocation of opam.ci.ocaml.org and ocaml.ci.dev2023-06-15T00:00:00+00:002023-06-15T00:00:00+00:00/2023/06/15/opam-repo-ci-ocaml-ciThe server toxis where Opam-Repo-CI and OCaml-CI were deployed suffered hardware difficulties yesterday, resulting in BTRFS filesystem corruption and memory issues. These issues are tracked on ocaml/infrastructure#51. Services were restored temporarily using a spare spinning disk, but we continued to see ECC memory issues.

All services have now been redeployed on new ARM64 hardware. We retained the databases for Prometheus, OCaml-CI and Opam-Repo-CI, but unfortunately, older job logs have been lost.

The external URLs for these services are unchanged.

]]>Grafana Dashboard Changes2023-06-09T00:00:00+00:002023-06-09T00:00:00+00:00/2023/06/09/grafana-changesThe Grafana dashboards at status.ci3.ocamllabs.io and status.ci.ocamllabs.io have been merged into a single dashboard at status.ci.ocamllabs.io. HTTP redirects are in place for the old addresses.

]]>