<?xml version="1.0" encoding="utf-8"?><feed xmlns="http://www.w3.org/2005/Atom" ><generator uri="https://jekyllrb.com/" version="3.10.0">Jekyll</generator><link href="/feed.xml" rel="self" type="application/atom+xml" /><link href="/" rel="alternate" type="text/html" /><updated>2026-06-08T13:40:23+00:00</updated><id>/feed.xml</id><title type="html">OCaml Infrastructure</title><subtitle>WIki to hold the information about the machine resources available to OCaml.org</subtitle><entry><title type="html">Cluster workers upgraded to Ubuntu 26.04</title><link href="/2026/06/08/cluster-workers-ubuntu-26-04-upgrade.html" rel="alternate" type="text/html" title="Cluster workers upgraded to Ubuntu 26.04" /><published>2026-06-08T00:00:00+00:00</published><updated>2026-06-08T00:00:00+00:00</updated><id>/2026/06/08/cluster-workers-ubuntu-26-04-upgrade</id><content type="html" xml:base="/2026/06/08/cluster-workers-ubuntu-26-04-upgrade.html"><![CDATA[<p>All Linux cluster workers upgraded from Ubuntu 24.04 to 26.04 (kernel 6.8 → 7.0):</p>

<ul>
  <li><code class="language-plaintext highlighter-rouge">linux-x86_64</code>: asteria, bremusa, clete, doris, eumache, laodoke, odawa, phoebe, toxis</li>
  <li><code class="language-plaintext highlighter-rouge">linux-arm64</code>: ainia, kydoime, molpadia, okypous, ocaml-1, ocaml-2</li>
  <li><code class="language-plaintext highlighter-rouge">linux-ppc64</code>: orithia, scyleia</li>
  <li><code class="language-plaintext highlighter-rouge">linux-s390x</code>: s390x-worker-01</li>
</ul>

<p><code class="language-plaintext highlighter-rouge">riscv-bm-01</code>–<code class="language-plaintext highlighter-rouge">04</code> remain on Ubuntu 24.04; the hardware does not support 26.04.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[All Linux cluster workers upgraded from Ubuntu 24.04 to 26.04 (kernel 6.8 → 7.0):]]></summary></entry><entry><title type="html">Fermat upgraded to Ubuntu 24.04</title><link href="/2026/05/21/fermat-ubuntu-24-04-upgrade.html" rel="alternate" type="text/html" title="Fermat upgraded to Ubuntu 24.04" /><published>2026-05-21T00:00:00+00:00</published><updated>2026-05-21T00:00:00+00:00</updated><id>/2026/05/21/fermat-ubuntu-24-04-upgrade</id><content type="html" xml:base="/2026/05/21/fermat-ubuntu-24-04-upgrade.html"><![CDATA[<p>The benchmarking server <code class="language-plaintext highlighter-rouge">fermat.caelum.ci.dev</code> has been upgraded from Ubuntu
22.04 to 24.04, taking the kernel from 5.15 to 6.8. The trigger was a request
from Thomas, who was trying to benchmark the effect of some <code class="language-plaintext highlighter-rouge">io_uring</code> flags
that are not available on 5.15. The newer kernel covers every <code class="language-plaintext highlighter-rouge">io_uring</code>
addition made since 5.15, so those benchmarks can now proceed.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[The benchmarking server fermat.caelum.ci.dev has been upgraded from Ubuntu 22.04 to 24.04, taking the kernel from 5.15 to 6.8. The trigger was a request from Thomas, who was trying to benchmark the effect of some io_uring flags that are not available on 5.15. The newer kernel covers every io_uring addition made since 5.15, so those benchmarks can now proceed.]]></summary></entry><entry><title type="html">Revert Posthog Analytics on ocaml.org</title><link href="/2026/04/03/revert-posthog-analytics.html" rel="alternate" type="text/html" title="Revert Posthog Analytics on ocaml.org" /><published>2026-04-03T00:00:00+00:00</published><updated>2026-04-03T00:00:00+00:00</updated><id>/2026/04/03/revert-posthog-analytics</id><content type="html" xml:base="/2026/04/03/revert-posthog-analytics.html"><![CDATA[<p>The Posthog analytics tracking added in PR <a href="https://github.com/ocaml/ocaml.org/pull/3101">#3101</a>
has been reverted in PR <a href="https://github.com/ocaml/ocaml.org/pull/3594">#3594</a>.
The site returns to using Plausible as the sole analytics provider. Plausible
continued to collect metrics throughout the period that Posthog was active, so
there is a continuous set of statistics available with no gaps in coverage.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[The Posthog analytics tracking added in PR #3101 has been reverted in PR #3594. The site returns to using Plausible as the sole analytics provider. Plausible continued to collect metrics throughout the period that Posthog was active, so there is a continuous set of statistics available with no gaps in coverage.]]></summary></entry><entry><title type="html">OCaml CI services migration from Scaleway to Cambridge complete</title><link href="/2026/04/02/scaleway-to-cambridge-migration.html" rel="alternate" type="text/html" title="OCaml CI services migration from Scaleway to Cambridge complete" /><published>2026-04-02T00:00:00+00:00</published><updated>2026-04-02T00:00:00+00:00</updated><id>/2026/04/02/scaleway-to-cambridge-migration</id><content type="html" xml:base="/2026/04/02/scaleway-to-cambridge-migration.html"><![CDATA[<p>Several OCaml community and CI services have been migrated from Scaleway in Paris to the <a href="https://www.cl.cam.ac.uk">Cambridge Computer Laboratory</a>, as our Scaleway sponsorship credits have now expired. The affected services include:</p>

<ul>
  <li><a href="https://watch.ocaml.org">watch.ocaml.org</a> – PeerTube video hosting (<a href="/2026/03/27/watch-ocaml-org-migration.html">details</a>)</li>
  <li><a href="https://images.ci.ocaml.org">images.ci.ocaml.org</a> – Docker base image builder</li>
  <li><a href="https://scheduler.ci.dev">scheduler.ci.dev</a> – OCluster job scheduler</li>
  <li><a href="https://deploy.ci.dev">deploy.ci.dev</a> – OCaml CI deployment orchestration</li>
  <li><a href="https://ci.mirageos.org">ci.mirageos.org</a> and <a href="https://deploy.mirageos.org">deploy.mirageos.org</a> – MirageOS CI and deployment</li>
</ul>

<p>Multiple Scaleway VMs and baremetal servers have been consolidated onto fewer
machines in Cambridge.  Full technical details of the migration are available
in <a href="https://www.tunbury.org/2026/04/01/from-scaleway-to-cambridge/">Mark Elvers’ blog post</a>.</p>]]></content><author><name>Mark Elvers and Anil Madhavapeddy</name></author><summary type="html"><![CDATA[Several OCaml community and CI services have been migrated from Scaleway in Paris to the Cambridge Computer Laboratory, as our Scaleway sponsorship credits have now expired. The affected services include:]]></summary></entry><entry><title type="html">Migrating watch.ocaml.org from Scaleway to Cambridge</title><link href="/2026/03/27/watch-ocaml-org-migration.html" rel="alternate" type="text/html" title="Migrating watch.ocaml.org from Scaleway to Cambridge" /><published>2026-03-27T00:00:00+00:00</published><updated>2026-03-27T00:00:00+00:00</updated><id>/2026/03/27/watch-ocaml-org-migration</id><content type="html" xml:base="/2026/03/27/watch-ocaml-org-migration.html"><![CDATA[<p><a href="https://watch.ocaml.org">watch.ocaml.org</a> has been migrated from its Scaleway-hosted VM to a virtual machine at the <a href="https://www.cl.cam.ac.uk">Cambridge Computer Laboratory</a>. The primary motivation for this move is to reduce hosting costs, as our Scaleway sponsorship credits have now expired.</p>

<p>The new VM is hosted using <a href="https://xen-orchestra.com">Xen Orchestra</a>, which is itself a management toolstack written in OCaml via <a href="https://github.com/xapi-project/xen-api">XAPI</a> which is nice for a community OCaml video service!</p>

<p>The PeerTube instance and its data have been migrated across, and the service is now running as before with no user visible changes (we hope). The Ansible playbook and OCurrent-based deployment pipeline documented <a href="/2023/02/27/watch-ocaml-org.html">previously</a> continue to manage updates to the service.</p>

<p>This was part of a <a href="/2026/04/02/scaleway-to-cambridge-migration.html">broader migration</a> of OCaml CI services from Scaleway to Cambridge.</p>]]></content><author><name>Mark Elvers and Anil Madhavapeddy</name></author><summary type="html"><![CDATA[watch.ocaml.org has been migrated from its Scaleway-hosted VM to a virtual machine at the Cambridge Computer Laboratory. The primary motivation for this move is to reduce hosting costs, as our Scaleway sponsorship credits have now expired.]]></summary></entry><entry><title type="html">Current Bench Maintenance</title><link href="/2024/02/19/current-bench-maintenance.html" rel="alternate" type="text/html" title="Current Bench Maintenance" /><published>2024-02-19T00:00:00+00:00</published><updated>2024-02-19T00:00:00+00:00</updated><id>/2024/02/19/current-bench-maintenance</id><content type="html" xml:base="/2024/02/19/current-bench-maintenance.html"><![CDATA[<p>Due to storage related maintenance operations on Monday 19th February ‘24
between 6:30am and 9:30am (UTC+1), the current-bench benchmarking service will
be unavailable.</p>

<p>Thank you for your patience and understanding during this time.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[Due to storage related maintenance operations on Monday 19th February ‘24 between 6:30am and 9:30am (UTC+1), the current-bench benchmarking service will be unavailable.]]></summary></entry><entry><title type="html">Electrial Work in the Datacentre</title><link href="/2024/01/14/electrical-work.html" rel="alternate" type="text/html" title="Electrial Work in the Datacentre" /><published>2024-01-14T00:00:00+00:00</published><updated>2024-01-14T00:00:00+00:00</updated><id>/2024/01/14/electrical-work</id><content type="html" xml:base="/2024/01/14/electrical-work.html"><![CDATA[<p>There is an electrical shutdown for essential maintenance this weekend, which will affect large parts of the cluster.  All Power9, ARM64, RISC-V, Windows and FreeBSD workers and docs.ci.ocaml.org, docs-data.ocaml.org and the benchmarking servers will be unavailable.  x86_64 workers will be substantially reduced.  Service is expected to resume late on Sunday evening.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[There is an electrical shutdown for essential maintenance this weekend, which will affect large parts of the cluster. All Power9, ARM64, RISC-V, Windows and FreeBSD workers and docs.ci.ocaml.org, docs-data.ocaml.org and the benchmarking servers will be unavailable. x86_64 workers will be substantially reduced. Service is expected to resume late on Sunday evening.]]></summary></entry><entry><title type="html">Relocating opam.ci.ocaml.org and ocaml.ci.dev</title><link href="/2023/12/04/services-moved.html" rel="alternate" type="text/html" title="Relocating opam.ci.ocaml.org and ocaml.ci.dev" /><published>2023-12-04T00:00:00+00:00</published><updated>2023-12-04T00:00:00+00:00</updated><id>/2023/12/04/services-moved</id><content type="html" xml:base="/2023/12/04/services-moved.html"><![CDATA[<p>About six months ago, <a href="https://opam.ci.ocaml.org"><code class="language-plaintext highlighter-rouge">opam-repo-ci</code> (opam.ci.ocaml.org)</a> suffered from a lack of system memory (<a href="https://github.com/ocurrent/opam-repo-ci/issues/220">issue 220</a>) which caused it to be moved to the machine hosting <a href="https://ocaml.ci.dev"><code class="language-plaintext highlighter-rouge">ocaml-ci</code> (ocaml.ci.dev)</a>.</p>

<p>Subsequently, that machine suffered from BTRFS volume corruption (<a href="https://github.com/ocaml/infrastructure/issues/51">issue 51</a>). Therefore, we moved both services to a larger new server. The data was efficiently migrated using BTRFS tools: <code class="language-plaintext highlighter-rouge">btrfs send | btrfs receive</code>.</p>

<p>Since the move, we have seen issues with BTRFS metadata. Plus, we have suffered from a build-up of subvolumes, as reported by other users: <a href="https://github.com/moby/moby/issues/27653">Docker gradually exhausts disk space on BTRFS</a>.</p>

<p>Unfortunately, both services went down on Friday evening (<a href="https://github.com/ocaml/infrastructure/issues/85">issue 85</a>). Analysis showed over 500 BTRFS subvolumes, a shortage of metadata space, and insufficient space to perform a BTRFS <em>rebalance</em>.</p>

<p>Returning to the original configuration of splitting the <code class="language-plaintext highlighter-rouge">ci.dev</code> and OCaml.org services, they have been moved onto new and separate hardware. The underlying filesystem is now a RAID1-backed ext4, formatted with <code class="language-plaintext highlighter-rouge">-i 8192</code> in order to ensure the availability of sufficient inodes. Docker uses Overlayfs. RSYNC was used to copy the databases and logs from the old server. This change should add resilience and has doubled the capacity for storing history logs.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[About six months ago, opam-repo-ci (opam.ci.ocaml.org) suffered from a lack of system memory (issue 220) which caused it to be moved to the machine hosting ocaml-ci (ocaml.ci.dev).]]></summary></entry><entry><title type="html">macOS Sonoma Update</title><link href="/2023/11/09/macos-sonoma.html" rel="alternate" type="text/html" title="macOS Sonoma Update" /><published>2023-11-09T00:00:00+00:00</published><updated>2023-11-09T00:00:00+00:00</updated><id>/2023/11/09/macos-sonoma</id><content type="html" xml:base="/2023/11/09/macos-sonoma.html"><![CDATA[<p><a href="https://github.com/ocurrent/obuilder">OBuilder</a> workers for both Intel and Apple Silicon have been updated from macOS Ventura to <a href="https://www.apple.com/macos/sonoma/">macOS Sonoma</a> 14.1.1.</p>

<p>From today, <a href="https://ocaml.ci.dev">ocaml-ci</a> and <a href="https://opam.ci.ocaml.org">opam-repo-ci</a> will test against Sonoma rather than Ventura.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[OBuilder workers for both Intel and Apple Silicon have been updated from macOS Ventura to macOS Sonoma 14.1.1.]]></summary></entry><entry><title type="html">Current Bench Maintenance</title><link href="/2023/11/06/current-bench-maintenance.html" rel="alternate" type="text/html" title="Current Bench Maintenance" /><published>2023-11-06T00:00:00+00:00</published><updated>2023-11-06T00:00:00+00:00</updated><id>/2023/11/06/current-bench-maintenance</id><content type="html" xml:base="/2023/11/06/current-bench-maintenance.html"><![CDATA[<p>Due to storage related maintenance operations on Wednesday 8th November ‘23 between 12:00pm and 3:00pm (UTC+1), the current-bench benchmarking service will be unavailable.</p>

<p>Thank you for your patience and understanding during this time.</p>]]></content><author><name></name></author><summary type="html"><![CDATA[Due to storage related maintenance operations on Wednesday 8th November ‘23 between 12:00pm and 3:00pm (UTC+1), the current-bench benchmarking service will be unavailable.]]></summary></entry></feed>