Intro: HDF5 on hackage

I’ve been working on a little Haskell tool¹ that needs to read images from HDF5 files, which are pervasive in some research areas.

As usual, searching hackage for “hdf5” gives a few answers:

• hdf5-lite has its last commit in 2018 with the commit message “asdfasd” and the comment:

  Experimental, partly tested and incomplete, not meant for production use.

  as well as

  The HDF5 library headers must be correctly installed at /usr/local/hdf5/include.

  which were a few reason for me to rule that one out, since I’d definitely have to adapt the package.

• bindings-hdf5 is marked broken in nixpkgs. I haven’t figured out yet why that is.

• hdf5 looks ok, but is broken in nixpkgs (doesn’t compile) and depends on hdf5-serial, which I don’t have. I’m in active contact with the maintainer, who has been nothing but helpful. But I wanted a solution now, so…

…writing my own bindings

With Haskell, it’s pretty easy to write bindings to a C library (which HDF5 is). I only need a few functions, so I wrote my own little wrapper, which was fun! In the end, I had a file Hdf5Raw.hsc (note the ending: .hsc is like .hs, but with C extensions) and simply told cabal to search for hdf5 via pkg-config:

executable simplon-stub
  main-is: Main.hs
  other-modules: Simplon.Hdf5Raw
  hs-source-dirs: app
  build-depends:
      base >=4.7 && <5
  pkgconfig-depends: hdf5

I’m using Nix with Flakes, and cabal2nix, which converts a .cabal project description into a Nix derivation.

My flake.nix looked a little bit like this:

{
  description = "simplon-stub-hs";

  inputs = {
    nixpkgs.url = "nixpkgs/nixos-24.05";
    flake-utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, flake-utils }:
    flake-utils.lib.eachDefaultSystem
      (system:
        let
          pkgs = import nixpkgs {inherit system;};
          packageName = "simplon-stub-hs";
        in
        {
          packages.${packageName} =
            # 1. here we call cabal2nix, resolving the dependencies
            pkgs.haskellPackages.callCabal2nix packageName self {};
          devShells.default =
            pkgs.mkShell {
              buildInputs = with pkgs; [
                cabal-install
                # 2. what we need to compile hdf5 bindings
                pkg-config
              ];
              # 3. this will "import" the dependencies found by cabal2nix into the
              # current devshell
              inputsFrom = [ self.packages.${system}.${packageName}.env ];
            };
        });
}

Going through the source code:

1. We call the function callCabal2nix, which reads the .cabal file and spits out a Nix derivation for it, taking care to include all necessary dependencies.
2. I included pkg-config in my Nix shell build inputs, so cabal can find that in cabal build.
3. The inputsFrom takes the dependencies (the inputs) from the cabal2nix-generated derivation and uses them for the development shell.

However, this doesn’t work. In fact, with this flake.nix, nix develop will try to build haskellPackages.hdf5 (the one I described above, which is broken in nixpkgs). It took me a while and an issue on the cabal2nix GitHub repo to understand why.

Solution

My first solution to this strange problem was a hack: if cabal2nix is somehow inferring haskellPackages.hdf5 as a dependency, and that one is broken — why not just override (“redirect”) this package to a non-broken one? Like this:

packages.${packageName} =
  haskellPackages.callCabal2nix packageName self {hdf5 = haskellPackages.bytestring;};

bytestring is a package I’m using anyways, and it’s stable. This solution works, but feels weird, and it doesn’t help explaining what happened. To understand that, let’s call cabal2nix . (the command-line tool of the same name) manually and see the derivation it generates:

{ mkDerivation, aeson, base, bytestring, hdf5 }:
mkDerivation {
  pname = "simplon-stub-hs";
  version = "1.0.0";
  src = ./.;
  isLibrary = false;
  isExecutable = true;
  executableHaskellDepends = [ aeson base bytestring ];
  executablePkgconfigDepends = [ hdf5 ];
  mainProgram = "simplon-stub";
}

And here’s the — very subtle — problem: the derivation has a set of packages as its argument:

{ mkDerivation, aeson, base, bytestring, hdf5 }: ...

These derivations can be both packages from pkgs.haskellPackages (like bytestring) as well as packages from pkgs. But what about hdf5? This one is both. It’s in pkgs.hdf5 as well as pkgs.haskellPackage.hdf5. cabal2nix, as far as I understand it, cannot resolve this ambiguity, and gives precedence to haskellPackages.hdf5, which breaks the build.

The maintainer’s solution to that is similar to mine, but way less dubious:

haskellPackages.callCabal2nix packageName self {
  inherit (pkgs) hdf5;
};

So that’s it. It’s admittedly a corner case scenario, but if you encounter it now, maybe you’re not as surprised.

And of course, if I misunderstood, please write me!