Installing HDF5.jl on ARM M1

[tamasgal]

Using the latest 1.7.0-rc1, HDF5.jl cannot be installed due to a missing binary dependency:

(@v1.7) pkg> add HDF5
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.7/Project.toml`
  No Changes to `~/.julia/environments/v1.7/Manifest.toml`

(@v1.7) pkg> precompile
Precompiling project...
  ✗ HDF5
  0 dependencies successfully precompiled in 1 seconds (21 already precompiled)

ERROR: The following 1 direct dependency failed to precompile:

HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f]

Failed to precompile HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f] to /Users/tamasgal/.julia/compiled/v1.7/HDF5/jl_wRxaZM.
ERROR: LoadError: UndefVarError: libhdf5 not defined

I am not so familiar with the underlying BinaryBuilder.jl infrastructure, so I wanted to ask around what to do: how can I contribute to https://github.com/JuliaBinaryWrappers/HDF5_jll.jl ?

[musm]

Thanks for the report, the build script is actually https://github.com/JuliaPackaging/Yggdrasil/blob/master/H/HDF5/build_tarballs.jl let me know if you can't find the issue. Otherwise I can look into this as well.

[tamasgal]

Yes I just found it out a few seconds ago thanks!

As far as I can see, I only need to create a PR and update the list of platforms. I'll try!

[zahachtah]

Is this in the pipeline yet? Only package I use that does not wanna install on M1. If I knew anything about PR's or build scripts I'd love to help...

[tamasgal]

I am still struggling to get the compilation script working, help is appreciated ;)

[simonbyrne]

HDF5 doesn't support cross-compilation (see JuliaPackaging/Yggdrasil#567), so at the moment the build script just pulls in prebuilt binaries from elsewhere.

[musm]

The quickest solution would be for someone on M1 platform to compile the HDF5 library and upload the binaries somewhere so we can add them to https://github.com/JuliaPackaging/Yggdrasil/blob/master/H/HDF5/build_tarballs.jl, until we can sort out the more complicated cross-compilation process.

[tamasgal]

I built it on M1 with a standard configure:

http://www.tamasgal.com/downloads/hdf5/hdf5-1.12.1.tar.gz
http://www.tamasgal.com/downloads/hdf5/hdf5-1.12.1.tar.gz.md5

The MD5 checksum is 5b5085aee04af16c0d906b4b0129dfa7

Here is the configuration:

	    SUMMARY OF THE HDF5 CONFIGURATION
	    =================================

General Information:
-------------------
                   HDF5 Version: 1.12.1
                  Configured on: Wed Oct 27 16:00:30 CEST 2021
                  Configured by: tamasgal@silentbox
                    Host system: arm-apple-darwin20.6.0
              Uname information: Darwin silentbox 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:20 PDT 2021; root:xnu-7195.141.6~3/RELEASE_ARM64_T8101 arm64
                       Byte sex: little-endian
             Installation point: /Users/tamasgal/tmp/hdf5/hdf5-1.12.1/hdf5

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables:
                        LDFLAGS:
                     H5_LDFLAGS:
                     AM_LDFLAGS:
                Extra libraries: -lz -ldl -lm
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: /opt/homebrew/opt/llvm@11/bin/clang
                       CPPFLAGS:
                    H5_CPPFLAGS:   -DNDEBUG -UH5_DEBUG_API
                    AM_CPPFLAGS:
                        C Flags:
                     H5 C Flags:  -std=c99  -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wno-c++-compat -Wno-format-nonliteral -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var    -Wno-missing-noreturn  -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -O3
                     AM C Flags:
               Shared C Library: yes
               Static C Library: yes


                        Fortran: no

                            C++: no

                           Java: no


Features:
---------
                   Parallel HDF5: no
Parallel Filtered Dataset Writes: no
              Large Parallel I/O: no
              High-level library: yes
                Build HDF5 Tests: yes
                Build HDF5 Tools: yes
                    Threadsafety: no (recursive RW locks: no)
             Default API mapping: v112
  With deprecated public symbols: yes
          I/O filters (external): deflate(zlib)
                             MPE: no
                   Map (H5M) API: no
                      Direct VFD: no
                      Mirror VFD: no
              (Read-Only) S3 VFD: no
            (Read-Only) HDFS VFD: no
                         dmalloc: no
  Packages w/ extra debug output: none
                     API tracing: no
            Using memory checker: no
 Memory allocation sanity checks: no
          Function stack tracing: no
                Use file locking: best-effort
       Strict file format checks: no
    Optimization instrumentation: no

[musm]

Great thanks @tamasgal, do we know if there's a way to test M1 on CI or at least an M1 VM I or someone else could use to test the precompiled binaries?

[tamasgal]

I remember that Stefan Karpinski mentioned in a talk a few months ago that they are using set of Mac Minis for Julia for testing, which is kind of a shame for Apple, but at least a solution.

I cannot provide such a farm but at least I can manually check the test suite until we have a solution for an M1-CI 😕

[cortner]

Is there a workaround for this?

EDIT: I can install HDF5 via home-brew, then point HDF5.jl to that installation. But when using HDF5 I get

julia> using HDF5
[ Info: Precompiling HDF5 [f67ccb44-e63f-5c2f-98bd-6dc0ccc4ba2f]
ERROR: LoadError: InitError: UndefVarError: libblosc not defined
Stacktrace:
  [1] __init__()
    @ Blosc ~/.julia/packages/Blosc/vjmKP/src/Blosc.jl:6
...
 [15] top-level scope
    @ none:1
during initialization of module Blosc
in expression starting at /Users/ortner/.julia/packages/HDF5/pIJra/src/blosc_filter.jl:1
in expression starting at /Users/ortner/.julia/packages/HDF5/pIJra/src/HDF5.jl:1

Unfortunately I did not figure out how to fix Blosc in a similar way.

[abhayap]

I built it on M1 with a standard configure:

http://www.tamasgal.com/downloads/hdf5/hdf5-1.12.1.tar.gz http://www.tamasgal.com/downloads/hdf5/hdf5-1.12.1.tar.gz.md5

The MD5 checksum is 5b5085aee04af16c0d906b4b0129dfa7

Here is the configuration:

	    SUMMARY OF THE HDF5 CONFIGURATION
	    =================================

General Information:
-------------------
                   HDF5 Version: 1.12.1
                  Configured on: Wed Oct 27 16:00:30 CEST 2021
                  Configured by: tamasgal@silentbox
                    Host system: arm-apple-darwin20.6.0
              Uname information: Darwin silentbox 20.6.0 Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:20 PDT 2021; root:xnu-7195.141.6~3/RELEASE_ARM64_T8101 arm64
                       Byte sex: little-endian
             Installation point: /Users/tamasgal/tmp/hdf5/hdf5-1.12.1/hdf5

Compiling Options:
------------------
                     Build Mode: production
              Debugging Symbols: no
                        Asserts: no
                      Profiling: no
             Optimization Level: high

Linking Options:
----------------
                      Libraries: static, shared
  Statically Linked Executables:
                        LDFLAGS:
                     H5_LDFLAGS:
                     AM_LDFLAGS:
                Extra libraries: -lz -ldl -lm
                       Archiver: ar
                       AR_FLAGS: cr
                         Ranlib: ranlib

Languages:
----------
                              C: yes
                     C Compiler: /opt/homebrew/opt/llvm@11/bin/clang
                       CPPFLAGS:
                    H5_CPPFLAGS:   -DNDEBUG -UH5_DEBUG_API
                    AM_CPPFLAGS:
                        C Flags:
                     H5 C Flags:  -std=c99  -Wall -Warray-bounds -Wcast-qual -Wconversion -Wdouble-promotion -Wextra -Wformat=2 -Wframe-larger-than=16384 -Wimplicit-fallthrough -Wno-c++-compat -Wno-format-nonliteral -Wnull-dereference -Wunused-const-variable -Wwrite-strings -Wpedantic -Wvolatile-register-var    -Wno-missing-noreturn  -Wbad-function-cast -Wimplicit-function-declaration -Wincompatible-pointer-types -Wmissing-declarations -Wpacked -Wshadow -Wswitch -Wno-error=incompatible-pointer-types-discards-qualifiers -Wunused-function -Wunused-variable -Wunused-parameter -Wcast-align -Wformat -O3
                     AM C Flags:
               Shared C Library: yes
               Static C Library: yes


                        Fortran: no

                            C++: no

                           Java: no


Features:
---------
                   Parallel HDF5: no
Parallel Filtered Dataset Writes: no
              Large Parallel I/O: no
              High-level library: yes
                Build HDF5 Tests: yes
                Build HDF5 Tools: yes
                    Threadsafety: no (recursive RW locks: no)
             Default API mapping: v112
  With deprecated public symbols: yes
          I/O filters (external): deflate(zlib)
                             MPE: no
                   Map (H5M) API: no
                      Direct VFD: no
                      Mirror VFD: no
              (Read-Only) S3 VFD: no
            (Read-Only) HDFS VFD: no
                         dmalloc: no
  Packages w/ extra debug output: none
                     API tracing: no
            Using memory checker: no
 Memory allocation sanity checks: no
          Function stack tracing: no
                Use file locking: best-effort
       Strict file format checks: no
    Optimization instrumentation: no

@tamasgal I have downloaded the binary you built. Thank you! How do I use it so that I can get it working with Julia? Where should I copy it?