Scientific file formats: HDF5,FITS, NetCDF?
darenw
scientific file formats? Do libraries already exist?
Although I've read that Go might not be the best choice for massive
number crunching, many science and engineering apps tody have complex
GUIs, connect to other apps via dbus or other interconnection systems,
and do a lot of threading for example to let the user continue playing
with the GUI while data is saved to a file in the background. The
thing I'm working on currently has over 600 classes in a messy
hierarchy, many objects having pointers to other objects to access
methods that only return pointers to yet other objects, and most of
the real work done is done by short snips of code, one to several
lines, scattered hither thither and yon throughout a very large
directory tree.
I'm fantasizing the whole thing rewritten in Go might be 1/4 the size,
1/10th the bugs, and way easier to maintain. Never mind performance
of huge arrays of numbers - that can be dealt with in various ways.
I might try a small demo program in Go, but it'll be far easier if I
can at least read FITS files.
Andrew Gerrand
> How well would a Go program be able to read and write the major
> scientific file formats? Do libraries already exist?
Not that I am aware of. However, I have written file format
reader/writers (like archive/zip, and a WAVE format library) and in
both cases it was pretty easy to put together something that (at
least) works. If the file formats you need to work with are well
documented, you might consider doing that.
> Although I've read that Go might not be the best choice for massive
> number crunching
From where did you get that idea? Go's compiled code is fast to
execute, and Go doesn't have a lot of the overheads associated with
some other languages (like Java). There's no substantial reason Go
can't crunch numbers as fast as anything else.
Andrew
Sonia Keys
On 28 December 2010 11:47, darenw <darensc...@gmail.com> wrote:
> Although I've read that Go might not be the best choice for massive
> number crunching
From where did you get that idea? Go's compiled code is fast to
execute, and Go doesn't have a lot of the overheads associated with
some other languages (like Java). There's no substantial reason Go
can't crunch numbers as fast as anything else.
I might try a small demo program in Go, but it'll be far easier if I
can at least read FITS files.
darenw
On Dec 27, 10:01 pm, Andrew Gerrand <a...@golang.org> wrote:
> > Although I've read that Go might not be the best choice for massive
> > number crunching
>
> From where did you get that idea?
Go's compiled code is fast to
> execute, and Go doesn't have a lot of the overheads associated with
> some other languages (like Java). There's no substantial reason Go
> can't crunch numbers as fast as anything else.
>
> Andrew
on huge images, needing mostly pure number crunching, nothing fancy.
Comparing programs written in Go and D will be educational. Nothing
beats a careful reality check and measurement.
darenw
On Dec 28, 9:47 am, Sonia Keys <soniak...@gmail.com> wrote:
> On Monday, December 27, 2010 10:01:21 PM UTC-5, Andrew Gerrand wrote:
> > On 28 December 2010 11:47, darenw <darensc...@gmail.com> wrote:
>
> > > Although I've read that Go might not be the best choice for massive
> > > number crunching
>
> > From where did you get that idea? Go's compiled code is fast to
> > execute, and Go doesn't have a lot of the overheads associated with
> > some other languages (like Java). There's no substantial reason Go
> > can't crunch numbers as fast as anything else.
>
> Andrew
>
> Totally agree with Andrew. Go works great for number crunching. You might
>
>
> I suspect a general FITS reader would be a substantial project. A small
> demo program that only had to deal with one particular source or style of
> FITS files should be much simpler and sounds like an excellent project.
>
> Sonia
kind of programming I prefer, compared to apps programming and most
other areas that aren't image and number crunching.
Given a choice between some kind of wrapper around an existing
library, and a genuine from the ground-up library in Go, I'd take the
latter even if it's a lot of work to create. Done right, if it runs
well, it'll help promote Go.
pymo...@gmail.com
Fazlul Shahriar
I haven't had time to work on it further, but I'm very much interested.
I also want to work on hdf5 but I don't deal with hdf5 as often as
hdf4. NetCDF4 format is pretty much a stripped down version of HDF5 as
far as I know, and I think NetCDF3 is the simplest format to
implement.
Of course, you can always use cgo. Both pytables and pyhdf are
bindings to the C libraries.
fhs
Rémy Oudompheng
> I started on hdf4 a while ago: https://bitbucket.org/fhs/gohdf
> I haven't had time to work on it further, but I'm very much interested.
>
> I also want to work on hdf5 but I don't deal with hdf5 as often as
> hdf4. NetCDF4 format is pretty much a stripped down version of HDF5 as
> far as I know, and I think NetCDF3 is the simplest format to
> implement.
>
> Of course, you can always use cgo. Both pytables and pyhdf are
> bindings to the C libraries.
Sébastien Binet has Go bindings for libhdf5 at
https://bitbucket.org/binet/go-hdf5/
I didn't try them and I don't know if it uses reflection for dataset
reading/writing.
At least, the interface looks usable:
http://gopkgdoc.appspot.com/pkg/bitbucket.org/binet/go-hdf5/pkg/hdf5
Rémy.
Sebastien Binet
> Le 31 mars 2012 22:30, Fazlul Shahriar <fsha...@gmail.com> a écrit :
>> I started on hdf4 a while ago: https://bitbucket.org/fhs/gohdf
>> I haven't had time to work on it further, but I'm very much interested.
>>
>> I also want to work on hdf5 but I don't deal with hdf5 as often as
>> hdf4. NetCDF4 format is pretty much a stripped down version of HDF5 as
>> far as I know, and I think NetCDF3 is the simplest format to
>> implement.
>>
>> Of course, you can always use cgo. Both pytables and pyhdf are
>> bindings to the C libraries.
>
> Sébastien Binet has Go bindings for libhdf5 at
> https://bitbucket.org/binet/go-hdf5/
> I didn't try them and I don't know if it uses reflection for dataset
> reading/writing.
it does:
https://bitbucket.org/binet/go-hdf5/src/50c6c6f0bdc4/pkg/hdf5/h5t.go#cl-386
-s
--
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################
a.ko...@gmail.com
Replying to this 9-year-old thread because the original question has finally been answered.
Back in 2015, the consensus was that HDF5 is "so complicated that there is only one implementation" and too difficult for pure Go. I'm happy to report that's no longer true.
WHAT EXISTS NOW (2025):
A pure Go HDF5 library with full read support and beta write support:
Repository: https://github.com/scigolib/hdf5
Read: Feature-complete (superblock v0/2/3, all layouts, compression, attributes)
Write: Beta (v0.11.1-beta - chunked datasets, GZIP, dense groups, attributes)
Example - Reading:
file, _ := hdf5.Open("data.h5")
dataset := file.Datasets["/temperature"]
data := dataset.Data() // []float64, []int32, etc.
Example - Writing (beta, but functional):
file, _ := hdf5.CreateForWrite("output.h5", hdf5.Truncate)
file.CreateDataset("data", myData,
hdf5.WithChunked([]uint64{100, 100}),
hdf5.WithCompression(6),
)
HOW IT WAS DONE:
The C library (https://github.com/HDFGroup/hdf5) served as reference implementation. Instead of "figuring out" the format from scratch, we ported proven algorithms to Go. Format spec + reference code = solvable problem.
Development time: ~1 year from concept to write MVP (with AI assistance for rapid prototyping).
WHY IT MATTERS:
- No CGo = actual cross-compilation, no C dependencies
- Type safety = Go's compiler catches HDF5 format errors at compile time
- Standard library integration (io.ReaderAt, encoding/binary)
- Single binary deployment
CURRENT STATUS:
- Test coverage: 70-88% depending on package
- Platforms: Linux, macOS, Windows
- Recognition: HDF Group acknowledged it on their forum
(https://forum.hdfgroup.org/t/loking-for-an-hdf5-version-compatible-with-go1-9-2/10021/7)
- Beta limitations: Some write features in progress (dense storage
read-modify-write, h5dump compatibility)
FOR THE SCIENTIFIC GO COMMUNITY:
If you're working with HDF5 files and want to avoid CGo, this is now viable. Looking for beta testers with real-world datasets (astronomy, climate, genomics, etc.).
Installation:
go get github.com/scigolib/hd...@v0.11.1-beta
The format is indeed complex, but it's been tackled. Thought this group might appreciate the update after 9 years.
Best,
[Your name]
P.S. NetCDF4 being "stripped down HDF5" (as mentioned in the original thread) means this library could potentially support it too, though that's not implemented yet.