The problem is this guy tried to slam a circular peg into a square hole. Although the spec is huge, there is plenty of sample code online to get it working. I have to ask though, why not just make a cached file of your data and onces its done, integrate into the final HDF file? Free forum by Nabble. I remember when someone dropped the specification on my desk and said, “should be a good read. It is much much faster to convert endianness again at least 10x faster than to parse text.

Uploader: Shadal
Date Added: 26 November 2012
File Size: 40.51 Mb
Operating Systems: Windows NT/2000/XP/2003/2003/7/8/10 MacOS 10/X
Downloads: 47938
Price: Free* [*Free Regsitration Required]

Moving away from HDF5 | Hacker News

For feature-finding, this is straightforward: A fair comparison shows a mixed picture — hdf5 is faster in some cases, and numpy. Message 5 of 7. Based on our observation, there is one data set, mydataset in the file. Strictly proprietary technology which gives a real competitive advantage.

HDF5 supports parallelism from distinct programs, but its multithreaded story was is?

This opens an HDF5 file. HDF5 is widely used ; you can be sure it will be around for many, many years to streaj. Funny enough, we started out with a file-based system like the one you described, and moved to an HDF5-based system. You can also see h5py maintainer Andrew Collette’s response here: Got an SGI Indigo somewhere in your lab?


Also, if efficiency isn’t paramount — just use SQLite!

How do you stream data to a HDF5 file? – Discussion Forums – National Instruments

High complexity of the specification and the implementation – Opacity of the development and slow reactivity of the development srteam – slow reactivity to what? Amazing tool when it’s in its sweetspot.

HDF5 was never intended be be a container used to stream data into — it was meant for sharing data. These are tools that serialize buffers with razor-sharp binary specifications.

I’m just getting started with HDF5, and your streaming example should be most helpful. Yes, you can use H5Literate to iterate over all links in a group. Message 4 hdg5 7. If you are working on some esoteric platform that is big endian you will know it. Agreed, HDF could benefit from journaling.

For more, see Attributes. We had some strict requirements about data corruption e.

It is right that converting endianness is faster strram reading text, but it still requires more code than calling printf and scanf. Message Edited by DFGray on Thank you for getting back to me. I am surprised the original poster did not run into this issue. But HDF is not a simple file format, so expect some overhead.


HDF5 file as a binary stream ?

I have to ask though, why not just make a cached file hcf5 your data and onces its done, integrate into the final HDF file? Deleting the link to a group should remove all the links to the datasets within it and therefore the datasets themselves. For linking, the problem is more challenging, but trackpy handles all this complexity for you, using as little memory as possible throughout.

The idea of a general format serving all research just sounds silly to me. Try telling a customer to tar up a directory and send it to you — things get lost so easily! It’s not even that HDF5 is that badit’s just strdam filesystems are, in many respects, so much better. In the former case, one file approach HDF5 makes sense.

I will consider your proposal, although I did not want to write a complete server. We can get by just fine with: