Stack Overflow
Stack Overflow

Questions tagged [h5py]

h5py is a NumPy-compatible Python module for handling The Hierarchical Data Format (HDF5) files.

h5py is a NumPy-compatible Python module for handling The Hierarchical Data Format (HDF5) files.

Main features

  • Free (BSD licensed)
  • limited dependencies (Python, NumPy, HDF5 libs.)
  • includes both a low level c-like HDF5 interface and a high level Python/NumPy style interface
  • directly interact with datasets using NumPy metaphors, such as slicing
  • datatypes specified using standard NumPy dtype objects

Some links to get started

1070 questions
15
votes
3 answers

How convert this type of data to something more readable in the python?

I have quite big dataset. All information stored in the hdf5 format file. I found h5py library for python. All works properly except of the [] I have no idea how to convert it in something more readable. Can I do it at all ?…
Dmytro Chasovskyi
  • 2,169
  • 3
  • 18
  • 44
15
votes
3 answers

Can h5py load a file from a byte array in memory?

My python code is receiving a byte array which represents the bytes of the hdf5 file. I'd like to read this byte array to an in-memory h5py file object without first writing the byte array to disk. This page says that I can open a memory mapped…
mahonya
  • 7,648
  • 5
  • 37
  • 64
14
votes
1 answer

Writing a large hdf5 dataset using h5py

At the moment, I am using h5py to generate hdf5 datasets. I have something like this import h5py import numpy as np my_data=np.genfromtxt("/tmp/data.csv",delimiter=",",dtype=None,names=True) myFile="/tmp/f.hdf" with h5py.File(myFile,"a") as f: …
NinjaGaiden
  • 1
  • 2
  • 20
  • 43
14
votes
1 answer

Updating h5py Datasets

Does any one have an idea for updating hdf5 datasets from h5py? Assuming we create a dataset like: import h5py import numpy f = h5py.File('myfile.hdf5') dset = f.create_dataset('mydataset',…
George Monet
  • 187
  • 1
  • 2
  • 7
13
votes
2 answers

How to resize an HDF5 array with `h5py`

How can I resize an HDF5 array using the h5py Python library ? I've tried using the .resize method and on an array with chunks set to True. Alas, I'm still missing something. In [1]: import h5py In [2]: f = h5py.File('foo.hdf5', 'w') In [3]: d =…
MRocklin
  • 48,441
  • 20
  • 124
  • 196
12
votes
2 answers

Saving with h5py arrays of different sizes

I am trying to store about 3000 numpy arrays using HDF5 data format. Arrays vary in length from 5306 to 121999 np.float64 I am getting Object dtype dtype('O') has no native HDF5 equivalent error since due to the irregular nature of the data numpy…
Jose Javier Gonzalez Ortiz
  • 278
  • 1
  • 2
  • 13
12
votes
1 answer

What is the recommended compression for HDF5 for fast read/write performance (in Python/pandas)?

I have read several times that turning on compression in HDF5 can lead to better read/write performance. I wonder what ideal settings can be to achieve good read/write performance at: data_df.to_hdf(..., format='fixed', complib=..., complevel=...,…
Mark Horvath
  • 934
  • 1
  • 8
  • 19
12
votes
2 answers

h5py setup.py on Mac: hdf5.h file not found

I am building h5py on Mac, following instructions "Building against Parallel HDF5" in this link: http://docs.h5py.org/en/latest/build.html $ export CC=mpicc $ python setup.py configure --mpi $ sudo python setup.py build I get this…
yanggao
  • 191
  • 1
  • 7
11
votes
2 answers

Storing scipy sparse matrix as HDF5

I want to compress and store a humongous Scipy matrix in HDF5 format. How do I do this? I've tried the below code: a = csr_matrix((dat, (row, col)), shape=(947969, 36039)) f = h5py.File('foo.h5','w') dset = f.create_dataset("init", data=a, dtype…
Rama
  • 909
  • 1
  • 12
  • 28
10
votes
2 answers

Is there a way to get a numpy-style view to a slice of an array stored in a hdf5 file?

I have to work on large 3D cubes of data. I want to store them in HDF5 files (using h5py or maybe pytables). I often want to perform analysis on just a section of these cubes. This section is too large to hold in memory. I would like to have a numpy…
Caleb
  • 3,135
  • 5
  • 20
  • 32
10
votes
2 answers

Store datetimes in HDF5 with H5Py

How can I store NumPy datetime objects in HDF5 using h5py? In [1]: import h5py In [2]: import numpy as np In [3]: f = h5py.File('foo.hdfs', 'w') In [4]: d = f.create_dataset('data', shape=(2, 2), dtype=np.datetime64) TypeError: No conversion path…
MRocklin
  • 48,441
  • 20
  • 124
  • 196
9
votes
1 answer

Open .h5 file in Python

I am trying to read a h5 file in Python. The file can be found in this link and it is called 'vstoxx_data_31032014.h5'. The code I am trying to run is from the book Python for Finance, by Yves Hilpisch and goes like this: import pandas as pd h5…
python_enthusiast
  • 806
  • 2
  • 7
  • 22
9
votes
2 answers

Visible Deprecation warning...?

I have some data that Im reading from a h5 file as a numpy array and am doing some analysis with. For context, the data plots a spectral response curve. I am indexing the data (and a subsequent array I have made for my x axis) to get a specific…
Nathan Thomas
  • 1,344
  • 6
  • 21
  • 44
9
votes
2 answers

Convert HDF5 file to other formats

I am having a few big files sets of HDF5 files and I am looking for an efficient way of converting the data in these files into XML, TXT or some other easily readable format. I tried working with the Python package (www.h5py.org), but I was not able…
visakh
  • 2,333
  • 6
  • 25
  • 50
9
votes
3 answers

Python-created HDF5 dataset transposed in Matlab

I have some data that I share between Python and Matlab. I used to do it by saving NumPy arrays in MATLAB-style .mat files but would like to switch to HDF5 datasets. However, I've noticed a funny feature: when I save a NumPy array in an HDF5 file…
John Manak
  • 12,620
  • 27
  • 73
  • 114
1 2
3
…
71 72