c++ - 无法写入 vector 大于 2^13 的 HDF5 文件

Question

我正在使用 C++ 和 HDF5 来编写文件。但遇到了问题。这是我使用的代码:

void fileRead::writeFile(string name, const vector<double>* data) {
int dimn = data->size();

hsize_t dim[1] = {data->size()}; //-> 2^13!!!

hid_t sid = H5Pcreate(H5P_DATASET_CREATE);
hid_t didProp = H5Screate_simple(1,dim,NULL);
H5Pset_layout(sid, H5D_COMPACT);

hid_t did = H5Dcreate(fid, name.c_str(),H5T_IEEE_F64LE, didProp, H5P_DEFAULT, sid,H5P_DEFAULT);
H5Dwrite (did, H5T_NATIVE_DOUBLE, H5S_ALL, H5S_ALL, H5P_DEFAULT, &(data->at(0)));
H5Dclose(did);

H5Sclose(didProp);
H5Pclose(sid);
}

但这给了我这个错误消息:

HDF5-DIAG: Error detected in HDF5 (1.8.10) thread 0:   #000: /pub/devel/hdf5/hdf5-1.8.10-1/src/hdf5-1.8.10/src/H5D.c line 170 in H5Dcreate2(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object   #001: /pub/devel/hdf5/hdf5-1.8.10-1/src/hdf5-1.8.10/src/H5Dint.c line 439 in H5D__create_named(): unable to create and link to dataset
    major: Dataset
    minor: Unable to initialize object   #002: /pub/devel/hdf5/hdf5-1.8.10-1/src/hdf5-1.8.10/src/H5L.c line 1638 in H5L_link_object(): unable to create new link to object
    major: Links
    minor: Unable to initialize object   #003: /pub/devel/hdf5/hdf5-1.8.10-1/src/hdf5-1.8.10/src/H5L.c line 1882 in H5L_create_real(): can't insert link
    major: Symbol table
    minor: Unable to insert object   #004: /pub/devel/hdf5/hdf5-1.8.10-1/src/hdf5-1.8.10/src/H5Gtraverse.c line 861 in H5G_traverse(): internal path traversal failed
    major: Symbol table
    minor: Object not found   #005: /pub/devel/hdf5/hdf5-1.8.10-1/src/hdf5-1.8.10/src/H5Gtraverse.c line 641 in H5G_traverse_real(): traversal operator failed
    major: Symbol table
    minor: Callback failed   #006: /pub/devel/hdf5/hdf5-1.8.10-1/src/hdf5-1.8.10/src/H5L.c line 1685 in H5L_link_cb(): unable to create object
    major: Object header
    minor: Unable to initialize object   #007: /pub/devel/hdf5/hdf5-1.8.10-1/src/hdf5-1.8.10/src/H5O.c line 3015 in H5O_obj_create(): unable to open object
    major: Object header
    minor: Can't open object   #008: /pub/devel/hdf5/hdf5-1.8.10-1/src/hdf5-1.8.10/src/H5Doh.c line 293 in H5O__dset_create(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object   #009: /pub/devel/hdf5/hdf5-1.8.10-1/src/hdf5-1.8.10/src/H5Dint.c line 1044 in H5D__create(): unable to construct layout information
    major: Dataset
    minor: Unable to initialize object   #010: /pub/devel/hdf5/hdf5-1.8.10-1/src/hdf5-1.8.10/src/H5Dcompact.c line 212 in H5D__compact_construct(): compact dataset size is bigger than header message maximum size
    major: Dataset
    minor: Unable to initialize object HDF5-DIAG: Error detected in HDF5 (1.8.10) thread 0:   #000: /pub/devel/hdf5/hdf5-1.8.10-1/src/hdf5-1.8.10/src/H5D.c line 391 in H5Dclose(): not a dataset
    major: Invalid arguments to routine
    minor: Inappropriate type

所有 vector 大小 >= 2^13 (8192) 都会发生这种情况。这让我很困惑，因为读入更大的文件没有问题，而且 2^13 仍然是一个相当小的数字，所以我的代码一定有问题。

如有任何帮助，我们将不胜感激。 你的 麻姑_

Answer 1

来自H5Pset_layout的H5D_COMPACT参数的文档:

Store raw data in the dataset object header in file. This should only be used for datasets with small amounts of raw data. The raw data size limit is 64K (65520 bytes). Attempting to create a dataset with raw data larger than this limit will cause the H5Dcreate call to fail.

因此，如果您的 double 是 8 个字节，那么您就遇到了该限制。

您需要使用其他存储选项之一，连续或分块。