java - 如何在 Java 中增量增长无限的 HDF5 数据集

Question

我的基本用例非常简单。我不知道数据集会变得有多大。我正在循环运行并且必须存储数据。为此，我尝试创建一个实用程序类，我希望能够像这样使用它:

    H5File file = new H5File( Paths.get( file ) );

    file.create();
    file.createDataset( dataset ); // empty

    file.extendDataset( dataset, data ); // append some data
    file.extendDataset( dataset, data ); // append some data
    // ... do this some more

    file.close();

我的问题是在 extendDataset 成员函数的实现中。据我了解，要执行此操作，我必须使用 H5.H5Sselect_hyperslab 创建一个 hyperslab，然后使用 H5.H5Dwrite 写入新选择。我的代码的相应部分如下所示。

boolean extendDataset( String param, float[] data )
{
    try
    {
        long[] extdims = new long[1];

        H5ScalarDS dataset = (H5ScalarDS) h5File.get( param );

        int dataset_id = dataset.open();
        int dataspace_id = H5.H5Dget_space( dataset_id );

        H5.H5Sget_simple_extent_dims( dataspace_id, extdims, null );

        long[] start = extdims.clone();
        extdims[0] += data.length;

        H5.H5Sclose(dataspace_id);

        dataset.extend( extdims );

        dataspace_id = H5.H5Dget_space(dataset_id);

        long[] count = { data.length };
        start[ 0 ] = extdims[0] - data.length;

        H5.H5Sselect_hyperslab(
                dataspace_id,
                HDF5Constants.H5S_SELECT_SET,
                start, null,
                count, null );

        float [] extData = new float[ (int)extdims[0] ];
        System.arraycopy( data, 0, extData, (int)start[0], data.length );

        // Write the data to the selected portion of the dataset.
        H5.H5Dwrite(
                dataset_id,
                HDF5Constants.H5T_NATIVE_FLOAT,
                HDF5Constants.H5S_ALL,
                dataspace_id,
                HDF5Constants.H5P_DEFAULT,
                extData );

        dataset.close( dataset_id );
    }
    catch( Exception e )
    {
        e.printStackTrace();
        return false;
    }

    return true;
}

我对这段代码的问题是线条。

        float [] extData = new float[ (int)extdims[0] ];
        System.arraycopy( data, 0, extData, (int)start[0], data.length );

如果不让缓冲区extData与数据集一样大，我就无法让它工作，这是我想避免的。 有什么方法可以处理传递给 H5.H5Dwrite 的数据，这只是超板的大小。

这是完整的源代码，应该很容易运行。

  package h5;

  import ncsa.hdf.hdf5lib.H5;
  import ncsa.hdf.hdf5lib.HDF5Constants;
  import ncsa.hdf.hdf5lib.exceptions.HDF5Exception;
  import ncsa.hdf.object.Datatype;
  import ncsa.hdf.object.FileFormat;
  import ncsa.hdf.object.Group;
  import ncsa.hdf.object.h5.H5Datatype;
  import ncsa.hdf.object.h5.H5ScalarDS;

  import java.nio.file.Path;
  import java.nio.file.Paths;

  /**
   * Created by Thomas on 24.07.2015.
   */
  public class H5File
  {

      @FunctionalInterface
      public interface IFloatGenerator
      {
          float generate( int t );
      }

      public static
      float[]
      generate( float [] vs, IFloatGenerator gen )
      {
          for( int i = 0; i < vs.length; ++i  )
          {
              vs[i] = gen.generate( i );
          }

          return vs;
      }


      Path path;
      ncsa.hdf.object.h5.H5File h5File;
      final H5Datatype floatType = new H5Datatype(Datatype.CLASS_FLOAT, 4, Datatype.NATIVE, -1);

      private static final long[] dims = { 0 };
      private static final long[] maxdims = { HDF5Constants.H5S_UNLIMITED };
      private static final long[] chunks = { 16384 };

      public H5File( Path path )
      {
          this.path = path;
      }

      boolean open()
      {
          h5File = new ncsa.hdf.object.h5.H5File( path.toString(), FileFormat.WRITE );
          try
          {
              h5File.open();
          }
          catch( Exception e )
          {
              e.printStackTrace();
              return  false;
          }

          return true;
      }

      boolean create()
      {
          h5File = new ncsa.hdf.object.h5.H5File( path.toString(), FileFormat.CREATE);
          try
          {
              h5File.open();
          }
          catch( Exception e )
          {
              e.printStackTrace();
              return false;
          }

          return true;
      }

      boolean close()
      {
          try
          {
              h5File.close();
          }
          catch( HDF5Exception e )
          {
              e.printStackTrace();
              return false;
          }

          return true;
      }

      boolean createDataset( String name )
      {
          Group root = (Group) ((javax.swing.tree.DefaultMutableTreeNode) h5File.getRootNode()).getUserObject();

          try
          {
              h5File.createScalarDS( name, root, floatType, dims, maxdims, chunks, 0, null);
          }
          catch( Exception e )
          {
              e.printStackTrace();
              return false;
          }

          return true;
      }

      boolean extendDataset( String param, float[] data )
      {
          try
          {
              long[] extdims = new long[1];

              H5ScalarDS dataset = (H5ScalarDS) h5File.get( param );

              int dataset_id = dataset.open();
              int dataspace_id = H5.H5Dget_space( dataset_id );

              H5.H5Sget_simple_extent_dims( dataspace_id, extdims, null );

              long[] start = extdims.clone();
              extdims[0] += data.length;

              H5.H5Sclose(dataspace_id);

              dataset.extend( extdims );

              dataspace_id = H5.H5Dget_space(dataset_id);

              long[] count = { data.length };
              start[ 0 ] = extdims[0] - data.length;

              H5.H5Sselect_hyperslab(
                      dataspace_id,
                      HDF5Constants.H5S_SELECT_SET,
                      start, null,
                      count, null );

              float [] extData = new float[ (int)extdims[0] ];
              System.arraycopy( data, 0, extData, (int)start[0], data.length );

              // Write the data to the selected portion of the dataset.
              H5.H5Dwrite(
                      dataset_id,
                      HDF5Constants.H5T_NATIVE_FLOAT,
                      HDF5Constants.H5S_ALL,
                      dataspace_id,
                      HDF5Constants.H5P_DEFAULT,
                      extData );

              dataset.close( dataset_id );
          }
          catch( Exception e )
          {
              e.printStackTrace();
              return false;
          }

          return true;
      }



      public static void main(String[] argv)
      {
          H5File file = new H5File( Paths.get( "test.h5" ) );

          String name = "floats";
          float[] data = new float[ 10 ];

          file.create();
          file.createDataset( name );
          generate( data, i -> i );
          file.extendDataset( name, data );
          generate( data, i -> 10 + i );
          file.extendDataset( name, data );
          file.close();
      }


  }

Answer 1

Is there any way to work with data passed to H5.H5Dwrite, which is just the size of the hyperslab.

是的。传入H5Dwrite的数据必须与数据集匹配的原因是由于参数H5S_ALL。相反，必须使用 data 的数据集 ID。这可以使用 H5Screate_simple

创建

 long[] mem_dim = { data.length };
 int mem_dataset_id = H5.H5Screate_simple( 1, mem_dim, mem_dim );

 H5.H5Dwrite(
          dataset_id,
          HDF5Constants.H5T_NATIVE_FLOAT,
          mem_dataset_id, 
          dataspace_id,
          HDF5Constants.H5P_DEFAULT,
          data );

 H5.H5Sclose( mem_dataset_id );