java - 使用 Java 8 Stream(列表中的大量数据)时的性能问题

标签 java performance java-8 java-stream

我在一个方法中有以下 2 个代码/逻辑:

代码 1:返回大约 10K++ 的数据

    //myObjectRepository extends the spring CrudRepository (hibernate)
    final List<MyObject> myObjectList =
            myObjectRepository.getObjectsByIdAndReportDate();

代码2:过滤10k++数据

    //Only return valid objects
    final List<MyObject> objectsWithStrings = myObjectList
            //parallelStream() randomly throw an exception related to hibernate
            .stream()
            .filter(d -> d.getMyObjectStrings() != null && !d.getMyObjectStrings().isEmpty())
            .collect(Collectors.toList());

注意:MyObject 是一个具有以下结构的 hibernate 实体:

@JsonBackReference
//Jackson – Bidirectional Relationships. To avoid "Could not write content: Infinite recursion (StackOverflowError)"
@OneToMany(mappedBy = "myObject")
@Cache(usage = CacheConcurrencyStrategy.NONSTRICT_READ_WRITE)
private List<MyObjectString> myObjectStrings;

它按原样工作，但我在代码 2 中遇到以下情况:

  • 我的代码 2 中存在性能问题。
  • 每当我使用parallelStream()时都会出现随机 hibernate 问题，因此，我选择使用顺序stream()

在这种情况下，我有办法优化代码 2，以便它可以更快地处理数据或过滤数据吗？

我遇到的 hibernate 错误的片段:

2018-03-05 15:35:28.431 ERROR 10056 --- [nio-8082-exec-1] e.b.MyServiceExceptionControllerAdvice : An generic error is encountered [400] [Error accessing field [private java.util.Date my.entity.AnotherObjectPK.reportDate] by reflection for persistent property [my.entity.AnotherObjectPK#reportDate] : null]

org.hibernate.property.access.spi.PropertyAccessException: Error accessing field [private java.util.Date my.entity.AnotherObjectPK.reportDate] by reflection for persistent property [my.entity.AnotherObjectPK#reportDate] : null
at org.hibernate.property.access.spi.GetterFieldImpl.get(GetterFieldImpl.java:43) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.tuple.component.AbstractComponentTuplizer.getPropertyValue(AbstractComponentTuplizer.java:58) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.ComponentType.getPropertyValue(ComponentType.java:419) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.ComponentType.getHashCode(ComponentType.java:242) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.cache.internal.CacheKeyImplementation.calculateHashCode(CacheKeyImplementation.java:57) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.cache.internal.CacheKeyImplementation.<init>(CacheKeyImplementation.java:53) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.cache.internal.DefaultCacheKeysFactory.staticCreateEntityKey(DefaultCacheKeysFactory.java:50) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.cache.redis.hibernate5.strategy.ReadWriteRedisEntityRegionAccessStrategy.generateCacheKey(ReadWriteRedisEntityRegionAccessStrategy.java:54) ~[hibernate-redis-2.4.0.jar:na]
at org.hibernate.event.internal.DefaultLoadEventListener.getFromSharedCache(DefaultLoadEventListener.java:644) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:595) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:462) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:219) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.returnNarrowedProxy(DefaultLoadEventListener.java:310) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:270) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:121) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:89) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1129) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.internal.SessionImpl.internalLoad(SessionImpl.java:1022) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:639) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.EntityType.resolve(EntityType.java:431) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.EntityType.nullSafeGet(EntityType.java:262) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.persister.collection.AbstractCollectionPersister.readElement(AbstractCollectionPersister.java:839) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.PersistentBag.readFrom(PersistentBag.java:95) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.process.internal.CollectionReferenceInitializerImpl.finishUpRow(CollectionReferenceInitializerImpl.java:77) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.readRow(AbstractRowReader.java:121) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractResults(ResultSetProcessorImpl.java:122) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:122) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:86) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.collection.plan.AbstractLoadPlanBasedCollectionInitializer.initialize(AbstractLoadPlanBasedCollectionInitializer.java:88) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:688) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:75) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.internal.SessionImpl.initializeCollection(SessionImpl.java:1991) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:570) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:252) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:566) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:135) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:164) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:149) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:252) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:148) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.PersistentBag.isEmpty(PersistentBag.java:266) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at my.service.impl.MyObjectServiceImpl.lambda$getObjectsWithObjectStringByKeyAndReportDate$0(MyObjectServiceImpl.java:126) ~[classes/:na]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) ~[na:1.8.0_121]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[na:1.8.0_121]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_121]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_121]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747) ~[na:1.8.0_121]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721) ~[na:1.8.0_121]
at java.util.stream.AbstractTask.compute(AbstractTask.java:316) ~[na:1.8.0_121]
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) ~[na:1.8.0_121]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[na:1.8.0_121]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[na:1.8.0_121]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[na:1.8.0_121]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[na:1.8.0_121]
Caused by: java.lang.NullPointerException: null
at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:57) ~[na:1.8.0_121]
at sun.reflect.UnsafeObjectFieldAccessorImpl.get(UnsafeObjectFieldAccessorImpl.java:36) ~[na:1.8.0_121]
at java.lang.reflect.Field.get(Field.java:393) ~[na:1.8.0_121]
at org.hibernate.property.access.spi.GetterFieldImpl.get(GetterFieldImpl.java:39) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
... 53 common frames omitted

我有以下 hibernate 实体结构:

MyObjectEntity:

@Data
@Entity
@Table(name = "MY_OBJECT_TABLE", schema = "MYSCHEMA")
@Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
@IdClass(MyObjectEntityPK.class)
@JsonAutoDetect(fieldVisibility = JsonAutoDetect.Visibility.ANY)
@JsonIgnoreProperties({"hibernateLazyInitializer", "handler"})
public class MyObjectEntity implements Serializable {

private static final long serialVersionUID = 1L;

@Id
@ManyToOne(fetch = FetchType.LAZY)
@Fetch(FetchMode.SELECT)
@JoinColumns(value = {
        @JoinColumn(name = "OBJECTID", referencedColumnName = "OBJECTID", insertable = false,
                updatable = false),
        @JoinColumn(name = "REPORT_DATE", referencedColumnName = "REPORT_DATE", insertable = false,
                updatable = false),
        @JoinColumn(name = "LASTGENERATION", referencedColumnName = "GENERATION", insertable = false,
                updatable = false)},
        foreignKey = @ForeignKey(value = ConstraintMode.CONSTRAINT))
private AnotherObject anotherObject;

@JsonBackReference
//Jackson – Bidirectional Relationships. To avoid "Could not write content: Infinite recursion (StackOverflowError)"
@OneToMany(mappedBy = "myObject")
@Cache(usage = CacheConcurrencyStrategy.NONSTRICT_READ_WRITE)
private List<MyObjectString> myObjectStrings;

嵌入式主键:MyObjectEntityPK

@Data
class MyObjectEntityPK implements Serializable {

private static final long serialVersionUID = 1L;

@Id
@ManyToOne(fetch = FetchType.LAZY, targetEntity = AnotherObject.class)
@JoinColumns(value = {
        @JoinColumn(name = "OBJECTID", referencedColumnName = "OBJECTID", insertable = false,
                updatable = false),
        @JoinColumn(name = "REPORT_DATE", referencedColumnName = "REPORT_DATE", insertable = false,
                updatable = false)},
        foreignKey = @ForeignKey(value = ConstraintMode.CONSTRAINT))
private AnotherObject anotherObject;

}

AnotherObject 实体:

@Data
@Entity
@Table(name = "ANOTHER_OBJECT", schema = "MYSCHEMA")
@Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
@IdClass(AnotherObjectPK.class)
@JsonAutoDetect(fieldVisibility = JsonAutoDetect.Visibility.ANY)
@JsonIgnoreProperties({"hibernateLazyInitializer", "handler"})
public class AnotherObject implements Serializable {
...
 }

嵌入式主键:AnotherObjectPK

@Data
class AnotherObjectPK implements Serializable {

private static final long serialVersionUID = 1L;

@Id
@Column(name = "OBJECTID", nullable = false, unique = true)
private Integer id;

@Id
@Column(name = "REPORT_DATE")
private Date reportDate;
}

最佳答案

在我看来，问题在于您试图同时从多个线程初始化您的案例 myObjectStrings 中的惰性集合。在我看来，你需要初始化两者

List<MyObject> myObjectList

和

d.getMyObjectStrings()

在任何并行性开始之前提前。在我的第一个答案中，我要求您提前初始化 myObjectList 现在我告诉您另外初始化 d.getMyObjectStrings()

关于java - 使用 Java 8 Stream(列表中的大量数据)时的性能问题，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/49112577/

上一篇：java - 如何在 Java 中计算给定约束的对象数组的最大值组合？

下一篇：java - 第一个 while 循环和第二个 while 循环有什么区别？

相关文章：

java - apache Camel Web 服务客户端

java - Java徒手绘图，添加绘图到JPanel

java - 重写compareTo()方法无法按预期工作

python - 如何管理服务器上的 CPU 密集型进程

c++ - 如果使用 unordered_set，O(NLogN) 显示出比 O(N) 更好的性能

java - 无法找出 FileReader 的问题

c# - Windows 窗体 : using BackgroundImage slows down drawing of the Form's controls

java-8 - 带有 lambda 参数的 kotlin 日志记录

java - DoubleStream 和 LongStream 的范围方法

lambda - 在 Java 8 中将不同类型映射合并为一个

©2024 IT工具网  联系我们