Performance Issue when using Java 8 Stream (Bulk of Data in a List)

Question

I have the following 2 codes/logics in a single method:

Code 1: Returns around 10K++ of data

    //myObjectRepository extends the spring CrudRepository (hibernate)
    final List<MyObject> myObjectList =
            myObjectRepository.getObjectsByIdAndReportDate();

Code 2: Filters the 10k++ of data

    //Only return valid objects
    final List<MyObject> objectsWithStrings = myObjectList
            //parallelStream() randomly throw an exception related to hibernate
            .stream()
            .filter(d -> d.getMyObjectStrings() != null && !d.getMyObjectStrings().isEmpty())
            .collect(Collectors.toList());

Note: MyObject is a hibernate entity with the following structure:

@JsonBackReference
//Jackson – Bidirectional Relationships. To avoid "Could not write content: Infinite recursion (StackOverflowError)"
@OneToMany(mappedBy = "myObject")
@Cache(usage = CacheConcurrencyStrategy.NONSTRICT_READ_WRITE)
private List<MyObjectString> myObjectStrings;

It is working as it is but I am encountering the following in Code 2:

a performance problem in my Code 2.
a random hibernate problem whenever I used the parallelStream(), thus, I opted to use the sequential stream() instead

In this case, is there a way for me to optimize code 2 so it can process the data or filter the data faster?

Snippet of the hibernate error I was encountering:

2018-03-05 15:35:28.431 ERROR 10056 --- [nio-8082-exec-1] e.b.MyServiceExceptionControllerAdvice : An generic error is encountered [400] [Error accessing field [private java.util.Date my.entity.AnotherObjectPK.reportDate] by reflection for persistent property [my.entity.AnotherObjectPK#reportDate] : null]

org.hibernate.property.access.spi.PropertyAccessException: Error accessing field [private java.util.Date my.entity.AnotherObjectPK.reportDate] by reflection for persistent property [my.entity.AnotherObjectPK#reportDate] : null
at org.hibernate.property.access.spi.GetterFieldImpl.get(GetterFieldImpl.java:43) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.tuple.component.AbstractComponentTuplizer.getPropertyValue(AbstractComponentTuplizer.java:58) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.ComponentType.getPropertyValue(ComponentType.java:419) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.ComponentType.getHashCode(ComponentType.java:242) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.cache.internal.CacheKeyImplementation.calculateHashCode(CacheKeyImplementation.java:57) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.cache.internal.CacheKeyImplementation.<init>(CacheKeyImplementation.java:53) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.cache.internal.DefaultCacheKeysFactory.staticCreateEntityKey(DefaultCacheKeysFactory.java:50) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.cache.redis.hibernate5.strategy.ReadWriteRedisEntityRegionAccessStrategy.generateCacheKey(ReadWriteRedisEntityRegionAccessStrategy.java:54) ~[hibernate-redis-2.4.0.jar:na]
at org.hibernate.event.internal.DefaultLoadEventListener.getFromSharedCache(DefaultLoadEventListener.java:644) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.loadFromSecondLevelCache(DefaultLoadEventListener.java:595) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:462) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:219) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.returnNarrowedProxy(DefaultLoadEventListener.java:310) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.proxyOrLoad(DefaultLoadEventListener.java:270) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:121) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:89) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1129) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.internal.SessionImpl.internalLoad(SessionImpl.java:1022) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.EntityType.resolveIdentifier(EntityType.java:639) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.EntityType.resolve(EntityType.java:431) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.type.EntityType.nullSafeGet(EntityType.java:262) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.persister.collection.AbstractCollectionPersister.readElement(AbstractCollectionPersister.java:839) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.PersistentBag.readFrom(PersistentBag.java:95) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.process.internal.CollectionReferenceInitializerImpl.finishUpRow(CollectionReferenceInitializerImpl.java:77) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.readRow(AbstractRowReader.java:121) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractResults(ResultSetProcessorImpl.java:122) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:122) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:86) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.loader.collection.plan.AbstractLoadPlanBasedCollectionInitializer.initialize(AbstractLoadPlanBasedCollectionInitializer.java:88) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:688) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.event.internal.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:75) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.internal.SessionImpl.initializeCollection(SessionImpl.java:1991) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:570) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:252) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:566) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:135) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:164) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection$1.doWork(AbstractPersistentCollection.java:149) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:252) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.AbstractPersistentCollection.readSize(AbstractPersistentCollection.java:148) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at org.hibernate.collection.internal.PersistentBag.isEmpty(PersistentBag.java:266) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
at my.service.impl.MyObjectServiceImpl.lambda$getObjectsWithObjectStringByKeyAndReportDate$0(MyObjectServiceImpl.java:126) ~[classes/:na]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174) ~[na:1.8.0_121]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374) ~[na:1.8.0_121]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) ~[na:1.8.0_121]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) ~[na:1.8.0_121]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:747) ~[na:1.8.0_121]
at java.util.stream.ReduceOps$ReduceTask.doLeaf(ReduceOps.java:721) ~[na:1.8.0_121]
at java.util.stream.AbstractTask.compute(AbstractTask.java:316) ~[na:1.8.0_121]
at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) ~[na:1.8.0_121]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[na:1.8.0_121]
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[na:1.8.0_121]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[na:1.8.0_121]
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[na:1.8.0_121]
Caused by: java.lang.NullPointerException: null
at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:57) ~[na:1.8.0_121]
at sun.reflect.UnsafeObjectFieldAccessorImpl.get(UnsafeObjectFieldAccessorImpl.java:36) ~[na:1.8.0_121]
at java.lang.reflect.Field.get(Field.java:393) ~[na:1.8.0_121]
at org.hibernate.property.access.spi.GetterFieldImpl.get(GetterFieldImpl.java:39) ~[hibernate-core-5.0.12.Final.jar:5.0.12.Final]
... 53 common frames omitted

I have the following hibernate entities structure:

MyObjectEntity:

@Data
@Entity
@Table(name = "MY_OBJECT_TABLE", schema = "MYSCHEMA")
@Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
@IdClass(MyObjectEntityPK.class)
@JsonAutoDetect(fieldVisibility = JsonAutoDetect.Visibility.ANY)
@JsonIgnoreProperties({"hibernateLazyInitializer", "handler"})
public class MyObjectEntity implements Serializable {

private static final long serialVersionUID = 1L;

@Id
@ManyToOne(fetch = FetchType.LAZY)
@Fetch(FetchMode.SELECT)
@JoinColumns(value = {
        @JoinColumn(name = "OBJECTID", referencedColumnName = "OBJECTID", insertable = false,
                updatable = false),
        @JoinColumn(name = "REPORT_DATE", referencedColumnName = "REPORT_DATE", insertable = false,
                updatable = false),
        @JoinColumn(name = "LASTGENERATION", referencedColumnName = "GENERATION", insertable = false,
                updatable = false)},
        foreignKey = @ForeignKey(value = ConstraintMode.CONSTRAINT))
private AnotherObject anotherObject;

@JsonBackReference
//Jackson – Bidirectional Relationships. To avoid "Could not write content: Infinite recursion (StackOverflowError)"
@OneToMany(mappedBy = "myObject")
@Cache(usage = CacheConcurrencyStrategy.NONSTRICT_READ_WRITE)
private List<MyObjectString> myObjectStrings;

EmbeddedPrimary Key: MyObjectEntityPK

@Data
class MyObjectEntityPK implements Serializable {

private static final long serialVersionUID = 1L;

@Id
@ManyToOne(fetch = FetchType.LAZY, targetEntity = AnotherObject.class)
@JoinColumns(value = {
        @JoinColumn(name = "OBJECTID", referencedColumnName = "OBJECTID", insertable = false,
                updatable = false),
        @JoinColumn(name = "REPORT_DATE", referencedColumnName = "REPORT_DATE", insertable = false,
                updatable = false)},
        foreignKey = @ForeignKey(value = ConstraintMode.CONSTRAINT))
private AnotherObject anotherObject;

}

AnotherObject entity:

@Data
@Entity
@Table(name = "ANOTHER_OBJECT", schema = "MYSCHEMA")
@Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
@IdClass(AnotherObjectPK.class)
@JsonAutoDetect(fieldVisibility = JsonAutoDetect.Visibility.ANY)
@JsonIgnoreProperties({"hibernateLazyInitializer", "handler"})
public class AnotherObject implements Serializable {
...
 }

Embedded Primary key: AnotherObjectPK

@Data
class AnotherObjectPK implements Serializable {

private static final long serialVersionUID = 1L;

@Id
@Column(name = "OBJECTID", nullable = false, unique = true)
private Integer id;

@Id
@Column(name = "REPORT_DATE")
private Date reportDate;
}

There is no database operaton in code 2 ? Right? All is in memory processing. — Alexander Petrov, Mar 05 '18 at 14:24
`performance problem` - which? `random hibernate problem` - which? — Eugene, Mar 05 '18 at 14:25
@Eugene as I mentioned, Code 2 is encountering performance problem as it takes around 2 minutes to process the 10k++ data mentioned. — Jown, Mar 05 '18 at 14:28
Possible duplicate of [What is the solution for the N+1 issue in hibernate?](https://stackoverflow.com/questions/32453989/what-is-the-solution-for-the-n1-issue-in-hibernate) — user158037, Mar 05 '18 at 14:49
@Eugene edited the original post to include a snippet of the hibernate problem whenever the parallelStream() is invoked. — Jown, Mar 05 '18 at 15:08
use @Batch(size=10) on the myObjectStrings relationship declaration. — Alexander Petrov, Mar 05 '18 at 15:23
@AlexandarPetrov I have updated the original post to include the whole stacktrace of the error including the structure of my hibernate entities — Jown, Mar 05 '18 at 15:25
If you want performance, don’t use entities linked/attached to the DB. Collect them, detach them (eg make copies), do your processing, then write back any changes if any. — Bohemian, Jun 01 '18 at 09:18

Alexander Petrov · Answer 1 · 2018-03-05T15:41:24.390

0

In my opinion the problem is that you are trying to initialize a Lazy collection in your case myObjectStrings from multiple threads at the same time. In my opinion you need to initialize both

List<MyObject> myObjectList

and

d.getMyObjectStrings()

In advance before any paralelism kicks in. In my first answer I asked you to initialize in advance myObjectList now I am telling you to initialize in addition d.getMyObjectStrings()

edited Mar 05 '18 at 15:41

answered Mar 05 '18 at 15:33

Alexander Petrov

9,723
24
57

how can I initialize the second one, d.getMyObjectStrings()? This is for the reason that this is called after the stream()/parallelStream() :-) – Jown Mar 05 '18 at 15:36
You can use JPAQL and prefetch it. Or mark it as EAGER just for the experiment. – Alexander Petrov Mar 05 '18 at 15:39
Then later you can decide what to do with it if it work out. – Alexander Petrov Mar 05 '18 at 15:40
Here is a link with some options https://www.thoughts-on-java.org/5-ways-to-initialize-lazy-relations-and-when-to-use-them/ – Alexander Petrov Mar 05 '18 at 15:40

Performance Issue when using Java 8 Stream (Bulk of Data in a List)

1 Answers1