Intermittent error org.hibernate.PersistentObjectException: detached entity passed to persist

Question

I am getting the exception org.hibernate.PersistentObjectException: detached entity passed to persist. From the numerous posts on this forum and elsewhere, I understand that this happens in two cases(not considering One-One annotations etc),

1. There is an issue with the transaction going out of scope
2. An id is set where it should be automatically generated.

I see neither of these happening with my code. I am unable to reproduce the error, because I don't have the data which initially triggered it. On other data it runs perfectly fine. I have provided an SCCE below:

public class MyProcessor {
    private MyImportEJB myEJB = MyImportEJB.getInstance();
    private List<MyClass> saveQueue = new ArrayList<MyClass>();
    public void process() {
        List<X> rawData = getListOfX();
        for(X x:rawData) {
             processX();
         }
         saveFoos(saveQueue);   
     }

     public void saveOrUpdateFoos(List<Foo> foos) {

        for(MyClass foo:foos) {
              MyClass existingFoo = myEJB.getFoosForWidAndDates(foo.getWid(), foo.getEffBeginDt(),foo.getEffEndDt());
              if(existingFoo == null) saveQueue.add(foo);
              else {
                   existingFoo.updateIfDifferent(foo);
                   saveQueue.add(existingFoo);
              }
         }

         if(saveQueue.size() > 5000) {
             myEJB.saveObjects(saveQueue);
             saveQueue.clear();
         }
     }

     public void processX() {
          ArrayList<MyClass> foos = new ArrayList<MyClass>();

          if(x.reportPeriod != null && x.gravity != null){
              MyClass foo = new MyClass();
              foo.setId(null);
              foo.setWId(x.getWid());
              foo.setEffBeginDt(x.reportPeriod);
              foo.setEffEndDt(addOneMonth(x.reportPeriod));
              foo.setGravity(x.gravity);

              foos.add(foo);
          }
          saveOrUpdateFoos(foos);

     }
 }

MyImportEJB.java:

@Stateless
@EJB(name = "MyImportEJB", beanInterface = MyImportEJB.class)
@TransactionAttribute(TransactionAttributeType.SUPPORTS)
@PermitAll
public class MyImportEJB{
    @Override
    @TransactionAttribute(TransactionAttributeType.REQUIRED)
    public void saveObjects(List<? extends P> mappedObjects) 
    {
        for (P mappedObject : mappedObjects)
        {
            this.saveObject(mappedObject);
        }
    }


    @Override
    @TransactionAttribute(TransactionAttributeType.REQUIRED)
    public void saveObject(P mappedObject) 
    {
        EntityManager entityManager = this.getEntityManager();

        Object identifier = this.getEntityManagerFactory().getPersistenceUnitUtil().getIdentifier(mappedObject);
        if (identifier != null) {
            Object existingObject = entityManager.find(mappedObject.getClass(), identifier);

            if (existingObject != null) {
                entityManager.merge(mappedObject);
                return;
            }
         }

         entityManager.persist(mappedObject);
    }

    public MyClass getFoosForWidAndDates(Integer wid, Calendar effBeginDt, Calendar effEndDt) {
         try {
            return (MyClass)((this.entityManager
        .createQuery("select M from MyClass M where wid = :wid and effBeginDt = :effBeginDt and effEndDt = :effEndDt ", MyClass.class)
        .setParameter("wid",wid)
        .setParameter("effBeginDt", effBeginDt)
        .setParameter("effEndDt", effEndDt)).getSingleResult());
         } catch(NoResultException | NonUniqueResultException e) {
            return null;
        }
    }
 }

MyClass.java

public MyClass{

      @Id
      @GeneratedValue(strategy=GenerationType.IDENTITY)
      @Column(name = "Id")
      private Integer id;

      @Column(name = "wid")
      private Integer wId;

      @Column(name = "eff_begin_dt")
      private Calendar effBeginDt;

      @Column(name = "eff_end_dt")
      private Calendar effEndDt;

      @Column(name = "gravity")
      private Double gravity;

      private Integer dataDownloadId;

      public void updateIfDifferent(MyClass other) {
          if(other.gravity != null && other.gravity != this.gravity) this.gravity = other.gravity;
          //same for effBeginDt andeffEndDt

      }

 }

persistence.xml

  <?xml version="1.0" encoding="UTF-8" ?>

 <persistence xmlns="http://java.sun.com/xml/ns/persistence"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://java.sun.com/xml/ns/persistence
http://java.sun.com/xml/ns/persistence/persistence_2_0.xsd"
version="2.0">
    <persistence-unit name="ProdData">
        <description>ProdData Database Persistence Unit</description>
        <provider>org.hibernate.ejb.HibernatePersistence</provider>
        <jta-data-source>java:jboss/ProdDataJNDI</jta-data-source>
        <class>path.to.MyClass</class>
        <class>path.to.X</class>
        <exclude-unlisted-classes>true</exclude-unlisted-classes>

        <properties>
            <property name="hibernate.show_sql" value="false" />
         </properties>
    </persistence-unit>
 </persistence>

The exception is thrown on calling entityManager.persist(mappedObject) <- MyImportEJB.saveObject <-MyImportEJB.saveObjects. I dont have the line number

I have tried writing a sample program where I get an existingFoo object from the database, update and save it, because that was the most likely source of the error. But I could not reproduce the error. Please help.

EDIT: Here are the details of getListofX() as requested

from MyProcessor.java:

public List<X> getListOfX() {
    return myImportEJB.getUnprocessedIds(X.class, 30495);
}

from the file MyImportEJB.java:

@TransactionAttribute(TransactionAttributeType.SUPPORTS)
public List<Integer> getUnprocessedIds(Class<? extends ProductionRawData> clazz, Integer dataDownloadId) {
    String canonicalName = clazz.getCanonicalName();

    String queryStr = "select id from " + canonicalName + " where datadownloadId = :dataDownloadId and isProcessed != 1";

    TypedQuery<Integer> query = this.entityManager.createQuery(queryStr, Integer.class)
        .setParameter("dataDownloadId", dataDownloadId);
    try {
        return query.getResultList();
    } catch(NoResultException nre) {
         return new ArrayList<T>();
    }
}

EDIT: Also added the details of getFoosForWidAndDates(). It was suggested to me that I set the id on a new Foo to null before adding it to the save queue. I would like to know if it is possible that the id is being set "under the hood" by Hibernate to an unacceptable value

Answer 1

I think this could be your issue

 if(saveQueue.size() > 5000) {
     myEJB.saveObjects(saveQueue);
     saveQueue.clear();
 }

You are filling the saveQueue up to 5000, and then you try to persist / merge them. I'm not sure about Hibernate, but EclipseLink's default shared cache size is 1000. That means, you could fill up saveQueue with, for example, 2000 existing Foos and, by the time you call persist() on them, they are no longer in entity manager's cache. This leads to calling persist() on detached instance which has id already set.

In your case, I think it would be perfectly OK to call entityManager.merge(mappedObject) on all objects, regardless of weather it is an existing object or not, because it will update existing ones, and persist new ones.

Answer 2

I hope you are using latest Hibernate. And I expect that it is a typo here:

if(existingFoo == null) saveQueue.add(existingFoo); // if it is a null, you add it?

Otherwise it looks quite interesting, you could try to do following things:

1. Flush session after each (10, 1000) persisted objects, like that:

   public void saveObjects(List<? extends P> mappedObjects) 
   {
       for (P mappedObject : mappedObjects)
       {
           this.saveObject(mappedObject);
           // some condition if you have a lot of objects
           this.entityManager.flush();
       }
   }
   

2. Use less exotic way to check if entity was persisted already:

   public void saveObject(P mappedObject) {
       EntityManager entityManager = this.getEntityManager();
   
       if (mappedObject.getId() != null) {
           entityManager.merge(mappedObject);
           return;
        }
   
        entityManager.persist(mappedObject);
   }
   

   Or it is better completely avoid merge and just work with managed objects, this will require a single transaction for all operations, thus will require redesign of the things.

Answer 3

I ran the code on the production server again and I got a TransactionRollbackException right before the detached entity exceptions. I think that this might be the actual cause of the exception I observed.

The transaction timed out and was rolled back due to the huge number of objects I was trying to save, and this led to the Foo objects becoming detached. Somehow the TransactionRollbackException didnt show up in the logs the first time around, probably because it happened exactly at 12:0am UT. In support of this theory, the program hasn't crashed so far after I decreased the batch size to ~1000

For those interested, a TransactionRollbackException does lead to entities getting detached: entityManager.getTransaction().rollback() detaches entities?

There are ways to handle this when it occurs, but I opted against implementing them for simplicity's sake