Why does a parallel stream processing with lambda in the static initializer block with forEachOrdered produces a deadlock, but not with forEach?

Question

While playing with Java parallel streams, I experienced deadlocks when some parallel operations are done within a static initializer block.

When using a sequential Stream, everything works fine:

import java.util.Arrays;
public class Example1 {
    static {
        // displays the numbers from 1 to 10 ordered => no thread issue
        Arrays.asList(1,2,3,4,5,6,7,8,9,10)
             .forEach(s->System.out.println(s));
    }
    public static final void main(String[] args) {}
}

When processing the stream in parallel, everyting work (the numbers are displayed without order):

import java.util.Arrays;
public class Example2 {
    static {
        // displays the numbers from 1 to 10 unordered => no thread issue
        Arrays.asList(1,2,3,4,5,6,7,8,9,10).parallelStream()
             .forEach(s->System.out.println(s));
    }
    public static final void main(String[] args) {}
}

However, when processing the Stream with forEachOrdered(), a deadlock occurs (I suppose this is related to the interaction between the main thread and the ForkJoinPool management):

import java.util.Arrays;
public class Example3 {
    static {
        // hangs forever (deadlock between the main thread which loads the class and the underlying ForkJoinPool which join several tasks)
        Arrays.asList(1,2,3,4,5,6,7,8,9,10).parallelStream()
                .forEachOrdered(s->System.out.println(s));
    }
    public static final void main(String[] args) {}
}

But when spawning the Stream processing in a separate Thread, everything goes well:

import java.util.Arrays;
public class Example4 {
    static {
        // displays the numbers from 1 to 10 ordered => no thread issue
        new Thread(()->
            Arrays.asList(1,2,3,4,5,6,7,8,9,10).parallelStream()
                 .forEachOrdered(s->System.out.println(s))
        ).start();
    }
    public static final void main(String[] args) {}
}

From what I've seen from the Thread Dump, the main Thread is waiting on the ForkJoinPool used in the .forEachOrdered() to finish his work, but the first worker Thread in the pool is blocked waiting for something (most probably blocked by the main thread).

I would really appreciate to understand why the deadlock occurs in some cases and not in other cases. This is obviously not due only to the usage of static initializer block, parallel stream and lambda because Example2, Example3 and Example4 use these three concepts, but only Example3 causes a deadlock.

While this question may look like a duplicate of Why does parallel stream with lambda in static initializer cause a deadlock?, it is not. My question goes beyond the linked one as it provide Example2 for which we have static initializer block, parallel stream and lambda, but no deadlock. This is why the question title contains "may lead to deadlock but not necessarily".

Answer 1

This deadlock behavior has two root causes:

1. The main Thread is waiting that another Thread (let's say OtherThread) finishes its work (in the Example3, the OtherThread is one of the Thread of the ForkJoinPool used by the forEachOrdered() operation)
2. The OtherThread uses a Lambda expression which will be defined by the main Thread but later (recall: Lambdas are created at runtime, not at compile time). In the Example3, this Lambda is the one in the .forEachOrdered().

Let's review the examples and explain why they produce or not a deadlock.

Example1

Only one Thread (main) does the following operations:

1. processes the static initializer block
2. do the foreach on each element
3. creates the lambda expression at runtime when processing the first stream element

Since there is only one thread, no deadlock can occur.

Example2

In order to have a better understanding of the processing, we can rewrite it as :

import java.util.Arrays;
public class Example2Instrumented {
    static {
        // displays the numbers from 1 to 10 unordered => no thread issue
        System.out.println(Thread.currentThread().getName()+" : "+"static initializer");
        Arrays.asList(1,2,3,4,5,6,7,8,9,10)
             .parallelStream()
             .forEach(s->System.out.println(Thread.currentThread().getName()+" : "+s));
    }
    public static final void main(String[] args) {}
}

This produces the following result:

main : static initializer
main : 7
main : 6
ForkJoinPool.commonPool-worker-2 : 9
ForkJoinPool.commonPool-worker-4 : 5
ForkJoinPool.commonPool-worker-9 : 3
ForkJoinPool.commonPool-worker-11 : 2
ForkJoinPool.commonPool-worker-2 : 10
ForkJoinPool.commonPool-worker-4 : 4
ForkJoinPool.commonPool-worker-9 : 1
ForkJoinPool.commonPool-worker-13 : 8

The main Thread processes the static initializer, then starts the forEach and build the lambda at runtime when processing the first element. The other stream elements are processed by the workers Threads from the ForkJoinPool. There is no deadlock because the main Thread processed the first element and built the lambda.

Example3

We can rewrite Example3 without the Lambda to break the deadlock:

import java.util.Arrays;
import java.util.function.Consumer;
public class Example3NoDeadlock {
    static {
        // displays the numbers from 1 to 10 ordered => no thread issue anymore
        Arrays.asList(1,2,3,4,5,6,7,8,9,10).parallelStream()
                .forEachOrdered(
                    new Consumer<Integer>() {
                        @Override
                        public void accept(Integer t) {
                            System.out.println(t);          
                    }});
    }
    public static final void main(String[] args) {}
}

Since the Consumer Class is constructed at compile time (contrary to lambdas that are built at runtime), this breaks the deadlock cycle. This prooves that at least the lambda is involved in the deadlock.

To have a better understanding, we could instrument the code as follow:

import java.util.Arrays;
import java.util.function.Consumer;
public class Example3Instrumented {
    static {
        System.out.println("static initializer");
        // hangs forever (deadlock between the main thread which loads the class and the underlying ForkJoinPool which join several tasks)
        Arrays.asList(1,2,3,4,5,6,7,8,9,10).parallelStream()
            .peek(new Consumer<Integer>() {
                @Override
                public void accept(Integer t) {
                        System.out.println(Thread.currentThread().getName()+" "+t);
            }})
                .forEachOrdered(s->System.out.println(s));
    }
    public static final void main(String[] args) {}
}

This produces the following output:

main : static initializer
ForkJoinPool.commonPool-worker-6 1
ForkJoinPool.commonPool-worker-9 3
main 7
ForkJoinPool.commonPool-worker-4 2
ForkJoinPool.commonPool-worker-13 6
ForkJoinPool.commonPool-worker-11 8
ForkJoinPool.commonPool-worker-15 5
ForkJoinPool.commonPool-worker-2 9
ForkJoinPool.commonPool-worker-4 10
ForkJoinPool.commonPool-worker-9 4

The main Thread processes the static initializer, then starts processing the forEachOrdered by creating a Task for each element in the stream (to maintain the order, a complex tree-based algorithm is used, see ForEachOps.ForEachOrderedTask: tasks are created and it looks from the code that there is each task is waiting that another task is completed to run). All the tasks are submitted to the ForkJoinPool. I think the deadlock occures because the first Task is processed by a worker Thread from the ForkJoinPool and this Thread waits on the main Thread to build the lambda. And the main Thread has already started processing its Task and is waiting for another worker thread to complete its Task to run. Hence the deadlock.

Example4

In the Example4, we spawn a new Thread that is ran asynchronously (i.e. we don't wait for the result). This is why the main Thread is not locked and has now the time to build the Lambdas at runtime.

Conclusion

The takeaway lesson is : if you mix static initializers, threads and lambdas, you should really understand how these concepts are implemented, otherwise you may have deadlocks.