Deadlock happens if I use lambda in parallel stream but it doesn't happen if I use anonymous class instead?

Question

The following code leads to deadlock(on my pc):

public class Test {
    static {
        final int SUM = IntStream.range(0, 100)
                .parallel()
                .reduce((n, m) -> n + m)
                .getAsInt();
    }

    public static void main(String[] args) {
        System.out.println("Finished");
    }
}

But if I replace reduce lambda argument with anonymous class it doesn't lead to deadlock:

public class Test {
    static {
        final int SUM = IntStream.range(0, 100)
                .parallel()
                .reduce(new IntBinaryOperator() {
                    @Override
                    public int applyAsInt(int n, int m) {
                        return n + m;
                    }
                })
                .getAsInt();
    }

    public static void main(String[] args) {
        System.out.println("Finished");
    }
}

Could you explain that situation?

P.S.

I found that code(a bit different from previous):

public class Test {
    static {
        final int SUM = IntStream.range(0, 100)
                .parallel()
                .reduce(new IntBinaryOperator() {
                    @Override
                    public int applyAsInt(int n, int m) {
                        return sum(n, m);
                    }
                })
                .getAsInt();
    }

    private static int sum(int n, int m) {
        return n + m;
    }

    public static void main(String[] args) {
        System.out.println("Finished");
    }
}

works not stable. In most cases it hangs buts sometimes it finishes successfully:

I really not able to understand why this behaviour is not stable. Actually I retest first code snippet and behaviour the same. So the latest code is equals the first one.

To understand which threads are used I added following "logging":

public class Test {
    static {
        final int SUM = IntStream.range(0, 100)
                .parallel()
                .reduce((n, m) -> {
                    System.out.println(Thread.currentThread().getName());
                    return (n + m);
                })
                .getAsInt();
    }

    public static void main(String[] args) {
        System.out.println("Finished");
    }
}

For case when application finishes successfully I see following log:

main
main
main
main
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
main
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
main
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
ForkJoinPool.commonPool-worker-1
Finished

P.S. 2

I Undestand that reduce is enough complex operations. I found a simpler example to show that problem:

public class Test {
    static {
        System.out.println("static initializer: " + Thread.currentThread().getName());

        final long SUM = IntStream.range(0, 2)
                .parallel()
                .mapToObj(i -> {
                    System.out.println("map: " + Thread.currentThread().getName() + " " + i);
                    return i;
                })
                .count();
    }

    public static void main(String[] args) {
        System.out.println("Finished");
    }
}

for happy case(rare case) I seee following output:

static initializer: main
map: main 1
map: main 0
Finished

example of happy case for extended stream range:

static initializer: main
map: main 2
map: main 3
map: ForkJoinPool.commonPool-worker-2 4
map: ForkJoinPool.commonPool-worker-1 1
map: ForkJoinPool.commonPool-worker-3 0
Finished

example of case which leads to deadlock:

static initializer: main
map: main 1

It also leads to deadlock but not for each start.

Had doubts for a second there, but yes, it's an exact duplicate. In the question he says if he removes the lambda then the code works, so it's exactly the same problem you've described. — Michael, Dec 10 '18 at 13:04
@gstackoverflow it is explicit in that duplicate - as soon as you use multi threaded code in static initializer, you are in deep trouble; exactly what happens to you. there is not "work-around" btw - just don't use `parallel` for that stream operation, it does not increase your performance anyway... — Eugene, Dec 10 '18 at 13:05
@Michael but there are no explanation to this behaviour in answer set — gstackoverflow, Dec 10 '18 at 13:06
@gstackoverflow well there *is* an answer that explains that, doesn't it? — Eugene, Dec 10 '18 at 13:08
@gstackoverflow *point you out*? what is that suppose to mean? — Eugene, Dec 10 '18 at 13:10
@Eugene, please provide me link to the answer which explains difference in behaviour between using lambda and anonymous class in my example — gstackoverflow, Dec 10 '18 at 13:13
@gstackoverflow All of the information is there. We should not need to hold your hand through it. I can't believe that you've possibly read every single answer and both JDK bug tickets thoroughly in this time. — Michael, Dec 10 '18 at 13:16
@Michael, I read all answes to that topic but I didn't read JDK bug tickets — gstackoverflow, Dec 10 '18 at 13:19
@Michael, if you mean https://stackoverflow.com/a/34820539/2674303 that answer - both code snippets works without deadlock on my mashine — gstackoverflow, Dec 10 '18 at 13:22

score 6 · Accepted Answer · answered Dec 10 '18 at 15:55

6

The difference is that lambda body is written in the same Test class, i.e. a synthetic method

private static int lambda$static$0(int n, int m) {
    return n + m;
}

In the second case the implementation of the interface resides in a different Test$1 class. So the threads of a parallel stream do not call static methods of Test and thus do not depend on Test initialization.

answered Dec 10 '18 at 15:55

apangin

79,047
9
168
200

Thanks for your response. I have additional related question. I added it to the topic - I will appreciate your explanation of that race condition. – gstackoverflow Dec 10 '18 at 16:01
1

@gstackoverflow Just by definition of a [race condition](https://en.wikipedia.org/wiki/Race_condition). The outcome depends on thread timings, their relative execution order, which is unspecified. – apangin Dec 10 '18 at 19:21
Absolutely agree. It was a sign why I decided that it is a race condition. But I can't imagine "happy case" for that situation. Anyway **Test** initialization class starts first and thread from fork join pool starts before initialization finish.So from my point of view I expect stable deadlock) But looks like I wrong somewhere, and I am very curious to know where;) – gstackoverflow Dec 10 '18 at 20:54
@gstackoverflow Hint: one of the "parallel" threads is the `main` thread itself. – apangin Dec 10 '18 at 21:24
So we have 2 threads: Main and fork-join-thread **main()** method invocation leads to the **Test** class initialization by main thread. So main thread starts to calculate stream expression. Because of **parallel()** stream will calculate **reduce** lambda by thread from fork join pool. reduce body lambda is a static method of **Test** class. So to calculate lambda fork-join-thread has to access lambda. But lambda is not available because Test class is not initialized yet. From other hand Test class is not able to be initialized because of stream can't be calculated – gstackoverflow Dec 11 '18 at 07:30
Please, correct where I wrong – gstackoverflow Dec 11 '18 at 07:30
@gstackoverflow you are the first one to use the F/J pool in this JVM. So there is no worker thread yet. Starting a new thread takes time. So there are chances that the main thread completes this trivial stream operation, before the worker thread gets to the point to pick up a work chunk. To be more precise, the main thread only needs to have started processing of the last chunk that has not divided further, to prevent other threads from picking up a chunk. – Holger Dec 11 '18 at 07:49
@Holger, I added thread name logging.When application finishes succesfully I see that 2 threads are used for lambda calculation – gstackoverflow Dec 11 '18 at 08:58
@gstackoverflow it seems, a successful invocation from the main thread makes the invocation instruction eligible to proceed even from other threads. For that, it’s helpful that `reduce` allows threads to proceed without waiting for each other. Compare with [this answer](https://stackoverflow.com/a/53055952/2711488). – Holger Dec 11 '18 at 09:02
@Holger, anyway I can't understand it. I found a simpler example with the simple example. Could you please provide trace for happy case and deadlock case ? – gstackoverflow Dec 11 '18 at 10:06
@gstackoverflow your simpler case is an exact match of what has been described in [this comment](https://stackoverflow.com/questions/53706189/deadlock-happens-if-i-use-lambda-in-parallel-stream-but-it-doesnt-happen-if-i-u/53709217?noredirect=1#comment94292452_53709217) already. In the first output, your main thread has processed both elements before another worker thread arrived, in the second, the worker thread was slightly faster and got blocked. – Holger Dec 11 '18 at 10:12
@Holger, I extended stream range a bit and I see following output: `static initializer: main map: main 2 map: main 3 map: ForkJoinPool.commonPool-worker-2 4 map: ForkJoinPool.commonPool-worker-1 1 map: ForkJoinPool.commonPool-worker-3 0 Finished` – gstackoverflow Dec 11 '18 at 10:27
So it might work without deadlock even when the worker thread was started – gstackoverflow Dec 11 '18 at 10:29
1

@gstackoverflow I already acknowledged the phenomenon in [this comment](https://stackoverflow.com/questions/53706189/deadlock-happens-if-i-use-lambda-in-parallel-stream-but-it-doesnt-happen-if-i-u/53709217?noredirect=1#comment94294627_53709217) and linked to an older answer describing a similar behavior. But the question why worker threads can suddenly access an uninitialized class in this scenario, when the main thread has already executed the same code, is way to broad to be discussed in the “P.S.” section of another question. – Holger Dec 11 '18 at 10:32
1

@gstackoverflow According to [JVMS §5.5](https://docs.oracle.com/javase/specs/jvms/se8/html/jvms-5.html#jvms-5.5), `invokedynamic` instruction does not trigger class initialization, but the **first** invocation of MethodHandle does. The key is which thread invokes the MethodHandle first. – apangin Dec 11 '18 at 10:51
2

If you have further questions, please ask a new one rather than discussing in the comments. – apangin Dec 11 '18 at 10:53
2

@apangin the `invokedynamic` instruction is always executed by the main thread, as it will create the `IntBinaryOperator` instance which is passed to `reduce`, before the parallel reduction even starts. But I agree that this should be discussed in a dedicated Q&A rather than comments. – Holger Dec 11 '18 at 11:22
@apangin, I will create separated topic but I afraid it will be marked as duplicate 5 seconds after publication despite the fact it is not a deuplicate) – gstackoverflow Dec 11 '18 at 12:45
@Holger, please elaborate a bit your comment – gstackoverflow Dec 11 '18 at 13:01

Deadlock happens if I use lambda in parallel stream but it doesn't happen if I use anonymous class instead?

P.S.

P.S. 2

1 Answers1

Linked