Find the median from a stream of integers

Question

Given an unsorted sequence of integers that flows into your program as a stream.

The integers are too many to fit into memory.

Imagine there is a function:

int getNext() throws NoSuchElementException;

It returns the next integer from the stream.

Write a function to find the median.

Solve the problem in O(n).

Any ideas?

Hint is given (use heap the data structure..)

See http://stackoverflow.com/questions/3378506/probability-of-finding-the-median-with-finite-space — deinst, Aug 09 '10 at 14:29
Or [this stackoverflow question](http://stackoverflow.com/questions/2579912/how-do-i-find-the-median-of-numbers-in-linear-time-using-heaps). Which also references prior SO questions... — NealB, Aug 09 '10 at 16:52

score 10 · Answer 1 · answered Aug 09 '10 at 16:05

You have to maintain two heaps one max heap ( which contains the smallest N/2 elements seen till now) and one min heap ( which contains the largest N/2 elements). Store the extra element aside if N is odd.

Whenever you call the function getNext(),

If N becomes odd, save the new element as the extra element. If necessary, exchange that element with one from the min-heap or max-heap to satisfy the following condition

max(max-heap) <= extra element <= min(min-heap).

If N becomes even, do the same as above to get a 2nd extra element. Then, add the smaller one to the max-heap and the larger one to the min-heap. Insert should be O(log N)

Get Median: O(1)
If N is odd, the median is the extra element.
If N is even, the median is the average between the tops of the 2 heaps

How would you cater to "the integers are too many to fit in memory"? — , Aug 09 '10 at 16:08
Not to mention the O(n) time constraint. Even with a Fibonacci heap the running time will be O(n lg n) — deinst, Aug 09 '10 at 17:04

score 2 · Accepted Answer · edited Aug 09 '10 at 18:01

2

See this paper. It will (likely) take more than one pass. The idea is that in each pass upper and lower bounds are computed such that the median lies between them.

A fundamental result here is N = size of data, P = number of passes

Theorem 2) A P-pass algorithm which selects the K^th highest of N elements requires storage at most O(N^(¹/_P)(log N)^(2-²/_P)).

Also, for very small amounts of storage S, i.e., for 2 <= S <= O((log N)²), there is a class of selection algorithms which use at most O((log N)^3/S) passes.

Read the paper. I'm not really sure what the heap has to do with it

edited Aug 09 '10 at 18:01

Svante

46,788
11
77
118

answered Aug 09 '10 at 14:31

deinst

16,749
3
43
45

3

Seems a bit heavy for an interview question. – Henk Holterman Aug 09 '10 at 14:39
1

I suspect that the purpose of the interview question was to figure out how the interviewee approached problems, and not to find a solution. (At least I hope so.) – deinst Aug 09 '10 at 14:42

score 0 · Answer 3 · answered Jun 20 '12 at 08:39

0

Using selection algorithm, we can achieve this in O(n) complexity. But I still dont understand the use of a Heap in this case clearly.

answered Jun 20 '12 at 08:39

Pavan Dittakavi

2,604
5
23
36

score 0 · Answer 4 · answered Mar 27 '12 at 21:20

0

Suppose the window for the median to maintain is K. Construct a binary search tree for K number. O(K); Do in-order traversal and find the (K/2)th element.O(K/2);

The overall time is O(K).

answered Mar 27 '12 at 21:20

willawill

450
6
4

Find the median from a stream of integers

4 Answers4

Linked