5

I missed the class where big-O was introduced thinking that it was pretty straight forward. It still seems to be however the teacher said something about O(n) deviating from the function when n gets very small? I couldn't find this anywhere in the book. Could someone enlighten me? Our exploration of O(n) has been in the context of sorting algorithms if that is of any significance.

Thanks Gene

edit: Thanks for the help guys it has been illuminating. I have a follow-up question. Is there a relatively simple mathematical way to figure out the point where n is too small for O(n)?

Related questions

are there any O(1/n) algorithms?
What is the difference between Θ(n) and O(n)?

Community
  • 1
  • 1
  • I think this would make sense if you omitted the word "very". Are you sure about that? – allyourcode May 08 '09 at 23:36
  • I was taught to always remove the modifier "very" unless it can be replaced with a "damn". I forget whose advice this was.. ;-) – SplittingField May 08 '09 at 23:47
  • Your auxilliary question - look at the constants on the different terms that make up the cost. O(N) is a shorthand that saves time with the constants, but as my example makes clear, you have to know the constants to determine break-even points. Jon Bentley makes the point beautifully in one of his columns in 'Programming Pearls'; he runs a Cray with a cubic algorithm against a TRS-80 with a linear algorithm. Up to about size 5,000 (IIRC), the Cray wins; thereafter, the TRS-80 does - by a large margin. – Jonathan Leffler May 09 '09 at 00:36
  • Thanks Jon. So O(n) is basically the leading coefficient( is that the right word? ) of f(x)[ function for exact number of operations in algorithm ]. Therefore I would have to set f(x) = 0 and solve for x to find the set size where O(n) is no longer useful. Is this correct? –  May 09 '09 at 01:04
  • @Gene: more or less. The notation O(n) notation means that as the size of a problem, N, increases towards infinity, the expression C = a * O(n) becomes more and more accurate, because the contributions of the other terms are dwarfed by the dominant (leading) term. O(n) is not the coefficient itself; it is the expression that is multiplied by a coefficient. Thus, if the the cost C(n) = an^3 + bn^2 + cn + d, as n increases, the coefficient a and the term in n^3 mean that the expression is O(N^3) - the cubic term dominates the cost; C(n) ~= an^3 for big enough values of n. ...continued... – Jonathan Leffler May 09 '09 at 18:38
  • @Gene: continued. To see when O(n) is no longer useful, you have to look at the other terms in the equation and see when they make a significant contribution. In the cubic example (previous comment), the cost C(n) is no longer accurate enough when bn^2 is large enough to be a significant part of the cost. If 1% accuracy was required, then an^3 > 100 bn^2 or n > 100 b / a. There is, of course, no guarantee that the terms are simple polynomials; there could be logs and roots and so on too. But the analysis still applies. – Jonathan Leffler May 09 '09 at 18:42

7 Answers7

22

Big O doesn't describe the execution time of a function, just the growth. All functions have some constant factor or overhead that needs to be added in. When n is low, this overhead can greatly dwarf any improvements to the algorithm - an algorithm that requires 50ms per operation but has O(n) will perform worse for small n than an algorithm that requires 5 ms per operation, but has O(n*n).

This is why, in general, for small sets big O doesn't matter. For most objects with simple comparisons, a quick sort on 10 items will not be noticiably faster than a bubble sort, a linear search on 100 items will probably be faster than a binary tree, and so on.

Michael
  • 51,314
  • 5
  • 111
  • 139
  • 1
    The linear search performance depends on the cost of comparison; for complex tests, binary search can be quicker than linear far quicker than N=100. For simple integer comparisons, you could be right. – Jonathan Leffler May 08 '09 at 23:51
  • I was just going to second the comment above. Often it is taken for granted that comparisons are easy, constant time operations where this constant is always small.. which is not always true. – SplittingField May 08 '09 at 23:59
  • Right, but even then it is possible for linear to outperform since linear search would incur fewer page faults. For the simple integer case, the cost of walking the tree and all the cache misses and page faults that it would incur over an array could easily outpace the cost of the comparisons. – Michael May 09 '09 at 00:00
  • Updated answer with a "simple comparison" disclaimer. – Michael May 09 '09 at 00:06
  • Even with integer comparison, around N=10, binary search gets faster, based on some of my quick test. – Eamon Nerbonne Nov 23 '09 at 18:52
  • With N=1, binary search is just twice as slow - not exactly hugely noticeable. – Eamon Nerbonne Nov 23 '09 at 18:54
11

The concept behind Big-O notation is the asymptotic performance of the algorithm. As N gets bigger, the term in the Big-O notation comes to dominate the total time. For example, in an O(N^2) algorithm, the total time T(N) might be:

T(N) = a * N * N + b * N + c

As N gets bigger, and bigger, the term in N^2 dominates, regardless of the value of b or c.

When N is small, though, the b and c terms matter.

For example, if a = 0.001, b = 1,000, and c = 1,000,000.

 N                ~ T(N) [1 significant figure]
 1                1,000,000                (almost all c)
 1,000            2,000,000                (50:50 split on b and c)
 1,000,000        2,000,000,000            (50:50 split on a and b)
 1,000,000,000    1,000,000,000,000,000    (almost all a)

So, if you ignored the low-order terms, the performance at low N would be completely misrepresented. At high N, the low-order terms don't matter.

Jonathan Leffler
  • 666,971
  • 126
  • 813
  • 1,185
7

The course material gets harder to understand as the number of lectures attended (N) becomes very small.

Daniel Earwicker
  • 108,589
  • 35
  • 194
  • 274
  • I just read your profile. Why would you delete an answer with 60 upvotes? – Unknown May 13 '09 at 07:43
  • I can't remember the details but I have a mildly acerbic sense of humour, and I think it was just an answer to a very lame question. 60 people obviously liked it, but a couple of people clicked the "flag as offensive" link. Until people understand what that link is for (i.e. until pigs fly) I'm not going to bother to have the argument, I'm just going to delete my lame joke answer. Anyway I can't even find the question now! – Daniel Earwicker May 13 '09 at 09:48
2

Maybe the following is an example of "O(n) deviating from the function when n gets very small":

Consider an operation which requires, for example, time "50 times n, plus n squared".

When n is large then the "n squared" term will dominate, and so the operation can be said to be "O(n squared)".

When n is small, however, the "n squared" term will be negligible, and the "50 times n" term will dominate, and so when (and only when) n is small then it could be said to be O(n).

ChrisW
  • 51,820
  • 11
  • 101
  • 201
  • ((50 * N) + (N * N)) is an algorithm operating on a single value. Big O notation used in way the question is asked is talking about an algorithm that operates on N different values, not the length of the array. – Jherico May 25 '09 at 18:22
  • I was saying, "consider an algorithm which requires ((50 * N) + (N * N)) units of time to process N items." – ChrisW May 25 '09 at 18:40
1

To expand on the answer above, Big-Oh notation measures the eventual growth of the function or its limiting behavior.

f = O(g) if and only there exists an N and a constant c (which can be a function of N) such that for all n > N,
f(n) <= c*g(n)

Lets say f = 10000000*n and g = n^2.

f = O(g), however if you look at too small values of n, say less than 10000000 and set c = 1, you will see that g(n) <= f(n).

To add a more extreme example, would you rather deal with an algorithm with complexity n^100000 or an algorithm with complexity of 2^(.0000000001n). For reasonable problem sizes, the latter is better. What makes a lot of CS so beautiful is that it allows for this type of abuse, however, most natural algorithms do not take advantage of it. Most polynomial time algorithms have small constants (at least after a little of work).

Good luck!

SplittingField
  • 723
  • 5
  • 11
0

A big off topic but for the sake of completeness I want to mention some other notations which are related to the Big_o notation and commonly used in theoretical computer science and which you may find referred to in computer science literature: The Big-Θ notation, the Big-Ω notation and the little-o notation. These are simply other (and tighter) descriptions of growth rates. The little-o notation is easily mistaken for the big-O notation.

The little-o is also a relation between two functions f(x) and g(x). Saying that 'f(x) is little-o of g(x)' means that f(x) grows much faster than g(x). In more mathematical tearms is says that the limit of f(x)/g(x) is zero, as x approaches infinity.

As mentioned in the previous answers the big-O notation is used to describe the upper bound of the growth rate of an algorithm. It is really a relation between two functions f(x) and g(x), where f(x) = O(g(x)) as x goes to infinity.

See the Big-o wikipedia page for a nice and concise presentation of the different notations.

AnnaR
  • 3,116
  • 6
  • 31
  • 38
0

According to the definition:
f(n)=Θ(g(n)) means the set of all the functions f(n) such that there needs to be constants c1 and c2 and also n0 where all of these cases are true:

  • c1 . g(n) is a non-negative term or 0.
  • c1 . g(n) <= f(n) [g(n) needs to be the lower bound for certain n]
  • f(n) <= c2 . g(n) [g(n) needs to be the upper bound too since we are defining Θ].
  • for all n greater than our selected n0

So all we need to do is select such a c1, c2 and n0 that makes ALL the conditions true. Therefore for certain combinations of c1 c2, if you select n < n0, you cannot guarantee that your bound works. I think this is what your teacher meant by "the deviation".

unj2
  • 47,759
  • 80
  • 235
  • 362
  • Your criteria do not use 'c2'; I suspect that one of the first two bulletted formulae should reference c2. – Jonathan Leffler May 25 '09 at 17:31
  • Sorry it took so many attempts to get around the Markdown editor showing the information one way and the final display showing it another. It was very frustrating. – Jonathan Leffler May 25 '09 at 17:36