Questions tagged [benchmarking]

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them. The benchmarking tag should be used for questions about how to perform benchmarking tasks or theory questions, not for lists of benchmarking results or requests for benchmarking data; those questions are off-topic for Stack Overflow.

Benchmarking is the process of comparing two or more systems or processes under controlled circumstances in order to have a quantitative measure with which to compare or rank them.

For hardware, benchmarking typically involves either performing a simple task many times or a complex task to ascertain the performance characteristics desired (often speed, but power draw, heat, memory usage, and other characteristics may be of interest as well). Common benchmark operations include FLOPS (Floating Point Operations Per Second), write or read time for a large file, write or read time for many small files, rendering large images, and downloading or uploading large files over a network.

For software, benchmarking typically involves running the different software of interest (different versions of a program, different programs that accomplish a similar task, etc.) on an identical system (either one system, or two identical systems) and performing tasks that take sufficient time to notice a difference. This is often performed on very small differences in code, such as to verify which approach is superior to solving a particular problem.

Benchmarking also includes industry standard benchmarks and common benchmarking suites, used to assist users in making purchasing decisions or otherwise comparing their current systems to other available systems. However, it should be used in this context for issues with building or understanding the code and behavior of these benchmarks, not for general recommendations on the benchmarks suites or on the tested products.

Include additional tags to your question to indicate what type of benchmarking the question is about, such as , or what type of programming language if software, etc.

3156 questions
1981
votes
10 answers

Why is reading lines from stdin much slower in C++ than Python?

I wanted to compare reading lines of string input from stdin using Python and C++ and was shocked to see my C++ code run an order of magnitude slower than the equivalent Python code. Since my C++ is rusty and I'm not yet an expert Pythonista, please…
JJC
  • 7,445
  • 6
  • 39
  • 51
1923
votes
7 answers

What do 'real', 'user' and 'sys' mean in the output of time(1)?

$ time foo real 0m0.003s user 0m0.000s sys 0m0.004s $ What do 'real', 'user' and 'sys' mean in the output of time? Which one is meaningful when benchmarking my app?
Iraimbilanja
909
votes
11 answers

How do I write a correct micro-benchmark in Java?

How do you write (and run) a correct micro-benchmark in Java? I'm looking for some code samples and comments illustrating various things to think about. Example: Should the benchmark measure time/iteration or iterations/time, and why? Related: Is…
John Nilsson
  • 16,089
  • 8
  • 29
  • 41
874
votes
10 answers

Why is 2 * (i * i) faster than 2 * i * i in Java?

The following Java program takes on average between 0.50 secs and 0.55 secs to run: public static void main(String[] args) { long startTime = System.nanoTime(); int n = 0; for (int i = 0; i < 1000000000; i++) { n += 2 * (i * i); …
Stefan
  • 4,691
  • 3
  • 5
  • 12
867
votes
3 answers

Why does Python code run faster in a function?

def main(): for i in xrange(10**8): pass main() This piece of code in Python runs in (Note: The timing is done with the time function in BASH in Linux.) real 0m1.841s user 0m1.828s sys 0m0.012s However, if the for loop isn't…
thedoctar
  • 8,353
  • 3
  • 18
  • 30
270
votes
30 answers

How much faster is C++ than C#?

Or is it now the other way around? From what I've heard there are some areas in which C# proves to be faster than C++, but I've never had the guts to test it by myself. Thought any of you could explain these differences in detail or point me to the…
Trap
  • 11,199
  • 15
  • 53
  • 64
267
votes
7 answers

Clang vs GCC - which produces faster binaries?

I'm currently using GCC, but I discovered Clang recently and I'm pondering switching. There is one deciding factor though - quality (speed, memory footprint, reliability) of binaries it produces - if gcc -O3can produce a binary that runs 1% faster,…
SF.
  • 12,380
  • 11
  • 65
  • 102
231
votes
16 answers

Execution time of C program

I have a C program that aims to be run in parallel on several processors. I need to be able to record the execution time (which could be anywhere from 1 second to several minutes). I have searched for answers, but they all seem to suggest using the…
Roger
  • 2,903
  • 5
  • 21
  • 20
218
votes
3 answers

Load Testing with AB ... fake failed requests (length)

To do some load testing, for my own curiosity, on my server I ran: ab -kc 50 -t 200 http://localhost/index.php This opens up 50 keep-alive connections for 200 seconds and just slams my server with requests for index.php In my results, I…
Franky
210
votes
6 answers

ab load testing

Can someone please walk me through the process of how I can load test my website using apache bench tool (ab)? I want to know the following: How many people per minute can the site handle? Please walk me through the commands I should run to figure…
jewelwast
  • 2,327
  • 4
  • 15
  • 13
208
votes
7 answers

How much faster is Redis than mongoDB?

It's widely mentioned that Redis is "Blazing Fast" and mongoDB is fast too. But, I'm having trouble finding actual numbers comparing the results of the two. Given similar configurations, features and operations (and maybe showing how the factor…
Homer6
  • 13,933
  • 10
  • 53
  • 77
197
votes
12 answers

Which is faster: multiple single INSERTs or one multiple-row INSERT?

I am trying to optimize one part of my code that inserts data into MySQL. Should I chain INSERTs to make one huge multiple-row INSERT or are multiple separate INSERTs faster?
dusoft
  • 10,672
  • 5
  • 35
  • 40
148
votes
4 answers

Is MATLAB OOP slow or am I doing something wrong?

I'm experimenting with MATLAB OOP, as a start I mimicked my C++'s Logger classes and I'm putting all my string helper functions in a String class, thinking it would be great to be able to do things like a + b, a == b, a.find( b ) instead of strcat(…
stijn
  • 31,563
  • 13
  • 95
  • 145
140
votes
2 answers

Why is C++ initial allocation so much larger than C's?

When using the same code, simply changing the compiler (from a C compiler to a C++ compiler) will change how much memory is allocated. I'm not quite sure why this is and would like to understand it more. So far the best response I've gotten is…
Rerumu
  • 1,223
  • 1
  • 6
  • 8
136
votes
8 answers

How to benchmark efficiency of PHP script

I want to know what is the best way to benchmark my PHP scripts. Does not matter if a cron job, or webpage or web service. I know i can use microtime but is it really giving me the real time of a PHP script? I want to test and benchmark different…
eric
  • 2,318
  • 3
  • 16
  • 21
1
2 3
99 100