nr_cpus boot parameter in Linux kernel

Question

I was browsing Linux kernel code to understand the nr_cpus boot parameter. As per the documentation, (https://www.kernel.org/doc/Documentation/kernel-parameters.txt)

[SMP] Maximum number of processors that an SMP kernel
            could support.  nr_cpus=n : n >= 1 limits the kernel to
            supporting 'n' processors. Later in runtime you can not
            use hotplug cpu feature to put more cpu back to online.
            just like you compile the kernel NR_CPUS=n

In the smp.c code, the value is set to nr_cpu_ids which is then used everywhere in kernel.
http://lxr.free-electrons.com/source/kernel/smp.c

527 static int __init nrcpus(char *str)
528 {
529         int nr_cpus;
530 
531         get_option(&str, &nr_cpus);
532         if (nr_cpus > 0 && nr_cpus < nr_cpu_ids)
533                 nr_cpu_ids = nr_cpus;
534 
535         return 0;
536 }
537 
538 early_param("nr_cpus", nrcpus);

What I do not understand the nr_cpu_ids is also set by setup_nr_cpu_ids.

555 /* An arch may set nr_cpu_ids earlier if needed, so this would be redundant */
556 void __init setup_nr_cpu_ids(void)
557 {
558         nr_cpu_ids = find_last_bit(cpumask_bits(cpu_possible_mask),NR_CPUS) + 1;
559 }

Initially, I thought this is called before the early_param invocation. After adding logs, I found that setup_nr_cpu_ids() is called after nr_cpus(). nr_cpu_ids is always set to the value sets in setup_nr_cpu_ids() in stead of nr_cpus(). I even verified its value in smp_init().

Can anybody please clarify if my observation is correct or not?
What is the exact usage of nr_cpu_ids?

Answer 1

As part of documentation describes from your question:

Maximum number of processors that an SMP kernel could support

Actually both of these function do the same. The early_param() provides ability to search the first parameter in the kernel command line and if the search was successful, the function which is noted in the second parameter of the early_param() will be called.

All functions which are marked with early_param will be called in the do_early_param() from the init/main.c which will be called from the setup_arch function. The setup_arch function is architecture specific and each architecture provides own implementation of the setup_arch(). So after the call of the nrcpus() function, the nr_cpu_ids will contain number of processors that kernel could support.

If you will look at the Linux kernel source code, you will note that the setup_nr_cpu_ids() function will be called from the init/main.c after functions which are marked with early_param. So in this case it is redundant. But sometimes it might be useful to get number of processors earlier.

For example, you can see it in the powerpc architecture. As described in the comment of the smp_setup_cpu_maps() functions where the setup_nr_cpu_ids() is called:

Having the possible map set up early allows us to restrict allocations of things like irqstacks to nr_cpu_ids rather than NR_CPUS.

Answer 2

Typically arch detects the number of cpus available on the system. But, its possible to reduce the number of cpus you want to use. And, for that reason nr_cpus parameter was introduced. By default no one uses this parameter and in that case arch code is responsible for detecting the number of cpus available on the system i.e. for x86 arch look at prefill_possible_map, where a check is made to see whether nr_cpus were passed or not. If nr_cpus were passed then that value is used. After arch detects the number of possible cpus available then setup_nr_cpu_ids in kenrel/smp.c finalizes the value of nr_cpu_ids. Do note that, it might sounds redundant but since it works so no one complains.

So, your observation is partially correct due to the fact that you missed the point, how arch smpboot code integrates nr_cpus. Hope this clarifies you understanding.

Answer 3

Checkout cpumask.h, in particular this...

787 #define for_each_cpu_mask_nr(cpu, mask)                 \
788         for ((cpu) = -1;                                \
789                 (cpu) = __next_cpu_nr((cpu), &(mask)),  \
790                 (cpu) < nr_cpu_ids; )

nr_cpu_ids is the max usable cpus and nr_cpus that you pass as a boot param is used to set it. This is what it's doing for me in kernel 3.16.

The comment here

555 /* An arch may set nr_cpu_ids earlier if needed, so this would be redundant */
556 void __init setup_nr_cpu_ids(void)
557 {
558         nr_cpu_ids = find_last_bit(cpumask_bits(cpu_possible_mask),NR_CPUS) + 1;
559 }

is saying that if you have already set nr_cpu_ids that this call is redundant. The reason being the cpu_possible_mask has already been set to nr_cpu_id.