CPU Affinity (The GNU C Library)

#### 23.3.6 Limiting execution to certain CPUs ¶

On a multi-processor system the operating system usually distributes the different processes which are runnable on all available CPUs in a way which allows the system to work most efficiently. Which processes and threads run can to some extend be controlled with the scheduling functionality described in the last sections. But which CPU finally executes which process or thread is not covered.

There are a number of reasons why a program might want to have control over this aspect of the system as well:

- One thread or process is responsible for absolutely critical work which under no circumstances must be interrupted or hindered from making progress by other processes or threads using CPU resources. In this case the special process would be confined to a CPU which no other process or thread is allowed to use.
- The access to certain resources (RAM, I/O ports) has different costs from different CPUs. This is the case in NUMA (Non-Uniform Memory Architecture) machines. Preferably memory should be accessed locally but this requirement is usually not visible to the scheduler. Therefore forcing a process or thread to the CPUs which have local access to the most-used memory helps to significantly boost the performance.
- In controlled runtimes resource allocation and book-keeping work (for instance garbage collection) is performance local to processors. This can help to reduce locking costs if the resources do not have to be protected from concurrent accesses from different processors.

The POSIX standard up to this date is of not much help to solve this problem. The Linux kernel provides a set of interfaces to allow specifying affinity sets for a process. The scheduler will schedule the thread or process on CPUs specified by the affinity masks. The interfaces which the GNU C Library define follow to some extent the Linux kernel interface.

Data Type: cpu_set_t ¶

  This data set is a bitset where each bit represents a CPU. How the system’s CPUs are mapped to bits in the bitset is system dependent. The data type has a fixed size; it is strongly recommended to allocate a dynamically sized set based on the actual number of CPUs detected, such as via `get_nprocs_conf()`, and use the `CPU_*_S` variants instead of the fixed-size ones.

  This type is a GNU extension and is defined in `sched.h`.

To manipulate the bitset, to set and reset bits, and thus add and remove CPUs from the sets, a number of macros are defined. Some of the macros take a CPU number as a parameter. Here it is important to never exceed the size of the bitset, either `CPU_SETSIZE` for fixed sets or the allocated size for dynamic sets. For each macro there is a fixed-size version (documented below) and a dynamic-sized version (with a `_S` suffix).

Macro: `int` CPU_SETSIZE ¶

  The value of this macro is the maximum number of CPUs which can be handled with a fixed `cpu_set_t` object.

For applications that require CPU sets larger than the built-in size, a set of macros that support dynamically-sized sets are defined.

Macro: `size_t` CPU_ALLOC_SIZE `(size_t count)` ¶

  Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety Concepts.

  Given a count of CPUs to hold, returns the size of the set to allocate. This return value is appropriate to be used in the *_S macros.

  This macro is a GNU extension and is defined in `sched.h`.

Macro: `cpu_set_t *` CPU_ALLOC `(size_t count)` ¶

  Preliminary: | MT-Safe | AS-Unsafe lock | AC-Unsafe lock fd mem | See POSIX Safety Concepts.

  Given the count of CPUs to hold, returns a set large enough to hold them; that is, the resulting set will be valid for CPUs numbered 0 through count-1, inclusive. This set must be freed via `CPU_FREE` to avoid memory leaks. Warning: the argument is the CPU count and not the size returned by `CPU_ALLOC_SIZE`.

  This macro is a GNU extension and is defined in `sched.h`.

Macro: `void` CPU_FREE `(cpu_set_t *set)` ¶

  Preliminary: | MT-Safe | AS-Unsafe lock | AC-Unsafe lock fd mem | See POSIX Safety Concepts.

  Frees a CPU set previously allocated by `CPU_ALLOC`.

  This macro is a GNU extension and is defined in `sched.h`.

The type `cpu_set_t` should be considered opaque; all manipulation should happen via the `CPU_*` macros described below.

Macro: `void` CPU_ZERO `(cpu_set_t *set)` ¶

  Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety Concepts.

  This macro initializes the CPU set set to be the empty set.

  This macro is a GNU extension and is defined in `sched.h`.

Macro: `void` CPU_SET `(int cpu, cpu_set_t *set)` ¶

  Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety Concepts.

  This macro adds cpu to the CPU set set.

  The cpu parameter must not have side effects since it is evaluated more than once.

  This macro is a GNU extension and is defined in `sched.h`.

Macro: `void` CPU_CLR `(int cpu, cpu_set_t *set)` ¶

  Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety Concepts.

  This macro removes cpu from the CPU set set.

  The cpu parameter must not have side effects since it is evaluated more than once.

  This macro is a GNU extension and is defined in `sched.h`.

Macro: `cpu_set_t *` CPU_AND `(cpu_set_t *dest, cpu_set_t *src1, cpu_set_t *src2)` ¶

  Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety Concepts.

  This macro populates dest with only those CPUs included in both src1 and src2. Its value is dest.

  This macro is a GNU extension and is defined in `sched.h`.

Macro: `cpu_set_t *` CPU_OR `(cpu_set_t *dest, cpu_set_t *src1, cpu_set_t *src2)` ¶

  Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety Concepts.

  This macro populates dest with those CPUs included in either src1 or src2. Its value is dest.

  This macro is a GNU extension and is defined in `sched.h`.

Macro: `cpu_set_t *` CPU_XOR `(cpu_set_t *dest, cpu_set_t *src1, cpu_set_t *src2)` ¶

  Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety Concepts.

  This macro populates dest with those CPUs included in either src1 or src2, but not both. Its value is dest.

  This macro is a GNU extension and is defined in `sched.h`.

Macro: `int` CPU_ISSET `(int cpu, const cpu_set_t *set)` ¶

  Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety Concepts.

  This macro returns a nonzero value (true) if cpu is a member of the CPU set set, and zero (false) otherwise.

  The cpu parameter must not have side effects since it is evaluated more than once.

  This macro is a GNU extension and is defined in `sched.h`.

Macro: `int` CPU_COUNT `(const cpu_set_t *set)` ¶

  Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety Concepts.

  This macro returns the count of CPUs (bits) set in set.

  This macro is a GNU extension and is defined in `sched.h`.

Macro: `int` CPU_EQUAL `(cpu_set_t *src1, cpu_set_t *src2)` ¶

  Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety Concepts.

  This macro returns nonzero if the two sets set1 and set2 have the same contents; that is, the set of CPUs represented by both sets is identical.

  This macro is a GNU extension and is defined in `sched.h`.

Macro: `void` CPU_ZERO_S `(size_t size, cpu_set_t *set)` ¶

Macro: `void` CPU_SET_S `(int cpu, size_t size, cpu_set_t *set)` ¶

Macro: `void` CPU_CLR_S `(int cpu, size_t size, cpu_set_t *set)` ¶

Macro: `cpu_set_t *` CPU_AND_S `(size_t size, cpu_set_t *dest, cpu_set_t *src1, cpu_set_t *src2)` ¶

Macro: `cpu_set_t *` CPU_OR_S `(size_t size, cpu_set_t *dest, cpu_set_t *src1, cpu_set_t *src2)` ¶

Macro: `cpu_set_t *` CPU_XOR_S `(size_t size, cpu_set_t *dest, cpu_set_t *src1, cpu_set_t *src2)` ¶

Macro: `int` CPU_ISSET_S `(int cpu, size_t size, const cpu_set_t *set)` ¶

Macro: `int` CPU_COUNT_S `(size_t size, const cpu_set_t *set)` ¶

Macro: `int` CPU_EQUAL_S `(size_t size, cpu_set_t *src1, cpu_set_t *src2)` ¶

Each of these macros performs the same action as its non-`_S` variant, but takes a size argument to specify the set size. This size argument is as returned by the `CPU_ALLOC_SIZE` macro, defined above.

CPU bitsets can be constructed from scratch or the currently installed affinity mask can be retrieved from the system.

Function: `int` sched_getaffinity `(pid_t pid, size_t cpusetsize, cpu_set_t *cpuset)` ¶

  Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety Concepts.

  This function stores the CPU affinity mask for the process or thread with the ID pid in the cpusetsize bytes long bitmap pointed to by cpuset. If successful, the function always initializes all bits in the `cpu_set_t` object and returns zero.

  If pid does not correspond to a process or thread on the system the or the function fails for some other reason, it returns `-1` and `errno` is set to represent the error condition.

  `ESRCH`

    No process or thread with the given ID found.

  `EFAULT`

    The pointer cpuset does not point to a valid object.

  This function is a GNU extension and is declared in `sched.h`.

Note that it is not portably possible to use this information to retrieve the information for different POSIX threads. A separate interface must be provided for that.

Function: `int` sched_setaffinity `(pid_t pid, size_t cpusetsize, const cpu_set_t *cpuset)` ¶

  Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety Concepts.

  This function installs the cpusetsize bytes long affinity mask pointed to by cpuset for the process or thread with the ID pid. If successful the function returns zero and the scheduler will in the future take the affinity information into account.

  If the function fails it will return `-1` and `errno` is set to the error code:

  `ESRCH`

    No process or thread with the given ID found.

  `EFAULT`

    The pointer cpuset does not point to a valid object.

  `EINVAL`

    The bitset is not valid. This might mean that the affinity set might not leave a processor for the process or thread to run on.

  This function is a GNU extension and is declared in `sched.h`.

Function: `int` getcpu `(unsigned int *cpu, unsigned int *node)` ¶

  Preliminary: | MT-Safe | AS-Safe | AC-Safe | See POSIX Safety Concepts.

  The `getcpu` function identifies the processor and node on which the calling thread or process is currently running and writes them into the integers pointed to by the cpu and node arguments. The processor is a unique nonnegative integer identifying a CPU. The node is a unique nonnegative integer identifying a NUMA node. When either cpu or node is `NULL`, nothing is written to the respective pointer.

  The return value is `0` on success and `-1` on failure. The following `errno` error condition is defined for this function:

  `ENOSYS`

    The operating system does not support this function.

  This function is Linux-specific and is declared in `sched.h`.

Function: `int` sched_getcpu `(void)` ¶

  Similar to `getcpu` but with a simpler interface. On success, returns a nonnegative number identifying the CPU on which the current thread is running. Returns `-1` on failure. The following `errno` error condition is defined for this function:

  `ENOSYS`

    The operating system does not support this function.

  This function is Linux-specific and is declared in `sched.h`.

Here’s an example of how to use most of the above to limit the number of CPUs a process runs on, not including error handling or good logic on CPU choices:

    #define _GNU_SOURCE
    #include <sched.h>
    #include <sys/sysinfo.h>
    #include <unistd.h>
    void
    limit_cpus (void)
    {
      unsigned int mycpu;
      size_t nproc, cssz, cpu;
      cpu_set_t *cs;
      getcpu (&mycpu, NULL);
      nproc = get_nprocs_conf ();
      cssz = CPU_ALLOC_SIZE (nproc);
      cs = CPU_ALLOC (nproc);
      sched_getaffinity (0, cssz, cs);
      if (CPU_COUNT_S (cssz, cs) > nproc / 2)
        {
          for (cpu = nproc / 2; cpu < nproc; cpu ++)
            if (cpu != mycpu)
              CPU_CLR_S (cpu, cssz, cs);
          sched_setaffinity (0, cssz, cs);
        }
      CPU_FREE (cs);
    }