A.9 Numerical reproducibility errors

Collective MPI algorithms are communication routines that coordinate data movement or global computation across all processes within a specified communicator, such as MPI_Bcast (which copies data from one "root" process to all other processes) and MPI_Allreduce (which applies a reduction operation, e.g. a sum, to data across all processes and distributes the result back to all processes). Such routines are used extensively throughout FHI-aims, for example in the eigensolver ELPA. By default Intel MPI chooses the collective communication algorithms at runtime from message size, communicator size, and node topology, from the multiple possible algorithms that are available ^[7]. On AMD CPUs, for certain task counts and matrix sizes, the topology-aware, shared-memory variants of these algorithms appear to be problematic, returning different numerical results for a fixed input. One of the ELPA developers, Petr Karpov, has produced a minimal reproducer illustrating this problem with MPI_Bcast + MPI_Allreduce^[8].

One method to disable these topology-aware shared memory collective algorithms is to set the environment variable:

export I_MPI_COLL_INTRANODE = pt2pt

which uses only point-to-point communication based collectives, and does not use shared memory ones.

If this does not resolve the problem, then one can instead completely disable topology aware collectives via the environment variable

export I_MPI_CBWR=1

This enforces Conditional Numerical Reproducibility (CNR), which provides controls for obtaining reproducible floating-point results on collective operations. With this feature, Intel MPI collective operations are designed to return the same floating-point results from run to run in the case of the same number of MPI ranks. The value 1 disables topology-aware collectives, so that the result of a collective operation does not depend on the rank placement. The mode guarantees reproducibility of results across different runs on the same cluster (independent of the rank placement). Our tests proved it to be sufficient to avoid charge integration errors popping up randomly for a given system and a given number of cores. This can incur some performance penalty, which based on limited testing is greater than that imposed by I_MPI_COLL_INTRANODE, and is why we recommend attempting that approach first.

CNR is also possible to enforce with the Intel MKL library, and will be discussed in Section A.9.2, as it has been observed to help the non-positive definite overlap matrix error. If setting I_MPI_CBWR=1 does not remedy the charge integration error on its own, we would recommend combining this with the Intel MKL CNR discussed in Section A.9.2 to have CNR across the whole software stack.

If none of these approaches work, then next we would recommend to control MPI process pinning.

Process pinning (also called process affinity or CPU binding) is the binding of each MPI rank to a fixed set of logical CPUs, so that the operating-system scheduler is not free to move (“migrate”) the rank onto other cores while it runs.^[9]

Where a rank runs matters on a modern multi-socket compute node for several reasons:

• •

  Non-uniform memory access (NUMA) locality. Memory that a rank allocates is placed physically close to the core that first writes to it (the Linux “first-touch” policy). If the rank later migrates to a core on a different socket / NUMA domain, every access to that memory has to cross the inter-socket link, which is substantially slower.

• •

  Caches. Migration cold-starts the L1 / L2 caches the rank was using.

On a dedicated node, poor pinning normally only costs performance. On a shared node the failure modes are more severe: ranks can be squeezed onto a handful of cores, migrated unpredictably as other jobs start and finish, or massively oversubscribe the cores allotted to the job. On the affected platforms this coincided with the errors described above, and enforcing correct one-rank-per-physical-core pinning resolved them.

Before touching pinning, make sure each rank is not silently launching extra threads. FHI-aims can link a threaded BLAS / LAPACK (typically Intel MKL); left at its defaults, each rank may start one MKL thread per core, so $N$ ranks $\times N$ threads badly oversubscribe the node. For the standard pure-MPI FHI-aims setup, force single-threaded math:

MPI can use different internal algorithms for carrying out operations based on the number of cores, matrix sizes and types involved, inner topology of the MPI process map, etc (as discussed above in relation to the topology aware MPI collectives). This can legally reorder floating-point operations and be the reason for large floating-point errors resulting in the charge integration error. If you are free to choose, just changing the number of cores can be sufficient to change the sizes of the matrices involved in the computation in one MPI process, and thus shift the algorithm choice from a faulty one to one that works.

When fewer ranks are launched than there are cores on the node, Intel MPI does not pin each rank to a single core by default; instead it gives every rank a domain spanning several logical CPUs, shown in a pinning table (which is printed to stdout if I_MPI_DEBUG is more than 4) as a range in curly braces:

Here 32 ranks on a 96-core node each receive a 3-core domain. A rank is then free to migrate within its domain, and — more importantly — Intel MPI’s algorithms make assumptions about where each rank sits that a loose, multi-core domain can violate. We also observed situations where the pinning table showed 1-to-1 pinning on logical cores on a system with hyper-threading; this too was sufficient to get the charge integration error.

To force strict one-rank-per-physical-core (1-to-1) pinning with mpirun, set the domain size to a single physical core:

Then confirm from the debug output that every rank shows a single-CPU {n} mask.

If you launch through Slurm, do not rely on a bare srun. Bind the tasks to cores explicitly: