3.47 Large-scale, massively parallel: Memory use, sparsity, communication, etc.

For memory efficiency reasons, this keyword should be set to .false. whenever possible. Otherwise, especially the largest and most demanding calculations will run out of memory. When .true., this keyword creates a complete and up-to-date copy of every eigenvector for the k-point assigned to it on every MPI task. When .false., this information is communicated only when unavoidable. In principle, every operation in FHI-aims should be implemented so as to keep this keyword set to .false. and it is only human time that keeps us from doing it.

This flag is .false. by default because the complete splined density might be needed for some postprocessing or output option.

If requested, enables to cache the (deduced) topology of the nodes to the default communicator. In principle, the communication layer should then obtain more information on the network topology and organize the communication pattern more efficiently. However, nothing is guaranteed. The standard does not force MPI to respect the cached information, and in all decent MPI library implementations, this information should already be provided by the system.

In short: Perhaps try this if a truly strange communication pattern is observed, but probably, there will be no effect.

This feature, which was implemented by Rainer Johanni, eliminates the negative impact on performance incurred by the use_local_index keyword. It should always be used whenever memory becomes a bottleneck for a calculation. For more information about what this keyword does, please see the documentation for the use_local_index keyword, as the two keywords are closely intertwined.

Load balancing requires that KS_method parallel be used: that is, there is more than one CPU for cluster calculations or more CPUs than k-points for periodic calculations. If the if_scalapack option is supplied, load balancing will be turned on when KS_method parallel is being used and will be turned off when KS_method serial is being used.

When this keyword is set to .true. or if_scalapack, the use_local_index keyword will be set to the same value by default.

The following options exist for type:

• •

  none - no packing is used

• •

  index - matrices are packed by strictly eliminating all near-zero elements of all three matrices. Elements $ij$ are eliminated if both the overlap matrix element and the initial Hamiltonian matrix element are smaller than a threshold set by keyword packed_matrix_threshold.

Packing the overlap, Hamiltonian and density matrices reduces the size of these arrays during matrix integration, at the expense of some small effort to correctly sort intermediate results during the integration appropriately.

From a technical point of view, packing only makes sense if the full Hamiltonian is not required later anyway, during the eigenvalue solution. This is the case:

• •

  In the cluster case: if KS_method parallel is used.

• •

  In the periodic case: for more than one k-point, and/or if KS_method parallel is used.

We have set the new default to 0.0 instead of ${10}^{-13}$. In our experience, this change doesn’t introduce any significant computational or memory overhead. More importantly, it fixes several problems we noticed in the past, including SCF instabilities and incorrect results for quantities depending on the real-space momentum matrix or the dipole matrix.

All operations for the integrations and the electron density update are $O(N)$ operations, but verifying which basis functions are non-zero for each integration batch requires to check each basis function, and is thus an $O(N^2)$ operation with a small prefactor. This step can be avoided by checking for the non-zero basis functions once, and then storing their indices in memory for each batch of integration points. For very large systems and restricted memory, it may be worth trying to switch this feature off to save some memory, otherwise this should always be done.

During relaxation (see relax_geometry), geometry steps can be rejected if the total energy increased unexpectedly. In order to revert to a previous step, it is necessary to access the Kohn-Sham eigenvectors used to initialize that step, which must hence be stored for that purpose. In normal calculations, eigenvector storage is not a problem, but their size grows as $O(N^2)$ with system size. For very large system sizes, their storage in memory can become a bottleneck, which is here circumvented by the option to store them to disk.

This keyword is not related to the restart functionality of the restart keyword, since it is an old, not a current, set of Kohn-Sham eigenvectors that may be needed when reverting a geometry step.

Only relevant for correlated beyond-hybrid methods.

Only relevant for KS_method parallel.

When running on a system with many CPUs, it may be faster to use “all-to-all” communcation (mpi_alltoallv) than doing n_tasks times a sendrecv call. This should be tested on your particular hardware and MPI library though, as it could be the case that no benefit is offered, or it could come with worse performance. It was, for example, noticeably more performant on the BlueGene/P with $\approx$10⁵ MPI tasks at a time. For much fewer tasks, the effect is not usually relevant.

Also note that “all-to-all” costs a significant amount of memory, and this memory pressure will be felt for much fewer MPI tasks (e.g., Intel architecture with hundreds of CPU cores at a time.

Only relevant for parallel runs.

When running on a system with many CPUs, it can be a bit more memory efficient to use “MPI_IN_PLACE” communication. Then, instead of separate send- and receive buffers, a single buffer is used and information is updated “in place”.

However, not all MPI implementations seem to implement this feature correctly. Thus, this choice can sometimes lead to problems or even errors in the results, depending on the MPI library and version that was used.

The code checks for specific problems related to “MPI_IN_PLACE” and switches to use_mpi_in_place .false. if problems are encountered.

If anyone identifies further problems related to the use of “MPI_IN_PLACE”, please let us know.

Originally, FHI-aims stored its real-space Hamiltonian/overlap matrices in a non-distributed fashion, i.e. every CPU has the full copies of the real-space Hamiltonian/overlap matrices. While this makes the math easier internally, it will lead to a memory bottleneck for large calculations: as the number of atoms in a calculation increases, the sizes of the real-space matrices increase, and eventually the real-space matrices are too large to fit in memory. Adding more CPUs to a calculation does not fix this problem, since each CPU has the full copies of the matrices.

For very large systems with many CPUs, it thus becomes necessary to spread the real-space matrices across CPUs. This is done using a method known as “domain decomposition” (or, informally, local indexing), where we assign batches of integration grid points close to one another to the same CPU. Each CPU then stores only the portions of the real-space Hamiltonian/overlap matrices which have non-zero support on the integration points which it possesses. The real-space matrices are thus distributed across CPUs, considerably decreasing their sizes. Should a calculation still suffer from memory issues associated with the sizes of the real-space matrices, we can increase the number of CPUs to reduce the memory overhead on each individual CPU.

While the domain decomposition method will spread the real-space matrices across CPUs and eliminates the memory bottleneck previously mentioned, eventually we will need to solve the Kohn-Sham eigenvalue problem. To solve the Kohn-Sham eigenvalue problem, we need to generate the Hamiltonian entering into the eigensolver from the real-space Hamiltonian, which is now distributed across CPUs. The subsequent merge of all results into the BLACS infrastructure used by the eigensolvers supported by FHI-aims then becomes more difficult, and some performance overhead results from the altered load on each CPU. This option is switched on by default for systems containing 100 or more atoms, but not all current functionality supports it. If any conflicting keywords are found at runtime, local indexing is not set.

To overcome the performance overhead associated with use_local_index keyword, it is *strongly* recommended that the user also try using the load_balancing keyword, which enables load balancing. Load balancing will eliminate the overhead associated with the use_local_index keyword, but it is not enabled for all non-standard functionality in FHI-aims, hence why we do not enable it by default.

Domain decomposition requires KS_method parallel: that is, there is more than one CPU for cluster calculations or more CPUs than k-points for periodic calculations. If the if_scalapack option is supplied, domain decomposition will be turned on when KS_method parallel is being used and will be turned off when KS_method serial is being used.

Domain composition also requires prune_basis_once and a parallel
grid_partitioning_method. These are enabled by default for FHI-aims calculations, so you shouldn’t worry about setting them yourself.

In order to reach the postprocessing phase and write information required at the end of a calculation in an organized manner, FHI-aims can force a stop before a certain amount of real time (wall clock time) is exceeded. In order to achieve this safely, FHI-aims uses an internal estimate of the duration of a single s.c.f. iteration within the calculation, and stops if it estimates that the next iteration will take more time than what remains available.