Mpi, Python, Scatterv, And Overlapping Data

The MPI standard, 3.0, says about mpi_scatterv: The specification of counts, types, and displacements should not cause any location on the root to be read more than once.' Howeve

Solution 1:

You cannot rely on this.

This assumption stems directly from the MPI standard. Since mpi4py upper-case functions are just a thin layer on top of MPI, this is what matters. The standard also states:

Rationale. Though not needed, the last restriction is imposed so as to achieve symmetry with MPI_GATHER, where the corresponding restriction (a multiple-write restriction) is necessary. (End of rationale.)

Considering it is in the standard, an MPI implementation may make use of that:

• Ignore violations
• Issue a warning when violated
• Fail when violated
• Use this assumptions for any kind of optimization that could lead to undefined behavior when violated

The last point is most scary as it may introduce subtle bugs. Considering the read-only nature of the send buffer, it is difficult to imagine such an optimization, but that doesn't mean it does/will not exist. Just as an idea consider strict aliasing optimizations. Also note that MPI implementations are very complex - their behavior may change seemingly erratic between versions, configurations, data sizes or other environmental changes.

There is also an infamous example with memcpy: The standard forbids overlapping memory inputs, and at some point the glibc implementation made use of that for a tiny disputed optimization. Code that did not satisfied the requirement started to fail, and users started to hear strange sound on mp3 flash websites, followed by a heated debate involving Linus Torvalds and Ulrich Drepper.

The morale of the story is: Follow the requirements imposed by the standard, even if it works right now and the requirement doesn't make sense to you. Also be glad that there is such a detailed standard.

Solution 2:

The MPI standard includes many requirements that are often not strictly checked by the implementations, mainly for performance reasons. The rationale is that any program that is correct according to the standard will also be correct given a set of relaxed constraints. Relying on such implementation-specific behaviour results in non-portable code and goes against the standard.

There are many valid reasons to require disjoint send segments. The immediately visible one is the symmetry with MPI_Gatherv. For the latter the segments must be disjoint, otherwise the content of the memory after the gather will depend on the order of the underlying receive operations. Since in a typical MPI program scatters are usually mirrored by gathers, the computed offset and count arrays can be reused if the same constraints apply to both the gather and the scatter. A less obvious reason is that on some architectures the network equipment might not allow simultaneous reads from overlapping memory regions.

As it is very easy for non-standard MPI behaviour to creep into the program code during development, one might want to use tools like our MUST to check the correctness of the program.