27162

Why does this MPI code execute out of order? [duplicate]

Question:

This question already has an answer here:

<ul><li> <a href="/questions/22943663/why-is-my-mpi-program-outputting-incorrectly" dir="ltr" rel="nofollow">Why is my MPI program outputting incorrectly</a> <span class="question-originals-answer-count"> 2 answers </span> </li> </ul>

I'm trying to create a "Hello, world!" application in (Open)MPI such that each process will print out in order.

My idea was to have the first process send a message to the second when it's finished, then the second to the third, etc.:

#include <mpi.h> #include <stdio.h> int main(int argc,char **argv) { int rank, size; MPI_Init(&argc, &argv); MPI_Comm_rank(MPI_COMM_WORLD, &rank); MPI_Comm_size(MPI_COMM_WORLD, &size); // See: http://mpitutorial.com/mpi-send-and-receive/ if (rank == 0) { // This is the first process. // Print out immediately. printf("Hello, World! I am rank %d of %d.\n", rank, size); MPI_Send(&rank, 1, MPI_INT, 1, 0, MPI_COMM_WORLD); } else { // Wait until the previous one finishes. int receivedData; MPI_Recv(&receivedData, 1, MPI_INT, rank - 1, 0, MPI_COMM_WORLD, MPI_STATUS_IGNORE); printf("Hello, World! I am rank %d of %d (message: %d).\n", rank, size, receivedData); if (rank + 1 < size) { // We're not the last one. Send out a message. MPI_Send(&rank, 1, MPI_INT, rank + 1, 0, MPI_COMM_WORLD); } else { printf("Hello world completed!\n"); } } MPI_Finalize(); return 0; }

When I run this on an eight-core cluster, it runs perfectly every time. However, when I run it on a sixteen-core cluster, sometimes it works, and sometimes it outputs something like this:

Hello, world, I am rank 0 of 16. Hello, world, I am rank 1 of 16 (message: 0). Hello, world, I am rank 2 of 16 (message: 1). Hello, world, I am rank 3 of 16 (message: 2). Hello, world, I am rank 4 of 16 (message: 3). Hello, world, I am rank 5 of 16 (message: 4). Hello, world, I am rank 6 of 16 (message: 5). Hello, world, I am rank 7 of 16 (message: 6). Hello, world, I am rank 10 of 16 (message: 9). Hello, world, I am rank 11 of 16 (message: 10). Hello, world, I am rank 8 of 16 (message: 7). Hello, world, I am rank 9 of 16 (message: 8). Hello, world, I am rank 12 of 16 (message: 11). Hello, world, I am rank 13 of 16 (message: 12). Hello, world, I am rank 14 of 16 (message: 13). Hello, world, I am rank 15 of 16 (message: 14). Hello world completed!

That is, most of the output is in order, but some is out of place.

Why is this happening? How is it even possible? How can I fix it?

Answer1:

MPI codes are not guaranteed to complete in any specific order. This is especially true when running on multiple nodes, but still true even on one node.

While you are enforcing some sort of ordering by adding the sequential sends and receives, the output messages are still forwarded from the application process to the MPI layer and back up to the mpiexec/mpirun process to be printed to the screen. This message forwarding can happen in any order and is interleaved with other communication (since it uses a completely different communication topology). If you really must ensure that messages are printed in order, you have to make sure that the same MPI rank prints all of them out.

Recommend

  • What is the right way to “notify” processors without blocking?
  • MPI_Barrier() does not work on a small cluster
  • C, MPI: Program not terminating and not printing numbers
  • Signal: Segmentation fault (11) in MPI C++
  • MPI_Scatter - not working as expected
  • Redirecting stdout from children spawned via MPI_Comm_spawn
  • R + Fortran + MPI memory not mapped error
  • Fortran90 derived types with mpi, alignment issue?
  • MPI_Allreduce mix elements in the sum
  • MPI_Gather() the central elements into a global matrix
  • Eliminate redundancy with CRTP and multiple inheritance
  • Modifying a array in a function in C
  • Implicit declaration of function 'sum' is invalid in C99
  • friend class with forward class declaration does not compile
  • Doubt regarding a tail optimized code under 'gdb'
  • Passing arguments to executable from command line
  • rsa encryption decryption using c
  • Calling main function from another function in C
  • A simple question about type coercion in C++
  • MPICH communication failed
  • OPENCV : CUDA context initialization for different methods
  • Pass Arbitrary Sized 2 Dimension Array
  • a simple person class in objective c
  • Implicit cast from const string to bool [duplicate]
  • std::system Exception when instantiating a singleton object
  • Call a macro with parameters : Python win32com API
  • Get the App path without the app name at the end of the app
  • Getting short path in python
  • How to read piped content in C?
  • C: Incompatible pointer type initializing
  • AES padding and writing the ciphertext to a disk file
  • Convert array of 8 bytes to signed long in C++
  • Rearranging Cells in UITableView Bug & Saving Changes
  • Circular dependency while pushing http interceptor
  • How to get icons for entities from eclipse?
  • Linker errors when using intrinsic function via function pointer
  • FormattedException instead of throw new Exception(string.Format(…)) in .NET
  • Turn off referential integrity in Derby? is it possible?
  • LevelDB C iterator
  • JaxB to read class hierarchy