10[1].1.1.39.5046

Outstanding Issues in the MPIBLACS � R. Clint Whaley y November 6, 1997 DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT Contents 1 Introduction 1 2 BLACS to MPI interface 1 3 Globally-blocking context operations 1 4 Inter-language operations 2 4.1 Interim solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 5 Integer system contexts 3 5.1 The interim routines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 5.1.1 SYS2BLACS HANDLE . . . . . . . . . . . . . . . . . . . . . . . . . 3 5.1.2 BLACS2SYS HANDLE . . . . . . . . . . . . . . . . . . . . . . . . . 4 5.1.3 FREE BLACS SYSTEM HANDLE . . . . . . . . . . . . . . . . . . 4 5.2 Example of usage for interim routines . . . . . . . . . . . . . . . . . . . . . 4 6 MPI INIT 5 7 ANSI C 5 8 Testing 6 9 Conclusion 6 REFERENCES 7 � This work was supported in part by DARPA and ARO under contract number DAAL03-91-C-0047, and in part by the Applied Mathematical Sciences subprogram of the O�ce of Energy Research, U.S. Department of Energy, under Contract DE-AC05-84OR21400. y Dept. of Computer Sciences, Univ. of TN, Knoxville, TN 37996, rwhaley@cs.utk.edu i DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT 1 1 Introduction This note discusses some outstanding issues for the alpha release of the MPIBLACS. We hope that user input and the continuing MPI standardization e�ort will allow us to settle these issues, and thus proceed to a full release. Most of these problems are closely related to issues being studied by the MPI committee. We have drafted a small note [4] describing our proposals for MPI, which expand on the sketches given here. 2 BLACS to MPI interface It is important that users wishing to use the BLACS in conjunction with native MPI calls be able to do so. For this reason, the BLACS accept an MPI communicator when setting up the BLACS, allowing for an MPI user to call the BLACS. The reverse interface is also required. I.e., a user may be using the BLACS for the majority of his code, and then �nd he needs access to a MPI speci�c routine. The BLACS have therefore de�ned that the user can obtain an MPI communicator from a BLACS context through the support routine BLACS GET. What has not so far been de�ned is the speci�cs of this communicator. At the moment, the user knows that the returned communicator will have ranks corresponding to those returned by BLACS PNUM, and little else. Further re�nement of this interface is probably required. It may be convenient, for instance, to de�ne that two additional communicators will be cached on the communicator returned by BLACS GET. These cached communicators would correspond to the process's process row and column. Similarly, the BLACS/MPI interface might be further aided by specifying that the BLACS will have made the appropriate calls to MPI CART CREATE so that the communicator is set up such that calls to MPI CART RANK and MPI CART COORD will provide mappings between the BLACS' 2D view of the processes, and MPI's 1D view. It is not clear this will be required, as the BLACS have routines which can perform these operations (BLACS PNUM and BLACS PCOORD). At any rate, we have not yet written code that requires this interface, and so it is di�cult for us to specify exactly what this interface should be. We hope our continuing e�orts in MPI usage, along with user input, will allow us to de�ne this interface more completely. 3 Globally-blocking context operations MPI has de�ned that when the user creates a new communicator from a previous com- municator, the communicator creation routine will block until all processes in the original communicator call the routine. This means that if a new communicator is formed based on MPI COMM WORLD, for instance, then all processes must call the communicator creation routine. Communicator creation occurs in the BLACS in the grid creation routines (BLACS GRIDINIT and BLACS GRIDMAP). These routines were originally designed so that only the those pro- DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT 2 cesses who are de�ned in the new grid need call the routines. This does not match what is required by MPI. For now, we have little choice but to de�ne that on MPI, all processes in the context passed to the grid creation routines must call the routine. It is our hope that MPI will relax this restriction. 4 Inter-language operations The BLACS allow the user to mix Fortran77 and C code (via the Fortran77 and C inter- faces). This means that you can, for instance, de�ne a context using the Fortran77 routine BLACS GRIDINIT, and then use that context as input for a C interface broadcast. This leads to problems on MPI, where inter-language operations are unde�ned. In particular, the translation between C and Fortran77 communicators is problematic. At present, the BLACS use a variety of ways to handle this problem. First, regardless of whether the BLACS C or Fortran77 interface is called, the same MPI internals are used. The BLACS can call either the C or Fortran77 interface to MPI, based on user input. The problem of translating one communicator type to another still remains, however. The present (and unsatisfactory) solution to this problem is described below. We anticipate that the next MPI standard will make allowance of inter-language oper- ations, and that this will solve our inter-language problems. 4.1 Interim solution The �rst thing that should be understood is when a communicator translation must take place. Communicators correspond to BLACS contexts, and are thus formed in the BLACS context creation routines, BLACS GRIDMAP and BLACS GRIDINIT. As previously mentioned, the BLACS call the same MPI interface regardless of what BLACS interface is being called. Therefore, a communicator translation will only take place when the user calls BLACS GRIDMAP or BLACS GRIDINIT from a di�erent language interface than the MPI interface the BLACS are internally using. Therefore, if the user calls Cblacs gridinit, but the BLACS are calling the Fortran77 interface to MPI, a translation will have to take place. Similarly, if the user calls BLACS GRIDINIT, and the BLACS are calling the C interface to MPI, a translation will have to take place. The BLACS therefore o�er two methods of getting around the translation problems. First, the BLACS support several methods of doing communicator translation. Second, we allow the user to indicate which internals the BLACS should call. The idea is that if translation can take place naturally on a user's machine, he is urged to set the translation method, and let the BLACS use the default internals. However, the user can also tell the BLACS to use a particular MPI interface. Thus, if he knows he will be calling mainly the Fortran77 interface, he can tell the BLACS to call the Fortran77 interface MPI, and thus avoid the translation altogether. The BLACS o�er preprocessor macros (which can be set in the make�le include �le Bmake.inc) which allow the user to tune the BLACS to his system. These macros can be used to indicate how to translate communicators (the macro name is TRANSCOMM), or which DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT 3 MPI interface to use (the macro is WHATMPI) The BLACS installation and testing guide [3] provides details describing these macros. The BLACS have adopted the following strategy for the translation of communicators between C and Fortran77. If the user knows something about his MPI, he can tell the BLACS how to perform the translation. If the user is using MPICH or one of its derivities, the BLACS can call some MPICH internal internal routines to carry out the translation. On many systems, the C and Fortran77 communicators are the same anyway, and the BLACS can also handle this. If neither of these translation schemes work, the BLACS do the following: when a translation must occur, all ranks in the context are translated to MPI COMM WORLD, and the new communicator is formed based on MPI COMM WORLD (it is assumed that ranks in MPI COMM WORLD are the same for both languages). This has the unfortunate side e�ect of making the routine doing the translation block on MPI COMM WORLD. Thus in situations where such a translation must take place, the user should be sure to have all processes call the grid formation routine. 5 Integer system contexts A related issue is that a C communicator handle is not constrained to be an integer. This means that the user cannot standardly give it as input to the grid creation routines (these routines take integer system context handles as input). We hope that the next release of MPI will solve this problem by allowing translation of communicators between languages. If a language-neutral communicator is adopted, we may be able to take this as input. Otherwise, we may tell MPI users to translate to a Fortran77 communicator, which is constrained to be an integer. As an interim solution, we have added routines which simply map C communicators to integer handles which can then be passed to the grid creation routines. These mapping routines are explained in turn below. Note that only user's using both MPI and the C interface BLACS will need these routines. User's using purely the BLACS will obtain a default system context (MPI COMM WORLD) via BLACS GET, while user's using the Fortran77 interface already have integer context handles. Thus, in this scheme, users obtaining system contexts from Cblacs get will get integer mapping handles, while users calling Fortran77's BLACS GET will get the actual communicators. This means that users should be sure to call the same interface for their call to BLACS GET and the grid creation routine. 5.1 The interim routines 5.1.1 SYS2BLACS HANDLE int Csys2blacs handle(MPI Comm SysCtxt) SysCtxt (input) The system context to be mapped to an integer BLACS handle. This function returns an integer handle which can be passed into the grid creation routines to indicate the MPI communicator (system context) SysCtxt. DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT 4 5.1.2 BLACS2SYS HANDLE MPI Comm Cblacs2sys handle(int BlacsHandle) BlacsHandle (input) The integer BLACS handle who's mapped system context is to be determined. This function returns the MPI communicator (system context) which the integer handle BlacsHandle corresponds to. 5.1.3 FREE BLACS SYSTEM HANDLE void Cfree system handle(int BlacsHandle) BlacsHandle (input) The integer BLACS handle who's mapping is no longer needed. This routine may be called to dissassociate a BlacsHandle with a system context. Note that there is no need to do this, other than to keep memory usage down. 5.2 Example of usage for interim routines int bhandle; MPI_Comm mycomm, bcomm; . . . /* * Translate mycomm to a BLACS handle */ bhandle = Csys2blacs_handle(mycomm); /* * Form BLACS context based on mycomm */ icontxt = bhandle; Cblacs_gridinit(&icontxt, "r", 2, 2); . . . /* * Free the handle associated with mycomm */ Cfree_blacs_system_handle(bhandle); . . . /* DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT 5 * Find the communicator the ranks returned by Cblacs_pnum are based on */ Cblacs_get(icontxt, 10, &bhandle); bcomm = blacs2sys_handle(bhandle); . . . 6 MPI INIT In the present BLACS, there are some problems with MPI Init. The problems are twofold. First, the standard states that the C interface MPI Init takes pointers to argc and argv (arguments to main). It does not mention what happens if you wish to call MPI Init from anything other than the main program. Many compilers de�ne globally-addressable macros such as argc argv which allow the programmer to access these variables without explicitly passing them, but this is not ANSI standard. We are therefore unable to call MPI Init from the BLACS, and so user's who's main is in C must explicitly call MPI Init themselves. This is unsatisfactory, because the user no longer has a portable code, in the sense that if he wished to run on another BLACS platform, the MPI Init call must be removed. The second problem also involves language issues. In MPICH (the MPI implemen- tation which has been used for BLACS testing), you must call the language version of MPI INIT which matches your main program. I.e., if your main is in C, you must call MPI Init(int *argc, char **argv), and if it is in Fortran77, the programmer must call MPI INIT(IERROR). This feature makes it tough for a library such as the BLACS to operate correctly, as we cannot know in which language the user has written his main routine. If the user's main routine is in C, but the BLACS go ahead and call Fortran77's MPI INIT, MPICH calls some Fortran77 speci�c routines, which cause link errors. We have therefore split o� the only BLACS routine which calls MPI INIT, and place it into two seperate libraries. The C interface initialization library (the default name for this li- brary is blacsCinit --.a) should be linked to if the user's main routine is in C; otherwise the user should link to the Fortran77 interface initialization library blacsF77init --.a). We hope that the next draft of the MPI standard will de�ne a way to cleanly call MPI INIT in a non-language dependant fashion, as well as clearing up the argv, argc problems when the C version is called. 7 ANSI C The MPIBLACS are released in ANSI C. We believe that most users have access to an ANSI standard compiler, and we can thus take advantage of the greater type and param- eter checking available through ANSI C. This also avoids having to change our codes to avoid bugs in old C compilers which are no longer being actively supported, due to their replacement by ANSI C counterparts (an example is SUN's cc, replaced by acc or gcc). DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT 6 These reasons, coupled with the fact that the MPI standard is speci�ed in terms of ANSI C, have motivated us to provide the BLACS in ANSI C. User's unable to obtain ANSI C compilers should register their problem by mailing to blacs@cs.utk.edu. 8 Testing The �nal reason we classify the MPIBLACS as an alpha release is simply that of testing. MPI itself is still relatively new, and �nding a fully-featured and correct MPI is not always easy. However, MPI is already available on enough platforms that testing the MPIBLACS everywhere they might be ran is an impossible task. Further, the MPIBLACS have quite a few user-tunable parameters to allow them to acheive greater e�ciency on the many di�erent platforms supporting MPI. This means that user testing of codes will be absolutely necessary for us to become relatively con�dent in our implementation. 9 Conclusion If you wish to o�er suggestions, give a bug report, or have questions involving the BLACS, send mail to blacs@cs.utk.edu. Bug reports should be as concise as possible, with the smallest amount of code which can be used to demonstrate the error. You should consult the BLACS homepage (URL = http://www.netlib.org/blacs/Blacs.html), and the fol- lowing papers for further insight into the BLACS: [1, 3, 4]. The MPICH homepage (URL = http://www.mcs.anl.gov/mpi/mpich/index.html privides good on-line MPI help, and [2] should be examined as well. DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT DRAFT 7 References [1] Jack Dongarra and R. Clint Whaley. \A User's Guide to the BLACS v1.0". Technical Report UT CS-95-281, LAPACK Working Note #94, University of Tennessee, 1995. [2] Message Passing Interface Forum. MPI: A Message-Passing Interface Standard . In- ternational Journal of Supercomputer Applications and High Performance Comput- ing, 8(3/4), 1994. Special issue on MPI. Also available electronically, the url is ftp://www.netlib.org/mpi/mpi-report.ps. [3] R. Clint Whaley. \Installing and testing the BLACS". Technical Report UT CS-95- YYY, LAPACK Working Note #XXX, University of Tennessee, 1995. [4] R. Clint Whaley. \Some Plebian Extensions to MPI", 1995.