<br><font size=2 face="sans-serif">On Mon, </font><font size=2 face="Courier New">2009-11-30 at</font><font size=2 face="sans-serif"> 17:31 </font><font size=1 face="sans-serif"><b><ashley@pittman.co.uk></b></font><font size=2 face="sans-serif"> wrote:</font>
<br>
<br><font size=2 face="Courier New">>I knew you had to do this when running OpenMPI with slurm however I'd<br>
>never done it myself. My test cluster has both installed so I should be<br>
>able to try it, do you happen to know if you need and special configure<br>
>options to either to allow this?</font>
<br>
<br><font size=2 face="Courier New">I used slurm 2.0.1 and openmpi_1.3.3, uppers versions should work also.</font>
<br><font size=2 face="Courier New">I don't know the special configure except in my $PATH, I have added the PATH</font>
<br><font size=2 face="Courier New">to where is installed my openMPI_1.3.3 binaries and libs.</font>
<br><font size=2 face="Courier New">Check the path with "type mpirun" command,it should show the PATH to openmpi.</font>
<br>
<br><font size=2 face="Courier New">>Does the mpirun job (i.e. the processes we want) have it's own slurm job<br>
>step or does it share the job step with the allocation?</font>
<br>
<br><font size=2 face="Courier New">Just after salloc step is:</font>
<br><font size=2 face="Courier New">[thipa@vb0 openmpi]$ salloc.sh</font>
<br><font size=2 face="Courier New">salloc: Granted job allocation 27828</font>
<br><font size=2 face="Courier New">[thipa@vb0 openmpi]$</font>
<br><font size=2 face="Courier New">[thipa@vb0 openmpi]$ squeue -s</font>
<br><font size=2 face="Courier New">STEPID NAME PARTITION USER TIME NODELIST</font>
<br><font size=2 face="Courier New">[thipa@vb0 openmpi]$ squeue</font>
<br><font size=2 face="Courier New"> JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)</font>
<br><font size=2 face="Courier New"> 27828 jlg bash thipa R 0:25 2 vb[8,10]</font>
<br>
<br><font size=2 face="Courier New">After mpirun step is:</font>
<br><font size=2 face="Courier New">[thipa@vb0 openmpi]$ squeue -s</font>
<br><font size=2 face="Courier New">STEPID NAME PARTITION USER TIME NODELIST</font>
<br><font size=2 face="Courier New">27828.0 orted jlg thipa 1:02 vb[8,10]</font>
<br><font size=2 face="Courier New">[thipa@vb0 openmpi]$</font>
<br>
<br><font size=2 face="Courier New">I believe it can't share job step, each job step is its own.</font>
<br><font size=2 face="Courier New"><br>
>I also notice the /proc/version in the patch, does this mean the patch<br>
>works on an OS other than Linux?</font>
<br>
<br><font size=2 face="Courier New">It is not complete, to run on other OS that linux you must have two branches:</font>
<br><font size=2 face="Courier New">1 - with /proc/version using readdir /proc and /proc/$pid/cmdline</font>
<br><font size=2 face="Courier New">2 - with "ps -edf | grep slurmstepd" something like this.</font>
<br>
<br><font size=2 face="Courier New">>What happens if you run salloc... srun? Does this work with the<br>
>existing support and how should users know which resource manager plugin<br>
>to pick (Ideally padb could do the right thing).</font>
<br>
<br><font size=2 face="Courier New">You mean salloc ... srun ...mpirun prog ?</font>
<br><font size=2 face="Courier New">That's what I have experimented:</font>
<br>
<br><font size=2 face="Courier New">[thipa@vb0 openmpi]$ salloc.sh</font>
<br><font size=2 face="Courier New">salloc: Granted job allocation 27830</font>
<br><font size=2 face="Courier New">[thipa@vb0 openmpi]$ </font>
<br><font size=2 face="Courier New">[thipa@vb0 openmpi]$ srun -n1 mpirun -bynode -n 6 ./pp_sndrcv_spbl</font>
<br><font size=2 face="Courier New">srun: Warning: can't run 1 processes on 2 nodes, setting nnodes to 1</font>
<br><font size=2 face="Courier New">I am, process 0 starting on vb8, total by srun 6</font>
<br><font size=2 face="Courier New">Me, process 0, send 1000 to process 2</font>
<br><font size=2 face="Courier New">I am, process 2 starting on vb8, total by srun 6</font>
<br><font size=2 face="Courier New">I am, process 4 starting on vb8, total by srun 6</font>
<br><font size=2 face="Courier New">I am, process 1 starting on vb10, total by srun 6</font>
<br><font size=2 face="Courier New">I am, process 5 starting on vb10, total by srun 6</font>
<br><font size=2 face="Courier New">I am, process 3 starting on vb10, total by srun 6</font>
<br>
<br><font size=2 face="Courier New">There are 2 steps:</font>
<br><font size=2 face="Courier New">[thipa@vb0 openmpi]$ squeue -s</font>
<br><font size=2 face="Courier New">STEPID NAME PARTITION USER TIME NODELIST</font>
<br><font size=2 face="Courier New">27830.0 mpirun jlg thipa 0:22 vb8</font>
<br><font size=2 face="Courier New">27830.1 orted jlg thipa 0:22 vb10</font>
<br><font size=2 face="Courier New">[thipa@vb0 openmpi]$</font>
<br>
<br><font size=2 face="Courier New">And rmgr=slurm doesn't work (existing support)</font>
<br><font size=2 face="Courier New">You just catch the stack of orted:</font>
<br>
<br><font size=2 face="Courier New">[thipa@vb0 openmpi]$ padb_r341 -O stack-shows-locals=no -O stack-shows-params=no -O rmgr=slurm --verbose -tx 27830</font>
<br><font size=2 face="Courier New">Loading config from "/etc/padb.conf"</font>
<br><font size=2 face="Courier New">Loading config from "/home_nfs/thipa/.padbrc"</font>
<br><font size=2 face="Courier New">Loading config from environment</font>
<br><font size=2 face="Courier New">Loading config from command line</font>
<br><font size=2 face="Courier New">Setting 'rmgr' to 'slurm'</font>
<br><font size=2 face="Courier New">Setting 'stack_shows_locals' to 'no'</font>
<br><font size=2 face="Courier New">Setting 'stack_shows_params' to 'no'</font>
<br>
<br><font size=2 face="Courier New">Collecting information for job '27830'</font>
<br>
<br><font size=2 face="Courier New">Attaching to job 27830</font>
<br><font size=2 face="Courier New">Job has 1 process(es)</font>
<br><font size=2 face="Courier New">Job spans 2 host(s)</font>
<br><font size=2 face="Courier New">Mode 'stack' mode specific options:</font>
<br><font size=2 face="Courier New"> gdb_retry_count : '3'</font>
<br><font size=2 face="Courier New"> max_distinct_values : '3'</font>
<br><font size=2 face="Courier New"> stack_shows_locals : '0'</font>
<br><font size=2 face="Courier New"> stack_shows_params : '0'</font>
<br><font size=2 face="Courier New"> stack_strip_above : 'elan_waitWord,elan_pollWord,elan_deviceCheck,opal_condition_wait,opal_progress'</font>
<br><font size=2 face="Courier New"> stack_strip_below : 'main,__libc_start_main,start_thread'</font>
<br><font size=2 face="Courier New"> strip_above_wait : '1'</font>
<br><font size=2 face="Courier New"> strip_below_main : '1'</font>
<br><font size=2 face="Courier New">-----------------</font>
<br><font size=2 face="Courier New">[0] (1 processes)</font>
<br><font size=2 face="Courier New">-----------------</font>
<br><font size=2 face="Courier New">main() at main.c:13</font>
<br><font size=2 face="Courier New"> orterun() at orterun.c:686</font>
<br><font size=2 face="Courier New"> opal_event_dispatch() at ?:?</font>
<br><font size=2 face="Courier New"> opal_event_base_loop() at ?:?</font>
<br><font size=2 face="Courier New"> poll_dispatch() at ?:?</font>
<br><font size=2 face="Courier New"> poll() at ?:?</font>
<br><font size=2 face="Courier New"> ??() at ?:?</font>
<br><font size=2 face="Courier New">result from parallel command is 0 (state=shutdown)</font>
<br><font size=2 face="Courier New">[thipa@vb0 openmpi]$</font>
<br>
<br><font size=2 face="Courier New">>> [thipa@machu0 padb_open]$ ./padb -O rmgr="sl-orte" -O<br>
>> stack-shows-locals=no -O stack-shows-params=no --debug=verbose=all<br>
>> -tx 8324 <br>
>> DEBUG (verbose): 0: There are 1 processes over 3 hosts <br>
<br>
>This isn't great, the number of processes expected is so far only used<br>
>to check for missing processes but there are other potential uses for it<br>
>so I'd rather it was correct.</font>
<br>
<br><font size=2 face="Courier New">I will dig it more, I don't know the meaning of processes number actually you do with.</font>
<br>
<br><font size=2 face="Courier New">>> I don't use scontrol listpids, because I found this command not a<br>
>> universal method (some version doesn't have it), <br>
>> and may issued error message such as : <br>
>> slurmd[machu139]: proctrack/pgid does not implement<br>
>> slurm_container_get_pids <br>
<br>
>I'd prefer to use this if at all possible, this option was added at a<br>
>request my be several years ago so I'd have thought most versions have<br>
>it by now, can you be clearer on the versions where it doesn't work?</font>
<br>
<br><font size=2 face="Courier New">It work only for slurm upper from 1.2, may be some clients have it still ?</font>
<br><font size=2 face="Courier New">If you can get rid of messages above (slurmd[hostnn]: proctrack/pgid does not implement)</font>
<br><font size=2 face="Courier New">I am ok.</font>
<br>
<br><font size=2 face="Courier New">Thipadin.</font>
<br>
<br>