Difference between revisions of "Configuring Ipython for Parallel Computing"

Jump to navigation Jump to search

Configuring Ipython for Parallel Computing (view source)

Revision as of 11:16, 13 October 2022

28 bytes removed ,  11:16, 13 October 2022
m
no edit summary
m
 
(One intermediate revision by the same user not shown)
 
==== Step 2. We modify ipcluster_config.py so that engines are launched in remote machines and controller is launched on local machine ====
$ cd <NetID>/.ipython/profile_myprofile
 
$ vi ipcluster_config.py
 
Now in this file we modify the following (uncomment the line and make the changes as shown):
 
c.Cluster.engine_launcher_class = 'ssh'                                               -->line 528
 
c.SSHControllerLauncher.controller_args = ['--ip=*']                            -->line 1689
 
c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2}           -->line 2423
 
Save the file and exit. Note that in the c.SSHEngineSetLauncher.engines field, you need to put which machines (ramsey,fibonacci,etc) or nodes (pnode01,02,etc) and how many kernels you want from each. In my example I have taken 2 each from pnode10 and pnode11
Next, in file ipcontroller_config.py in the same location:
 
c.IPController.ip = '0.0.0.0'
 
Save and exit file. This step is important because otherwise the controller won't be able to listen on the engines and no parallel computation will happen.
 
==== Step 3. With this you should have parallel computing set up. Examples with IPython and MPI ====
 
====== Only IPython ======
Launch IPython with the profile you created and edited:
 
$ ipython3 --profile=myprofile
 
Code:
 
import time
 
import ipyparallel as ipp
 
def parallel_example():
 
    return f"Hello World!!"
 
<nowiki>#</nowiki> request a cluster
 
with ipp.Cluster() as rc:
 
    # get a view on the cluster
 
    view = rc.load_balanced_view()
 
    # submit the tasks
 
    asyncresult = view.map_async(parallel_example)
 
    # wait interactively for results
 
    asyncresult.wait_interactive()
 
    # retrieve actual results
 
    result = asyncresult.get()
 
    print("\n".join(result))
 
[[File:IPython_Parallel_Computing_Example.png|alt=|frameless|635x635px]]
 
[[File:IPython_Parallel_Computing_Result_1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|alt=|655x655px]][[File:IPython_Parallel_Computing_Result_2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|alt=|frameless|658x658px]]
 
====== Example with MPI ======
Code (taken from <nowiki>[https://ipyparallel.readthedocs.io/en/latest/</nowiki> here]):
 
import ipyparallel as ipp▼
 
def mpi_example():▼
 
    from mpi4py import MPI▼
 
    comm = MPI.COMM_WORLD▼
 
    return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"▼
 
<nowiki>#</nowiki> request an MPI cluster with 4 engines▼
 
with ipp.Cluster(engines='mpi', n=4) as rc:▼
 
    # get a broadcast_view on the cluster which is best▼
 
    # suited for MPI style computation▼
 
    view = rc.broadcast_view()▼
 
    # run the mpi_example function on all engines in parallel▼
 
    r = view.apply_sync(mpi_example)▼
 
    # Retrieve and print the result from the engines▼
 
▲ import ipyparallel as ipp
    print("\n".join(r))   ▼
▲ def mpi_example():
▲    from mpi4py import MPI
▲    comm = MPI.COMM_WORLD
▲    return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"
▲ <nowiki>#</nowiki> request an MPI cluster with 4 engines
▲ with ipp.Cluster(engines='mpi', n=4) as rc:
▲    # get a broadcast_view on the cluster which is best
▲    # suited for MPI style computation
▲    view = rc.broadcast_view()
▲    # run the mpi_example function on all engines in parallel
▲    r = view.apply_sync(mpi_example)
▲    # Retrieve and print the result from the engines
▲    print("\n".join(r))   
 
Result:
Sl2625
53

edits

Retrieved from "https://e.math.cornell.edu/wiki/index.php/Special:MobileDiff/216...232"

Navigation menu