Configuring Ipython for Parallel Computing - Revision history

Sl2625 at 15:16, 13 October 2022

2022-10-13T15:16:13Z

Sl2625 at 15:15, 13 October 2022

2022-10-13T15:15:11Z

Sl2625 at 14:08, 13 October 2022

2022-10-13T14:08:06Z

Sl2625 at 13:56, 13 October 2022

2022-10-13T13:56:32Z

Sl2625 at 13:55, 13 October 2022

2022-10-13T13:55:44Z

Sl2625: /* Example with MPI: */

2022-10-13T13:53:04Z

Example with MPI:

Sl2625 at 13:51, 13 October 2022

2022-10-13T13:51:43Z

Sl2625 at 13:51, 13 October 2022

2022-10-13T13:51:07Z

Sl2625 at 13:49, 13 October 2022

2022-10-13T13:49:51Z

Sl2625: Added description of changes to configure IPython for parallel computing using the cluster

2022-10-13T13:49:16Z

Added description of changes to configure IPython for parallel computing using the cluster

New page

==== Step 1. From Terminal run following command to create a parallel profile: ====
$ ipython3 profile create --parallel --profile=myprofile

If this is completed successfully, then there should be some config files like ipcontroller_config.py, ipcluster_config.py, etc in the folder <NetID>/.ipython/profile_myprofile. Check this folder, if you are missing these config files, there might be some issue with the installation of IPython on the system.

==== Step 2. We modify ipcluster_config.py so that engines are launched in remote machines and controller is launched on local machine: ====
$ cd <NetID>/.ipython/profile_myprofile

$ vi ipcluster_config.py

Now in this file we modify the following (uncomment the line and make the changes as shown):

c.Cluster.engine_launcher_class = 'ssh' -->line 528

c.SSHControllerLauncher.controller_args = ['--ip=*'] -->line 1689

c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2} -->line 2423

Save the file and exit. Note that in the c.SSHEngineSetLauncher.engines field, you need to put which machines (ramsey,fibonacci,etc) or nodes (pnode01,02,etc) and how many kernels you want from each. In my example I have taken 2 each from pnode10 and pnode11

Next, in file ipcontroller_config.py in the same location:

c.IPController.ip = '0.0.0.0'

Save and exit file. This step is important because otherwise the controller won't be able to listen on the engines and no parallel computation will happen.

==== Step 3. With this you should have parallel computing set up. Examples with IPython and MPI: ====

====== Only IPython: ======
Launch IPython with the profile you created and edited:

$ ipython3 --profile=myprofile

Code:

import time

import ipyparallel as ipp

def parallel_example():

return f"Hello World!!"

<nowiki>#</nowiki> request a cluster

with ipp.Cluster() as rc:

# get a view on the cluster

view = rc.load_balanced_view()

# submit the tasks

asyncresult = view.map_async(parallel_example)

# wait interactively for results

asyncresult.wait_interactive()

# retrieve actual results

result = asyncresult.get()

print("\n".join(result))
[[File:IPython Parallel Computing Example.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Example.png|left|662x662px|IPython Parallel Computing Example]]

On execution of this code:
[[File:IPython Parallel Computing Result 1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|left|655x655px|IPython Parallel Computing Result-1]]
[[File:IPython Parallel Computing Result 2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|left|658x658px|IPython Parallel Computing Result-2]]

====== Example with MPI: ======
Code (taken from <nowiki>https://ipyparallel.readthedocs.io/en/latest/</nowiki>):

import ipyparallel as ipp

def mpi_example():

from mpi4py import MPI

comm = MPI.COMM_WORLD

return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"

<nowiki>#</nowiki> request an MPI cluster with 4 engines

with ipp.Cluster(engines='mpi', n=4) as rc:

# get a broadcast_view on the cluster which is best

# suited for MPI style computation

view = rc.broadcast_view()

# run the mpi_example function on all engines in parallel

r = view.apply_sync(mpi_example)

# Retrieve and print the result from the engines

print("\n".join(r))

Result:
[[File:IPython Parallel with MPI.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20with%20MPI.png|left|frameless|768x768px]]

@@ Line 73: / Line 73: @@
 ======Example with MPI======
-Code (taken from <nowiki>https://ipyparallel.readthedocs.io/en/latest/</nowiki>):
+Code (taken from [https://ipyparallel.readthedocs.io/en/latest/ here]):
  import ipyparallel as ipp

← Older revision		Revision as of 13:56, 13 October 2022
Line 68:		Line 68:


		+	On execution of this code:

		+	[[File:IPython_Parallel_Computing_Result_1.png\|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png\|alt=\|655x655px]][[File:IPython_Parallel_Computing_Result_2.png\|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png\|alt=\|frameless\|658x658px]]
−
−
−
−
−
−
−
−
−
−
−
−
−	On execution of this code:[[File:IPython Parallel Computing Result 1.png\|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png\|left\|655x655px\|IPython Parallel Computing Result-1]]
−	[[File:IPython Parallel Computing Result 2.png\|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png\|left\|658x658px\|IPython Parallel Computing Result-2]]
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−
−

	====== Example with MPI: ======		====== Example with MPI: ======

← Older revision		Revision as of 13:55, 13 October 2022
Line 63:		Line 63:

	print("\n".join(result))		print("\n".join(result))
		+
−	[[File:IPython Parallel Computing Example.png\|left\|frameless\|635x635px\|IPython Parallel Computing Example]]
		+	[[File:IPython_Parallel_Computing_Example.png\|alt=\|frameless\|635x635px]]
		+

← Older revision		Revision as of 13:53, 13 October 2022
Line 63:		Line 63:

	print("\n".join(result))		print("\n".join(result))
−	[[File:IPython Parallel Computing Example~~.png\|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Example~~.png\|left\|~~662x662px~~\|IPython Parallel Computing Example]]	+	[[File:IPython Parallel Computing Example.png\|left\|frameless\|635x635px\|IPython Parallel Computing Example]]
−

← Older revision		Revision as of 13:51, 13 October 2022
Line 79:		Line 79:


		⚫	On execution of this code:[[File:IPython Parallel Computing Result 1.png\|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png\|left\|655x655px\|IPython Parallel Computing Result-1]]
−	On execution of this code:
⚫	[[File:IPython Parallel Computing Result 1.png\|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png\|left\|655x655px\|IPython Parallel Computing Result-1]]
	[[File:IPython Parallel Computing Result 2.png\|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png\|left\|658x658px\|IPython Parallel Computing Result-2]]		[[File:IPython Parallel Computing Result 2.png\|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png\|left\|658x658px\|IPython Parallel Computing Result-2]]
		+
		+

@@ Line 5: / Line 5: @@
 ==== Step 2. We modify ipcluster_config.py so that engines are launched in remote machines and controller is launched on local machine ====
-$ cd <NetID>/.ipython/profile_myprofile
+ $ cd <NetID>/.ipython/profile_myprofile
-$ vi ipcluster_config.py
+ $ vi ipcluster_config.py
 Now in this file we modify the following (uncomment the line and make the changes as shown):
-c.Cluster.engine_launcher_class = 'ssh'                                               -->line 528
+ c.Cluster.engine_launcher_class = 'ssh'                              -->line 528
-c.SSHControllerLauncher.controller_args = ['--ip=*']                            -->line 1689
+ c.SSHControllerLauncher.controller_args = ['--ip=*']                 -->line 1689
-c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2}           -->line 2423
+ c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2}           -->line 2423
 Save the file and exit. Note that in the c.SSHEngineSetLauncher.engines field, you need to put which machines (ramsey,fibonacci,etc) or nodes (pnode01,02,etc) and how many kernels you want from each. In my example I have taken 2 each from pnode10 and pnode11
@@ Line 21: / Line 21: @@
 Next, in file ipcontroller_config.py in the same location:
-c.IPController.ip = '0.0.0.0'
+ c.IPController.ip = '0.0.0.0'
 Save and exit file. This step is important because otherwise the controller won't be able to listen on the engines and no parallel computation will happen.
-==== Step 3. With this you should have parallel computing set up. Examples with IPython and MPI ====
+====Step 3. With this you should have parallel computing set up. Examples with IPython and MPI====
-====== Only IPython ======
+======Only IPython======
 Launch IPython with the profile you created and edited:
-$ ipython3 --profile=myprofile
+ $ ipython3 --profile=myprofile
 Code:
-import time
+ import time
-import ipyparallel as ipp
+ import ipyparallel as ipp
-def parallel_example():
+ def parallel_example():
-    return f"Hello World!!"
+    return f"Hello World!!"
-<nowiki>#</nowiki> request a cluster
+ <nowiki>#</nowiki> request a cluster
-with ipp.Cluster() as rc:
+ with ipp.Cluster() as rc:
-    # get a view on the cluster
+     # get a view on the cluster
-    view = rc.load_balanced_view()
+    view = rc.load_balanced_view()
-    # submit the tasks
+    # submit the tasks
-    asyncresult = view.map_async(parallel_example)
+    asyncresult = view.map_async(parallel_example)
-    # wait interactively for results
+    # wait interactively for results
-    asyncresult.wait_interactive()
+    asyncresult.wait_interactive()
-    # retrieve actual results
+    # retrieve actual results
-    result = asyncresult.get()
+    result = asyncresult.get()
-    print("\n".join(result))
+    print("\n".join(result))
 [[File:IPython_Parallel_Computing_Example.png|alt=|frameless|635x635px]]
@@ Line 72: / Line 72: @@
 [[File:IPython_Parallel_Computing_Result_1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|alt=|655x655px]][[File:IPython_Parallel_Computing_Result_2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|alt=|frameless|658x658px]]
-====== Example with MPI ======
+======Example with MPI======
 Code (taken from <nowiki>https://ipyparallel.readthedocs.io/en/latest/</nowiki>):
-import ipyparallel as ipp
+ import ipyparallel as ipp
-def mpi_example():
+ def mpi_example():
-    from mpi4py import MPI
+    from mpi4py import MPI
-    comm = MPI.COMM_WORLD
+    comm = MPI.COMM_WORLD
-    return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"
+    return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"
-<nowiki>#</nowiki> request an MPI cluster with 4 engines
+ <nowiki>#</nowiki> request an MPI cluster with 4 engines
-with ipp.Cluster(engines='mpi', n=4) as rc:
+ with ipp.Cluster(engines='mpi', n=4) as rc:
-    # get a broadcast_view on the cluster which is best
+    # get a broadcast_view on the cluster which is best
-    # suited for MPI style computation
+    # suited for MPI style computation
-    view = rc.broadcast_view()
+    view = rc.broadcast_view()
-    # run the mpi_example function on all engines in parallel
+    # run the mpi_example function on all engines in parallel
-    r = view.apply_sync(mpi_example)
+    r = view.apply_sync(mpi_example)
-    # Retrieve and print the result from the engines
+    # Retrieve and print the result from the engines
-    print("\n".join(r))
+    print("\n".join(r))
 Result:

@@ Line 64: / Line 64: @@
     print("\n".join(result))
 [[File:IPython Parallel Computing Example.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Example.png|left|662x662px|IPython Parallel Computing Example]]
@@ Line 81: / Line 82: @@
 On execution of this code:[[File:IPython Parallel Computing Result 1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|left|655x655px|IPython Parallel Computing Result-1]]
 [[File:IPython Parallel Computing Result 2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|left|658x658px|IPython Parallel Computing Result-2]]