mathpub - User contributions [en]

Slurm Quick Start

2022-11-29T16:08:20Z

Sl2625: /* Run Script and Check Output */

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|alt=|frameless|451x451px]]

 
 
 

In case the nodes show up in STATE "down", run the following command to put them in idle state (then check again using sinfo command):
$ scontrol update nodename=pnode[01-64] state=idle

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

 

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]
 
 
 
 

For more information: [https://slurm.schedmd.com/ Slurm Documentation]

Slurm Quick Start

2022-11-29T16:07:58Z

Sl2625: /* Run Script and Check Output */

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|alt=|frameless|451x451px]]

 
 
 

In case the nodes show up in STATE "down", run the following command to put them in idle state (then check again using sinfo command):
$ scontrol update nodename=pnode[01-64] state=idle

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

 

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]
 
 
 
For more information: [https://slurm.schedmd.com/ Slurm Documentation]

Slurm Quick Start

2022-11-29T16:07:29Z

Sl2625: /* Run Script and Check Output */

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|alt=|frameless|451x451px]]

 
 
 

In case the nodes show up in STATE "down", run the following command to put them in idle state (then check again using sinfo command):
$ scontrol update nodename=pnode[01-64] state=idle

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

 

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]
 
 
 

For more information: [https://slurm.schedmd.com/ Slurm Documentation]

Slurm Quick Start

2022-11-29T16:07:11Z

Sl2625: /* Run Script and Check Output */

Slurm Quick Start

2022-11-29T16:06:54Z

Sl2625: /* Run Script and Check Output */

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|alt=|frameless|451x451px]]

 
 
 

In case the nodes show up in STATE "down", run the following command to put them in idle state (then check again using sinfo command):
$ scontrol update nodename=pnode[01-64] state=idle

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

 

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]
 
 
 

For more information: [https://slurm.schedmd.com/ Slurm Documentation]

Slurm Quick Start

2022-11-29T16:06:33Z

Sl2625: /* Run Script and Check Output */

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|alt=|frameless|451x451px]]

 
 
 

In case the nodes show up in STATE "down", run the following command to put them in idle state (then check again using sinfo command):
$ scontrol update nodename=pnode[01-64] state=idle

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

 

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]
 
 

For more information: [https://slurm.schedmd.com/ Slurm Documentation]

Slurm Quick Start

2022-11-29T16:05:54Z

Sl2625: /* Run Script and Check Output */

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|alt=|frameless|451x451px]]

 
 
 

In case the nodes show up in STATE "down", run the following command to put them in idle state (then check again using sinfo command):
$ scontrol update nodename=pnode[01-64] state=idle

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

 

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]
 

<blockquote>For more information: [https://slurm.schedmd.com/ Slurm Documentation] </blockquote>

Slurm Quick Start

2022-11-29T16:05:36Z

Sl2625: /* Run Script and Check Output */

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|alt=|frameless|451x451px]]

 
 
 

In case the nodes show up in STATE "down", run the following command to put them in idle state (then check again using sinfo command):
$ scontrol update nodename=pnode[01-64] state=idle

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

 

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]

<blockquote>For more information: [https://slurm.schedmd.com/ Slurm Documentation] </blockquote>

Slurm Quick Start

2022-11-29T15:59:07Z

Sl2625: /* 1. Configuring SLURM */

Test Page

2022-11-29T15:55:17Z

Sl2625: /* Handy Links */ Added link for Slurm configuration and example

== Test Page ==

This is a place to doodle, and test out formatting without having to temporarily mess up an actual page.

= Handy Links =

* [https://e.math.cornell.edu/wiki/index.php/Slurm_Quick_Start Slurm Quick Start Guide]
* [https://e.math.cornell.edu/wiki/index.php/Mathematica_Parallel_Computing_Configuration Mathematica Remote Kernel Configuration]
* [https://e.math.cornell.edu/wiki/index.php/Configuring_Ipython_for_Parallel_Computing Configuring IPython] for Parallel Computing
*The [https://math.cornell.edu Math Department] Homepage.
*The Math Department [https://math.cornell.edu/people People Pages].
*[https://people.as.cornell.edu/saml_login Link to edit] your People page.
*The [https://webwork2.math.cornell.edu/ WeBWorK] Math Homework System.

For instructors and researchers:
*Log in to [http://outlook.cornell.edu Cornell Email] on the web.
*Reset your [https://accounts.math.cornell.edu/panel/ Math Account] Password.
*Instructors can [https://accounts.math.cornell.edu/panel/invite.php send an invitation] to set up a Math Account to any NetID.
*[https://pi.math.cornell.edu View] the old server, pi.
*[https://pi.math.cornell.edu/m/ADMIN/Protected Log in] to pi.
*The 'Syllabus File' [https://e.math.cornell.edu/apps/courseinfo/ Course Materials] database.
*[https://e.math.cornell.edu/webdisk Access your files] on the Math system Webdisk.
*Other ways to access your Math files.
*Use the Math Department computation machines.
*How to print at the Math department.
*Printer activity and availability.
*How to scan.
* View the status of the Math systems.

Links for Staff:
*The [https://dynomite.math.cornell.edu Department Database].
*How to connect to the staff file share.
*How to connect to your work computer from home.
* Math Department Student [https://e.math.cornell.edu/apps/emp Employment] Site.
**Student Employee [https://e.math.cornell.edu/apps/emp-review/ Performance Review] Site.
**[https://e.math.cornell.edu/apps/emp/admin Administrative] Login

[[Math How Tos]]

Slurm Quick Start

2022-11-29T15:53:47Z

Sl2625: /* 2. Running a Slurm Batch Job on Multiple Nodes */

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|alt=|frameless|451x451px]]

 
 
 

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

 

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]

Slurm Quick Start

2022-11-29T15:50:43Z

Sl2625: /* 2. Running a Slurm Batch Job on Multiple Nodes */

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|alt=|frame]]<blockquote>

</blockquote>

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]

Slurm Quick Start

2022-11-29T15:49:58Z

Sl2625: /* 2. Running a Slurm Batch Job on Multiple Nodes */

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|frameless|509x509px]]<blockquote>

</blockquote>

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]

Slurm Quick Start

2022-11-29T15:49:33Z

Sl2625:

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|frameless|509x509px]]<blockquote>

</blockquote>

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]

Slurm Quick Start

2022-11-29T15:47:51Z

Sl2625: /* 2. Running a Slurm Batch Job on Multiple Nodes */

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|frameless|509x509px]]

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]

Slurm Quick Start

2022-11-29T15:47:38Z

Sl2625: /* 2. Running a Slurm Batch Job on Multiple Nodes */

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|frameless|509x509px]]

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]

Slurm Quick Start

2022-11-29T15:47:17Z

Sl2625: /* 1. Configuring SLURM */

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|frameless|509x509px]]

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]

Slurm Quick Start

2022-11-29T15:45:42Z

Sl2625: Added description of changes to configure Slurm for running batch jobs using the cluster

This page explains how to get the configure and run a batch job on the SLURM cluster from the head node.

== 1. Configuring SLURM ==
Slurm has 2 entities: a slurmctld controller node and multiple slurmd host nodes where we can run jobs in parallel.

To start the slurm controller on a machine we need to give this command from root:
$ systemctl start slurmctld.service
After running this, we can verify that slurm controller is running by viewing the log with the command:
$ tail /var/log/slurmctld.log
The nodes should be configured to run with the controller. You can see the information about available nodes using this command:
$ sinfo
[[File:Sinfo.png|left|frameless|509x509px]]

== 2. Running a Slurm Batch Job on Multiple Nodes ==
We can create python scripts and run these scripts in parallel on the cluster nodes. This is a simple example where we will print the task number from each node.

=== Create a Python Script ===
First we create a Python script which prints the system task number:
#!/usr/bin/python
# import sys library (needed for accepted command line args)
import sys
# print task number
print('Hello! I am a task number: ', sys.argv[1])
We will save this python script as hello-parallel.py

=== Create Slurm Script ===
Next, we need to create a Slurm script to run the python program we just created:
#!/bin/bash

# Example of running python script with a job array

#SBATCH -J hello
#SBATCH -p debug
#SBATCH --array=1-10 # how many tasks in the array
#SBATCH -c 1 # one CPU core per task
#SBATCH -t 10:00
#SBATCH -o hello-%j-%a.out

# Run python script with a command line argument
srun python3 hello-parallel.py $SLURM_ARRAY_TASK_ID
We will save this Slurm script as hello-parallel.slurm

The first few lines of this file (with #SBATCH) are used to configure different parameters we want for the execution of the python script on the cluster.

For example, -J specifies job name, -p specifies the partition on which the cluster nodes are (for us the partition is named debug), --array specifies how many tasks we want, -c specifies number of CPU cores per task.

For more details on other command line options for sbatch for configuring the cluster, please visit [https://slurm.schedmd.com/sbatch.html Slurm Sbatch Documentation]

=== Run Script and Check Output ===
Now we are ready to run the script on the cluster, specifically on 10 nodes of the cluster. To run the slurm script, simply give the command:
$ sbatch hello-parallel.slurm
This should run our python script on 10 different nodes and generate output files in the same location. The output files will have an extension .out.
[[File:Slurm op.png|left|frameless|709x709px]]

We can view the output using the less command:
[[File:Slrm.png|left|frameless|447x447px]]
[[File:Slurm output.png|left|frameless]]

File:Slrm.png

2022-11-29T15:43:54Z

Sl2625:

slrm

File:Slurm output.png

2022-11-29T15:42:27Z

Sl2625:

Slurm output

File:Slurm op.png

2022-11-29T15:40:37Z

Sl2625:

Slurm output

Slurm Quick Start

2022-11-29T15:24:09Z

Sl2625: /* Slurm Cluster Quick Start Guide */

File:Sinfo.png

2022-11-29T15:18:08Z

Sl2625:

sinfo output

Slurm Quick Start

2022-11-29T15:08:14Z

Sl2625: Created page with "== Slurm Cluster Quick Start Guide =="

== Slurm Cluster Quick Start Guide ==

Configuring Ipython for Parallel Computing

2022-10-13T15:16:13Z

Sl2625:

==== Step 1. From Terminal run following command to create a parallel profile ====
$ ipython3 profile create --parallel --profile=myprofile

If this is completed successfully, then there should be some config files like ipcontroller_config.py, ipcluster_config.py, etc in the folder <NetID>/.ipython/profile_myprofile. Check this folder, if you are missing these config files, there might be some issue with the installation of IPython on the system.

==== Step 2. We modify ipcluster_config.py so that engines are launched in remote machines and controller is launched on local machine ====
$ cd <NetID>/.ipython/profile_myprofile

$ vi ipcluster_config.py

Now in this file we modify the following (uncomment the line and make the changes as shown):

c.Cluster.engine_launcher_class = 'ssh' -->line 528

c.SSHControllerLauncher.controller_args = ['--ip=*'] -->line 1689

c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2} -->line 2423

Save the file and exit. Note that in the c.SSHEngineSetLauncher.engines field, you need to put which machines (ramsey,fibonacci,etc) or nodes (pnode01,02,etc) and how many kernels you want from each. In my example I have taken 2 each from pnode10 and pnode11

Next, in file ipcontroller_config.py in the same location:

c.IPController.ip = '0.0.0.0'

Save and exit file. This step is important because otherwise the controller won't be able to listen on the engines and no parallel computation will happen.

====Step 3. With this you should have parallel computing set up. Examples with IPython and MPI====

======Only IPython======
Launch IPython with the profile you created and edited:

$ ipython3 --profile=myprofile

Code:

import time

import ipyparallel as ipp

def parallel_example():

return f"Hello World!!"

<nowiki>#</nowiki> request a cluster

with ipp.Cluster() as rc:

# get a view on the cluster

view = rc.load_balanced_view()

# submit the tasks

asyncresult = view.map_async(parallel_example)

# wait interactively for results

asyncresult.wait_interactive()

# retrieve actual results

result = asyncresult.get()

print("\n".join(result))

[[File:IPython_Parallel_Computing_Example.png|alt=|frameless|635x635px]]

On execution of this code:

[[File:IPython_Parallel_Computing_Result_1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|alt=|655x655px]][[File:IPython_Parallel_Computing_Result_2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|alt=|frameless|658x658px]]

======Example with MPI======
Code (taken from [https://ipyparallel.readthedocs.io/en/latest/ here]):

import ipyparallel as ipp

def mpi_example():

from mpi4py import MPI

comm = MPI.COMM_WORLD

return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"

<nowiki>#</nowiki> request an MPI cluster with 4 engines

with ipp.Cluster(engines='mpi', n=4) as rc:

# get a broadcast_view on the cluster which is best

# suited for MPI style computation

view = rc.broadcast_view()

# run the mpi_example function on all engines in parallel

r = view.apply_sync(mpi_example)

# Retrieve and print the result from the engines

print("\n".join(r))

Result:
[[File:IPython Parallel with MPI.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20with%20MPI.png|left|frameless|768x768px]]

Configuring Ipython for Parallel Computing

2022-10-13T15:15:11Z

Sl2625:

==== Step 1. From Terminal run following command to create a parallel profile ====
$ ipython3 profile create --parallel --profile=myprofile

If this is completed successfully, then there should be some config files like ipcontroller_config.py, ipcluster_config.py, etc in the folder <NetID>/.ipython/profile_myprofile. Check this folder, if you are missing these config files, there might be some issue with the installation of IPython on the system.

==== Step 2. We modify ipcluster_config.py so that engines are launched in remote machines and controller is launched on local machine ====
$ cd <NetID>/.ipython/profile_myprofile

$ vi ipcluster_config.py

Now in this file we modify the following (uncomment the line and make the changes as shown):

c.Cluster.engine_launcher_class = 'ssh' -->line 528

c.SSHControllerLauncher.controller_args = ['--ip=*'] -->line 1689

c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2} -->line 2423

Save the file and exit. Note that in the c.SSHEngineSetLauncher.engines field, you need to put which machines (ramsey,fibonacci,etc) or nodes (pnode01,02,etc) and how many kernels you want from each. In my example I have taken 2 each from pnode10 and pnode11

Next, in file ipcontroller_config.py in the same location:

c.IPController.ip = '0.0.0.0'

Save and exit file. This step is important because otherwise the controller won't be able to listen on the engines and no parallel computation will happen.

====Step 3. With this you should have parallel computing set up. Examples with IPython and MPI====

======Only IPython======
Launch IPython with the profile you created and edited:

$ ipython3 --profile=myprofile

Code:

import time

import ipyparallel as ipp

def parallel_example():

return f"Hello World!!"

<nowiki>#</nowiki> request a cluster

with ipp.Cluster() as rc:

# get a view on the cluster

view = rc.load_balanced_view()

# submit the tasks

asyncresult = view.map_async(parallel_example)

# wait interactively for results

asyncresult.wait_interactive()

# retrieve actual results

result = asyncresult.get()

print("\n".join(result))

[[File:IPython_Parallel_Computing_Example.png|alt=|frameless|635x635px]]

On execution of this code:

[[File:IPython_Parallel_Computing_Result_1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|alt=|655x655px]][[File:IPython_Parallel_Computing_Result_2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|alt=|frameless|658x658px]]

======Example with MPI======
Code (taken from <nowiki>https://ipyparallel.readthedocs.io/en/latest/</nowiki>):

import ipyparallel as ipp

def mpi_example():

from mpi4py import MPI

comm = MPI.COMM_WORLD

return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"

<nowiki>#</nowiki> request an MPI cluster with 4 engines

with ipp.Cluster(engines='mpi', n=4) as rc:

# get a broadcast_view on the cluster which is best

# suited for MPI style computation

view = rc.broadcast_view()

# run the mpi_example function on all engines in parallel

r = view.apply_sync(mpi_example)

# Retrieve and print the result from the engines

print("\n".join(r))

Result:
[[File:IPython Parallel with MPI.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20with%20MPI.png|left|frameless|768x768px]]

Test Page

2022-10-13T15:11:03Z

Sl2625:

== Test Page ==

This is a place to doodle, and test out formatting without having to temporarily mess up an actual page.

= Handy Links =

* [https://e.math.cornell.edu/wiki/index.php/Mathematica_Parallel_Computing_Configuration Mathematica Remote Kernel Configuration]
* [https://e.math.cornell.edu/wiki/index.php/Configuring_Ipython_for_Parallel_Computing Configuring IPython] for Parallel Computing
*The [https://math.cornell.edu Math Department] Homepage.
*The Math Department [https://math.cornell.edu/people People Pages].
*[https://people.as.cornell.edu/saml_login Link to edit] your People page.
*The [https://webwork2.math.cornell.edu/ WeBWorK] Math Homework System.

For instructors and researchers:
*Log in to [http://outlook.cornell.edu Cornell Email] on the web.
*Reset your [https://accounts.math.cornell.edu/panel/ Math Account] Password.
*Instructors can [https://accounts.math.cornell.edu/panel/invite.php send an invitation] to set up a Math Account to any NetID.
*[https://pi.math.cornell.edu View] the old server, pi.
*[https://pi.math.cornell.edu/m/ADMIN/Protected Log in] to pi.
*The 'Syllabus File' [https://e.math.cornell.edu/apps/courseinfo/ Course Materials] database.
*[https://e.math.cornell.edu/webdisk Access your files] on the Math system Webdisk.
*Other ways to access your Math files.
*Use the Math Department computation machines.
*How to print at the Math department.
*Printer activity and availability.
*How to scan.
* View the status of the Math systems.

Links for Staff:
*The [https://dynomite.math.cornell.edu Department Database].
*How to connect to the staff file share.
*How to connect to your work computer from home.
* Math Department Student [https://e.math.cornell.edu/apps/emp Employment] Site.
**Student Employee [https://e.math.cornell.edu/apps/emp-review/ Performance Review] Site.
**[https://e.math.cornell.edu/apps/emp/admin Administrative] Login

Mathematica Parallel Computing Configuration

2022-10-13T15:08:24Z

Sl2625: Added description of changes to configure Mathematica for parallel computing using the cluster nodes as Remote Kernels

This page contains information regarding how to setup nodes in the cluster as Remote Kernels in different methods to run parallel computation

===First Method : Launch the kernels using commands in the notebook===
Use the code given below in a Mathematica notebook. Change "kernels" variable to the number of kernels wanted for parallel computation
Needs["SubKernels`RemoteKernels`"]
kernels = 4; (*Number of kernels on each machine*)
configureKernel[host_] :=
SubKernels`RemoteKernels`RemoteMachine[host, kernels];
LaunchKernels[
Map[configureKernel, {"<nowiki>ssh://ramsey/</nowiki>", "<nowiki>ssh://fibonacci/</nowiki>",
"<nowiki>ssh://hopper/</nowiki>", "<nowiki>ssh://boole/</nowiki>", "<nowiki>ssh://heaviside/</nowiki>",
"<nowiki>ssh://squid1/</nowiki>", "<nowiki>ssh://squid2/</nowiki>"}]]; (*all the available machines, change to whatever is needed*)

Printing "Hello World" in parallel after setting up 4 kernels per machine with above code

[[File:Image1.png|frameless|663x663px]]

[[File:Image.png|frameless|667x667px]]

===Second Method: Configure Remote Kernels in Mathematica Menus===
In this method, we will set up the nodes in the cluster as Remote Kernels from the Mathematica Parallelize menu. This setup is required only once, every subsequent time, simply running LaunchKernels

command should set up the remote kernels for parallel evaluation.

====Step 1====
Open Mathematica and open Parallel Kernel Configuration from the Evaluation tab

[[File:Mathematica parallel.png|frameless|660x660px]]

====Step 2====
Under Parallel, choose Remote Kernels, click on add hosts and add as shown below. Tick enable next to the number of kernels. For each, you can configure the number of kernels you want running on the machine.

[[File:Image2.png|frameless|669x669px]]

Add all the machines and number of kernels you want to add. This is a one time setup, you won't need to do this again and again.

[[File:Image3.png|frameless|406x406px]]

====Step 3====
Disable RemoteKernel Object here (if it is enabled)

[[File:Image4.png|frameless|699x699px]]

If you don't disable this, subkernels will be created on the local kernel and not on the remote hosts.

====Step 4====
Once all these settings are done, running LaunchKernels[] from any notebook will launch the specification in the menu.

In this example 3 machines were configured in menu as shown above:

[[File:Image6.png|frameless|732x732px]]

[[File:Image5.png|frameless|732x732px]]

===Third Method: Launching the kernels in a loop===
For this method, the kernels will be launched by running the below code:

Needs["SubKernels`

RemoteKernels`"]

kernels = 1;(*number of kernels on each machine*)

configureKernel[host_] :=

SubKernels`RemoteKernels`RemoteMachine[host, kernels];

For[i = 0, i < 15, i++;

If[i < 10,

LaunchKernels[

Map[configureKernel, {StringJoin[{"<nowiki>ssh://pnode0</nowiki>", ToString[i],

"/"}]}]],

LaunchKernels[

Map[configureKernel, {StringJoin[{"<nowiki>ssh://pnode</nowiki>", ToString[i],

"/"}]}]]]]
We are launching 15 kernels here. Change the number in the loop condition to launch that number of kernels.

===Parallel Evaluation Example: Generating and Plotting Mandelbrot Set===

In this section we will take a look at how the time for Mandelbrot set generation is greatly improved when we run on parallel remote kernels.

Code for generating Mandelbrot set without parallel kernels:

[[File:Image7.png|frameless|527x527px]]

Output and Timing:

[[File:Image8.png|frameless|547x547px]]

To parallelize this, we will launch 15 cores using the third method:

[[File:Image9.png|frameless|557x557px]]

Output and Timing for Parallel Evaluation:

[[File:Image10.png|frameless|566x566px]]

Code for running parallel evaluation of Mandelbrot set:

f[c_, maxiter_Integer : 10^4] :=

Abs[NestWhile[#^2 + c &, 0, Abs[#] < 2 &, 1, maxiter]] < 2

f[0]

(*True*)

f[1]

(*False*)

f[I]

(*True*)

f[I, 10^6]

(*True*)

ArrayPlot[

ParallelTable[

Boole[f[N[x + I y], 10^3]], {y, -2, 2, 1/50}, {x, -2, 2, 1/50}]]

File:Image10.png

2022-10-13T15:04:57Z

Sl2625:

Mandelbrot Parallel

File:Image9.png

2022-10-13T15:03:22Z

Sl2625:

Parallel Kernels

File:Image8.png

2022-10-13T15:01:52Z

Sl2625:

Mandelbrot output and timing

File:Image7.png

2022-10-13T15:00:05Z

Sl2625:

Mandelbrot Code

File:Image6.png

2022-10-13T14:46:41Z

Sl2625:

Res

File:Image5.png

2022-10-13T14:46:01Z

Sl2625:

Result

File:Image4.png

2022-10-13T14:43:29Z

Sl2625:

RemoteKernel Object

File:Image3.png

2022-10-13T14:41:44Z

Sl2625:

Configuring settings

File:Image2.png

2022-10-13T14:40:20Z

Sl2625:

Cofiguring

File:Mathematica parallel.png

2022-10-13T14:38:12Z

Sl2625:

Mathematica Parallel

File:Image.png

2022-10-13T14:35:44Z

Sl2625:

Hello World Example

File:Image1.png

2022-10-13T14:31:34Z

Sl2625:

Launch Remote Kernels

Configuring Ipython for Parallel Computing

2022-10-13T14:08:06Z

Sl2625:

==== Step 1. From Terminal run following command to create a parallel profile ====
$ ipython3 profile create --parallel --profile=myprofile

If this is completed successfully, then there should be some config files like ipcontroller_config.py, ipcluster_config.py, etc in the folder <NetID>/.ipython/profile_myprofile. Check this folder, if you are missing these config files, there might be some issue with the installation of IPython on the system.

==== Step 2. We modify ipcluster_config.py so that engines are launched in remote machines and controller is launched on local machine ====
$ cd <NetID>/.ipython/profile_myprofile

$ vi ipcluster_config.py

Now in this file we modify the following (uncomment the line and make the changes as shown):

c.Cluster.engine_launcher_class = 'ssh' -->line 528

c.SSHControllerLauncher.controller_args = ['--ip=*'] -->line 1689

c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2} -->line 2423

Save the file and exit. Note that in the c.SSHEngineSetLauncher.engines field, you need to put which machines (ramsey,fibonacci,etc) or nodes (pnode01,02,etc) and how many kernels you want from each. In my example I have taken 2 each from pnode10 and pnode11

Next, in file ipcontroller_config.py in the same location:

c.IPController.ip = '0.0.0.0'

Save and exit file. This step is important because otherwise the controller won't be able to listen on the engines and no parallel computation will happen.

==== Step 3. With this you should have parallel computing set up. Examples with IPython and MPI ====

====== Only IPython ======
Launch IPython with the profile you created and edited:

$ ipython3 --profile=myprofile

Code:

import time

import ipyparallel as ipp

def parallel_example():

return f"Hello World!!"

<nowiki>#</nowiki> request a cluster

with ipp.Cluster() as rc:

# get a view on the cluster

view = rc.load_balanced_view()

# submit the tasks

asyncresult = view.map_async(parallel_example)

# wait interactively for results

asyncresult.wait_interactive()

# retrieve actual results

result = asyncresult.get()

print("\n".join(result))

[[File:IPython_Parallel_Computing_Example.png|alt=|frameless|635x635px]]

On execution of this code:

[[File:IPython_Parallel_Computing_Result_1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|alt=|655x655px]][[File:IPython_Parallel_Computing_Result_2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|alt=|frameless|658x658px]]

====== Example with MPI ======
Code (taken from <nowiki>https://ipyparallel.readthedocs.io/en/latest/</nowiki>):

import ipyparallel as ipp

def mpi_example():

from mpi4py import MPI

comm = MPI.COMM_WORLD

return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"

<nowiki>#</nowiki> request an MPI cluster with 4 engines

with ipp.Cluster(engines='mpi', n=4) as rc:

# get a broadcast_view on the cluster which is best

# suited for MPI style computation

view = rc.broadcast_view()

# run the mpi_example function on all engines in parallel

r = view.apply_sync(mpi_example)

# Retrieve and print the result from the engines

print("\n".join(r))

Result:
[[File:IPython Parallel with MPI.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20with%20MPI.png|left|frameless|768x768px]]

Test Page

2022-10-13T14:07:25Z

Sl2625: Added link for IPython Config

== Test Page ==

This is a place to doodle, and test out formatting without having to temporarily mess up an actual page.

= Handy Links =

* [https://e.math.cornell.edu/wiki/index.php/Configuring_Ipython_for_Parallel_Computing Configuring IPython] for Parallel Computing
*The [https://math.cornell.edu Math Department] Homepage.
*The Math Department [https://math.cornell.edu/people People Pages].
*[https://people.as.cornell.edu/saml_login Link to edit] your People page.
*The [https://webwork2.math.cornell.edu/ WeBWorK] Math Homework System.

For instructors and researchers:
*Log in to [http://outlook.cornell.edu Cornell Email] on the web.
*Reset your [https://accounts.math.cornell.edu/panel/ Math Account] Password.
*Instructors can [https://accounts.math.cornell.edu/panel/invite.php send an invitation] to set up a Math Account to any NetID.
*[https://pi.math.cornell.edu View] the old server, pi.
*[https://pi.math.cornell.edu/m/ADMIN/Protected Log in] to pi.
*The 'Syllabus File' [https://e.math.cornell.edu/apps/courseinfo/ Course Materials] database.
*[https://e.math.cornell.edu/webdisk Access your files] on the Math system Webdisk.
*Other ways to access your Math files.
*Use the Math Department computation machines.
*How to print at the Math department.
*Printer activity and availability.
*How to scan.
* View the status of the Math systems.

Links for Staff:
*The [https://dynomite.math.cornell.edu Department Database].
*How to connect to the staff file share.
*How to connect to your work computer from home.
* Math Department Student [https://e.math.cornell.edu/apps/emp Employment] Site.
**Student Employee [https://e.math.cornell.edu/apps/emp-review/ Performance Review] Site.
**[https://e.math.cornell.edu/apps/emp/admin Administrative] Login

Configuring Ipython for Parallel Computing

2022-10-13T13:56:32Z

Sl2625:

==== Step 1. From Terminal run following command to create a parallel profile: ====
$ ipython3 profile create --parallel --profile=myprofile

If this is completed successfully, then there should be some config files like ipcontroller_config.py, ipcluster_config.py, etc in the folder <NetID>/.ipython/profile_myprofile. Check this folder, if you are missing these config files, there might be some issue with the installation of IPython on the system.

==== Step 2. We modify ipcluster_config.py so that engines are launched in remote machines and controller is launched on local machine: ====
$ cd <NetID>/.ipython/profile_myprofile

$ vi ipcluster_config.py

Now in this file we modify the following (uncomment the line and make the changes as shown):

c.Cluster.engine_launcher_class = 'ssh' -->line 528

c.SSHControllerLauncher.controller_args = ['--ip=*'] -->line 1689

c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2} -->line 2423

Save the file and exit. Note that in the c.SSHEngineSetLauncher.engines field, you need to put which machines (ramsey,fibonacci,etc) or nodes (pnode01,02,etc) and how many kernels you want from each. In my example I have taken 2 each from pnode10 and pnode11

Next, in file ipcontroller_config.py in the same location:

c.IPController.ip = '0.0.0.0'

Save and exit file. This step is important because otherwise the controller won't be able to listen on the engines and no parallel computation will happen.

==== Step 3. With this you should have parallel computing set up. Examples with IPython and MPI: ====

====== Only IPython: ======
Launch IPython with the profile you created and edited:

$ ipython3 --profile=myprofile

Code:

import time

import ipyparallel as ipp

def parallel_example():

return f"Hello World!!"

<nowiki>#</nowiki> request a cluster

with ipp.Cluster() as rc:

# get a view on the cluster

view = rc.load_balanced_view()

# submit the tasks

asyncresult = view.map_async(parallel_example)

# wait interactively for results

asyncresult.wait_interactive()

# retrieve actual results

result = asyncresult.get()

print("\n".join(result))

[[File:IPython_Parallel_Computing_Example.png|alt=|frameless|635x635px]]

On execution of this code:

[[File:IPython_Parallel_Computing_Result_1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|alt=|655x655px]][[File:IPython_Parallel_Computing_Result_2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|alt=|frameless|658x658px]]

====== Example with MPI: ======
Code (taken from <nowiki>https://ipyparallel.readthedocs.io/en/latest/</nowiki>):

import ipyparallel as ipp

def mpi_example():

from mpi4py import MPI

comm = MPI.COMM_WORLD

return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"

<nowiki>#</nowiki> request an MPI cluster with 4 engines

with ipp.Cluster(engines='mpi', n=4) as rc:

# get a broadcast_view on the cluster which is best

# suited for MPI style computation

view = rc.broadcast_view()

# run the mpi_example function on all engines in parallel

r = view.apply_sync(mpi_example)

# Retrieve and print the result from the engines

print("\n".join(r))

Result:
[[File:IPython Parallel with MPI.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20with%20MPI.png|left|frameless|768x768px]]

Configuring Ipython for Parallel Computing

2022-10-13T13:55:44Z

Sl2625:

==== Step 1. From Terminal run following command to create a parallel profile: ====
$ ipython3 profile create --parallel --profile=myprofile

If this is completed successfully, then there should be some config files like ipcontroller_config.py, ipcluster_config.py, etc in the folder <NetID>/.ipython/profile_myprofile. Check this folder, if you are missing these config files, there might be some issue with the installation of IPython on the system.

==== Step 2. We modify ipcluster_config.py so that engines are launched in remote machines and controller is launched on local machine: ====
$ cd <NetID>/.ipython/profile_myprofile

$ vi ipcluster_config.py

Now in this file we modify the following (uncomment the line and make the changes as shown):

c.Cluster.engine_launcher_class = 'ssh' -->line 528

c.SSHControllerLauncher.controller_args = ['--ip=*'] -->line 1689

c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2} -->line 2423

Save the file and exit. Note that in the c.SSHEngineSetLauncher.engines field, you need to put which machines (ramsey,fibonacci,etc) or nodes (pnode01,02,etc) and how many kernels you want from each. In my example I have taken 2 each from pnode10 and pnode11

Next, in file ipcontroller_config.py in the same location:

c.IPController.ip = '0.0.0.0'

Save and exit file. This step is important because otherwise the controller won't be able to listen on the engines and no parallel computation will happen.

==== Step 3. With this you should have parallel computing set up. Examples with IPython and MPI: ====

====== Only IPython: ======
Launch IPython with the profile you created and edited:

$ ipython3 --profile=myprofile

Code:

import time

import ipyparallel as ipp

def parallel_example():

return f"Hello World!!"

<nowiki>#</nowiki> request a cluster

with ipp.Cluster() as rc:

# get a view on the cluster

view = rc.load_balanced_view()

# submit the tasks

asyncresult = view.map_async(parallel_example)

# wait interactively for results

asyncresult.wait_interactive()

# retrieve actual results

result = asyncresult.get()

print("\n".join(result))

[[File:IPython_Parallel_Computing_Example.png|alt=|frameless|635x635px]]

On execution of this code:[[File:IPython Parallel Computing Result 1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|left|655x655px|IPython Parallel Computing Result-1]]
[[File:IPython Parallel Computing Result 2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|left|658x658px|IPython Parallel Computing Result-2]]

====== Example with MPI: ======
Code (taken from <nowiki>https://ipyparallel.readthedocs.io/en/latest/</nowiki>):

import ipyparallel as ipp

def mpi_example():

from mpi4py import MPI

comm = MPI.COMM_WORLD

return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"

<nowiki>#</nowiki> request an MPI cluster with 4 engines

with ipp.Cluster(engines='mpi', n=4) as rc:

# get a broadcast_view on the cluster which is best

# suited for MPI style computation

view = rc.broadcast_view()

# run the mpi_example function on all engines in parallel

r = view.apply_sync(mpi_example)

# Retrieve and print the result from the engines

print("\n".join(r))

Result:
[[File:IPython Parallel with MPI.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20with%20MPI.png|left|frameless|768x768px]]

Configuring Ipython for Parallel Computing

2022-10-13T13:53:04Z

Sl2625: /* Example with MPI: */

==== Step 1. From Terminal run following command to create a parallel profile: ====
$ ipython3 profile create --parallel --profile=myprofile

If this is completed successfully, then there should be some config files like ipcontroller_config.py, ipcluster_config.py, etc in the folder <NetID>/.ipython/profile_myprofile. Check this folder, if you are missing these config files, there might be some issue with the installation of IPython on the system.

==== Step 2. We modify ipcluster_config.py so that engines are launched in remote machines and controller is launched on local machine: ====
$ cd <NetID>/.ipython/profile_myprofile

$ vi ipcluster_config.py

Now in this file we modify the following (uncomment the line and make the changes as shown):

c.Cluster.engine_launcher_class = 'ssh' -->line 528

c.SSHControllerLauncher.controller_args = ['--ip=*'] -->line 1689

c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2} -->line 2423

Save the file and exit. Note that in the c.SSHEngineSetLauncher.engines field, you need to put which machines (ramsey,fibonacci,etc) or nodes (pnode01,02,etc) and how many kernels you want from each. In my example I have taken 2 each from pnode10 and pnode11

Next, in file ipcontroller_config.py in the same location:

c.IPController.ip = '0.0.0.0'

Save and exit file. This step is important because otherwise the controller won't be able to listen on the engines and no parallel computation will happen.

==== Step 3. With this you should have parallel computing set up. Examples with IPython and MPI: ====

====== Only IPython: ======
Launch IPython with the profile you created and edited:

$ ipython3 --profile=myprofile

Code:

import time

import ipyparallel as ipp

def parallel_example():

return f"Hello World!!"

<nowiki>#</nowiki> request a cluster

with ipp.Cluster() as rc:

# get a view on the cluster

view = rc.load_balanced_view()

# submit the tasks

asyncresult = view.map_async(parallel_example)

# wait interactively for results

asyncresult.wait_interactive()

# retrieve actual results

result = asyncresult.get()

print("\n".join(result))
[[File:IPython Parallel Computing Example.png|left|frameless|635x635px|IPython Parallel Computing Example]]

On execution of this code:[[File:IPython Parallel Computing Result 1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|left|655x655px|IPython Parallel Computing Result-1]]
[[File:IPython Parallel Computing Result 2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|left|658x658px|IPython Parallel Computing Result-2]]

====== Example with MPI: ======
Code (taken from <nowiki>https://ipyparallel.readthedocs.io/en/latest/</nowiki>):

import ipyparallel as ipp

def mpi_example():

from mpi4py import MPI

comm = MPI.COMM_WORLD

return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"

<nowiki>#</nowiki> request an MPI cluster with 4 engines

with ipp.Cluster(engines='mpi', n=4) as rc:

# get a broadcast_view on the cluster which is best

# suited for MPI style computation

view = rc.broadcast_view()

# run the mpi_example function on all engines in parallel

r = view.apply_sync(mpi_example)

# Retrieve and print the result from the engines

print("\n".join(r))

Result:
[[File:IPython Parallel with MPI.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20with%20MPI.png|left|frameless|768x768px]]

Configuring Ipython for Parallel Computing

2022-10-13T13:51:43Z

Sl2625:

==== Step 1. From Terminal run following command to create a parallel profile: ====
$ ipython3 profile create --parallel --profile=myprofile

If this is completed successfully, then there should be some config files like ipcontroller_config.py, ipcluster_config.py, etc in the folder <NetID>/.ipython/profile_myprofile. Check this folder, if you are missing these config files, there might be some issue with the installation of IPython on the system.

==== Step 2. We modify ipcluster_config.py so that engines are launched in remote machines and controller is launched on local machine: ====
$ cd <NetID>/.ipython/profile_myprofile

$ vi ipcluster_config.py

Now in this file we modify the following (uncomment the line and make the changes as shown):

c.Cluster.engine_launcher_class = 'ssh' -->line 528

c.SSHControllerLauncher.controller_args = ['--ip=*'] -->line 1689

c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2} -->line 2423

Save the file and exit. Note that in the c.SSHEngineSetLauncher.engines field, you need to put which machines (ramsey,fibonacci,etc) or nodes (pnode01,02,etc) and how many kernels you want from each. In my example I have taken 2 each from pnode10 and pnode11

Next, in file ipcontroller_config.py in the same location:

c.IPController.ip = '0.0.0.0'

Save and exit file. This step is important because otherwise the controller won't be able to listen on the engines and no parallel computation will happen.

==== Step 3. With this you should have parallel computing set up. Examples with IPython and MPI: ====

====== Only IPython: ======
Launch IPython with the profile you created and edited:

$ ipython3 --profile=myprofile

Code:

import time

import ipyparallel as ipp

def parallel_example():

return f"Hello World!!"

<nowiki>#</nowiki> request a cluster

with ipp.Cluster() as rc:

# get a view on the cluster

view = rc.load_balanced_view()

# submit the tasks

asyncresult = view.map_async(parallel_example)

# wait interactively for results

asyncresult.wait_interactive()

# retrieve actual results

result = asyncresult.get()

print("\n".join(result))
[[File:IPython Parallel Computing Example.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Example.png|left|662x662px|IPython Parallel Computing Example]]

On execution of this code:[[File:IPython Parallel Computing Result 1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|left|655x655px|IPython Parallel Computing Result-1]]
[[File:IPython Parallel Computing Result 2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|left|658x658px|IPython Parallel Computing Result-2]]

====== Example with MPI: ======
Code (taken from <nowiki>https://ipyparallel.readthedocs.io/en/latest/</nowiki>):

import ipyparallel as ipp

def mpi_example():

from mpi4py import MPI

comm = MPI.COMM_WORLD

return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"

<nowiki>#</nowiki> request an MPI cluster with 4 engines

with ipp.Cluster(engines='mpi', n=4) as rc:

# get a broadcast_view on the cluster which is best

# suited for MPI style computation

view = rc.broadcast_view()

# run the mpi_example function on all engines in parallel

r = view.apply_sync(mpi_example)

# Retrieve and print the result from the engines

print("\n".join(r))

Result:
[[File:IPython Parallel with MPI.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20with%20MPI.png|left|frameless|768x768px]]

Configuring Ipython for Parallel Computing

2022-10-13T13:51:07Z

Sl2625:

==== Step 1. From Terminal run following command to create a parallel profile: ====
$ ipython3 profile create --parallel --profile=myprofile

If this is completed successfully, then there should be some config files like ipcontroller_config.py, ipcluster_config.py, etc in the folder <NetID>/.ipython/profile_myprofile. Check this folder, if you are missing these config files, there might be some issue with the installation of IPython on the system.

==== Step 2. We modify ipcluster_config.py so that engines are launched in remote machines and controller is launched on local machine: ====
$ cd <NetID>/.ipython/profile_myprofile

$ vi ipcluster_config.py

Now in this file we modify the following (uncomment the line and make the changes as shown):

c.Cluster.engine_launcher_class = 'ssh' -->line 528

c.SSHControllerLauncher.controller_args = ['--ip=*'] -->line 1689

c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2} -->line 2423

Save the file and exit. Note that in the c.SSHEngineSetLauncher.engines field, you need to put which machines (ramsey,fibonacci,etc) or nodes (pnode01,02,etc) and how many kernels you want from each. In my example I have taken 2 each from pnode10 and pnode11

Next, in file ipcontroller_config.py in the same location:

c.IPController.ip = '0.0.0.0'

Save and exit file. This step is important because otherwise the controller won't be able to listen on the engines and no parallel computation will happen.

==== Step 3. With this you should have parallel computing set up. Examples with IPython and MPI: ====

====== Only IPython: ======
Launch IPython with the profile you created and edited:

$ ipython3 --profile=myprofile

Code:

import time

import ipyparallel as ipp

def parallel_example():

return f"Hello World!!"

<nowiki>#</nowiki> request a cluster

with ipp.Cluster() as rc:

# get a view on the cluster

view = rc.load_balanced_view()

# submit the tasks

asyncresult = view.map_async(parallel_example)

# wait interactively for results

asyncresult.wait_interactive()

# retrieve actual results

result = asyncresult.get()

print("\n".join(result))
[[File:IPython Parallel Computing Example.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Example.png|left|662x662px|IPython Parallel Computing Example]]

On execution of this code:[[File:IPython Parallel Computing Result 1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|left|655x655px|IPython Parallel Computing Result-1]]
[[File:IPython Parallel Computing Result 2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|left|658x658px|IPython Parallel Computing Result-2]]

====== Example with MPI: ======
Code (taken from <nowiki>https://ipyparallel.readthedocs.io/en/latest/</nowiki>):

import ipyparallel as ipp

def mpi_example():

from mpi4py import MPI

comm = MPI.COMM_WORLD

return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"

<nowiki>#</nowiki> request an MPI cluster with 4 engines

with ipp.Cluster(engines='mpi', n=4) as rc:

# get a broadcast_view on the cluster which is best

# suited for MPI style computation

view = rc.broadcast_view()

# run the mpi_example function on all engines in parallel

r = view.apply_sync(mpi_example)

# Retrieve and print the result from the engines

print("\n".join(r))

Result:
[[File:IPython Parallel with MPI.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20with%20MPI.png|left|frameless|768x768px]]

Configuring Ipython for Parallel Computing

2022-10-13T13:49:51Z

Sl2625:

==== Step 1. From Terminal run following command to create a parallel profile: ====
$ ipython3 profile create --parallel --profile=myprofile

If this is completed successfully, then there should be some config files like ipcontroller_config.py, ipcluster_config.py, etc in the folder <NetID>/.ipython/profile_myprofile. Check this folder, if you are missing these config files, there might be some issue with the installation of IPython on the system.

==== Step 2. We modify ipcluster_config.py so that engines are launched in remote machines and controller is launched on local machine: ====
$ cd <NetID>/.ipython/profile_myprofile

$ vi ipcluster_config.py

Now in this file we modify the following (uncomment the line and make the changes as shown):

c.Cluster.engine_launcher_class = 'ssh' -->line 528

c.SSHControllerLauncher.controller_args = ['--ip=*'] -->line 1689

c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2} -->line 2423

Save the file and exit. Note that in the c.SSHEngineSetLauncher.engines field, you need to put which machines (ramsey,fibonacci,etc) or nodes (pnode01,02,etc) and how many kernels you want from each. In my example I have taken 2 each from pnode10 and pnode11

Next, in file ipcontroller_config.py in the same location:

c.IPController.ip = '0.0.0.0'

Save and exit file. This step is important because otherwise the controller won't be able to listen on the engines and no parallel computation will happen.

==== Step 3. With this you should have parallel computing set up. Examples with IPython and MPI: ====

====== Only IPython: ======
Launch IPython with the profile you created and edited:

$ ipython3 --profile=myprofile

Code:

import time

import ipyparallel as ipp

def parallel_example():

return f"Hello World!!"

<nowiki>#</nowiki> request a cluster

with ipp.Cluster() as rc:

# get a view on the cluster

view = rc.load_balanced_view()

# submit the tasks

asyncresult = view.map_async(parallel_example)

# wait interactively for results

asyncresult.wait_interactive()

# retrieve actual results

result = asyncresult.get()

print("\n".join(result))
[[File:IPython Parallel Computing Example.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Example.png|left|662x662px|IPython Parallel Computing Example]]

On execution of this code:
[[File:IPython Parallel Computing Result 1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|left|655x655px|IPython Parallel Computing Result-1]]
[[File:IPython Parallel Computing Result 2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|left|658x658px|IPython Parallel Computing Result-2]]

====== Example with MPI: ======
Code (taken from <nowiki>https://ipyparallel.readthedocs.io/en/latest/</nowiki>):

import ipyparallel as ipp

def mpi_example():

from mpi4py import MPI

comm = MPI.COMM_WORLD

return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"

<nowiki>#</nowiki> request an MPI cluster with 4 engines

with ipp.Cluster(engines='mpi', n=4) as rc:

# get a broadcast_view on the cluster which is best

# suited for MPI style computation

view = rc.broadcast_view()

# run the mpi_example function on all engines in parallel

r = view.apply_sync(mpi_example)

# Retrieve and print the result from the engines

print("\n".join(r))

Result:
[[File:IPython Parallel with MPI.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20with%20MPI.png|left|frameless|768x768px]]

Configuring Ipython for Parallel Computing

2022-10-13T13:49:16Z

Sl2625: Added description of changes to configure IPython for parallel computing using the cluster

==== Step 1. From Terminal run following command to create a parallel profile: ====
$ ipython3 profile create --parallel --profile=myprofile

If this is completed successfully, then there should be some config files like ipcontroller_config.py, ipcluster_config.py, etc in the folder <NetID>/.ipython/profile_myprofile. Check this folder, if you are missing these config files, there might be some issue with the installation of IPython on the system.

==== Step 2. We modify ipcluster_config.py so that engines are launched in remote machines and controller is launched on local machine: ====
$ cd <NetID>/.ipython/profile_myprofile

$ vi ipcluster_config.py

Now in this file we modify the following (uncomment the line and make the changes as shown):

c.Cluster.engine_launcher_class = 'ssh' -->line 528

c.SSHControllerLauncher.controller_args = ['--ip=*'] -->line 1689

c.SSHEngineSetLauncher.engines = {'pnode10':2,'pnode11':2} -->line 2423

Save the file and exit. Note that in the c.SSHEngineSetLauncher.engines field, you need to put which machines (ramsey,fibonacci,etc) or nodes (pnode01,02,etc) and how many kernels you want from each. In my example I have taken 2 each from pnode10 and pnode11

Next, in file ipcontroller_config.py in the same location:

c.IPController.ip = '0.0.0.0'

Save and exit file. This step is important because otherwise the controller won't be able to listen on the engines and no parallel computation will happen.

==== Step 3. With this you should have parallel computing set up. Examples with IPython and MPI: ====

====== Only IPython: ======
Launch IPython with the profile you created and edited:

$ ipython3 --profile=myprofile

Code:

import time

import ipyparallel as ipp

def parallel_example():

return f"Hello World!!"

<nowiki>#</nowiki> request a cluster

with ipp.Cluster() as rc:

# get a view on the cluster

view = rc.load_balanced_view()

# submit the tasks

asyncresult = view.map_async(parallel_example)

# wait interactively for results

asyncresult.wait_interactive()

# retrieve actual results

result = asyncresult.get()

print("\n".join(result))
[[File:IPython Parallel Computing Example.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Example.png|left|662x662px|IPython Parallel Computing Example]]

On execution of this code:
[[File:IPython Parallel Computing Result 1.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%201.png|left|655x655px|IPython Parallel Computing Result-1]]
[[File:IPython Parallel Computing Result 2.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20Computing%20Result%202.png|left|658x658px|IPython Parallel Computing Result-2]]

====== Example with MPI: ======
Code (taken from <nowiki>https://ipyparallel.readthedocs.io/en/latest/</nowiki>):

import ipyparallel as ipp

def mpi_example():

from mpi4py import MPI

comm = MPI.COMM_WORLD

return f"Hello World from rank {comm.Get_rank()}. total ranks={comm.Get_size()}"

<nowiki>#</nowiki> request an MPI cluster with 4 engines

with ipp.Cluster(engines='mpi', n=4) as rc:

# get a broadcast_view on the cluster which is best

# suited for MPI style computation

view = rc.broadcast_view()

# run the mpi_example function on all engines in parallel

r = view.apply_sync(mpi_example)

# Retrieve and print the result from the engines

print("\n".join(r))

Result:
[[File:IPython Parallel with MPI.png|link=https://e.math.cornell.edu/wiki/index.php/File:IPython%20Parallel%20with%20MPI.png|left|frameless|768x768px]]

File:IPython Parallel with MPI.png

2022-10-13T13:38:25Z

Sl2625:

IPython Parallel with MPI