Mathematica Remote Kernels

2022-08-05T21:29:04Z

Admin: /* Power Consumption */

===Launching Remote Kernels in Mathematica===
This page assumes that you've already followed the instructions in [[Cluster SSH access]] and you have all of that working.

There are other ways to set this up, including lots of menus in Mathematica for the remote kernel settings. The instructions on this are vague and confusing, so for now we're just entering the information directly in the Mathematica code window, because it's simple and it works. Any suggestions for better ways to do this are welcome, and when we find better way to do it, we'll update the documentation.

Also, since this is written by someone who is not very familiar with Mathematica syntax, the large 'LaunchKernels' command for 40 nodes shown below could be done much more easily, for instance, using a loop to generate the string for the machine list passed to the LaunchKernels command, or some other loop that will do the same thing without having to cut and paste that giant command.

Here are the instructions:

On your Linux desktop, start up Mathematica.

We'll start by launching kernels on three of the other cruncher machines, since this is a simple example to start with.

For this example, we're going to start 4 kernels each on ramsey, fibonacci, and boole.

In the Mathematica window, type the following:

<nowiki>LaunchKernels[{"ssh://ramsey/?4","ssh://fibonacci/?4","ssh://boole/?4" }]</nowiki>

Then, right-click that cell and choose 'Evaluate Cell'. The system will launch the remote kernels, and show its progress while it does that. Once all the kernels are launched, you can run calculations on them. To close those kernels, do

<nowiki>CloseKernels[]</nowiki>

and evaluate the cell. The system will close the remote kernels. Please remember to do a CloseKernels[] at the end of your session. Closing Mathematica SHOULD close the remote kernels, but it's better to be sure.

Now, we'll launch four kernels each on the 40 node cluster. There's a better way to do this than copying and pasting this big command, like using a loop to generate the node names, but for now this works. Note that it will take a while to launch all of the kernels because it launches them one at a time, and this command should give you 160 remote kernels.

<nowiki>LaunchKernels[{"ssh://pnode01/?4","ssh://pnode02/?4","ssh://pnode03/?4","ssh://pnode04/?4","ssh://pnode05/?4","ssh://pnode06/?4","ssh://pnode07/?4","ssh://pnode08/?4","ssh://pnode09/?4","ssh://pnode10/?4","ssh://pnode11/?4","ssh://pnode12/?4","ssh://pnode13/?4","ssh://pnode14/?4","ssh://pnode15/?4","ssh://pnode16/?4","ssh://pnode17/?4","ssh://pnode18/?4","ssh://pnode19/?4","ssh://pnode20/?4","ssh://pnode21/?4","ssh://pnode22/?4","ssh://pnode23/?4","ssh://pnode24/?4","ssh://pnode25/?4","ssh://pnode26/?4","ssh://pnode27/?4","ssh://pnode28/?4","ssh://pnode29/?4","ssh://pnode30/?4","ssh://pnode31/?4","ssh://pnode32/?4","ssh://pnode33/?4","ssh://pnode34/?4","ssh://pnode35/?4","ssh://pnode36/?4","ssh://pnode37/?4","ssh://pnode38/?4","ssh://pnode39/?4","ssh://pnode40/?4"}]</nowiki>

Copy this command from this page and paste it into Mathematica. Note that if you're using X2go and you copy on the local machine, you may need to paste using the 'Paste' command under the 'Edit' menu, because different system handle cutting and pasting differently.

Once you have pasted that big command in there, right-click the cell and evaluate it. You'll see the progress as all of the remote kernels are started up. Once it's done, you'll have 160 remote kernels.

When you are finished using them, do a

CloseKernels[]

so the system can go and shut them all down properly.

==Tips==

Note that there is nothing stopping you from running remote kernels on both the cluster nodes and the crunchers, but this is not recommended. These machines have cores that are very different speeds, so the faster machine may end up waiting for the slower machines to finish, so this may just be a waste of resources. It's best to run things on the cluster, OR the crunchers, but not both.

The number of kernels you run on each machine may give you different outcomes depending on your job. So, if you run 8 kernels per node, maybe your job will run faster, maybe not. It's best to experiment with a subset of your job to find the optimum number of kernels per node for your job.

For the cluster nodes, each one has 8 cpu threads, so ideally if no one else is using the cluster, your job should load up each node to a load of 8 so that you're making full use of each node's CPU. If it's less than 8, you're not using the whole CPU, and if it's more than 8, some of your kernels are waiting for resources.

You can see the status of your the cluster nodes here:

[http://graph.math.cornell.edu:3000/d/BPzrVrznz/cluster-pnodes Cluster Pnodes] (You must be on the Cornell network or VPN to use this link)

Note that these graphs will usually show you the last 24 hours, so you can click on the little clock on the upper right and choose a different time interval, like the last hour, so you have a better picture of what is going on. The graphs update only once a minute, so be patient.

If you're running remote kernels on the Ryzen crunchers, they have 32 CPU threads each, so you would want to load them up to 32 to make full use of the CPU, if it was zero before you started. Note that other people are using these machines, so be a good neighbor and don't hog the entire machine.

The status of the crunchers is here:

[http://graph.math.cornell.edu:3000/d/f8zUy_L7k/crunchers Crunchers] (You must be on the Cornell network or VPN to use this link)

==Power Consumption==
If you really want to geek out, you can look at the server room power consumption when you're running your job.

[http://graph.math.cornell.edu:3000/d/_wfSvpjmz/server-room-power Server Room Power] (You must be on the Cornell network or VPN to use this link)

The orange and purple circuits are the cluster. The power usage is around 620W when nothing is going on, with an additional 100W being used by the two UPS units. It will go up when the machines get busy. If the orange circuit goes above 2200 watts or the purple circuit goes above 2800 watts, the circuit will shut down. At this time, the combined capacity of the two circuits should be more than the entire cluster can use. See if you can do enough math on the cluster to cause a shutdown.

Note the room temperature! The room has a large cooler that uses Cornell Lake-source cooling to cool the room, but your job will still probably warm up the room by a few degrees.

The blue circuit is the Ryzen GPU cruncher machines. That circuit also has a limit of 2700 watts total. There should be enough headroom there to be able to load up all of those machines including their GPUs without overloading the circuit. Again, you might be able to make it shut down. That's ok, as long as you tell the system administrator. It's better to cause this problem now so we can rearrange things to avoid it in the future. But, so far, there seems to be enough capacity on this circuit to handle all of the cruncher machines.

==Network Bottlenecks==

The crunchers are on a 10Gb/s network. The cluster nodes are each 1Gb/s, but their link as a group to the main network is 10Gb/s. There is a limitation at this time because the crunchers and the cluster are on different subnets, and the connection between the subnets is limited to 1G/s at this time. This should not impact your job, but if you go to crunchers and look at the machine where you are running your main Mathematica, and check the 'Network' graph, if that machine is maxing out at 1Gb/s during your job, then your job is being constrained by the subnet bottleneck. If this is happening, please email humphrey@cornell.edu and let me know about this. The bottleneck can be removed but we haven't gotten to that yet.

Magma Cluster

2022-07-25T20:57:57Z

Admin:

== Magma on the Cluster ==
The Math Cluster should support distributed Magma jobs.

We're still working on a good example, so please give feedback on this.

Here is some documentation on starting remote workers [https://magma.maths.usyd.edu.au/magma/handbook/text/64 StartWorkers Documentation]

To launch remote workers in Magma, you'll need to have setup your SSH keys: [[Cluster SSH access]]

So far it appears that you can do this on machines where you would have ssh access to the cluster nodes.

Here is an example of parallel factorization: [https://magma.maths.usyd.edu.au/magma/handbook/text/65 Integer Factorization]

Magma Cluster

2022-07-25T20:47:40Z

Admin:

2022-07-21T15:59:25Z

Admin: /* Launching Remote Kernels in Mathematica */

===Launching Remote Kernels in Mathematica===
This page assumes that you've already followed the instructions in [[Cluster SSH access]] and you have all of that working.

There are other ways to set this up, including lots of menus in Mathematica for the remote kernel settings. The instructions on this are vague and confusing, so for now we're just entering the information directly in the Mathematica code window, because it's simple and it works. Any suggestions for better ways to do this are welcome, and when we find better way to do it, we'll update the documentation.

Also, since this is written by someone who is not very familiar with Mathematica syntax, the large 'LaunchKernels' command for 40 nodes shown below could be done much more easily, for instance, using a loop to generate the string for the machine list passed to the LaunchKernels command, or some other loop that will do the same thing without having to cut and paste that giant command.

Here are the instructions:

On your Linux desktop, start up Mathematica.

We'll start by launching kernels on two of the other cruncher machines, since this is a simple example to start with.

For this example, we're going to start 4 kernels each on ramsey and fibonacci.

In the Mathematica window, type the following:

<nowiki>LaunchKernels[{"ssh://ramsey/?4","ssh://fibonacci/?4"}]</nowiki>

Then, right-click that cell and choose 'Evaluate Cell'. The system will launch the remote kernels, and show its progress while it does that. Once all the kernels are launched, you can run calculations on them. To close those kernels, do

<nowiki>CloseKernels[]</nowiki>

and evaluate the cell. The system will close the remote kernels. Please remember to do a CloseKernels[] at the end of your session. Closing Mathematica SHOULD close the remote kernels, but it's better to be sure.

Now, we'll launch four kernels each on the 40 node cluster. There's a better way to do this than copying and pasting this big command, like using a loop to generate the node names, but for now this works. Note that it will take a while to launch all of the kernels because it launches them one at a time, and this command should give you 160 remote kernels.

<nowiki>LaunchKernels[{"ssh://pnode01/?4","ssh://pnode02/?4","ssh://pnode03/?4","ssh://pnode04/?4","ssh://pnode05/?4","ssh://pnode06/?4","ssh://pnode07/?4","ssh://pnode08/?4","ssh://pnode09/?4","ssh://pnode10/?4","ssh://pnode11/?4","ssh://pnode12/?4","ssh://pnode13/?4","ssh://pnode14/?4","ssh://pnode15/?4","ssh://pnode16/?4","ssh://pnode17/?4","ssh://pnode18/?4","ssh://pnode19/?4","ssh://pnode20/?4","ssh://pnode21/?4","ssh://pnode22/?4","ssh://pnode23/?4","ssh://pnode24/?4","ssh://pnode25/?4","ssh://pnode26/?4","ssh://pnode27/?4","ssh://pnode28/?4","ssh://pnode29/?4","ssh://pnode30/?4","ssh://pnode31/?4","ssh://pnode32/?4","ssh://pnode33/?4","ssh://pnode34/?4","ssh://pnode35/?4","ssh://pnode36/?4","ssh://pnode37/?4","ssh://pnode38/?4","ssh://pnode39/?4","ssh://pnode40/?4"}]</nowiki>

Copy this command from this page and paste it into Mathematica. Note that if you're using X2go and you copy on the local machine, you may need to paste using the 'Paste' command under the 'Edit' menu, because different system handle cutting and pasting differently.

Once you have pasted that big command in there, right-click the cell and evaluate it. You'll see the progress as all of the remote kernels are started up. Once it's done, you'll have 160 remote kernels.

When you are finished using them, do a

CloseKernels[]

so the system can go and shut them all down properly.

==Tips==

Note that there is nothing stopping you from running remote kernels on both the cluster nodes and the crunchers, but this is not recommended. These machines have cores that are very different speeds, so the faster machine may end up waiting for the slower machines to finish, so this may just be a waste of resources. It's best to run things on the cluster, OR the crunchers, but not both.

The number of kernels you run on each machine may give you different outcomes depending on your job. So, if you run 8 kernels per node, maybe your job will run faster, maybe not. It's best to experiment with a subset of your job to find the optimum number of kernels per node for your job.

For the cluster nodes, each one has 8 cpu threads, so ideally if no one else is using the cluster, your job should load up each node to a load of 8 so that you're making full use of each node's CPU. If it's less than 8, you're not using the whole CPU, and if it's more than 8, some of your kernels are waiting for resources.

You can see the status of your the cluster nodes here:
[http://graph.math.cornell.edu:3000/d/BPzrVrznz/cluster-pnodes Cluster Pnodes]

If you're running remote kernels on the Ryzen crunchers, they have 32 CPU threads each, so you would want to load them up to 32 to make full use of the CPU, if it was zero before you started. Note that other people are using these machines, so be a good neighbor and don't hog the entire machine.

The status of the crunchers is here: [http://graph.math.cornell.edu:3000/d/f8zUy_L7k/crunchers Crunchers]

==Power Consumption==
If you really want to geek out, you can look at the server room power consumption when you're running your job.
[http://graph.math.cornell.edu:3000/d/_wfSvpjmz/server-room-power Server Room Power]

The purple circuit is the cluster. The power usage is around 650W when nothing is going on. It will go up when the machines get busy. Note that if the power on the purple circuit goes above 2700 watts, the cluster may shut down. I'm worried about going over this limit, but so far it hasn't happened. Can you send the cluster enough math to kill it? Try it and let me know! The solution is easy, we can put some of the nodes on another circuit. For now they're all on the same one so we can more easily measure the power usage of the cluster.

Note the room temperature! The room has a large cooler that uses Cornell Lake-source cooling to cool the room, but your job will still probably warm up the room by a few degrees.

The blue circuit is the Ryzen GPU cruncher machines. That circuit also has a limit of 2700 watts total. There should be enough headroom there to be able to load up all of those machines including their GPUs without overloading the circuit. Again, you might be able to make it shut down. That's ok, as long as you tell the system administrator. It's better to cause this problem now so we can rearrange things to avoid it in the future. But, so far, there seems to be enough capacity on this circuit to handle all of the cruncher machines.

==Network Bottlenecks==

The crunchers are on a 10Gb/s network. The cluster nodes are each 1Gb/s, but their link as a group to the main network is 10Gb/s. There is a limitation at this time because the crunchers and the cluster are on different subnets, and the connection between the subnets is limited to 1G/s at this time. This should not impact your job, but if you go to crunchers and look at the machine where you are running your main Mathematica, and check the 'Network' graph, if that machine is maxing out at 1Gb/s during your job, then your job is being constrained by the subnet bottleneck. If this is happening, please email humphrey@cornell.edu and let me know about this. The bottleneck can be removed but we haven't gotten to that yet.

Mathematica Remote Kernels

2022-07-21T15:28:02Z

Admin: /* Launching Remote Kernels in Mathematica */

Mathematica Remote Kernels

2022-07-21T15:26:48Z

Admin: Created page with "===Launching Remote Kernels in Mathematica=== This page assumes that you've already followed the instructions in Cluster SSH access and you have all of that working. Ther..."

2022-06-26T16:14:44Z

Admin:

== Math Department Linux Machines ==
This is a list of department machines that you may use remotely.
All of these machines have the standard set of packages.
This list is not complete and it is changing constantly, so check back from time to time.

You can connect to the machines using their complete domain name, such as squid2.math.cornell.edu

'''Dedicated Computation Machines'''
{| border="1"
|Hostname ||Processor ||Cores / Threads ||RAM ||GPU ||Net
|-
|ramsey || AMD Ryzen 9 5950x || 16/32 || 128GB || RTX 3080Ti || 10Gb
|-
|fibonacci|| AMD Ryzen 9 5950x || 16/32 || 128GB || RTX 3080Ti || 10Gb
|-
|squid2|| i7-6700k CPU @ 4.0GHz || 4/8 || 64GB || RTX 2080 Super || 10Gb
|-
|kraken|| Quad Opteron || 64/64 || 512GB || RTX 2080 || 1Gb
|-
|heaviside|| Xeon E5-2640 || 12/24 || 256GB || N/A || 10Gb
|-
|hopper||Xeon E5-2640 || 16/32 ||256GB || N/A || 10Gb
|-
|nautilus || Xeon X5560 || 8/16 || 96GB || N/A || 1Gb

|}

'''Shared Workstations'''
{| border="1"
|Hostname ||Processor ||Cores / Threads ||RAM ||GPU
|-
|aio01 || i5-6600 CPU @ 3.30GHz || 4/4 || 16GB || N/A
|-
|hank || i7-7700 CPU @ 3.60GHz || 4/8 || 16GB || N/A
|-
|feynman || i7-7500 CPU @ 3.40GHz ||4/8 || 16GB || N/A
|}

'''Private Machines'''
These machines are the property of faculty members and may only be used with their permission.
{| border="1"
|Hostname ||Processor ||Cores / Threads ||RAM ||GPU ||Net ||Owner
|-
|leo || Xeon E5-2698 || 20/40 || 256GB || N/A || 10Gb || A. Townsend
|-
|wooster || AMD Ryzen 9 5950x || 16/32 || 128GB || RTX 3080Ti || 1Gb || D. Barbasch
|-
|zeno || AMD Ryzen 9 5950x || 16/32 || 128GB || RTX 3080Ti || 10Gb || A. Vladimirsky
|}

MachineList

2022-06-21T16:57:25Z

Admin: