Mathematica Remote Kernels - Revision history

Admin: /* Power Consumption */

2022-08-05T21:29:04Z

Power Consumption

Admin: /* Launching Remote Kernels in Mathematica */

2022-07-25T19:33:58Z

Launching Remote Kernels in Mathematica

Admin: /* Tips */

2022-07-21T16:11:47Z

Tips

Admin: /* Power Consumption */

2022-07-21T16:03:47Z

Power Consumption

Admin: /* Tips */

2022-07-21T16:03:37Z

Tips

Admin: /* Tips */

2022-07-21T16:02:56Z

Tips

Admin: /* Tips */

2022-07-21T16:00:10Z

Tips

Admin: /* Power Consumption */

2022-07-21T15:59:54Z

Power Consumption

Admin: /* Launching Remote Kernels in Mathematica */

2022-07-21T15:59:25Z

Launching Remote Kernels in Mathematica

Admin: /* Launching Remote Kernels in Mathematica */

2022-07-21T15:28:02Z

Launching Remote Kernels in Mathematica

@@ Line 63: / Line 63: @@
 [http://graph.math.cornell.edu:3000/d/_wfSvpjmz/server-room-power Server Room Power] (You must be on the Cornell network or VPN to use this link)
-The purple circuit is the cluster. The power usage is around 650W when nothing is going on. It will go up when the machines get busy. Note that if the power on the purple circuit goes above 2700 watts, the cluster may shut down. I'm worried about going over this limit, but so far it hasn't happened. Can you send the cluster enough math to kill it? Try it and let me know! The solution is easy, we can put some of the nodes on another circuit. For now they're all on the same one so we can more easily measure the power usage of the cluster.
+The orange and purple circuits are the cluster. The power usage is around 620W when nothing is going on, with an additional 100W being used by the two UPS units. It will go up when the machines get busy. If the orange circuit goes above 2200 watts or the purple circuit goes above 2800 watts, the circuit will shut down. At this time, the combined capacity of the two circuits should be more than the entire cluster can use. See if you can do enough math on the cluster to cause a shutdown.
 Note the room temperature! The room has a large cooler that uses Cornell Lake-source cooling to cool the room, but your job will still probably warm up the room by a few degrees.

@@ Line 10: / Line 10: @@
 On your Linux desktop, start up Mathematica.
-We'll start by launching kernels on two of the other cruncher machines, since this is a simple example to start with.
+We'll start by launching kernels on three of the other cruncher machines, since this is a simple example to start with.
-For this example, we're going to start 4 kernels each on ramsey and fibonacci.
+For this example, we're going to start 4 kernels each on ramsey, fibonacci, and boole.
 In the Mathematica window, type the following:
- <nowiki>LaunchKernels[{"ssh://ramsey/?4","ssh://fibonacci/?4"}]</nowiki>
+ <nowiki>LaunchKernels[{"ssh://ramsey/?4","ssh://fibonacci/?4","ssh://boole/?4"  }]</nowiki>
 Then, right-click that cell and choose 'Evaluate Cell'. The system will launch the remote kernels, and show its progress while it does that. Once all the kernels are launched, you can run calculations on them. To close those kernels, do

← Older revision		Revision as of 16:11, 21 July 2022
Line 49:		Line 49:

	[http://graph.math.cornell.edu:3000/d/BPzrVrznz/cluster-pnodes Cluster Pnodes] (You must be on the Cornell network or VPN to use this link)		[http://graph.math.cornell.edu:3000/d/BPzrVrznz/cluster-pnodes Cluster Pnodes] (You must be on the Cornell network or VPN to use this link)
		+
		+	Note that these graphs will usually show you the last 24 hours, so you can click on the little clock on the upper right and choose a different time interval, like the last hour, so you have a better picture of what is going on. The graphs update only once a minute, so be patient.

	If you're running remote kernels on the Ryzen crunchers, they have 32 CPU threads each, so you would want to load them up to 32 to make full use of the CPU, if it was zero before you started. Note that other people are using these machines, so be a good neighbor and don't hog the entire machine.		If you're running remote kernels on the Ryzen crunchers, they have 32 CPU threads each, so you would want to load them up to 32 to make full use of the CPU, if it was zero before you started. Note that other people are using these machines, so be a good neighbor and don't hog the entire machine.

@@ Line 59: / Line 59: @@
 If you really want to geek out, you can look at the server room power consumption when you're running your job.
-[http://graph.math.cornell.edu:3000/d/_wfSvpjmz/server-room-power Server Room Power]
+[http://graph.math.cornell.edu:3000/d/_wfSvpjmz/server-room-power Server Room Power] (You must be on the Cornell network or VPN to use this link)
 The purple circuit is the cluster. The power usage is around 650W when nothing is going on. It will go up when the machines get busy. Note that if the power on the purple circuit goes above 2700 watts, the cluster may shut down. I'm worried about going over this limit, but so far it hasn't happened. Can you send the cluster enough math to kill it? Try it and let me know! The solution is easy, we can put some of the nodes on another circuit. For now they're all on the same one so we can more easily measure the power usage of the cluster.

@@ Line 46: / Line 46: @@
 For the cluster nodes, each one has 8 cpu threads, so ideally if no one else is using the cluster, your job should load up each node to a load of 8 so that you're making full use of each node's CPU. If it's less than 8, you're not using the whole CPU, and if it's more than 8, some of your kernels are waiting for resources.
-You can see the status of your the cluster nodes here: (You must be on the Cornell network or VPN to use this link)
+You can see the status of your the cluster nodes here:
-[http://graph.math.cornell.edu:3000/d/BPzrVrznz/cluster-pnodes Cluster Pnodes]
+[http://graph.math.cornell.edu:3000/d/BPzrVrznz/cluster-pnodes Cluster Pnodes] (You must be on the Cornell network or VPN to use this link)
 If you're running remote kernels on the Ryzen crunchers, they have 32 CPU threads each, so you would want to load them up to 32 to make full use of the CPU, if it was zero before you started. Note that other people are using these machines, so be a good neighbor and don't hog the entire machine.
-The status of the crunchers is here: (You must be on the Cornell network or VPN to use this link)
+The status of the crunchers is here:
-[http://graph.math.cornell.edu:3000/d/f8zUy_L7k/crunchers Crunchers]
+[http://graph.math.cornell.edu:3000/d/f8zUy_L7k/crunchers Crunchers] (You must be on the Cornell network or VPN to use this link)
 ==Power Consumption==

@@ Line 55: / Line 55: @@
 ==Power Consumption==
 If you really want to geek out, you can look at the server room power consumption when you're running your job.
 [http://graph.math.cornell.edu:3000/d/_wfSvpjmz/server-room-power Server Room Power]
@@ Line 62: / Line 63: @@
 The blue circuit is the Ryzen GPU cruncher machines. That circuit also has a limit of 2700 watts total. There should be enough headroom there to be able to load up all of those machines including their GPUs without overloading the circuit. Again, you might be able to make it shut down. That's ok, as long as you tell the system administrator. It's better to cause this problem now so we can rearrange things to avoid it in the future. But, so far, there seems to be enough capacity on this circuit to handle all of the cruncher machines.
 ==Network Bottlenecks==

← Older revision		Revision as of 15:59, 21 July 2022
Line 17:		Line 17:

	<nowiki>LaunchKernels[{"ssh://ramsey/?4","ssh://fibonacci/?4"}]</nowiki>		<nowiki>LaunchKernels[{"ssh://ramsey/?4","ssh://fibonacci/?4"}]</nowiki>
		+
		+	Then, right-click that cell and choose 'Evaluate Cell'. The system will launch the remote kernels, and show its progress while it does that. Once all the kernels are launched, you can run calculations on them. To close those kernels, do
		+
		+	<nowiki>CloseKernels[]</nowiki>
		+
		+	and evaluate the cell. The system will close the remote kernels. Please remember to do a CloseKernels[] at the end of your session. Closing Mathematica SHOULD close the remote kernels, but it's better to be sure.
		+
		+	Now, we'll launch four kernels each on the 40 node cluster. There's a better way to do this than copying and pasting this big command, like using a loop to generate the node names, but for now this works. Note that it will take a while to launch all of the kernels because it launches them one at a time, and this command should give you 160 remote kernels.
		+
		+	<nowiki>LaunchKernels[{"ssh://pnode01/?4","ssh://pnode02/?4","ssh://pnode03/?4","ssh://pnode04/?4","ssh://pnode05/?4","ssh://pnode06/?4","ssh://pnode07/?4","ssh://pnode08/?4","ssh://pnode09/?4","ssh://pnode10/?4","ssh://pnode11/?4","ssh://pnode12/?4","ssh://pnode13/?4","ssh://pnode14/?4","ssh://pnode15/?4","ssh://pnode16/?4","ssh://pnode17/?4","ssh://pnode18/?4","ssh://pnode19/?4","ssh://pnode20/?4","ssh://pnode21/?4","ssh://pnode22/?4","ssh://pnode23/?4","ssh://pnode24/?4","ssh://pnode25/?4","ssh://pnode26/?4","ssh://pnode27/?4","ssh://pnode28/?4","ssh://pnode29/?4","ssh://pnode30/?4","ssh://pnode31/?4","ssh://pnode32/?4","ssh://pnode33/?4","ssh://pnode34/?4","ssh://pnode35/?4","ssh://pnode36/?4","ssh://pnode37/?4","ssh://pnode38/?4","ssh://pnode39/?4","ssh://pnode40/?4"}]</nowiki>
		+
		+	Copy this command from this page and paste it into Mathematica. Note that if you're using X2go and you copy on the local machine, you may need to paste using the 'Paste' command under the 'Edit' menu, because different system handle cutting and pasting differently.
		+
		+	Once you have pasted that big command in there, right-click the cell and evaluate it. You'll see the progress as all of the remote kernels are started up. Once it's done, you'll have 160 remote kernels.
		+
		+	When you are finished using them, do a
		+
		+	CloseKernels[]
		+
		+	so the system can go and shut them all down properly.
		+
		+	==Tips==
		+
		+	Note that there is nothing stopping you from running remote kernels on both the cluster nodes and the crunchers, but this is not recommended. These machines have cores that are very different speeds, so the faster machine may end up waiting for the slower machines to finish, so this may just be a waste of resources. It's best to run things on the cluster, OR the crunchers, but not both.
		+
		+	The number of kernels you run on each machine may give you different outcomes depending on your job. So, if you run 8 kernels per node, maybe your job will run faster, maybe not. It's best to experiment with a subset of your job to find the optimum number of kernels per node for your job.
		+
		+	For the cluster nodes, each one has 8 cpu threads, so ideally if no one else is using the cluster, your job should load up each node to a load of 8 so that you're making full use of each node's CPU. If it's less than 8, you're not using the whole CPU, and if it's more than 8, some of your kernels are waiting for resources.
		+
		+	You can see the status of your the cluster nodes here:
		+	[http://graph.math.cornell.edu:3000/d/BPzrVrznz/cluster-pnodes Cluster Pnodes]
		+
		+	If you're running remote kernels on the Ryzen crunchers, they have 32 CPU threads each, so you would want to load them up to 32 to make full use of the CPU, if it was zero before you started. Note that other people are using these machines, so be a good neighbor and don't hog the entire machine.
		+
		+	The status of the crunchers is here: [http://graph.math.cornell.edu:3000/d/f8zUy_L7k/crunchers Crunchers]
		+
		+	==Power Consumption==
		+	If you really want to geek out, you can look at the server room power consumption when you're running your job.
		+	[http://graph.math.cornell.edu:3000/d/_wfSvpjmz/server-room-power Server Room Power]
		+
		+	The purple circuit is the cluster. The power usage is around 650W when nothing is going on. It will go up when the machines get busy. Note that if the power on the purple circuit goes above 2700 watts, the cluster may shut down. I'm worried about going over this limit, but so far it hasn't happened. Can you send the cluster enough math to kill it? Try it and let me know! The solution is easy, we can put some of the nodes on another circuit. For now they're all on the same one so we can more easily measure the power usage of the cluster.
		+
		+	Note the room temperature! The room has a large cooler that uses Cornell Lake-source cooling to cool the room, but your job will still probably warm up the room by a few degrees.
		+
		+	The blue circuit is the Ryzen GPU cruncher machines. That circuit also has a limit of 2700 watts total. There should be enough headroom there to be able to load up all of those machines including their GPUs without overloading the circuit. Again, you might be able to make it shut down. That's ok, as long as you tell the system administrator. It's better to cause this problem now so we can rearrange things to avoid it in the future. But, so far, there seems to be enough capacity on this circuit to handle all of the cruncher machines.
		+
		+
		+	==Network Bottlenecks==
		+
		+	The crunchers are on a 10Gb/s network. The cluster nodes are each 1Gb/s, but their link as a group to the main network is 10Gb/s. There is a limitation at this time because the crunchers and the cluster are on different subnets, and the connection between the subnets is limited to 1G/s at this time. This should not impact your job, but if you go to crunchers and look at the machine where you are running your main Mathematica, and check the 'Network' graph, if that machine is maxing out at 1Gb/s during your job, then your job is being constrained by the subnet bottleneck. If this is happening, please email humphrey@cornell.edu and let me know about this. The bottleneck can be removed but we haven't gotten to that yet.

← Older revision		Revision as of 15:28, 21 July 2022
Line 16:		Line 16:
	In the Mathematica window, type the following:		In the Mathematica window, type the following:

−	LaunchKernels[{"ssh://ramsey/?4","ssh://fibonacci/?4"}]	+	<nowiki>LaunchKernels[{"ssh://ramsey/?4","ssh://fibonacci/?4"}]</nowiki>