Amazon EC2: GPU Cluster - Revision history

Pew at 07:43, 3 January 2019

2019-01-03T07:43:39Z

Pew: Undo revision 5116 by imported>Pew (talk)

2019-01-03T07:43:26Z

Undo revision 5116 by [[Special:Contributions/imported>Pew|imported>Pew]] ([[User talk:imported>Pew|talk]])

imported>Pew: Very outdated & quickly available via google

2018-01-06T10:03:06Z

Very outdated & quickly available via google

imported>Pew at 09:15, 2 January 2018

2018-01-02T09:15:11Z

imported>Pew at 08:39, 2 January 2018

2018-01-02T08:39:50Z

imported>Pew at 08:39, 2 January 2018

2018-01-02T08:39:31Z

2001:468:C80:4281:2C0:CAFF:FE69:1E08: /* Creating a new image */

2013-04-10T13:37:51Z

‎Creating a new image

imported>Ubervert: /* Installing CUDA */ 'sudo' is redundant when logged in as root (only default login for SUSE)

2013-04-09T16:47:44Z

‎Installing CUDA: 'sudo' is redundant when logged in as root (only default login for SUSE)

imported>Timelord: /* Installing CUDA */

2013-04-07T18:23:24Z

‎Installing CUDA

imported>Timelord: added info about spot instances

2013-04-07T18:19:32Z

added info about spot instances

← Older revision		Revision as of 07:43, 3 January 2019
Line 67:		Line 67:
	[[Category:Howtos]]		[[Category:Howtos]]
	[[Category:Software]]		[[Category:Software]]
−	~~[[Category:Pending deletion]]~~

← Older revision		Revision as of 07:43, 3 January 2019
Line 1:		Line 1:
		+	An '''Amazon EC2: GPU cluster''' is a remote GPU cluster service.
		+
		+	=Creating a new image=
		+	Check which distributions are supported, [https://developer.nvidia.com/cuda-downloads here]. All you are concerned about are the developer drivers.
		+	From this point on, we will deal with a SUSE instance, but the general steps will be the same. '''Only select an instance that can work with CUDA!'''
		+
		+	* With the Classic Wizard, select Cluster Instances HVM SUSE Linux Enterprise 11
		+	* Change the Instance Type from M3 Extra Large to CG1 Cluster GPU (cg1.4xlarge).
		+	* Enable Termination protection.
		+	* Make sure you specify that your Elastic Block Storage volume NOT be destroyed when the instance is terminated (or your data will be lost when the instance is terminated, which may happen without your consent if you are using a spot instance and the variable price exceeds your maximum bid).
		+	* Generate your key pair and allow SSH access to the server.
		+	* Review the instance type and ensure all details are correct.
		+	* Pricing information may be found [https://aws.amazon.com/ec2/pricing/ here]. "Spot instances" are generally the way to go but be sure to bid with a high enough margin from the current price that you wont be shut down unintentionally. Even double the current spot price is usually lower than the on-demand price.
		+
		+	=Installing CUDA=
		+	Zypper is Suse's package manager; YaST could alternatively be used if more comfortable with a GUI.
		+	First, SSH into the system, update the package cache and upgrade:
		+	<pre>SSH -i /PATH/TO/PKEY root@ec2-...-amazonaws.com
		+	zypper ref
		+	zypper up</pre>
		+
		+	Install the needed packages and generate needed config:
		+	<pre>zypper in gcc kernel-ec2-devel kernel-docs
		+	cd /usr/src/linux
		+	make cloneconfig</pre>
		+
		+	Restart the instance:
		+	<pre>reboot</pre>
		+	You will be booted from the instance, just wait for it to resume again.
		+
		+	Now get the latest CUDA install and run the file:
		+	<pre>wget http://developer.download.nvidia.com/compute/cuda/$CUDA_MAJOR_VERSION/$CUDA_MINOR_VERSION/installers/cuda_5.0.35_$(Linux\|Windows)_$ARCH_$IMG.run
		+	chmod +x cuda_*.run
		+	./cuda_*.run
		+	</pre>
		+	We only need the developer drivers. Ensure that the drivers install correctly. Note that it needs to run as root. The cuda dev drivers are already installed in some AMIs (for example the CentOS-GPU image).
		+
		+	=oclHashCat-plus=
		+	Get oclHashCat:
		+	<pre>wget http://hashcat.net/files/oclHashcat-plus-0.14.7z</pre>
		+
		+	To extract and use oclHashCat, we must unpackage the 7zip archive. In many distributions 7z is provided by the p7zip package. SUSE users need to download the binaries and extract it:
		+	<pre>wget http://downloads.sourceforge.net/project/p7zip/p7zip/9.20.1/p7zip_9.20.1_x86_linux_bin.tar.bz2?r=&ts=1365352571&use_mirror=hivelocity
		+	tar xjvf p7zip_9.20.1_x86_linux_bin.tar.bz2</pre>
		+
		+	Unarchive file, extracting full path:
		+	<pre>7z x oclHashcat-plus-0.14.7z
		+	cd oclHashcat-plus-0.14</pre>
		+
		+	Running oclHashcat-plus (in a screen session):
		+	<pre>screen
		+	./cudaHashcat-plus64 $HASHES ($DICTIONARY \| -m $hashmask \| --rules=$RULES) </pre>
		+	oclHashcat-plus assumes unsalted md5 by default
		+
		+	=Tunneling=
		+	You can tunnel single ports or run a socks proxy with SSH. Google for SSH forwarding or see the howto below.
		+
		+	=See Also=
		+	* [[Proxies and VPN]]
		+
		+	=External Links=
		+	* [[w:Secure Shell\|Wikipedia Article]]
		+	* [http://dltj.org/article/ssh-as-socks-proxy/ SSH socks proxy howto]
		+	* [http://www.skullsecurity.org/wiki/index.php/Passwords Passwords Lists]
		+	* [http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/index.html oclHashcat-plus for Linux]
		+
		+	[[Category:Howtos]]
		+	[[Category:Software]]
	[[Category:Pending deletion]]		[[Category:Pending deletion]]

← Older revision		Revision as of 10:03, 6 January 2018
Line 1:		Line 1:
−	~~How is the related to VTLUUG~~
−
−	~~An '''Amazon EC2: GPU cluster''' is a remote GPU cluster serive.~~
−
−	~~=Creating a new image=~~
−	~~Check which distributions are supported, [https://developer.nvidia.com/cuda-downloads here]. All you are concerned about are the developer drivers.~~
−	~~From this point on, we will deal with a SUSE instance, but the general steps will be the same. '''Only select an instance that can work with CUDA!'''~~
−
−	* With the Classic Wizard, select Cluster Instances HVM SUSE Linux Enterprise 11
−	* Change the Instance Type from M3 Extra Large to CG1 Cluster GPU (cg1.4xlarge).
−	* Enable Termination protection.
−	* Make sure you specify that your Elastic Block Storage volume NOT be destroyed when the instance is terminated (or your data will be lost when the instance is terminated, which may happen without your consent if you are using a spot instance and the variable price exceeds your maximum bid).
−	* Generate your key pair and allow SSH access to the server.
−	* Review the instance type and ensure all details are correct.
−	* Pricing information may be found [https://aws.amazon.com/ec2/pricing/ here]. "Spot instances" are generally the way to go but be sure to bid with a high enough margin from the current price that you wont be shut down unintentionally. Even double the current spot price is usually lower than the on-demand price.
−
−	~~=Installing CUDA=~~
−	~~Zypper is Suse's package manager; YaST could alternatively be used if more comfortable with a GUI.~~
−	~~First, SSH into the system, update the package cache and upgrade:~~
−	~~<pre>SSH -i /PATH/TO/PKEY root@ec2-...-amazonaws.com~~
−	~~zypper ref~~
−	~~zypper up</pre>~~
−
−	~~Install the needed packages and generate needed config:~~
−	~~<pre>zypper in gcc kernel-ec2-devel kernel-docs~~
−	~~cd /usr/src/linux~~
−	~~make cloneconfig</pre>~~
−
−	~~Restart the instance:~~
−	~~<pre>reboot</pre>~~
−	~~You will be booted from the instance, just wait for it to resume again.~~
−
−	~~Now get the latest CUDA install and run the file:~~
−	~~<pre>wget http://developer.download.nvidia.com/compute/cuda/$CUDA_MAJOR_VERSION/$CUDA_MINOR_VERSION/installers/cuda_5.0.35_$(Linux\|Windows)_$ARCH_$IMG.run~~
−	~~chmod +x cuda_*.run~~
−	~~./cuda_*.run~~
−	~~</pre>~~
−	We only need the developer drivers. Ensure that the drivers install correctly. Note that it needs to run as root. The cuda dev drivers are already installed in some AMIs (for example the CentOS-GPU image).
−
−	~~=oclHashCat-plus=~~
−	~~Get oclHashCat:~~
−	~~<pre>wget http://hashcat.net/files/oclHashcat-plus-0.14.7z</pre>~~
−
−	~~To extract and use oclHashCat, we must unpackage the 7zip archive. In many distributions 7z is provided by the p7zip package. SUSE users need to download the binaries and extract it:~~
−	~~<pre>wget http://downloads.sourceforge.net/project/p7zip/p7zip/9.20.1/p7zip_9.20.1_x86_linux_bin.tar.bz2?r=&ts=1365352571&use_mirror=hivelocity~~
−	~~tar xjvf p7zip_9.20.1_x86_linux_bin.tar.bz2</pre>~~
−
−	~~Unarchive file, extracting full path:~~
−	~~<pre>7z x oclHashcat-plus-0.14.7z~~
−	~~cd oclHashcat-plus-0.14</pre>~~
−
−	~~Running oclHashcat-plus (in a screen session):~~
−	~~<pre>screen~~
−	~~./cudaHashcat-plus64 $HASHES ($DICTIONARY \| -m $hashmask \| --rules=$RULES) </pre>~~
−	~~oclHashcat-plus assumes unsalted md5 by default~~
−
−	~~=Tunneling=~~
−	~~You can tunnel single ports or run a socks proxy with SSH. Google for SSH forwarding or see the howto below.~~
−
−	~~=See Also=~~
−	* [[Proxies and VPN]]
−
−	~~=External Links=~~
−	* [[w:Secure Shell\|Wikipedia Article]]
−	* [http://dltj.org/article/ssh-as-socks-proxy/ SSH socks proxy howto]
−	* [http://www.skullsecurity.org/wiki/index.php/Passwords Passwords Lists]
−	* [http://docs.nvidia.com/cuda/cuda-getting-started-guide-for-linux/index.html oclHashcat-plus for Linux]
−
−	~~[[Category:Howtos]]~~
−	~~[[Category:Software]]~~
	[[Category:Pending deletion]]		[[Category:Pending deletion]]

@@ Line 69: / Line 69: @@
 [[Category:Howtos]]
 [[Category:Software]]
-[[Category:Pending Deletion]]
+[[Category:Pending deletion]]

← Older revision		Revision as of 08:39, 2 January 2018
Line 69:		Line 69:
	[[Category:Howtos]]		[[Category:Howtos]]
	[[Category:Software]]		[[Category:Software]]
		+	[[Category:Pending Deletion]]

@@ Line 5: / Line 5: @@
 From this point on, we will deal with a SUSE instance, but the general steps will be the same. '''Only select an instance that can work with CUDA!'''
-* With the Classic Wizard, select SUSE Linux Enterprise Server 11.
+* With the Classic Wizard, select Cluster Instances HVM SUSE Linux Enterprise 11
-* Change the Instance Type from T1 Micro to Cluster Compute Eight Extra Large Instance (cg1.4xlarge).
+* Change the Instance Type from M3 Extra Large to CG1 Cluster GPU (cg1.4xlarge).
 * Enable Termination protection.
 * Make sure you specify that your Elastic Block Storage volume NOT be destroyed when the instance is terminated (or your data will be lost when the instance is terminated, which may happen without your consent if you are using a spot instance and the variable price exceeds your maximum bid).

@@ Line 32: / Line 32: @@
 <pre>wget http://developer.download.nvidia.com/compute/cuda/$CUDA_MAJOR_VERSION/$CUDA_MINOR_VERSION/installers/cuda_5.0.35_$(Linux|Windows)_$ARCH_$IMG.run
 chmod +x cuda_*.run
-sudo ./cuda_*.run
+./cuda_*.run
 </pre>
 We only need the developer drivers. Ensure that the drivers install correctly. Note that it needs to run as root. The cuda dev drivers are already installed in some AMIs (for example the CentOS-GPU image).

@@ Line 8: / Line 8: @@
 * Change the Instance Type from T1 Micro to Cluster Compute Eight Extra Large Instance (cg1.4xlarge).
 * Enable Termination protection.
-* Make sure you specify that your Elastic Block Storage volume NOT be destroyed when the instance is terminated (or be absolutely sure you get everything you need off of it before you terminate the instance).
+* Make sure you specify that your Elastic Block Storage volume NOT be destroyed when the instance is terminated (or your data will be lost when the instance is terminated, which may happen without your consent if you are using a spot instance and the variable price exceeds your maximum bid).
 * Generate your key pair and allow SSH access to the server.
 * Review the instance type and ensure all details are correct.
-* Pricing information may be found [https://aws.amazon.com/ec2/pricing/ here].
+* Pricing information may be found [https://aws.amazon.com/ec2/pricing/ here]. "Spot instances" are generally the way to go but be sure to bid with a high enough margin from the current price that you wont be shut down unintentionally. Even double the current spot price is usually lower than the on-demand price.
 =Installing CUDA=