This article consist of two parts the first part will explain you how to setup multi-node Hadoop cluster on Amazon Cloud and second will explain you how client can access AWS EC2 Instances.
First you need to create AWS account for that follow these step
- Open http://aws.amazon.com
- Follow the on screen instructions.
- Procedure involves receiving a phone call and entering a PIN.
Sign into Amazon AWS Account:
- Once you successfully signup. Now use your credentials for Login to AWS account console.
Launch Amazon EC2 Instance:
- Open EC2 service of AWS.
- From EC2 screen click on Launch Instance.
Amazon Machine Images (AMIs):
- Choose Ubuntu Server 14.04 64-bit OS from a list of Configurations called Amazon Machine Images (AMIs).
Choose Instance Type:
- On this page t2.micro instance type is selected by default. This is the only instance type which is free.
Configure Instance Detail:
- Now click on Next: Configure Instance Details. Here mention number of instance you want and keep other option as default.
- Click on Next Add storage: Move on with default storage 8Gb.
- Click On Next: Add Tags. Here give your instance name.
Configure Security group:
- Click on Next: Configure Security group. Give name to security group later we modify it more.
Creating a new key pair:
- By Clicking on Review and Launch it will display a dialouge box for “Selecting an existing key pair or creating a new key pair”. A new key pair is generated and named it as “hadoopec2cluster” and it is required to be downloaded on the local machine. Be assure to download it because you will not be able to download it later once you pursue this step.
click Download Key Pair (.pem) file to your local machine.
- Click on Launch Instance.
- Instances will launch with pending state they will take time to be in a running state.
- Rename all the instances as below in given image.
Note Instance Details:
- Note down Instance ID, Public DNS and Public IP for each instance. We will need it later on to connect from Putty client.
Edit Security Group:
- Click on EC2 Security Group from left paned for more Edition.
- Select HadoopEC2SecurityGroup that we created above. Under inbound option click on Edit.
- Here We have to add two more rules “All TCP” and “All ICMP”.
- To access Linux Instances from window you need to download and install Putty and puttygen From Here
- Once download is completed open Puttygen.
- Load .pem private key file that you download previously into your system.
- Click Ok and save private key with the same name given to .pem file. PuTTY automatically adds the .ppk file extension.
- Now we will connect to Linux instance using PuTTY (Starting from HadoopNameNode) and all instances will follow the same process.
- In the Host Name box enter user_name@public_dns_name or public_ip. Default user name for Ubuntu machine is “ubuntu”
Under Connection type select SSH.Ensure that Port is 22.
- Now expand SSH from left category panel and click on Auth.
- Browse .ppk file and click open.
- If this is the first time you have connected to this instance, PuTTY displays a security alert dialog box that asks whether you trust the host you are connecting to. Click Yes. A window opens and you are connected to your instance.