In this article, I would show how to build a solid elasticsearch cluster on AWS. In the part 1 we would cover all the details of the AWS EC2 instance creation and preparing the environment to setup the elasticsearch cluster.
First of all to build the AWS cluster we should launch 3 EC2 instances. Why 3, not 2? After this small discussion I found that the cluster of 2 instances is not much better then the cluster of single instance.
As operation system we would use Ubuntu.
Creating the instances
Selecting the operation system
As our cluster would not be the real one, then we would use m3.medium kind of instance. If you’re building the real cluster, then, you probably want to have a lot of CPUs (so it would index everything faster) and a lot of RAM (so it can load your big indices to RAM during the search).
Selecting the instance type
Configuring the instance
On the step 4, it could be reasonable to add some space to your instatnce. As you’re going to index a lot and therefore store a lot of information. In our case we would live it as is, but have in mind that you can do that. Here you can read how to mount the EBS to your instance.
Adding the storage
Tagging the instance
It’s reasonable to add the elasticsearch cluster to the separate security group:
Adding the new security group
Configuring the security group
Next steps – reviewing the instances we are going to launch and pointing the key pair to access them.
Reviewing the instances
Pointing the key pair
The instances are launching:
The instances are launching
Overview of created instances
How we should create 3 elastic IPs and assign them to our instances, so we can restart the instances without the fear that they would receive new IP. Also, let’s agree, how we would name our instances, in my case those would be Homer, Marge and Bart.
Allocating the elastic IP
Allocating the new address
Associate address 1
Associate address 2
Associated addresses overview
The instances overview
One more thing, to make the configuration even more convinient – we would update our local /etc/hosts to associate the instance names with their IPs.
|
1 2 3 |
107.21.118.206 bart 107.22.161.139 homer 107.22.165.165 marge |
And now, we would state that hostname on each of the instances (you should do it while you are on the instance).
|
1 2 3 4 5 |
sudo bash hostname bart echo 127.0.1.1 bart >> /etc/hosts echo bart > /etc/hostname exit |
(the same should be done for the other instances)
After that, we would see the hostname, when we’re logged to the host:
Overview of the hostname on the instance
Another thing we should do in the AWS back-end – create the IAM user, so the cluster instances would receive the access to our AWS account and would be able to find each other.
To do that, open the IAM Management Console and create a new user.
Creating the user
Entering the user names
User credentials
You should write down those credentials, because we would use them during the instances configuration. IAM Management Console allows you to download them.
Plus, you should give the user read-only permissions, so it can read the EC2 instances list.
User permissions
User policy template
User policy overview
That’s all. Now we are ready to start the cluster configuration.
Elasticsearch cluster on AWS. Part 2 – configuring the elasticsearch.
Pingback: Elasticsearch cluster on AWS. Part 2 - configuring the elasticsearch. - Pavel Polyakov's blog()