create cluster with Ambari without Docker container successfully

[jentle]

Hi, guys. It is able to use ambari web UI or REST API to build a cluster in AWS. You can view the template at http://54.86.210.191:8080

The instance is CentOS 6.3 64 bits, 1GB RAM.

1. Download repo and setup Ambari server in one of the instance. http://docs.hortonworks.com/HDPDocuments/Ambari-2.0.1.0/bk_Installing_HDP_AMB/content/_download_the_ambari_repo.html
2. Create other instances as slave nodes, enable ssh login as root
3. When create the cluster with ambari, register hosts with AWS instance private DNS, paste the public key to enable SSH. Ambari will handle the installation and register all the host instances.
   or you can choose install the ambari-agent in each instance and register ambari-server hostname, then start.
4. Select componets and services , launch the cluster.

TIPS:
It would be much easier to find a way to hide the ambari-agent docker under a hostname known to agent-server. Simply register in Ambari-server.

Some point to care : Ambari 1.7.0 version seems stable at present, 2.1.0 version would cause some problem in different OS. Currently support and limit to CentOS 6, SLES11, Ubuntu 12. The RAM size of Ambari server would not sufficient to handle the operation of Cluster. It is better to use a enhanced instance. To running the Hadoop job, we have to attach storage to the instance.

[dragonfly90]

Where is the centos image ?

[jentle]

It is the instance OS，not docker image

在 2015年10月10日，上午10:47，Liang Dong notifications@github.com 写道：

Where is the centos image ?

―
Reply to this email directly or view it on GitHub.

[jentle]

Hi, guys. I have partially resolved the problem and wish you could contribute further work.

You can follow the script here to reproduce or view the cluster at 54.84.28.157:8080.

Ambari Server
In the server node, we will deploy the ambari server container and consul agent container (store all the cluster nodes). You can also deploy the consul in separate instance .

The consul and Ambari server is hosted at 172.31.55.213, public address 54.84.28.157

_launch the consul agent_

docker run -d -p 8500:8500 -p 8400:8400 -p 53:53/udp --name amb-consul -h amb-consul.service.consul  sequenceiq/consul -server -bootstrap -ui-dir /ui

-p 8500 , registering node using HTTP
-P 8400, RPC
-p 53, Expose DNS for other hosts
-ui-dir, Enable the Web UI, you can visit using http://54.84.28.157:8500

_Setup the Ambari server_

docker run -d -p 8080:8080 -p 8441:8441 -p 8440:8440 -e BRIDGE_IP=172.31.55.213 --name amb-server -h amb-server.service.consul sequenceiq/ambari /start-server

-p 8080, Web UI
-p 8440, 8441 for Ambari agent register and sent heatbeat
BRIDGE_IP, config as the DNS server address for all the node, here is the consul agent IP address.
here we only start the Ambari server

_Register Ambari server hostname to consul_

curl -X PUT -d "{\"Node\":\"amb-server\",\"Address\":\"172.31.55.213\", \"Service\":{\"Service\":\"amb-server\"}}" http://54.84.28.157:8500/v1/catalog/register

here we use the consul API to register the server node. We used the private IP address since the cluster is running inside my network.

next we create a duplicate for the server host named "ambari-8080", which we will used for ambari agent regisetering.

curl -X PUT -d "{\"Node\":\"ambari-8080\",\"Address\":\"172.31.55.213\", \"Service\":{\"Service\":\"ambari-8080\"}}" http://54.84.28.157:8500/v1/catalog/register

Ambari Agent
We launch a docker container as Ambari agent named amb-client1. This is much easier.

docker run -d -p 8670 -e BRIDGE_IP=172.31.55.213 --name amb-client1 -h amb-client1.service.consul sequenceiq/ambari /start-agent

-p 8670, used for Ambari server send configuration to ambari agent
BRIDGE_IP, config as the DNS server address for all the node, here is the consul agent IP address.
here we only start the Ambari server.

Let's take a look at how it works.

#ambari agent initialization
$ vi /etc/ambari-agent/conf/ambari-agent.ini

[server]
hostname=ambari-8080.service.consul
url_port=8440
secured_url_port=8441

When we launch the container

ambari-agent start

the container resolve the hostname "ambari-8080.service.consul". Obviously, it is blind to this hostname. Have no choice, it turn to 172.31.55.213 via interface 53 for help. Amazingly, that is where consul located and we just registered ambari-8080 before. Which could be found in

$ vi /etc/resolv.conf
nameserver 172.31.55.213
search service.consul node.dc1.consul                                       

Next step is similar,register to consul of our client node

curl -X PUT -d "{\"Node\":\"amb-client1\",\"Address\":\"172.31.55.215\", \"Service\":{\"Service\":\"amb-client1\"}}" http://54.84.28.157:8500/v1/catalog/register

Create the cluster
For testing, I just choose the web to install the cluster.
In "install options", type in the client node "amb-client1.service.consul", it will be resolved by the server.
Since we have ambari agent installed, do not choose SSH, and move next. We can see the node is successfully confirmed. Then you could install other service inside. You might receive many warnings with only 1 node and some ports not exposed.
You can view or add more nodes just follow above steps. Quite easy.

[jessexu20]

Okay... will look into it some time tmr ...

[jessexu20]

So I suppose we are using the latest version with consul instead of the 1.7 ? correct? @jentle

[maxlpy]

I remember he used 1.7.

[jentle]

the version of doesnt matter， the consul and Ambai are in different docker and independent. The key point is you have to start server and agent separately. but notice that in /etc/Ambari-agent/conf/Ambari-agent.init, configure the host name of server correctly. Well， you can use the lattest version without causing other problem.

在 2015年10月11日，上午11:24，Jesse Xu notifications@github.com 写道：

So I suppose we are using the latest version with consul instead of the 1.7 ? correct?

―
Reply to this email directly or view it on GitHub.

[maxlpy]

I can use my private IP to register to your consul.