1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
|
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
------------------------------------------------------------------------------------------------------------------------------------------------------
#BigTop Docker provisioner
## Overview
The Docker Compose definition and wrapper script that creates Bigtop virtual Hadoop cluster on top of Docker containers for you, by pulling from existing publishing bigtop repositories.
This cluster can be used:
- to test bigtop smoke tests
- to test bigtop puppet recipes
- to run integration test with your application
This has been verified on Docker Engine 1.9.1, with api version 1.15, and Docker Compose 1.5.2 on Amazon Linux 2015.09 release.
## Prerequisites
### OS X and Windows
* Install [Docker Toolbox](https://www.docker.com/docker-toolbox)
### Linux
* Install [Docker](https://docs.docker.com/installation/)
* Install [Docker Compose](https://docs.docker.com/compose/install/)
* Start the Docker daemon
```
service docker start
```
## USAGE
1) Create a Bigtop Hadoop cluster by given # of node.
```
./docker-hadoop.sh --create 3
```
2) Destroy the cluster.
```
./docker-hadoop.sh --destroy
```
3) Get into the first container (the master)
```
./docker-hadoop.sh --exec 1 bash
```
4) Execute a command on the second container
```
./docker-hadoop.sh --exec 2 hadoop fs -ls /
```
5) Update your cluster after doing configuration changes on ./config. (re-run puppet apply)
```
./docker-hadoop.sh --provision
```
6) Run Bigtop smoke tests
```
./docker-hadoop.sh --smoke-tests
```
7) Chain your operations with-in one command.
```
./docker-hadoop.sh --create 5 --smoke-tests --destroy
```
Commands will be executed by following order:
```
create 5 node cluster => run smoke tests => destroy the cluster
```
8) See helper message:
```
./docker-hadoop.sh -h
usage: docker-hadoop.sh [-C file ] args
-C file Use alternate file for config.yaml
commands:
-c NUM_INSTANCES, --create=NUM_INSTANCES Create a Docker based Bigtop Hadoop cluster
-d, --destroy Destroy the cluster
-e, --exec INSTANCE_NO|INSTANCE_NAME Execute command on a specific instance. Instance can be specified by name or number.
For example: docker-hadoop.sh --exec 1 bash
docker-hadoop.sh --exec docker_bigtop_1 bash
-p, --provision Deploy configuration changes
-s, --smoke-tests Run Bigtop smoke tests
-h, --help
```
##Configurations
* There are several parameters can be configured in config.yaml:
1) Modify memory limit for Docker containers
```
docker:
memory_size: "2048"
```
2) Use different host ports mapping for web UIs
```
namenode_ui_port: "50070"
yarn_ui_port: "8088"
hbase_ui_port: "60010"
```
Note: If running on OS X or Windows, the boot2docker VM should be reloaded after ports changed
##Configure Apache Hadoop ecosystem components
* Choose the ecosystem you want to be deployed by modifying components in config.yaml
```
components: "hadoop, hbase, yarn,..."
```
By default, Apache Hadoop and YARN will be installed.
|
