diff options
Diffstat (limited to 'bigtop-packages/src/charm/hadoop/layer-hadoop-plugin')
7 files changed, 140 insertions, 41 deletions
diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/README.md b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/README.md index cbea7f02..405c08ac 100644 --- a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/README.md +++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/README.md @@ -14,79 +14,109 @@ See the License for the specific language governing permissions and limitations under the License. --> -## Overview +# Overview The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model. -This charm facilitates communication between core Apache Bigtop cluster -components and workload charms. +This charm facilitates communication between Hadoop components of an +[Apache Bigtop][] cluster and workload applications. +[Apache Bigtop]: http://bigtop.apache.org/ -## Usage -This charm is intended to be deployed via one of the -[apache bigtop bundles](https://jujucharms.com/u/bigdata-dev/#bundles). -For example: +# Deploying - juju deploy hadoop-processing +A working Juju installation is assumed to be present. If Juju is not yet set +up, please follow the [getting-started][] instructions prior to deploying this +charm. -> Note: With Juju versions < 2.0, you will need to use [juju-deployer][] to -deploy the bundle. +This charm is intended to be deployed via one of the [apache bigtop bundles][]. +For example: -This will deploy the Apache Bigtop platform with a workload node -preconfigured to work with the cluster. + juju deploy hadoop-processing -You could extend this deployment, for example, to analyze data using Apache Pig. -Simply deploy Pig and attach it to the same plugin: +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart +hadoop-processing`. - juju deploy apache-pig pig - juju add-relation plugin pig +This will deploy an Apache Bigtop cluster with a client unit preconfigured to +work with the cluster. More information about this deployment can be found in the +[bundle readme](https://jujucharms.com/hadoop-processing/). +## Network-Restricted Environments +Charms can be deployed in environments with limited network access. To deploy +in this environment, configure a Juju model with appropriate proxy and/or +mirror options. See [Configuring Models][] for more information. -[juju-deployer]: https://pypi.python.org/pypi/juju-deployer/ +[getting-started]: https://jujucharms.com/docs/stable/getting-started +[apache bigtop bundles]: https://jujucharms.com/u/bigdata-charmers/#bundles +[juju-quickstart]: https://launchpad.net/juju-quickstart +[Configuring Models]: https://jujucharms.com/docs/stable/models-config -## Status and Smoke Test +# Verifying +## Status Apache Bigtop charms provide extended status reporting to indicate when they are ready: - juju status --format=tabular + juju status This is particularly useful when combined with `watch` to track the on-going progress of the deployment: - watch -n 0.5 juju status --format=tabular + watch -n 2 juju status + +The message column will provide information about a given unit's state. +This charm is ready for use once the status message indicates that it is +ready with hdfs and/or yarn. + +## Smoke Test +This charm provides a `smoke-test` action that can be used to verify the +application is functioning as expected. Run the action as follows: + + juju run-action plugin/0 smoke-test + +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action do plugin/0 smoke-test`. + +Watch the progress of the smoke test actions with: + + watch -n 2 juju show-action-status + +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action status`. -The message for each unit will provide information about that unit's state. -Once they all indicate that they are ready, you can perform a "smoke test" -to verify HDFS or YARN services are working as expected. Trigger the -`smoke-test` action by: +Eventually, the action should settle to `status: completed`. If it +reports `status: failed`, the application is not working as expected. Get +more information about a specific smoke test with: - juju action do namenode/0 smoke-test - juju action do resourcemanager/0 smoke-test + juju show-action-output <action-id> -After a few seconds or so, you can check the results of the smoke test: +> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version +of Juju, the syntax is `juju action fetch <action-id>`. - juju action status +## Utilities +This charm includes Hadoop command line utilities that can be used +to verify information about the cluster. -You will see `status: completed` if the smoke test was successful, or -`status: failed` if it was not. You can get more information on why it failed -via: +Show the dfsadmin report on the command line with the following: - juju action fetch <action-id> + juju run --application plugin "su hdfs -c 'hdfs dfsadmin -report'" -## Contact Information +# Contact Information - <bigdata@lists.ubuntu.com> -## Resources +# Resources - [Apache Bigtop](http://bigtop.apache.org/) home page - [Apache Bigtop issue tracking](http://bigtop.apache.org/issue-tracking.html) - [Apache Bigtop mailing lists](http://bigtop.apache.org/mail-lists.html) -- [Apache Bigtop charms](https://jujucharms.com/q/apache/bigtop) +- [Juju Bigtop charms](https://jujucharms.com/q/apache/bigtop) +- [Juju mailing list](https://lists.ubuntu.com/mailman/listinfo/juju) +- [Juju community](https://jujucharms.com/community) diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/actions.yaml b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/actions.yaml new file mode 100644 index 00000000..c2d65aec --- /dev/null +++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/actions.yaml @@ -0,0 +1,2 @@ +smoke-test: + description: Run a simple HDFS smoke test. diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/actions/smoke-test b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/actions/smoke-test new file mode 100755 index 00000000..65ba07c6 --- /dev/null +++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/actions/smoke-test @@ -0,0 +1,62 @@ +#!/usr/bin/env python3 + +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import sys + +from charmhelpers.core import hookenv +from jujubigdata.utils import run_as +from charms.reactive import is_state + +if not is_state('apache-bigtop-plugin.hdfs.ready'): + hookenv.action_fail('Charm is not yet ready') + + +# verify the hdfs-test directory does not already exist +output = run_as('ubuntu', 'hdfs', 'dfs', '-ls', '/tmp', capture_output=True) +if '/tmp/hdfs-test' in output: + run_as('ubuntu', 'hdfs', 'dfs', '-rm', '-R', '/tmp/hdfs-test') + output = run_as('ubuntu', 'hdfs', 'dfs', '-ls', '/tmp', capture_output=True) + if 'hdfs-test' in output: + hookenv.action_fail('Unable to remove existing hdfs-test directory') + sys.exit() + +# create the directory +run_as('ubuntu', 'hdfs', 'dfs', '-mkdir', '-p', '/tmp/hdfs-test') +run_as('ubuntu', 'hdfs', 'dfs', '-chmod', '-R', '777', '/tmp/hdfs-test') + +# verify the newly created hdfs-test subdirectory exists +output = run_as('ubuntu', 'hdfs', 'dfs', '-ls', '/tmp', capture_output=True) +for line in output.split('\n'): + if '/tmp/hdfs-test' in line: + if 'ubuntu' not in line or 'drwxrwxrwx' not in line: + hookenv.action_fail('Permissions incorrect for hdfs-test directory') + sys.exit() + break +else: + hookenv.action_fail('Unable to create hdfs-test directory') + sys.exit() + +# remove the directory +run_as('ubuntu', 'hdfs', 'dfs', '-rm', '-R', '/tmp/hdfs-test') + +# verify the hdfs-test subdirectory has been removed +output = run_as('ubuntu', 'hdfs', 'dfs', '-ls', '/tmp', capture_output=True) +if '/tmp/hdfs-test' in output: + hookenv.action_fail('Unable to remove hdfs-test directory') + sys.exit() + +hookenv.action_set({'outcome': 'success'}) diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/layer.yaml b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/layer.yaml index 5ddc2c9b..ceedad7f 100644 --- a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/layer.yaml +++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/layer.yaml @@ -1,8 +1,12 @@ -repo: git@github.com:juju-solutions/layer-hadoop-plugin.git -includes: ['layer:apache-bigtop-base', 'interface:hadoop-plugin', 'interface:dfs', 'interface:mapred'] +repo: https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin +includes: + - 'layer:apache-bigtop-base' + - 'interface:hadoop-plugin' + - 'interface:dfs' + - 'interface:mapred' options: basic: use_venv: true metadata: deletes: - - requires.java + - provides.java diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/metadata.yaml b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/metadata.yaml index a5fd4538..4df86f1b 100644 --- a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/metadata.yaml +++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/metadata.yaml @@ -1,5 +1,5 @@ name: hadoop-plugin -summary: Simplified connection point for Apache Bigtop platform +summary: Facilitates communication with an Apache Bigtop Hadoop cluster maintainer: Juju Big Data <bigdata@lists.ubuntu.com> description: > Hadoop is a software platform that lets one easily write and @@ -8,7 +8,7 @@ description: > This charm provides a simplified connection point for client / workload services which require access to Apache Hadoop. This connection is established via the Apache Bigtop gateway. -tags: ["applications", "bigdata", "hadoop", "apache"] +tags: [] subordinate: true requires: namenode: diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/reactive/apache_bigtop_plugin.py b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/reactive/apache_bigtop_plugin.py index e5b12758..e680002a 100644 --- a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/reactive/apache_bigtop_plugin.py +++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/reactive/apache_bigtop_plugin.py @@ -42,6 +42,7 @@ def install_hadoop_client_hdfs(principal, namenode): bigtop.render_site_yaml(hosts=hosts, roles='hadoop-client') bigtop.trigger_puppet() set_state('apache-bigtop-plugin.hdfs.installed') + hookenv.application_version_set(get_hadoop_version()) hookenv.status_set('maintenance', 'plugin (hdfs) installed') else: hookenv.status_set('waiting', 'waiting for namenode fqdn') diff --git a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/tests/01-basic-deployment.py b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/tests/01-basic-deployment.py index 512630df..815f9fbf 100755 --- a/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/tests/01-basic-deployment.py +++ b/bigtop-packages/src/charm/hadoop/layer-hadoop-plugin/tests/01-basic-deployment.py @@ -29,7 +29,7 @@ class TestDeploy(unittest.TestCase): """ def test_deploy(self): - self.d = amulet.Deployment(series='trusty') + self.d = amulet.Deployment(series='xenial') self.d.load({ 'services': { 'client': {'charm': 'hadoop-client'}, |