BIGTOP-2617: refresh juju spark-processing bundle (closes #168)

Signed-off-by: Kevin W Monroe <kevin.monroe@canonical.com>

6 files changed, 206 insertions, 134 deletions

diff --git a/bigtop-deploy/juju/spark-processing/README.md b/bigtop-deploy/juju/spark-processing/README.md
index 335566b5..36cf0298 100644
--- a/bigtop-deploy/juju/spark-processing/README.md
+++ b/bigtop-deploy/juju/spark-processing/README.md
@@ -17,17 +17,20 @@
 # Overview
 
 This bundle provides a complete deployment of
-[Apache Spark](https://spark.apache.org/) in standalone HA mode as provided
-by [Apache Bigtop](http://bigtop.apache.org/). Ganglia and rsyslog
+[Apache Spark][] in standalone HA mode as provided
+by [Apache Bigtop][]. Ganglia and rsyslog
 applications are included to monitor cluster health and syslog activity.
 
+[Apache Spark]: http://spark/apache.org/
+[Apache Bigtop]: http://bigtop.apache.org/
+
 ## Bundle Composition
 
-The applications that comprise this bundle are spread across 7 units as
+The applications that comprise this bundle are spread across 6 units as
 follows:
 
   * Spark (Master and Worker)
-    * 3 separate units
+    * 2 separate units
   * Zookeeper
     * 3 separate units
   * Ganglia (Web interface for monitoring cluster metrics)
@@ -42,61 +45,85 @@ demands.
 # Deploying
 
 A working Juju installation is assumed to be present. If Juju is not yet set
-up, please follow the
-[getting-started](https://jujucharms.com/docs/2.0/getting-started)
-instructions prior to deploying this bundle.
+up, please follow the [getting-started][] instructions prior to deploying this
+bundle.
+
+> **Note**: This bundle requires hardware resources that may exceed limits
+of Free-tier or Trial accounts on some clouds. To deploy to these
+environments, modify a local copy of [bundle.yaml][] with
+`zookeeper: num_units: 1` and `machines: 'X': constraints: mem=3G` as needed
+to satisfy account limits.
 
-Once ready, deploy this bundle with the `juju deploy` command:
+Deploy this bundle from the Juju charm store with the `juju deploy` command:
 
     juju deploy spark-processing
 
 > **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, use [juju-quickstart](https://launchpad.net/juju-quickstart) with the
-following syntax: `juju quickstart spark-processing`.
+of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
+spark-processing`.
+
+Alternatively, deploy a locally modified `bundle.yaml` with:
+
+    juju deploy /path/to/bundle.yaml
+
+> **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
+of Juju, use [juju-quickstart][] with the following syntax: `juju quickstart
+/path/to/bundle.yaml`.
 
 The charms in this bundle can also be built from their source layers in the
 [Bigtop charm repository][].  See the [Bigtop charm README][] for instructions
 on building and deploying these charms locally.
 
+## Network-Restricted Environments
+Charms can be deployed in environments with limited network access. To deploy
+in this environment, configure a Juju model with appropriate proxy and/or
+mirror options. See [Configuring Models][] for more information.
+
+[getting-started]: https://jujucharms.com/docs/stable/getting-started
+[bundle.yaml]: https://github.com/apache/bigtop/blob/master/bigtop-deploy/juju/spark-processing/bundle.yaml
+[juju-quickstart]: https://launchpad.net/juju-quickstart
 [Bigtop charm repository]: https://github.com/apache/bigtop/tree/master/bigtop-packages/src/charm
 [Bigtop charm README]: https://github.com/apache/bigtop/blob/master/bigtop-packages/src/charm/README.md
+[Configuring Models]: https://jujucharms.com/docs/stable/models-config
 
 
 # Verifying
 
 ## Status
-The applications that make up this bundle provide status messages to
-indicate when they are ready:
+The applications that make up this bundle provide status messages to indicate
+when they are ready:
 
     juju status
 
 This is particularly useful when combined with `watch` to track the on-going
 progress of the deployment:
 
-    watch -n 0.5 juju status
+    watch -n 2 juju status
 
 The message for each unit will provide information about that unit's state.
 Once they all indicate that they are ready, perform application smoke tests
 to verify that the bundle is working as expected.
 
 ## Smoke Test
-The spark charm provides a `smoke-test` action that can be used to verify the
-application is functioning as expected. Run it as follows:
+The spark and zookeeper charms provide a `smoke-test` action that can be used
+to verify the respective application is functioning as expected. Run these
+actions as follows:
 
     juju run-action spark/0 smoke-test
+    juju run-action zookeeper/0 smoke-test
 
 > **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
-of Juju, the syntax is `juju action do spark/0 smoke-test`.
+of Juju, the syntax is `juju action do <application>/0 smoke-test`.
 
-You can watch the progress of the smoke test action with:
+Watch the progress of the smoke test actions with:
 
-    watch -n 0.5 juju show-action-status
+    watch -n 2 juju show-action-status
 
 > **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
 of Juju, the syntax is `juju action status`.
 
-Eventually, the smoke test should settle to `status: completed`.  If
-it reports `status: failed`, Spark is not working as expected. Get
+Eventually, all of the actions should settle to `status: completed`.  If
+any report `status: failed`, that application is not working as expected. Get
 more information about the smoke-test action
 
     juju show-action-output <action-id>
@@ -104,29 +131,46 @@ more information about the smoke-test action
 > **Note**: The above assumes Juju 2.0 or greater. If using an earlier version
 of Juju, the syntax is `juju action fetch <action-id>`.
 
+## Utilities
+Applications in this bundle include Zookeeper command line and Spark web
+utilities that can be used to verify information about the cluster.
 
-# Monitoring
+From the command line, show the list of Zookeeper nodes with the following:
 
-This bundle includes Ganglia for system-level monitoring of the spark units.
-Metrics are sent to a centralized ganglia unit for easy viewing in a browser.
-To view the ganglia web interface, first expose the service:
+    juju run --unit zookeeper/0 'echo "ls /" | /usr/lib/zookeeper/bin/zkCli.sh'
 
-    juju expose ganglia
+To access the Spark web console, find the `PUBLIC-ADDRESS` of the spark
+application and expose it:
 
-Now find the ganglia public IP address:
+    juju status spark
+    juju expose spark
+
+The web interface will be available at the following URL:
+
+    http://SPARK_PUBLIC_IP:8080
+
+
+# Monitoring
+
+This bundle includes Ganglia for system-level monitoring of the spark and
+zookeeper units. Metrics are sent to a centralized ganglia unit for easy
+viewing in a browser. To view the ganglia web interface, find the
+`PUBLIC-ADDRESS` of the Ganglia application and expose it:
 
     juju status ganglia
+    juju expose ganglia
 
-The ganglia web interface will be available at:
+The web interface will be available at:
 
     http://GANGLIA_PUBLIC_IP/ganglia
 
 
 # Logging
 
-This bundle includes rsyslog to collect syslog data from the spark unit. These
-logs are sent to a centralized rsyslog unit for easy syslog analysis. One
-method of viewing this log data is to simply cat syslog from the rsyslog unit:
+This bundle includes rsyslog to collect syslog data from the spark and
+zookeeper units. These logs are sent to a centralized rsyslog unit for easy
+syslog analysis. One method of viewing this log data is to simply cat syslog
+from the rsyslog unit:
 
     juju run --unit rsyslog/0 'sudo cat /var/log/syslog'
 
@@ -142,16 +186,22 @@ the performance of the Spark cluster. Each benchmark is an action that can be
 run with `juju run-action`:
 
     $ juju actions spark | grep Bench
+    connectedcomponent                Run the Spark Bench ConnectedComponent benchmark.
+    decisiontree                      Run the Spark Bench DecisionTree benchmark.
+    kmeans                            Run the Spark Bench KMeans benchmark.
+    linearregression                  Run the Spark Bench LinearRegression benchmark.
     logisticregression                Run the Spark Bench LogisticRegression benchmark.
     matrixfactorization               Run the Spark Bench MatrixFactorization benchmark.
     pagerank                          Run the Spark Bench PageRank benchmark.
+    pca                               Run the Spark Bench PCA benchmark.
+    pregeloperation                   Run the Spark Bench PregelOperation benchmark.
+    shortestpaths                     Run the Spark Bench ShortestPaths benchmark.
     sql                               Run the Spark Bench SQL benchmark.
-    streaming                         Run the Spark Bench Streaming benchmark.
+    stronglyconnectedcomponent        Run the Spark Bench StronglyConnectedComponent benchmark.
     svdplusplus                       Run the Spark Bench SVDPlusPlus benchmark.
     svm                               Run the Spark Bench SVM benchmark.
-    trianglecount                     Run the Spark Bench TriangleCount benchmark.
 
-    $ juju run-action spark/0 pagerank
+    $ juju run-action spark/0 svdplusplus
     Action queued with id: 339cec1f-e903-4ee7-85ca-876fb0c3d28e
 
     $ juju show-action-output 339cec1f-e903-4ee7-85ca-876fb0c3d28e
@@ -160,48 +210,40 @@ run with `juju run-action`:
         composite:
           direction: asc
           units: secs
-          value: ".982000"
+          value: "200.754000"
         raw: |
-          PageRank,0,.982000,,,,PageRank-MLlibConfig,,,,,10,12,,200000,4.0,1.3,0.15
-        start: 2016-09-22T21:52:26Z
-        stop: 2016-09-22T21:52:33Z
+          SVDPlusPlus,2016-11-02-03:08:26,200.754000,85.974071,.428255,0,SVDPlusPlus-MLlibConfig,,,,,10,,,50000,4.0,1.3,
+        start: 2016-11-02T03:08:26Z
+        stop: 2016-11-02T03:11:47Z
       results:
         duration:
           direction: asc
           units: secs
-          value: ".982000"
+          value: "200.754000"
         throughput:
           direction: desc
           units: x/sec
-          value: ""
+          value: ".428255"
     status: completed
     timing:
-      completed: 2016-09-22 21:52:36 +0000 UTC
-      enqueued: 2016-09-22 21:52:09 +0000 UTC
-      started: 2016-09-22 21:52:13 +0000 UTC
+      completed: 2016-11-02 03:11:48 +0000 UTC
+      enqueued: 2016-11-02 03:08:21 +0000 UTC
+      started: 2016-11-02 03:08:26 +0000 UTC
 
 
 # Scaling
 
-By default, three spark units are deployed. To increase the amount of spark
-workers, simply add more units. To add one unit:
+By default, three spark and three zookeeper units are deployed. Scaling these
+applications is as simple as adding more units. To add one unit:
 
     juju add-unit spark
+    juju add-unit zookeeper
 
 Multiple units may be added at once.  For example, add four more spark units:
 
     juju add-unit -n4 spark
 
 
-# Network-Restricted Environments
-
-Charms can be deployed in environments with limited network access. To deploy
-in this environment, configure a Juju model with appropriate
-proxy and/or mirror options. See
-[Configuring Models](https://jujucharms.com/docs/2.0/models-config) for more
-information.
-
-
 # Contact Information
 
 - <bigdata@lists.ubuntu.com>
diff --git a/bigtop-deploy/juju/spark-processing/bundle-dev.yaml b/bigtop-deploy/juju/spark-processing/bundle-dev.yaml
index aaaf5143..c9689ec9 100644
--- a/bigtop-deploy/juju/spark-processing/bundle-dev.yaml
+++ b/bigtop-deploy/juju/spark-processing/bundle-dev.yaml
@@ -1,47 +1,46 @@
 services:
   spark:
     charm: "cs:~bigdata-dev/xenial/spark"
-    num_units: 3
+    num_units: 2
     annotations:
       gui-x: "500"
       gui-y: "0"
     to:
+      - "0"
       - "1"
-      - "2"
-      - "3"
   zookeeper:
-    charm: "cs:~charmers/trusty/zookeeper-1"
+    charm: "cs:xenial/zookeeper-10"
     num_units: 3
     annotations:
       gui-x: "500"
       gui-y: "400"
     to:
+      - "2"
+      - "3"
       - "4"
-      - "5"
-      - "6"
   ganglia:
-    charm: "cs:trusty/ganglia-2"
+    charm: "cs:~bigdata-dev/xenial/ganglia-5"
     num_units: 1
     annotations:
       gui-x: "0"
       gui-y: "800"
     to:
-      - "7"
+      - "5"
   ganglia-node:
-    charm: "cs:~bigdata-dev/xenial/ganglia-node-2"
+    charm: "cs:~bigdata-dev/xenial/ganglia-node-6"
     annotations:
       gui-x: "250"
       gui-y: "400"
   rsyslog:
-    charm: "cs:trusty/rsyslog-10"
+    charm: "cs:~bigdata-dev/xenial/rsyslog-6"
     num_units: 1
     annotations:
       gui-x: "1000"
       gui-y: "800"
     to:
-      - "7"
+      - "5"
   rsyslog-forwarder-ha:
-    charm: "cs:~bigdata-dev/xenial/rsyslog-forwarder-ha-2"
+    charm: "cs:~bigdata-dev/xenial/rsyslog-forwarder-ha-7"
     annotations:
       gui-x: "750"
       gui-y: "400"
@@ -49,28 +48,27 @@ series: xenial
 relations:
   - [spark, zookeeper]
   - ["ganglia-node:juju-info", "spark:juju-info"]
+  - ["ganglia-node:juju-info", "zookeeper:juju-info"]
   - ["ganglia:node", "ganglia-node:node"]
   - ["rsyslog-forwarder-ha:juju-info", "spark:juju-info"]
+  - ["rsyslog-forwarder-ha:juju-info", "zookeeper:juju-info"]
   - ["rsyslog:aggregator", "rsyslog-forwarder-ha:syslog"]
 machines:
+  "0":
+    constraints: "mem=7G root-disk=32G"
+    series: "xenial"
   "1":
-    constraints: "mem=7G"
+    constraints: "mem=7G root-disk=32G"
     series: "xenial"
   "2":
-    constraints: "mem=7G"
+    constraints: "mem=3G root-disk=32G"
     series: "xenial"
   "3":
-    constraints: "mem=7G"
+    constraints: "mem=3G root-disk=32G"
     series: "xenial"
   "4":
-    constraints: "mem=3G"
-    series: "trusty"
+    constraints: "mem=3G root-disk=32G"
+    series: "xenial"
   "5":
     constraints: "mem=3G"
-    series: "trusty"
-  "6":
-    constraints: "mem=3G"
-    series: "trusty"
-  "7":
-    constraints: "mem=3G"
-    series: "trusty"
+    series: "xenial"
diff --git a/bigtop-deploy/juju/spark-processing/bundle-local.yaml b/bigtop-deploy/juju/spark-processing/bundle-local.yaml
index b8a4d312..90e51e76 100644
--- a/bigtop-deploy/juju/spark-processing/bundle-local.yaml
+++ b/bigtop-deploy/juju/spark-processing/bundle-local.yaml
@@ -1,47 +1,46 @@
 services:
   spark:
     charm: "/home/ubuntu/charms/xenial/spark"
-    num_units: 3
+    num_units: 2
     annotations:
       gui-x: "500"
       gui-y: "0"
     to:
+      - "0"
       - "1"
-      - "2"
-      - "3"
   zookeeper:
-    charm: "cs:~charmers/trusty/zookeeper-1"
+    charm: "cs:xenial/zookeeper-10"
     num_units: 3
     annotations:
       gui-x: "500"
       gui-y: "400"
     to:
+      - "2"
+      - "3"
       - "4"
-      - "5"
-      - "6"
   ganglia:
-    charm: "cs:trusty/ganglia-2"
+    charm: "cs:~bigdata-dev/xenial/ganglia-5"
     num_units: 1
     annotations:
       gui-x: "0"
       gui-y: "800"
     to:
-      - "7"
+      - "5"
   ganglia-node:
-    charm: "cs:~bigdata-dev/xenial/ganglia-node-2"
+    charm: "cs:~bigdata-dev/xenial/ganglia-node-6"
     annotations:
       gui-x: "250"
       gui-y: "400"
   rsyslog:
-    charm: "cs:trusty/rsyslog-10"
+    charm: "cs:~bigdata-dev/xenial/rsyslog-6"
     num_units: 1
     annotations:
       gui-x: "1000"
       gui-y: "800"
     to:
-      - "7"
+      - "5"
   rsyslog-forwarder-ha:
-    charm: "cs:~bigdata-dev/xenial/rsyslog-forwarder-ha-2"
+    charm: "cs:~bigdata-dev/xenial/rsyslog-forwarder-ha-7"
     annotations:
       gui-x: "750"
       gui-y: "400"
@@ -49,28 +48,27 @@ series: xenial
 relations:
   - [spark, zookeeper]
   - ["ganglia-node:juju-info", "spark:juju-info"]
+  - ["ganglia-node:juju-info", "zookeeper:juju-info"]
   - ["ganglia:node", "ganglia-node:node"]
   - ["rsyslog-forwarder-ha:juju-info", "spark:juju-info"]
+  - ["rsyslog-forwarder-ha:juju-info", "zookeeper:juju-info"]
   - ["rsyslog:aggregator", "rsyslog-forwarder-ha:syslog"]
 machines:
+  "0":
+    constraints: "mem=7G root-disk=32G"
+    series: "xenial"
   "1":
-    constraints: "mem=7G"
+    constraints: "mem=7G root-disk=32G"
     series: "xenial"
   "2":
-    constraints: "mem=7G"
+    constraints: "mem=3G root-disk=32G"
     series: "xenial"
   "3":
-    constraints: "mem=7G"
+    constraints: "mem=3G root-disk=32G"
     series: "xenial"
   "4":
-    constraints: "mem=3G"
-    series: "trusty"
+    constraints: "mem=3G root-disk=32G"
+    series: "xenial"
   "5":
     constraints: "mem=3G"
-    series: "trusty"
-  "6":
-    constraints: "mem=3G"
-    series: "trusty"
-  "7":
-    constraints: "mem=3G"
-    series: "trusty"
+    series: "xenial"
diff --git a/bigtop-deploy/juju/spark-processing/bundle.yaml b/bigtop-deploy/juju/spark-processing/bundle.yaml
index d36ed434..c309b45d 100644
--- a/bigtop-deploy/juju/spark-processing/bundle.yaml
+++ b/bigtop-deploy/juju/spark-processing/bundle.yaml
@@ -1,47 +1,46 @@
 services:
   spark:
-    charm: "cs:xenial/spark-2"
-    num_units: 3
+    charm: "cs:xenial/spark-15"
+    num_units: 2
     annotations:
       gui-x: "500"
       gui-y: "0"
     to:
+      - "0"
       - "1"
-      - "2"
-      - "3"
   zookeeper:
-    charm: "cs:~charmers/trusty/zookeeper-1"
+    charm: "cs:xenial/zookeeper-10"
     num_units: 3
     annotations:
       gui-x: "500"
       gui-y: "400"
     to:
+      - "2"
+      - "3"
       - "4"
-      - "5"
-      - "6"
   ganglia:
-    charm: "cs:trusty/ganglia-2"
+    charm: "cs:~bigdata-dev/xenial/ganglia-5"
     num_units: 1
     annotations:
       gui-x: "0"
       gui-y: "800"
     to:
-      - "7"
+      - "5"
   ganglia-node:
-    charm: "cs:~bigdata-dev/xenial/ganglia-node-2"
+    charm: "cs:~bigdata-dev/xenial/ganglia-node-6"
     annotations:
       gui-x: "250"
       gui-y: "400"
   rsyslog:
-    charm: "cs:trusty/rsyslog-10"
+    charm: "cs:~bigdata-dev/xenial/rsyslog-6"
     num_units: 1
     annotations:
       gui-x: "1000"
       gui-y: "800"
     to:
-      - "7"
+      - "5"
   rsyslog-forwarder-ha:
-    charm: "cs:~bigdata-dev/xenial/rsyslog-forwarder-ha-2"
+    charm: "cs:~bigdata-dev/xenial/rsyslog-forwarder-ha-7"
     annotations:
       gui-x: "750"
       gui-y: "400"
@@ -49,28 +48,27 @@ series: xenial
 relations:
   - [spark, zookeeper]
   - ["ganglia-node:juju-info", "spark:juju-info"]
+  - ["ganglia-node:juju-info", "zookeeper:juju-info"]
   - ["ganglia:node", "ganglia-node:node"]
   - ["rsyslog-forwarder-ha:juju-info", "spark:juju-info"]
+  - ["rsyslog-forwarder-ha:juju-info", "zookeeper:juju-info"]
   - ["rsyslog:aggregator", "rsyslog-forwarder-ha:syslog"]
 machines:
+  "0":
+    constraints: "mem=7G root-disk=32G"
+    series: "xenial"
   "1":
-    constraints: "mem=7G"
+    constraints: "mem=7G root-disk=32G"
     series: "xenial"
   "2":
-    constraints: "mem=7G"
+    constraints: "mem=3G root-disk=32G"
     series: "xenial"
   "3":
-    constraints: "mem=7G"
+    constraints: "mem=3G root-disk=32G"
     series: "xenial"
   "4":
-    constraints: "mem=3G"
-    series: "trusty"
+    constraints: "mem=3G root-disk=32G"
+    series: "xenial"
   "5":
     constraints: "mem=3G"
-    series: "trusty"
-  "6":
-    constraints: "mem=3G"
-    series: "trusty"
-  "7":
-    constraints: "mem=3G"
-    series: "trusty"
+    series: "xenial"
diff --git a/bigtop-deploy/juju/spark-processing/tests/01-bundle.py b/bigtop-deploy/juju/spark-processing/tests/01-bundle.py
index 379778c9..fbb4ebfa 100755
--- a/bigtop-deploy/juju/spark-processing/tests/01-bundle.py
+++ b/bigtop-deploy/juju/spark-processing/tests/01-bundle.py
@@ -15,10 +15,9 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import amulet
 import os
 import unittest
-
-import amulet
 import yaml
 
 
@@ -31,22 +30,54 @@ class TestBundle(unittest.TestCase):
         with open(cls.bundle_file) as f:
             bun = f.read()
         bundle = yaml.safe_load(bun)
+
+        # NB: strip machine ('to') placement out. amulet loses our machine spec
+        # somewhere between yaml and json; without that spec, charms specifying
+        # machine placement will not deploy. This is ok for now because all
+        # charms in this bundle are using 'reset: false' so we'll already
+        # have our deployment just the way we want it by the time this test
+        # runs. However, it's bad. Remove once this is fixed:
+        #  https://github.com/juju/amulet/issues/148
+        for service, service_config in bundle['services'].items():
+            if 'to' in service_config:
+                del service_config['to']
+
         cls.d.load(bundle)
-        cls.d.setup(timeout=1800)
-        cls.d.sentry.wait_for_messages({'spark': 'ready (standalone - HA)'}, timeout=1800)
+        cls.d.setup(timeout=3600)
+        cls.d.sentry.wait_for_messages({'spark': 'ready (standalone - HA)'}, timeout=3600)
         cls.spark = cls.d.sentry['spark'][0]
+        cls.zookeeper = cls.d.sentry['zookeeper'][0]
 
     def test_components(self):
         """
         Confirm that all of the required components are up and running.
         """
-        spark, retcode = self.spark.run("pgrep -a java")
+        spark, rc = self.spark.run("pgrep -a java")
+        zk, rc = self.zookeeper.run("pgrep -a java")
 
-        assert 'spark' in spark, 'Spark should be running on spark'
+        assert 'Master' in spark, "Spark Master should be running"
+        assert 'QuorumPeerMain' in zk, "Zookeeper QuorumPeerMain should be running"
 
     def test_spark(self):
-        output, retcode = self.spark.run("su ubuntu -c 'bash -lc /home/ubuntu/sparkpi.sh 2>&1'")
-        assert 'Pi is roughly' in output, 'SparkPI test failed: %s' % output
+        """
+        Validates Spark with a simple sparkpi test.
+        """
+        uuid = self.spark.run_action('smoke-test')
+        result = self.d.action_fetch(uuid, timeout=600, full_output=True)
+        # action status=completed on success
+        if (result['status'] != "completed"):
+            self.fail('Spark smoke-test did not complete: %s' % result)
+
+    def test_zookeeper(self):
+        """
+        Validates Zookeeper using the Bigtop 'zookeeper' smoke test.
+        """
+        uuid = self.zookeeper.run_action('smoke-test')
+        # 'zookeeper' smoke takes a while (bigtop tests download lots of stuff)
+        result = self.d.action_fetch(uuid, timeout=1800, full_output=True)
+        # action status=completed on success
+        if (result['status'] != "completed"):
+            self.fail('Zookeeper smoke-test did not complete: %s' % result)
 
 
 if __name__ == '__main__':
diff --git a/bigtop-deploy/juju/spark-processing/tests/tests.yaml b/bigtop-deploy/juju/spark-processing/tests/tests.yaml
index 1ec8b824..84f78d7d 100644
--- a/bigtop-deploy/juju/spark-processing/tests/tests.yaml
+++ b/bigtop-deploy/juju/spark-processing/tests/tests.yaml
@@ -1,2 +1,7 @@
+reset: false
+deployment_timeout: 3600
+sources:
+  - 'ppa:juju/stable'
 packages:
   - amulet
+  - python3-yaml