Move requirements away from pypi tooling

Allow CI processes like LXC and Docker to obtain a list of
binary package names for a specified suite of a specified
distribution instead of duplicating the knowledge in the
Pypi requirements files. Include system dependencies which
are not Python.

Signed-off-by: Neil Williams <neil.williams@linaro.org>

10 files changed, 577 insertions, 33 deletions

diff --git a/MANIFEST.in b/MANIFEST.in
index 831330479..598f21d85 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -52,6 +52,7 @@ recursive-include doc/v2/images *.gif *.png *.jpg *.jpeg *.bmp *.svg *.odg
 recursive-include doc/v2/examples *.yaml *.txt *.py *.jinja2
 recursive-include doc/v2/_templates *.html
 recursive-include etc/dispatcher-config README
+recursive-include share/requirements *.yaml
 recursive-include lava_dispatcher/devices *.yaml
 recursive-include lava_dispatcher/test *.yaml *.txt
 recursive-include lava_results_app/templates *.html
diff --git a/common-requirements.txt b/common-requirements.txt
deleted file mode 100644
index c20441559..000000000
--- a/common-requirements.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-nose
-pyzmq
diff --git a/dispatcher-requirements.txt b/dispatcher-requirements.txt
deleted file mode 100644
index 5929c67df..000000000
--- a/dispatcher-requirements.txt
+++ /dev/null
@@ -1,15 +0,0 @@
-PyYAML
-configobj
-file-magic
-jinja2
-markdown>=2.0.3
-netifaces>=0.10.0
-nose
-pexpect>=4.2
-pyserial>=2.6
-pytz
-pyudev>=0.21
-pyzmq
-requests
-setproctitle>=1.1.8
-guestfs
diff --git a/doc/v2/debian.rst b/doc/v2/debian.rst
index 298b6bfd6..2a24427f1 100644
--- a/doc/v2/debian.rst
+++ b/doc/v2/debian.rst
@@ -408,8 +408,141 @@ Now the old database package can be removed::
 
  $ sudo apt remove postgresql-9.3
 
+.. index:: dependency requirements
+
+.. _dependency_requirements:
+
+Dependency Requirements
+***********************
+
+LAVA needs to control and output the list of dependencies in a variety
+of formats. Building Docker images and running unit tests in an LXC
+need an updated list of binary package names suitable for the
+distribution and suite of the LXC. Each needs to cope with dependencies
+outside the specified suite, e.g. stable releases which need backports.
+Building the LAVA Debian packages themselves also requires a properly
+up to date list of dependencies - including minimum versions. Each set
+of dependencies needs to be specific to each LAVA binary package -
+``lava-server`` has different dependencies to ``lava-dispatcher`` and
+``lava-common``.
+
+LAVA has several dependencies which are not available via PyPi or pip
+and the ``requirements.txt`` file is therefore misleading. However, the
+format of this file is still useful in building the LAVA packages.
+
+Therefore, LAVA has the ``./share/requires.py`` script which can be
+used to output the preferred format, depending on the arguments.
+
+The dependencies **MUST** be installed in the specified suite of the
+specified distribution for LAVA to work, so take care before pushing a
+merge request to add package names to the support. Make sure your merge
+request includes a change to the relevant requirement YAML files for
+**all** supported distributions or the CI will fail.
+
+Some distributions support ``Recommends`` level dependencies. These are
+typically intended to be installed by ~90% of installations but give
+flexibility for other use cases. ``Recommends`` are **not** handled by
+``requires.py`` at all. The packages must be listed explicitly by the
+maintainer of the packaging for the distribution. ``requires.py``
+exists so that automated processes, like CI, can have a reliable but
+minimal set of packages which must be installed for the specified
+package to be installable.
+
+.. note:: extra dependencies to enable unit tests and other CI actions
+   are not covered by this support. However, these tend to change less
+   often than the dependencies of the main source code.
+
+``requires.py`` does not currently support dependencies based on the
+architecture of the installation. (Currently, only ``Recommends``
+includes architecture-sensitive packages.)
+
+Outputting the requirements.txt format
+======================================
+
+Processes which need the version string can use the original output
+format which mimics ``requirements.txt``::
+
+    $ ./share/requires.py --package lava-server --distribution debian --suite stretch
+    django-auth-ldap>=1.2.12
+    PyYAML
+    dateutil
+    django-restricted-resource>=2016.8
+    django-tables2>=1.14.2
+    django>=1.10
+    docutils>=0.6
+    jinja2
+    markdown>=2.0.3
+    nose
+    psycopg2
+    pytz
+    pyzmq
+    requests
+    simplejson
+    voluptuous>=0.8.8
+
+Outputting a list of binary package names
+=========================================
+
+::
+
+    $ ./share/requires.py --package lava-server --distribution debian --suite stretch --names
+    python3-django-auth-ldap
+    python3-yaml
+    python3-dateutil
+    python3-django-restricted-resource
+    python3-django-tables2
+    python3-django
+    python3-docutils
+    python3-jinja2
+    python3-markdown
+    python3-nose
+    python3-psycopg2
+    python3-tz
+    python3-zmq
+    python3-requests
+    python3-simplejson
+    python3-voluptuous
+    apache2
+    adduser
+    gunicorn3
+    iproute2
+    python3-setuptools
+    libjs-excanvas
+    libjs-jquery-cookie
+    libjs-jquery
+    libjs-jquery-ui
+    libjs-jquery-watermark
+    libjs-jquery-flot
+    libjs-jquery-typeahead
+    systemd-sysv
+    postgresql
+    postgresql-client
+    postgresql-common
+    lava-common
+
+Outputting a single line of binary package names
+================================================
+
+This is intended to be passed directly to a package installer like
+``apt-get`` together with the other required commands and options.
+
+The caller determines the ``suite``, so to use with stretch-backports,
+the ``-t stretch-backports`` option would also be added to the
+other ``apt-get`` commands before appending the list of packages.
+
+(Line breaks are added for readability only)::
+
+    $ ./share/requires.py --package lava-server --distribution debian --suite stretch --names --inline
+    lava-common postgresql-common postgresql-client postgresql systemd-sysv libjs-jquery-typeahead libjs-jquery-flot \
+    libjs-jquery-watermark libjs-jquery-ui libjs-jquery libjs-jquery-cookie libjs-excanvas python3-setuptools iproute2 \
+    gunicorn3 adduser apache2 python3-django-auth-ldap python3-yaml python3-dateutil python3-django-restricted-resource \
+    python3-django-tables2 python3-django python3-docutils python3-jinja2 python3-markdown python3-nose python3-psycopg2 \
+    python3-tz python3-zmq python3-requests python3-simplejson python3-voluptuous
+
 .. index:: javascript
 
+.. _javascript_handling:
+
 Javascript handling
 *******************
 
diff --git a/requirements.txt b/requirements.txt
index 8f022e0fc..e0a9666eb 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,16 +1,4 @@
-PyYAML
-dateutil
-django-auth-ldap>=1.2.12
-django-restricted-resource>=2016.8
-django-tables2>=1.14.2
-django>=1.10
-docutils>=0.6
-jinja2
-markdown>=2.0.3
-nose
-psycopg2
-pytz
-pyzmq
-requests
-simplejson
-voluptuous>=0.8.8
+# DO NOT EDIT
+# content is managed via ./share/requires.py
+# Installing LAVA directly from git is NOT supported.
+
diff --git a/share/requirements/debian/buster.yaml b/share/requirements/debian/buster.yaml
new file mode 100644
index 000000000..0fdced8c6
--- /dev/null
+++ b/share/requirements/debian/buster.yaml
@@ -0,0 +1,89 @@
+# names must exist as binary packages in the suite
+# for this distribution at a version which matches the constraint.
+# else move the dependency to a new file for a suite which
+# contains the dependency, e.g. backports.
+
+lava-server:
+  django-auth-ldap:
+      version: ">=1.2.12"
+      name: python3-django-auth-ldap
+  PyYAML:
+      name: python3-yaml
+  dateutil:
+      name: python3-dateutil
+  django-restricted-resource:
+      version: ">=2016.8"
+      name: python3-django-restricted-resource
+  django-tables2:
+      version: ">=1.14.2"
+      name: python3-django-tables2
+  django:
+      version: ">=1.10"
+      name: python3-django
+  docutils:
+      version: ">=0.6"
+      name: python3-docutils
+  jinja2:
+      name: python3-jinja2
+  markdown:
+      version: ">=2.0.3"
+      name: python3-markdown
+  nose:
+      name: python3-nose
+  psycopg2:
+      name: python3-psycopg2
+  pytz:
+      name: python3-tz
+  pyzmq:
+      name: python3-zmq
+  requests:
+      name: python3-requests
+  simplejson:
+      name: python3-simplejson
+  voluptuous:
+      version: ">=0.8.8"
+      name: python3-voluptuous
+
+lava-dispatcher:
+    PyYAML:
+        name: python3-yaml
+    configobj:
+        name: python3-configobj
+    file-magic:
+        name: python3-magic
+    jinja2:
+        name: python3-jinja2
+    markdown:
+        version: ">=2.0.3"
+        name: python3-markdown
+    netifaces:
+        version: ">=0.10.0"
+        name: python3-netifaces
+    nose:
+        name: python3-nose
+    pexpect:
+        version: ">=4.2"
+        name: python3-pexpect
+    pyserial:
+        version: ">=2.6"
+        name: python3-serial
+    pytz:
+        name: python3-tz
+    pyudev:
+        version: ">=0.21"
+        name: python3-pyudev
+    pyzmq:
+        name: python3-zmq
+    requests:
+        name: python3-requests
+    setproctitle:
+        version: ">=1.1.8"
+        name: python3-setproctitle
+    guestfs:
+        name: python3-guestfs
+
+lava-common:
+    nose:
+        name: python3-nose
+    pyzmq:
+        name: python3-zmq
diff --git a/share/requirements/debian/sid.yaml b/share/requirements/debian/sid.yaml
new file mode 100644
index 000000000..0fdced8c6
--- /dev/null
+++ b/share/requirements/debian/sid.yaml
@@ -0,0 +1,89 @@
+# names must exist as binary packages in the suite
+# for this distribution at a version which matches the constraint.
+# else move the dependency to a new file for a suite which
+# contains the dependency, e.g. backports.
+
+lava-server:
+  django-auth-ldap:
+      version: ">=1.2.12"
+      name: python3-django-auth-ldap
+  PyYAML:
+      name: python3-yaml
+  dateutil:
+      name: python3-dateutil
+  django-restricted-resource:
+      version: ">=2016.8"
+      name: python3-django-restricted-resource
+  django-tables2:
+      version: ">=1.14.2"
+      name: python3-django-tables2
+  django:
+      version: ">=1.10"
+      name: python3-django
+  docutils:
+      version: ">=0.6"
+      name: python3-docutils
+  jinja2:
+      name: python3-jinja2
+  markdown:
+      version: ">=2.0.3"
+      name: python3-markdown
+  nose:
+      name: python3-nose
+  psycopg2:
+      name: python3-psycopg2
+  pytz:
+      name: python3-tz
+  pyzmq:
+      name: python3-zmq
+  requests:
+      name: python3-requests
+  simplejson:
+      name: python3-simplejson
+  voluptuous:
+      version: ">=0.8.8"
+      name: python3-voluptuous
+
+lava-dispatcher:
+    PyYAML:
+        name: python3-yaml
+    configobj:
+        name: python3-configobj
+    file-magic:
+        name: python3-magic
+    jinja2:
+        name: python3-jinja2
+    markdown:
+        version: ">=2.0.3"
+        name: python3-markdown
+    netifaces:
+        version: ">=0.10.0"
+        name: python3-netifaces
+    nose:
+        name: python3-nose
+    pexpect:
+        version: ">=4.2"
+        name: python3-pexpect
+    pyserial:
+        version: ">=2.6"
+        name: python3-serial
+    pytz:
+        name: python3-tz
+    pyudev:
+        version: ">=0.21"
+        name: python3-pyudev
+    pyzmq:
+        name: python3-zmq
+    requests:
+        name: python3-requests
+    setproctitle:
+        version: ">=1.1.8"
+        name: python3-setproctitle
+    guestfs:
+        name: python3-guestfs
+
+lava-common:
+    nose:
+        name: python3-nose
+    pyzmq:
+        name: python3-zmq
diff --git a/share/requirements/debian/stretch-backports.yaml b/share/requirements/debian/stretch-backports.yaml
new file mode 100644
index 000000000..4abd16862
--- /dev/null
+++ b/share/requirements/debian/stretch-backports.yaml
@@ -0,0 +1,16 @@
+# names must exist as binary packages in the suite
+# for this distribution at a version which matches the constraint.
+# else move the dependency to a new file for a suite which
+# contains the dependency, e.g. backports.
+
+lava-server:
+#  django:
+#      version: ">=1.11"
+#      name: python3-django
+#  django-allauth:
+#      version:
+#      name: python3-django-allauth
+
+lava-dispatcher:
+
+lava-common:
diff --git a/share/requirements/debian/stretch.yaml b/share/requirements/debian/stretch.yaml
new file mode 100644
index 000000000..1e7741279
--- /dev/null
+++ b/share/requirements/debian/stretch.yaml
@@ -0,0 +1,133 @@
+# names must exist as binary packages in the suite
+# for this distribution at a version which matches the constraint.
+# else move the dependency to a new file for a suite which
+# contains the dependency, e.g. backports.
+
+lava-server:
+  django-auth-ldap:
+      version: ">=1.2.12"
+      name: python3-django-auth-ldap
+  PyYAML:
+      name: python3-yaml
+  dateutil:
+      name: python3-dateutil
+  django-restricted-resource:
+      version: ">=2016.8"
+      name: python3-django-restricted-resource
+  django-tables2:
+      version: ">=1.14.2"
+      name: python3-django-tables2
+  # Remove if 1.11 is listed in stretch-backports
+  django:
+      version: ">=1.10"
+      name: python3-django
+  docutils:
+      version: ">=0.6"
+      name: python3-docutils
+  jinja2:
+      name: python3-jinja2
+  markdown:
+      version: ">=2.0.3"
+      name: python3-markdown
+  nose:
+      name: python3-nose
+  psycopg2:
+      name: python3-psycopg2
+  pytz:
+      name: python3-tz
+  pyzmq:
+      name: python3-zmq
+  requests:
+      name: python3-requests
+  simplejson:
+      name: python3-simplejson
+  voluptuous:
+      version: ">=0.8.8"
+      name: python3-voluptuous
+  apache2:
+      system: True
+  adduser:
+      system: True
+  gunicorn3:
+      system: True
+  iproute2:
+      system: True
+  python3-setuptools:
+      system: True
+  libjs-excanvas:
+      system: True
+  libjs-jquery-cookie:
+      system: True
+  libjs-jquery:
+      system: True
+  libjs-jquery-ui:
+      system: True
+  libjs-jquery-watermark:
+      system: True
+  libjs-jquery-flot:
+      system: True
+  libjs-jquery-typeahead:
+      system: True
+  systemd-sysv:
+      system: True
+  # may be removed when separate postgresql docker support is tested.
+  postgresql:
+      system: True
+  postgresql-client:
+      system: True
+  postgresql-common:
+      system: True
+  lava-common:
+      system: True
+
+lava-dispatcher:
+    PyYAML:
+        name: python3-yaml
+    configobj:
+        name: python3-configobj
+    file-magic:
+        name: python3-magic
+    jinja2:
+        name: python3-jinja2
+    markdown:
+        version: ">=2.0.3"
+        name: python3-markdown
+    netifaces:
+        version: ">=0.10.0"
+        name: python3-netifaces
+    nose:
+        name: python3-nose
+    pexpect:
+        version: ">=4.2"
+        name: python3-pexpect
+    pyserial:
+        version: ">=2.6"
+        name: python3-serial
+    pytz:
+        name: python3-tz
+    pyudev:
+        version: ">=0.21"
+        name: python3-pyudev
+    pyzmq:
+        name: python3-zmq
+    requests:
+        name: python3-requests
+    setproctitle:
+        version: ">=1.1.8"
+        name: python3-setproctitle
+    guestfs:
+        name: python3-guestfs
+    python3-setuptools:
+        system: True
+    systemd-sysv:
+        system: True
+    file:
+        system: True
+    lava-common:
+        system: True
+
+lava-common:
+    nose:
+        name: python3-nose
+    pyzmq:
+        name: python3-zmq
diff --git a/share/requires.py b/share/requires.py
new file mode 100755
index 000000000..68c8f72b8
--- /dev/null
+++ b/share/requires.py
@@ -0,0 +1,112 @@
+#!/usr/bin/env python3
+# -*- coding: utf-8 -*-
+#
+#  requires.py
+#
+#  Copyright 2018 Neil Williams <neil.williams@linaro.org>
+#
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation; either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import sys
+import yaml
+import argparse
+
+
+"""
+Goals:
+0: Convert a packaged list of dependencies and versions
+   to a distro-specific list of dependencies.
+1: Remove need for requirements.txt using Pip syntax as this is
+   misleading.
+2: output a list of binary package names for the requested distribution
+   and suite to pass to docker scripts and LXC unit test jobs.
+"""
+
+
+def main():
+    """
+    Parse options and load requirements files.
+    By default, outputs the same list as requirements.txt but
+    with the Pip name replaced by the distro|suite package name.
+    Use the -n option to only get the distro|suite package names.
+    """
+    parser = argparse.ArgumentParser(description="Handle dependency lists")
+    parser.add_argument(
+        "-p", "--package", required=True, help="Name of the LAVA package."
+    )
+    parser.add_argument(
+        "-d", "--distribution", required=True, help="Only Debian is supported currently"
+    )
+    parser.add_argument(
+        "-s", "--suite", required=True, help="The distribution suite / release"
+    )
+    parser.add_argument(
+        "-i",
+        "--inline",
+        action="store_true",
+        help="Print in a single line separated by whitespace. Requires --names",
+    )
+    parser.add_argument(
+        "-n",
+        "--names",
+        action="store_true",
+        help="Only output distribution package names, not versions",
+    )
+    args = parser.parse_args()
+    req = os.path.join(
+        os.path.dirname(__file__),
+        "requirements",
+        args.distribution,
+        "%s.yaml" % args.suite,
+    )
+    if not os.path.exists(req):
+        sys.stderr.write(
+            "Unsupported suite|distribution: %s %s\n\n"
+            % (args.distribution, args.suite)
+        )
+        return 1
+    with open(req, "r") as data:
+        depends = yaml.safe_load(data)
+    if args.package not in depends:
+        sys.stderr.write("Unknown package: %s\n\n" % args.package)
+        return 2
+    if args.names:
+        msg = []
+        for key, item in depends[args.package].items():
+            if depends[args.package][key].get("system"):
+                if args.inline:
+                    msg.insert(0, key)
+                else:
+                    print(key)
+                continue
+            if args.inline:
+                msg.append(item["name"])
+            else:
+                print(item["name"])
+        if args.inline:
+            print(" ".join(msg))
+        return 0
+    if not depends[args.package]:
+        return 0
+    for item in depends[args.package].keys():
+        if depends[args.package][item].get("system"):
+            continue
+        print("%s%s" % (item, depends[args.package][item].get("version", "")))
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())