From d23558902d18a3f59516319c10dc625d0fad35d1 Mon Sep 17 00:00:00 2001
From: Varac <varac@varac.net>
Date: Thu, 19 Dec 2019 14:44:43 +0100
Subject: [PATCH] Include command line tests in documentation
---
docs/testing_instructions.md | 5 +++
docs/troubleshooting.md | 84 ++++++++++++++++++++++++++++++++++++
test/README.md | 78 +--------------------------------
3 files changed, 90 insertions(+), 77 deletions(-)
diff --git a/docs/testing_instructions.md b/docs/testing_instructions.md
index ceca5b79a..b6bedbe8b 100644
--- a/docs/testing_instructions.md
+++ b/docs/testing_instructions.md
@@ -13,6 +13,11 @@ descibe how to give feedback via our issue tracker at the [end of these instruct
First we'd like you to setup an OpenAppStack cluster by yourself, following the
[installation tutorial](https://docs.openappstack.net/en/latest/installation_instructions.html).
+## Command line tests
+
+Please run the [command line tests](troubleshooting.md) which checks the overall
+functionality of your cluster and include the output in your feedback.
+
## Nextcloud
### Logging into Nextcloud
diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md
index 1dc5d425f..6716b44b8 100644
--- a/docs/troubleshooting.md
+++ b/docs/troubleshooting.md
@@ -2,6 +2,90 @@
Note: `cluster$` indicates that the commands should be run as root on your OAS cluster.
+## Run the cli tests
+
+To get an overall status of your cluster you can run the tests from the
+command line.
+
+There are two types of tests: [testinfra](https://testinfra.readthedocs.io/en/latest/)
+tests, and [behave](https://behave.readthedocs.io/en/latest/) tests.
+
+### Testinfra tests
+
+Testinfra tests are split into two groups, lets call them `blackbox` and
+`whitebox` tests. The blackbox tests run on your provisioning machine and test
+the OAS cluster from the outside. For example, the certificate check will check
+if the OAS will return valid certificates for the provided services.
+The whitebox tests run on the OAS host and check i.e. if docker is installed
+in the right version etc.
+
+To run the test against your cluster, first export the `CLUSTER_DIR` environment
+variabel with the location of your cluster config directory:
+
+ export CLUSTER_DIR="../clusters/CLUSTERNAME"
+
+Run all tests:
+
+ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'
+
+#### Advance usage
+
+Specify host manually:
+
+ py.test -s --hosts='ssh://root@example.openappstack.net'
+
+Run only tests tagged with `prometheus`:
+
+ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m prometheus
+
+Run cert test manually using the ansible inventory file:
+
+ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m certs
+
+Run cert test manually against a different cluster, not configured in any
+ansible inventory file, either by using pytest:
+
+ FQDN='example.openappstack.net' py.test -sv -m 'certs'
+
+or directly:
+
+ FQDN='example.openappstack.net' pytest/test_certs.py
+
+#### Known Issues
+
+- Default ssh backend for testinfra tests is `paramiko`, which doesn't work oout
+ of the box. It fails to connect to the host because the `ed25519` hostkey was
+ not verified. Therefore we need to force plain ssh:// with either
+ `connection=ssh` or `--hosts=ssh://…`
+
+#### Running tests with local gitlab-runner docker executor
+
+Export the following environment variables like this:
+
+ export CI_REGISTRY_IMAGE='open.greenhost.net:4567/openappstack/openappstack'
+ export SSH_PRIVATE_KEY="$(cat ~/.ssh/id_ed25519_oas_ci)"
+ export COSMOS_API_TOKEN='…'
+
+then:
+
+ gitlab-runner exec docker --env CI_REGISTRY_IMAGE="$CI_REGISTRY_IMAGE" --env SSH_PRIVATE_KEY="$SSH_PRIVATE_KEY" --env COSMOS_API_TOKEN="$COSMOS_API_TOKEN" bootstrap
+
+
+## Behave tests
+
+Behave tests run in a headless browser and test if all the interfaces are up
+and running and correctly connected to each other. They are integrated in the
+`openappstack` CLI command suite.
+To run the behave tests, run the following command in this repository:
+
+ python -m openappstack CLUSTERNAME test
+
+In the future, this command will run all tests, but now only *behave* is
+implemented. To learn more about the `test` subcommand, run:
+
+ python -m openappstack CLUSTERNAME test --help
+
+
## Upgrading
If you encounter problems when you upgrade your cluster, please make sure first
diff --git a/test/README.md b/test/README.md
index b91e561c2..bcb6c07b3 100644
--- a/test/README.md
+++ b/test/README.md
@@ -1,77 +1 @@
-# Tests
-
-There are two types of tests: "testinfra" tests, and "behave" tests.
-
-- Testinfra tests are split into two groups, lets call them blackbox- and
- whitebox tests. The blackbox tests run on your laptop and test the OAS
- cluster from the outside. For example, the certificate check will check if the
- OAS will return valid certificates for the provided services. The whitebox
- tests run on the OAS host and check i.e. if docker is installed in the right
- version etc.
-- Behave tests run in a browser and test if all the interfaces are up and
- running and correctly connected to each other.
-
-## Run *testinfra* tests
-
-Export `CLUSTER_DIR` env var with the location of your cluster config directory:
-
- export CLUSTER_DIR="../clusters/CLUSTERNAME"
-
-Run all tests:
-
- py.test -sv --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'
-
-Specify host manually:
-
- py.test -sv --hosts='ssh://root@example.openappstack.net'
-
-Run only tests tagged with `prometheus`:
-
- py.test -sv --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m prometheus
-
-### Cert tests
-
-Run cert test manually using the ansible inventory file:
-
- py.test -sv --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m certs
-
-Run cert test manually against a different cluster, not configured in any
-ansible inventory file, either by using pytest:
-
- FQDN='example.openappstack.net' py.test -sv -m 'certs'
-
-or directly:
-
- FQDN='example.openappstack.net' pytest/test_certs.py
-
-
-## Run *behave* tests
-
-The behave tests are controlled by the `openappstack` CLI command suite. To run
-the behave tests, run the following command in this repository:
-
- python -m openappstack CLUSTERNAME test
-
-In the future, this command will run all tests, but now only *behave* is
-implemented. To learn more about the `test` subcommand, run:
-
- python -m openappstack CLUSTERNAME test --help
-
-## Known Issues
-
-- Default ssh backend for testinfra tests is `paramiko`, which doesn't work oout
- of the box. It fails to connect to the host because the `ed25519` hostkey was
- not verified. Therefore we need to force plain ssh:// with either
- `connection=ssh` or `--hosts=ssh://…`
-
-# Running CI with local gitlab-runner docker executor
-
-Export the following environment variables like this:
-
- export CI_REGISTRY_IMAGE='open.greenhost.net:4567/openappstack/openappstack'
- export SSH_PRIVATE_KEY="$(cat ~/.ssh/id_ed25519_oas_ci)"
- export COSMOS_API_TOKEN='…'
-
-then:
-
- gitlab-runner exec docker --env CI_REGISTRY_IMAGE="$CI_REGISTRY_IMAGE" --env SSH_PRIVATE_KEY="$SSH_PRIVATE_KEY" --env COSMOS_API_TOKEN="$COSMOS_API_TOKEN" bootstrap
+Please refer to `../docs/troubleshooting.md` for running the tests.
--
GitLab