Include command line tests in documentation

docs/testing_instructions.md

@@ -13,6 +13,11 @@ descibe how to give feedback via our issue tracker at the [end of these instruct
 First we'd like you to setup an OpenAppStack cluster by yourself, following the
 [installation tutorial](https://docs.openappstack.net/en/latest/installation_instructions.html).
 
+## Command line tests
+
+Please run the [command line tests](troubleshooting.md) which checks the overall
+functionality of your cluster and include the output in your feedback.
+
 ## Nextcloud
 
 ### Logging into Nextcloud

docs/troubleshooting.md

@@ -2,6 +2,90 @@
 
 Note: `cluster$` indicates that the commands should be run as root on your OAS cluster.
 
+## Run the cli tests
+
+To get an overall status of your cluster you can run the tests from the
+command line.
+
+There are two types of tests: [testinfra](https://testinfra.readthedocs.io/en/latest/)
+tests, and [behave](https://behave.readthedocs.io/en/latest/) tests.
+
+### Testinfra tests
+
+Testinfra tests are split into two groups, lets call them `blackbox` and
+`whitebox` tests.  The blackbox tests run on your provisioning machine and test
+the OAS cluster from the outside. For example, the certificate check will check
+if the OAS will return valid certificates for the provided services.
+The whitebox tests run on the OAS host and check i.e. if docker is installed
+in the right version etc.
+
+To run the test against your cluster, first export the `CLUSTER_DIR` environment
+variabel with the location of your cluster config directory:
+
+    export CLUSTER_DIR="../clusters/CLUSTERNAME"
+
+Run all tests:
+
+    py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'
+
+#### Advance usage
+
+Specify host manually:
+
+    py.test -s --hosts='ssh://root@example.openappstack.net'
+
+Run only tests tagged with `prometheus`:
+
+    py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m prometheus
+
+Run cert test manually using the ansible inventory file:
+
+    py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m certs
+
+Run cert test manually against a different cluster, not configured in any
+ansible inventory file, either by using pytest:
+
+    FQDN='example.openappstack.net' py.test -sv -m 'certs'
+
+or directly:
+
+    FQDN='example.openappstack.net' pytest/test_certs.py
+
+#### Known Issues
+
+- Default ssh backend for testinfra tests is `paramiko`, which doesn't work oout
+  of the box. It fails to connect to the host because the `ed25519` hostkey was
+  not verified. Therefore we need to force plain ssh:// with either
+  `connection=ssh` or `--hosts=ssh://…`
+
+#### Running tests with local gitlab-runner docker executor
+
+Export the following environment variables like this:
+
+    export CI_REGISTRY_IMAGE='open.greenhost.net:4567/openappstack/openappstack'
+    export SSH_PRIVATE_KEY="$(cat ~/.ssh/id_ed25519_oas_ci)"
+    export COSMOS_API_TOKEN='…'
+
+then:
+
+    gitlab-runner exec docker --env CI_REGISTRY_IMAGE="$CI_REGISTRY_IMAGE" --env SSH_PRIVATE_KEY="$SSH_PRIVATE_KEY" --env COSMOS_API_TOKEN="$COSMOS_API_TOKEN" bootstrap
+
+
+## Behave tests
+
+Behave tests run in a headless browser and test if all the interfaces are up
+and running and correctly connected to each other. They are integrated in the
+`openappstack` CLI command suite.
+To run the behave tests, run the following command in this repository:
+
+    python -m openappstack CLUSTERNAME test
+
+In the future, this command will run all tests, but now only *behave* is
+implemented. To learn more about the `test` subcommand, run:
+
+    python -m openappstack CLUSTERNAME test --help
+
+
 ## Upgrading
 
 If you encounter problems when you upgrade your cluster, please make sure first

test/README.md

-# Tests
-
-There are two types of tests: "testinfra" tests, and "behave" tests.
-
-- Testinfra tests are split into two groups, lets call them blackbox- and
-  whitebox tests.  The blackbox tests run on your laptop and test the OAS
-  cluster from the outside. For example, the certificate check will check if the
-  OAS will return valid certificates for the provided services.  The whitebox
-  tests run on the OAS host and check i.e. if docker is installed in the right
-  version etc.
-- Behave tests run in a browser and test if all the interfaces are up and
-  running and correctly connected to each other.
-
-## Run *testinfra* tests
-
-Export `CLUSTER_DIR` env var with the location of your cluster config directory:
-
-    export CLUSTER_DIR="../clusters/CLUSTERNAME"
-
-Run all tests:
-
-    py.test -sv --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'
-
-Specify host manually:
-
-    py.test -sv --hosts='ssh://root@example.openappstack.net'
-
-Run only tests tagged with `prometheus`:
-
-    py.test -sv --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m prometheus
-
-### Cert tests
-
-Run cert test manually using the ansible inventory file:
-
-    py.test -sv --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m certs
-
-Run cert test manually against a different cluster, not configured in any
-ansible inventory file, either by using pytest:
-
-    FQDN='example.openappstack.net' py.test -sv -m 'certs'
-
-or directly:
-
-    FQDN='example.openappstack.net' pytest/test_certs.py
-
-
-## Run *behave* tests
-
-The behave tests are controlled by the `openappstack` CLI command suite. To run
-the behave tests, run the following command in this repository:
-
-    python -m openappstack CLUSTERNAME test
-
-In the future, this command will run all tests, but now only *behave* is
-implemented. To learn more about the `test` subcommand, run:
-
-    python -m openappstack CLUSTERNAME test --help
-
-## Known Issues
-
-- Default ssh backend for testinfra tests is `paramiko`, which doesn't work oout
-  of the box. It fails to connect to the host because the `ed25519` hostkey was
-  not verified. Therefore we need to force plain ssh:// with either
-  `connection=ssh` or `--hosts=ssh://…`
-
-# Running CI with local gitlab-runner docker executor
-
-Export the following environment variables like this:
-
-    export CI_REGISTRY_IMAGE='open.greenhost.net:4567/openappstack/openappstack'
-    export SSH_PRIVATE_KEY="$(cat ~/.ssh/id_ed25519_oas_ci)"
-    export COSMOS_API_TOKEN='…'
-
-then:
-
-    gitlab-runner exec docker --env CI_REGISTRY_IMAGE="$CI_REGISTRY_IMAGE" --env SSH_PRIVATE_KEY="$SSH_PRIVATE_KEY" --env COSMOS_API_TOKEN="$COSMOS_API_TOKEN" bootstrap
+Please refer to `../docs/troubleshooting.md` for running the tests.