design.md

Stackspin Design
===================

This article covers the basic design of Stackspin.

## Application build pipeline

The following diagram explains the process to go from an application's source
code to a deployment on Stackspin.

![Application build process](_static/diagrams/application-build-process.png)

These are the steps in more detail:

- Build container (this process should be maintained by application developer by
  providing a Dockerfile with the application)
   1. Get application package (source code, installation package, etc.)
      1. If not part of the package: get default configuration for the
         application
   1. Build container with application package installed
      1. Install application dependencies
      1. Install application package
      1. Setup default configuration
      1. Setup pluggable configuration override, can be:
         - Reading environment variables
         - Extra configuration file mounted into the container elsewhere
- Helm chart
   - Deployment configuration to specify:
     - The container(s) that should be deployed.
     - The port(s) that they expose.
     - Volume mounts for configuration files and secrets.
     - Live/readyness probes
     - Persistent storage locations and methods
     - A lot of other things
   - Service configuration to specify:
     - Ports exposed to the user of the application
   - Ingress configuration to specify:
     - How to proxy to the application (which hostname or URL)
     - Some authentication plugins (http auth, for example)
   - Custom files:
     - Add file templates for mountable application configuration files
     - Files that specify integrations with other services
- Deploy
   1. Create `values.yaml` file with the variables for the Helm deployment to
      the Kubernetes cluster
   1. "Manually" add secrets to the Kubernetes cluster.
   1. Run `helm install` to install the customised application.


## Configuration

As can be seen in the images, applications are expected to have two different
types of configuration. Containers should provide a default configuration,
that at least configures things like the port the application runs on, the
locations for log files, etc.

What we call the *external configuration* is provided by the user. This includes
overrides of the default application, as well as variables like the hostname
that the application will run on and listen to and the title of the web
interface.

Stackspin will use Helm charts to provide the external configuration for the
"Deploy" step. Helm charts can contain configuration file templates with
default values that can be overridden during the installation or upgrade of a
helm chart.

## Application containers

For inclusion in Stackspin, it is required that the application developers
provide Docker containers for their applications. There are several reasons for
this:

- If application developers do not provide a container, chances are they also do
  not think about how their application would update itself after a new
  container is deployed. This can lead to problems with things like database
  migrations.
- Maintaining the containerisation for an application can, in most cases, not be
  fully automated.

### Container updates

When an application update is available, these updates need to be rolled out to
Stackspin instances. This will be done according the following steps:

1. Application container is built with new application source and tagged for
   testing.
4. Helm chart for application is updated to provide new container.
3. Helm chart is deployed to an Stackspin test cluster following the steps in
   the diagram above.
2. Application is tested with automated tests
3. If tests succeed, new container is tagged for release.
4. Stackspin automated update job fetches new Helm chart and upgrades current
   instance using Helm.

Most of these steps can be developed by configuring a CI system and configuring
Kubernetes and Helm correctly. The automated update job that will run on
Stackspin clusters will be developed by us.

## Persistent data

Containerised applications are normally "stateless" (meaning no data is saved
inside the containers). However, it is possible to mount persistent volumes to
specific directories in the container, basically adding a persistent layer on
top of the containerised application. To provide this in Stackspin's simple setup, we
use a [local storage
provisioner](https://open.greenhost.net/stackspin/local-path-provisioner)
that automatically provides persistent data on the VPS running Stackspin to an
application that requests it.

## Automatic updates

Stackspin has an auto-update mechanism that performs unattended upgrades to
applications. [Flux 2](https://fluxcd.io/) is the system running in the cluster
that is responsible for these updates.

Flux 2 tracks all the files in the `flux2` directory of the [Stackspin code
repository](https://open.greenhost.net/stackspin/stackspin). Once changes
are pushd the branch that Flux tracks, the changes are applied to the cluster.

We use Flux 2 in "read only" mode, which means that your Stackspin cluster
does not push changes to our Git repository. You can read more about Flux 2 and
its components in the [flux 2 documentation](https://fluxcd.io/docs).

## APIs

Stackspin includes several APIs that are available for some pieces of the
software to talk to. Specifically the [Dashboard
application](https://open.greenhost.net/stackspin/admin-frontend) uses most
of these APIs.

<!-- Edit URL:
https://viewer.diagrams.net/?tags=%7B%7D&highlight=0000ff&edit=_blank&layers=1&nav=1#R5VnbcqM4EP0aPybFxbc8%2BjKZuFLZpOKa3X1VQMbaEYiVhC%2Fz9dsCcTHIGGds19TOSwIH0S26T5%2BW5J47C3dfOYrXL8zHtOdY%2Fq7nznuOY9uDMfxTyD5DRiMNBJz4elAJLMkPrEFLownxsTgYKBmjksSHoMeiCHvyAEOcs%2B3hsBWjh15jFOAGsPQQbaJ%2FEV%2BuM3Q8sEr8CZNgnXu2Lf0kRPlgDYg18tm2Arlfeu6MMyazq3A3w1QFL49L9t7jkafFxDiOZJcX4m9%2F%2FL183VijPh5L6%2BXbzvvn6U5b2SCa6A9%2B5XsAnjmSTMDF5G2hpy%2F3eUzEloQURXA3bU4it4i5xLsKpCf1FbMQy9SHfjrU8dEEcfPMb8twO3lQ15VQFyDSKQ4K04W3d6AEigKYeOHOPe3uweTNPXSGqMQ8QhJPWRL5ohp5uKh8Zwml%2BTgjN%2F1GbnrOkMJ8pj7ZqKlQEkTpg%2BG%2FiWLRlOKVLO%2FgKtD%2F07cSWkcoyZE3VQZQpVh4nMSSsAi%2Bc5KPg%2BmXQytg02IXpGkq%2FZ5DSMQoyjHKAhJV3FafVWCDFePEW%2F0vUpHgAUQQanifkluILeP%2BpT29PE4ubfL%2B%2Fh4mjKV3%2FwnLxny2unvjbEXS2gIEhTFA2V9FTs%2BDwkiDiCIgV6gE4sKfOyl8cOwxUJy9iT2XI44xQvX3a1oJIqjwtQyh9ue2CpEu27RY3akSSgL9ZqLhkPi%2BennKsSA%2F0EdqyIL7mJFIpuozmPYGc2UpAY3OOqYyLCRn3%2FGMUcYBiVgq0JAgWoM4k0hVuLZ7AQnv1zS1P2hKuGMQVbvfIuEHcnqudg6P9LWnvc%2FR6baW58iDmGB%2BmSA5o9OdxzZF6ZONrou%2F0a%2FR6UZHO12bIr0u5jOVFM42sEbk55Ru5yItCHD7Mm1SrpXpnXl459R4YaCha%2BDF4Fq1Om5k%2Fzn5AMphiTusQK9Rqn3rdOn0%2B4YYuW2CdrxUu%2FgzS8Pta%2FWhka0%2FMcWc%2FWSOWpnROXF2LY7NPmQPTGEcD67E7XxCN1nET5H3%2FS6BjfEFlu3%2Fk7VOR6I9nEu0uojaBqbZtxRRu7mTf6TJ7teoyvpKxLA8NK5ErleWzpFoWc%2BJkCwEMqYr5CtEzz67G50WteIM5SB61%2BOaeyx6T5iG79AOkMBXiZ3zk7EbNfuqY2oI9vhqsRvcsCFM4piC3mZUthaRkIhSfdtp0%2F2OkVpFWDMWrUjwglRvUccjap5x3EuPXeFBws8xusRSmUnitNwqK70QRWSFhcydnDiAavvyRRRAT2nvhCUyxzFl%2B%2ByIwlL5dawl7JHxKqECy45G3pjf1d8S8w3xcNfhtQOd7gcTv1EHLzTp0y3cMWiDqSvZw6tpw%2FET3zauL5MPIYlMsiISMBlVmBQaWbkp%2FuDGc7TOanMHUcHp1DhRWRbHCPVb7sHP70sdyGebfob4BPfgtvyRKdsFlj%2FVuV%2F%2BAw%3D%3D -->
![API graph](_static/diagrams/apis.svg)

- [Ory Kratos](https://www.ory.sh/kratos/docs/quickstart/) is the user
  management API that we use. It comes with several handy features, like sending
  password reset emails and (soon) MFA.
- [Ory Hydra](https://www.ory.sh/hydra/docs/) provides us with an OpenID Connect
  workflow.
- [Velero](https://velero.io/docs/v1.7/) is a backup mechanism that integrates
  well with Kubernetes: it includes "Custom Resource Definitions" for
  backups and backup locations. As such, we can access it through the Kubernetes
  API.
- [Flux 2](https://fluxcd.io/docs/) is used to configure and auto-update
  applications. App configuration happens mostly through Kubernetes `ConfigMaps`
  and `Secrets` and are applied through the Flux CRDs
  [`Kustomization`](https://fluxcd.io/docs/components/kustomize/kustomization/)
  and [`HelmRelease`](https://fluxcd.io/docs/components/helm/helmreleases/).