Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • xeruf/dashboard
  • stackspin/dashboard
2 results
Show changes
Commits on Source (588)
Hide whitespace changes
Inline Side-by-side
Showing
with 854 additions and 250 deletions
.gitignore
View file @ 9d43b6f4
......@@ -2,6 +2,10 @@
# dependencies
/node_modules
/frontend/node_modules
/backend/venv
/backend/web/node_modules
/backend/venv
/.pnp
.pnp.js
......@@ -13,6 +17,7 @@
# local environment
/frontend/local.env
/.vscode
# misc
.DS_Store
......@@ -25,6 +30,9 @@ yarn-error.log*
.eslintcache
cypress/videos/
# KUBECONFIG values
backend/kubeconfig/*
# Helm dependencies
deployment/helmchart/charts/
......
.gitlab-ci.yml
View file @ 9d43b6f4
......@@ -4,18 +4,18 @@ include:
stages:
- build-project
- build-container
- build-image
- lint-helm-chart
- package-helm-chart
- release-helm-chart
image: node:18-alpine
image: node:20-alpine
variables:
CHART_NAME: stackspin-dashboard
CHART_DIR: deployment/helmchart/
build-project:
yarn:
stage: build-project
before_script: []
script:
......@@ -25,13 +25,14 @@ build-project:
- echo "REACT_APP_API_URL=/api/v1" > .env
- echo "EXTEND_ESLINT=true" >> .env
- yarn build
- mv build web-build
- mkdir docker
- mv build docker/html
- echo "Build successful"
artifacts:
expire_in: 1 hour
name: web-build
paths:
- frontend/web-build
- frontend/docker/html
.kaniko-build:
script:
......@@ -40,8 +41,10 @@ build-project:
- export CONTAINER_TAG=${CI_COMMIT_TAG:-${CI_COMMIT_REF_SLUG}}
- /kaniko/executor --cache=true --context ${CI_PROJECT_DIR}/${DIRECTORY} --destination ${CI_REGISTRY_IMAGE}/${KANIKO_BUILD_IMAGENAME}:${CONTAINER_TAG}
build-frontend-container:
stage: build-container
build-frontend-image:
stage: build-image
needs:
- yarn
image:
# We need a shell to provide the registry credentials, so we need to use the
# kaniko debug image (https://github.com/GoogleContainerTools/kaniko#debug-image)
......@@ -49,15 +52,16 @@ build-frontend-container:
entrypoint: [""]
variables:
KANIKO_BUILD_IMAGENAME: dashboard
DIRECTORY: frontend/web-build
DIRECTORY: frontend/docker
before_script:
- cp deployment/Dockerfile $DIRECTORY
- cp deployment/nginx.conf $DIRECTORY
extends:
.kaniko-build
build-backend-container:
stage: build-container
build-backend-image:
stage: build-image
needs: []
variables:
KANIKO_BUILD_IMAGENAME: dashboard-backend
DIRECTORY: backend
......@@ -68,3 +72,17 @@ build-backend-container:
entrypoint: [""]
extends:
.kaniko-build
build-test-image:
stage: build-image
needs: []
variables:
KANIKO_BUILD_IMAGENAME: cypress-test
DIRECTORY: tests
image:
# We need a shell to provide the registry credentials, so we need to use the
# kaniko debug image (https://github.com/GoogleContainerTools/kaniko#debug-image)
name: gcr.io/kaniko-project/executor:debug
entrypoint: [""]
extends:
.kaniko-build
.vscode/settings.json 0 → 100644
View file @ 9d43b6f4
{}
CHANGELOG.md
View file @ 9d43b6f4
# Changelog
## [0.6.0]
## 0.13.3
- Fix creating app roles for users created via the CLI.
## 0.13.2
- Update hydra client library to v2 and adapt to its changed API.
- Change the jwt identity claim because that's strictly required to be a string
now, and we put a json object in there before.
## 0.13.1
- Add the `cryptography` python library as dependency of the backend. This is
necessary for sha256_password and caching_sha2_password auth methods.
## 0.13.0
- Fix the initial recovery flow created automatically for new users, which was
broken by the kratos client lib upgrade.
- Add support for serving arbitrary files from the frontend pod, provided by a
persistent volume claim. Reorganize the assets to make it easier to include
custom icons this way.
- Add support for theming. Customizing colours, logo and background image is
now particularly easy, but mostly anything can be changed through a custom
stylesheet.
- Only show Stackspin version info to admin users.
## 0.12.4
- Prevent database connections from being shared between worker processes.
That sharing may cause intermittent database errors.
## 0.12.3
- Fix broken kratos hooks tracking last recovery and login times.
- Upgrade python to 3.13.
## 0.12.2
- Fix consent page for `ENFORCE_2FA` instances.
## 0.12.1
- Add back `jinja2-base64-filters` to backend for templating kustomizations
during app installation.
## 0.12.0
- Add basic support for WebAuthn as second-factor authentication.
- Only show app version numbers in the dashboard tiles to admin users.
- Do not serve Dockerfile and nginx config from frontend.
- Start renovating python dependencies of the backend. Upgrade many direct and
indirect dependencies.
## 0.11.1
- Fix password reset form in case no email address is pre-filled.
## 0.11.0
- Allow pre-filling user's email address in a link to the password (re)set
form. This is useful when creating new user accounts.
- Fix user provisioning after installing new apps.
## 0.10.5
- Look up users from Kratos by email address using the proper (new) API
mechanism for that, instead of iterating over all users.
- Compare email addresses case insensitively to deal with Stackspin apps
changing case of email address strings.
- Fix broken user accounts when created via the flask CLI.
- Replace slightly off-spec usage of `__repr__` by `__str__`.
## 0.10.4
- Disable Zulip accounts when deleting users, because Zulip doesn't allow
deleting accounts via SCIM.
## 0.10.3
- Fix setting successful provisioning status.
## 0.10.2
- Fine-tune logging levels, and introduce a new environment variable
`LOG_LEVEL` to set the log level at runtime.
- Track when a user's full name has been changed, and only include the name in
the SCIM provisioning call when it has changed, or for newly provisioned
users.
## 0.10.1
- Watch dashboard configmaps with lists of apps and oauthclients, and
reload config on changes. This also makes sure that we always load the config
at dashboard start-up, even when there are no (SCIM-supporting) apps
installed.
## 0.10.0
- Include new "System resources" module with basic stats.
- Implement basic (manual/static) SCIM functionality for automatic user provisioning.
- Implement dynamic (i.e., arbitrary apps) SCIM functionality, tested and
tailored for Nextcloud and Zulip.
- Upgrade to tailwind v3, and update several other javascript dependencies.
- Make info modals slightly wider, to make sure you can see the full contents
also for slightly larger fonts. In particular, this fixes a partially
invisible reset link.
- Add a CLI command for deleting older unused accounts.
- Add logo for Gitea.
## 0.9.2
- Fix saving user properties, which was broken because of the partial tags
implementation.
## 0.9.1
- Do not autocomplete totp input field.
- Allow removing user app roles from CLI.
## 0.9.0
- Improve user listing: show label for admin users, show last login and
password reset times, improved layout.
- Fix rare bug in frontend's idea of admin status in the face of custom apps.
- Prepare backend for user tags.
## 0.8.4
- Allow enforcing 2fa.
- Add button for admin users to reset 2FA of users. Also improve UX of this and
other dangerous operations in the user edit screen.
- Fix logout to include hydra post-logout.
- Do not show link to recovery on TOTP form.
- Fix css of demo sign-up.
- Upgrade to python 3.12.
## 0.8.3
- Introduce backend code for resetting 2FA, and add cli command for that.
- Upgrade Kratos api library `ory-kratos-client` to 1.0.0.
- Patch our usage of Kratos api pagination of identities list.
## 0.8.2
- End the Kratos session in prelogout. This makes sure that we end the "SSO
session" also when logging out from an app. We used to rely on hydra's
post-logout url to get at the kratos logout, but apps sometimes override that
url via an oidc parameter.
## 0.8.1
- Add a couple of attributes to our OIDC tokens to support our switch to
another Nextcloud app for OIDC.
## 0.8.0
- Add feature to easily edit app permissions for multiple users at once.
- Change the way secrets are created for apps, creating them in the stackspin
project (using an existing secrets controller). So remove support for
generating app secrets in the dashboard.
- Fix password reset when 2FA is enabled.
- Fix bug that all Wekan users get admin permissions in Wekan regardless of
role set in Stackspin.
- Enable "pre-ping" for all database connections managed by sqlalchemy in the
dashboard backend, hoping to eliminate or reduce dropped database
connections.
- Fix listing of Velero in app permissions when batch-creating users.
## 0.7.6
- Add Forgejo metadata for use as custom app.
## 0.7.5
- Add Jitsi and Mattermost metadata for use as custom apps.
## 0.7.4
- Make the sign-in UI less wide.
## 0.7.3
Only changes to the helm chart.
## 0.7.2
- Apply Stackspin styling to the login component. This covers the login pages,
recovery page, and profile/authentication settings.
## 0.7.1
- Load the flask_migrate flask extension in dev/cli mode so we may run `flask
db` commands from the cli again.
## 0.7.0
- Improve the UX of the dashboard tiles: adding help texts in modals, add a
status dropdown with version info, add alerts before and after automatic
upgrades, show greeting, show tag when logged in as admin user.
- Make sure we run the initialisation code in the backend only once per run,
both in development and production mode. Also, do not run the init code on
flask cli runs.
- Remember the active tab in the authentication settings when saving.
- No longer send emails to addresses that do not match an existing account.
This was fixed by upgrading Kratos; we're happy to see that the default
Kratos behaviour was changed in this regard.
## 0.6.7
Only changes to the helm chart.
## 0.6.6
Only changes to the helm chart.
## 0.6.5
- Further improve (error) message handling. In particular, show feedback when
saving profile settings. Some of the previous error message changes have been
reverted pending further consideration of the design.
- Disable changing the email address as this is not supported right now.
## 0.6.4
- Fix error messages that were not shown, in particular when providing wrong
credentials when logging in. We redesigned the error handling, considering
that these messages may be translated later on.
## 0.6.3
- Add support for Hedgedoc.
- Add a button for admins for creating a recovery link for a user.
- Automatically log in to dashboard if already authenticated.
- Redirect to dashboard if not redirect login is set, on successful login.
- Fix deletion of apps via the CLI.
- Add special features (sign-up form) for the Stackspin demo instance.
- Show the user UUID in user modal.
- Only show installed apps when configuring roles.
## 0.6.2
- Fix submit button label in the form for verifying your TOTP code.
## 0.6.1
- Add TOTP as second factor authentication. Please note that you'll need to set
a `backend.dashboardUrl` value instead of the old `backend.loginPanelUrl` one
-- typically dropping the `/web` suffix to get the new value.
- Create a new backend endpoint for providing some environment variables to the
frontend, with the URLs of the Kratos and Hydra APIs.
## 0.6.0
- Make it easier to add apps, by reading apps and oauthclients from configmaps
at startup.
- Reset alembic migration history.
## [0.5.2]
## 0.5.2
- Fix login welcome message
- Clarify "set new password" button (#94)
......@@ -14,11 +266,11 @@
entered (#96)
- Fix access checking for monitoring (#105)
## [0.5.1]
## 0.5.1
- Fix bug of missing "Monitoring" app access when creating a new user.
- Add Velero to the list of installable apps, but hide it from the dashboard
## [0.5.0]
## 0.5.0
- Merge dashboard-backend repository into this repository, released as 0.5.0
README.md
View file @ 9d43b6f4
......@@ -17,25 +17,25 @@ identity provider, login, consent and logout endpoints for the OpenID Connect
The application relies on the following components:
- **Hydra**: Hydra is an open source OIDC server.
It means applications can connect to Hydra to start a session with a user.
Hydra provides the application with the username
and other roles/claims for the application.
Hydra is developed by Ory and has security as one of their top priorities.
- **Kratos**: This is Identity Manager
and contains all the user profiles and secrets (passwords).
Kratos is designed to work mostly between UI (browser) and kratos directly,
over a public API endpoint.
Authentication, form-validation, etc. are all handled by Kratos.
Kratos only provides an API and not UI itself.
Kratos provides an admin API as well,
which is only used from the server-side flask app to create/delete users.
- **MariaDB**: The login application, as well as Hydra and Kratos, need to store data.
This is done in a MariaDB database server.
There is one instance with three databases.
As all databases are very small we do not foresee resource limitation problems.
- **Hydra**: Hydra is an open source OIDC server.
It means applications can connect to Hydra to start a session with a user.
Hydra provides the application with the username
and other roles/claims for the application.
Hydra is developed by Ory and has security as one of their top priorities.
- **Kratos**: This is Identity Manager
and contains all the user profiles and secrets (passwords).
Kratos is designed to work mostly between UI (browser) and kratos directly,
over a public API endpoint.
Authentication, form-validation, etc. are all handled by Kratos.
Kratos only provides an API and not UI itself.
Kratos provides an admin API as well,
which is only used from the server-side flask app to create/delete users.
- **MariaDB**: The login application, as well as Hydra and Kratos, need to store data.
This is done in a MariaDB database server.
There is one instance with three databases.
As all databases are very small we do not foresee resource limitation problems.
If Hydra hits a new session/user, it has to know if this user has access.
To do so, the user has to login through a login application.
......@@ -52,76 +52,114 @@ it is based on traditional Bootstrap + JQuery.
## Development environment
After this process is finished, the following will run in local docker containers:
- the dashboard frontend
- the dashboard backend
The following will be available through proxies running in local docker containers and port-forwards:
- Hydra admin API
- Kratos admin API and public API
- The MariaDB database
These need to be available locally, because Kratos wants to run on the same
domain as the front-end that serves the login interface.
### Setup
Please read through all subsections to set up your environment before
attempting to run the dashboard locally.
#### 1. Stackspin cluster
To develop the Dashboard, you need a Stackspin cluster that is set up as a
development environment. Follow the instructions [in the
dashboard-dev-overrides
repository](https://open.greenhost.net/stackspin/dashboard-dev-overrides#dashboard-dev-overrides)
in order to set up a development-capable cluster. The Dashboard, as well as
Kratos and Hydra, will be configured to point their endpoints to
`http://stackspin_proxy:8081` in that cluster. As a result, you can run
components using the `docker-compose.yml` file in this repository, and still log
into Stackspin applications that run on the cluster.
#### 2. Environment for frontend
The frontend needs to know where the backend API and hydra can be reached. To
configure it, create a `local.env` file in the `frontend` directory:
cp local.env.example local.env
and adjust the `REACT_APP_HYDRA_PUBLIC_URL` to the SSO URL of your cluster.
#### 3. Setup hosts file
The application will run on `http://stackspin_proxy`. Add the following line to
`/etc/hosts` to be able to access that from your browser:
```
127.0.0.1 stackspin_proxy
```
#### 4. Kubernetes access
The script needs you to have access to the Kubernetes cluster that runs
Stackspin. Point the `KUBECONFIG` environment variable to a kubectl config.
Attention points:
* The kubeconfig will be mounted inside docker containers, so also make sure
your Docker user can read it.
* The bind-mount done by docker might not work if the file pointed to is
part of a filesystem such as sshfs. In that case, copy the file to a local
drive first.
### Build and run
After you've finished all setup steps, you can run everything using
./run_app.sh
This sets a few environment variables based on what is in your cluster
secrets, and run `docker compose up` to build and run all necessary components,
including a reverse proxy and the backend flask application.
The development environment is a hybrid one, where one or both of the dashboard
frontend and backend run locally, but the rest of the cluster runs on a remote
machine.
The remote should be a regular Stackspin cluster, though preferably one that's
dedicated to development purposes.
The local dashboard frontend and/or backend can run in a docker container or
directly ("native mode"). (At this time it's not possible to mix the two, for
example by having the dashboard backend run directly and the frontend in a
docker container.)
The connection between the local and remote parts is set up by a tool called
telepresence. If you want to develop the frontend for example, telepresence
intercepts traffic that goes into the remote's frontend pod and redirects it to
your copy that's running locally on your machine; responses from your local
frontend are led back via the remote. This interception happens invisibly to
your browser, which you just point at the remote cluster.
### Prerequisites
#### Set up telepresence on your local development machine
You need to do this once for every development machine you're using
(workstation, laptop).
* You need root on your machine and at some point allow telepresence to perform
actions as root, in order to make network changes to allow the two-way
tunnel. If this is not possible or not desirable, you can try to run your
local dashboard in a docker container instead.
* Set `user_allow_other` in `/etc/fuse.conf`. This is necessary when
telepresence adds (FUSE-based) sshfs mounts so your local code can access
volumes from the kubernetes cluster, in particular the one with the service
account token (credentials for calling the kubernetes api), to let the
dashboard interact with the cluster.
- MacOS users may have to do a little extra work to get a working current
sshfs: see [telepresence
docs](https://www.getambassador.io/docs/telepresence-oss/latest/troubleshooting#volume-mounts-are-not-working-on-macos).
* Download and install the telepresence binary on your development machine:
https://www.getambassador.io/docs/telepresence-oss/latest/install
#### Access to development cluster
You need `kubectl` and `helm` binaries, and a `kubectl` configuration file
(often called "kubeconfig") containing credentials needed to authenticate
against your cluster. If the `KUBECONFIG` environment variable is set and
points to the config file, this will be picked up by the various programs.
#### Set up telepresence on your development cluster
You need to do this once for every cluster you want to use as a development cluster.
* Install telepresence on your development cluster:
```
telepresence helm install -f telepresence-values.yaml
```
#### Install local dependencies
Before running the frontend in native mode:
* Make sure you have nodejs installed. You may want to use [Node Version
Manager](https://github.com/nvm-sh/nvm) to make it easy to install several
versions side by side.
* Install necessary javascript dependencies (will be placed in
`frontend/node_modules`) using `./dev.sh frontend setup`.
Before running the backend in native mode:
* Make sure you have python3 installed.
* Install necessary python dependencies (in a virtualenv in `backend/venv`)
using `./dev.sh backend setup`.
### Run
From the root `dashboard` directory, run for example `./dev.sh frontend`. This
will set up the telepresence tunnel to the cluster, and start the dashboard
frontend server in native mode. `./dev.sh backend` will do the same but for the
backend. You can run both at the same time (in separate terminal windows) if
you want to make changes to both frontend and backend.
If you want to run the local dashboard in docker instead, use `./dev.sh
frontend docker` and/or `./dev.sh backend docker`. Please note that due to a
telepresence limitation it's not currently possible to run the frontend
natively and the backend in docker at the same time, or vice versa.
#### Known issues
* Running the dashboard backend locally with telepresence in docker mode
currently doesn't work because of dns resolution issues in the docker
container: https://github.com/telepresenceio/telepresence/issues/1492 . We
could work around this by using a fully qualified domain name for the
database service -- which doesn't agree with the goal of making the stackspin
namespace variable -- or using the service env vars, but we're hoping that
telepresence will fix this in time.
* Telepresence intercepts traffic to a pod, but the original pod is still
running. In case of the backend, this is sometimes problematic, for example
when you're adding database migrations which the original pod then doesn't
know about and crashes, or with SCIM which involves timer-based actions which
are then performed both by your modified local instance and by the original
remote one. There is some work in progress to allow scaling down the
intercepted pod: https://github.com/telepresenceio/telepresence/issues/1608 .
* If telepresence is giving errors, in particular ones about "an intercept with
the same name already existing" on repeated runs, it may help to reset the
telepresence state by doing `./dev.sh reset`. This will stop the local
telepresence daemon so it can be cleanly restarted on the next try, and will
also restart the "traffic manager" on the remote so it will discard any old
lingering intercepts.
---
## Testing as a part of Stackspin
......@@ -130,15 +168,40 @@ might behave differently in the local development environment compared to a
regular Stackspin instance, i.e., one that's not a local/cluster hybrid. In
this case, you'll want to run your new version in a regular Stackspin cluster.
To do that, make sure to increase the chart version number in `Chart.yaml`, and
push your work to a MR. The CI pipeline should then publish your new chart
version in the Gitlab helm chart repo for the dashboard project, but in the
`unstable` channel -- the `stable` channel is reserved for chart versions that
have been merged to the `main` branch.
To do that:
* Push your work to an MR.
* Set the image tags in `values.yaml` to the one created for your branch; if
unsure, check the available tags in the Gitlab container registry for the
dashboard project.
* Make sure to increase the chart version number in `Chart.yaml`, preferably
with a suffix to denote that it's not a stable version. For example, if the
last stable release is 1.2.3, make the version 1.2.4-myawesomefeature in your
branch.
The CI pipeline should then publish your new chart version in the Gitlab helm
chart repo for the dashboard project, but in the `unstable` channel -- the
`stable` channel is reserved for chart versions that have been merged to the
`main` branch.
Once your package is published, use it by
1. changing the `spec.url` field of the `flux-system/dashboard`
`HelmRepository` object in the cluster where you want to run this, replacing
`stable` by `unstable`; and
2. changing the `spec.chart.spec.version` field of the `stackspin/dashboard`
`HelmRelease` to your chart version (the one from this chart's `Chart.yaml`).
## Release process
To publish a new version of the helm chart:
1. Increase the docker image tag in `deployment/helmchart/values.yaml` so it uses the new tag (to be
created in a later step of this release).
2. Update the appVersion in `deployment/helmchart/Chart.yaml` to match that new tag version.
3. Increase the chart version in `deployment/helmchart/Chart.yaml`.
4. Update `CHANGELOG.md` and/or `deployment/helmchart/CHANGELOG.md` and check
that it includes relevant changes, including ones added by renovatebot.
5. Commit and push these changes to `main`.
6. Create a new git tag for the new release and push it to gitlab as well.
The last step will trigger a CI run that will package and publish the helm chart.
backend/Dockerfile
View file @ 9d43b6f4
FROM python:3.11-slim
FROM python:3.13-slim
# set "app" as the working directory from which CMD, RUN, ADD references
WORKDIR /app
......@@ -20,8 +20,7 @@ RUN pip install --no-cache-dir -r requirements.txt
# now copy all the files in this directory to /app
COPY . .
# Listen to port 80 at runtime
EXPOSE 5000
# Define our command to be run when launching the container
CMD ["gunicorn", "app:app", "-b", "0.0.0.0:5000", "--workers", "4", "--reload", "--capture-output", "--enable-stdio-inheritance", "--log-level", "DEBUG"]
CMD ["gunicorn", "app:app", "-b", "0.0.0.0:5000", "--workers", "8", "--preload", "--capture-output", "--enable-stdio-inheritance", "--log-level", "DEBUG"]
backend/MESSAGE_CODES 0 → 100644
View file @ 9d43b6f4
List of message codes used in the frontend
Kratos codes:
=============
4000006 The provided credentials are invalid, check for spelling mistakes
in your password or username, email address, or phone number.
1010003 Please confirm this action by verifying that it is you.
Stackspin codes:
================
S_CONFIRM_CREDENTIALS Please confirm your credentials to complete this action.
backend/app.py
View file @ 9d43b6f4
from apscheduler.schedulers.background import BackgroundScheduler
from flask import Flask, jsonify
from flask_cors import CORS
from flask_jwt_extended import JWTManager
from flask_migrate import Migrate
import flask_migrate
from jsonschema.exceptions import ValidationError
from NamedAtomicLock import NamedAtomicLock
import threading
import traceback
from werkzeug.exceptions import BadRequest
# These imports are required
......@@ -11,10 +15,16 @@ from cliapp import cli
from web import web
from areas import users
from areas import apps
from areas.apps.apps import *
from areas import auth
from areas import resources
from areas import roles
from areas import tags
from cliapp import cliapp
import config
import helpers.kubernetes
import helpers.provision
import helpers.threads
from web import login
from database import db
......@@ -32,39 +42,149 @@ from helpers import (
unauthorized_error,
)
import cluster_config
from config import *
import logging
import migration_reset
import os
import sys
# Configure logging.
log_level = logging.getLevelName(config.LOG_LEVEL or 'INFO')
from logging.config import dictConfig
dictConfig({
'version': 1,
'formatters': {'default': {
'format': '[%(asctime)s] %(levelname)s in %(module)s: %(message)s',
'format': '[%(asctime)s] %(levelname)s in %(name)s (%(filename)s+%(lineno)d): %(message)s',
}},
'handlers': {'wsgi': {
'class': 'logging.StreamHandler',
'stream': 'ext://flask.logging.wsgi_errors_stream',
'formatter': 'default',
'level': log_level,
}},
'root': {
'level': 'INFO',
'handlers': ['wsgi'],
}
'level': log_level,
},
# Loggers are created also by alembic, flask_migrate, etc. Without this
# setting, those loggers seem to be ignored.
'disable_existing_loggers': False,
})
logging.getLogger("kubernetes.client.rest").setLevel(logging.WARNING)
app = Flask(__name__)
app.config["SECRET_KEY"] = SECRET_KEY
app.config["SQLALCHEMY_DATABASE_URI"] = SQLALCHEMY_DATABASE_URI
app.config["SQLALCHEMY_ENGINE_OPTIONS"] = {'pool_pre_ping': True}
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = SQLALCHEMY_TRACK_MODIFICATIONS
cors = CORS(app)
Migrate(app, db)
db.init_app(app)
db.init_app(app)
app.logger.setLevel(logging.INFO)
with app.app_context():
provisioner = helpers.provision.Provision()
def init_routines():
with app.app_context():
# We have reset the alembic migration history at Stackspin version 2.2.
# This checks whether we need to prepare the database to follow that
# change.
migration_reset.reset()
app.logger.info("Loading flask_migrate.")
# flask_migrate exits the program when it encounters an error, for example
# when the version set in the database is not found in the
# `migrations/versions` directory. We could prevent that by catching the
# `SystemExit` exception here, but actually it's not safe to continue in
# that case.
flask_migrate.Migrate(app, db)
app.logger.info("Attempting flask_migrate database upgrade.")
try:
with app.app_context():
flask_migrate.upgrade()
# TODO: actually flask_migrate.upgrade will catch any errors and
# exit the program :/
except Exception as e:
app.logger.info(f"upgrade failed: {type(e)}: {e}")
sys.exit(2)
def reload():
# We need this app context in order to talk the database, which is managed by
# flask-sqlalchemy, which assumes a flask app context.
with app.app_context():
app.logger.info("Reloading dashboard config from cluster resources.")
# Load the list of apps from a configmap and store any missing ones in the
# database.
app_slugs = cluster_config.populate_apps()
# Same for the list of oauthclients.
cluster_config.populate_oauthclients()
# Load per-app scim config if present.
cluster_config.populate_scim_config(app_slugs)
# We could call `reload` here manually, but actually the watch also at its
# start creates events for existing secrets so we don't need to.
with app.app_context():
# Set watch for dashboard SCIM config secrets. Any time those change,
# we reload so we can do SCIM for newly installed apps.
try:
helpers.kubernetes.watch_dashboard_config(app, reload)
except Exception as e:
app.logger.error(f"Error watching dashboard config: {e}")
# Set up a generic task scheduler (cron-like).
scheduler = BackgroundScheduler()
scheduler.start()
# Add a job to run the provisioning reconciliation routine regularly.
# We'll also run it when we make changes that should be propagated
# immediately.
scheduler.add_job(helpers.threads.request_provision, 'interval', id='provision', hours=24)
# We'll run this in a separate thread so it can be done in the background.
# We have this single "provisioning worker" so there will be only one
# provisioning operation at a time. We use an Event to signal a
# provisioning request.
def provisioning_loop():
while True:
app.logger.info("Waiting for next provisioning run.")
# helpers.threads.provision_event.wait()
# helpers.threads.provision_event.clear()
helpers.threads.wait_provision()
app.logger.info("Starting provisioning.")
with app.app_context():
try:
provisioner.reconcile()
except Exception as e:
app.logger.warn(f"Exception during user provisioning:")
app.logger.warn(traceback.format_exc())
threading.Thread(target=provisioning_loop).start()
# `init_routines` should only run once per dashboard instance. To enforce this
# we have different behaviour for production and development mode:
# * We have "preload" on for gunicorn, so this file is loaded only once, before
# workers are forked (production).
# * We make sure that in development mode we run this only once, even though
# this file is loaded twice by flask for some reason.
if RUN_BY_GUNICORN:
logging.info("Running initialization code (production mode).")
init_routines()
else:
logging.info("WERKZEUG_RUN_MAIN: {}".format(os.environ.get("WERKZEUG_RUN_MAIN", "unset")))
if os.environ.get("WERKZEUG_RUN_MAIN") == "true":
logging.info("Running initialization code (dev mode).")
init_routines()
else:
logging.info("Not running initialization code (dev or cli mode).")
# This should not perform any actual migration, just load the
# flask_migrate extension so we can use `flask db` commands from the
# cli.
flask_migrate.Migrate(app, db)
# Now that we've done all database interactions in the initialisation routines,
# we need to drop all connections to the database to prevent those from being
# shared among different worker processes.
logging.info("Disposing of database connections.")
with app.app_context():
db.engine.dispose()
app.register_blueprint(api_v1)
app.register_blueprint(web)
......@@ -82,11 +202,19 @@ jwt = JWTManager(app)
# When token is not valid or missing handler
@jwt.invalid_token_loader
def invalid_token_callback(reason):
logging.info(f"Invalid token: {reason}.")
return jsonify({"errorMessage": "Unauthorized (invalid token)"}), 401
@jwt.unauthorized_loader
@jwt.expired_token_loader
def expired_token_callback(*args):
return jsonify({"errorMessage": "Unauthorized"}), 401
def unauthorized_callback(reason):
logging.info(f"No token: {reason}.")
return jsonify({"errorMessage": "Unauthorized (no token)"}), 401
@jwt.expired_token_loader
def expired_token_callback(reason):
logging.info(f"Expired token: {reason}.")
return jsonify({"errorMessage": "Unauthorized (expired token)"}), 401
@app.route("/")
def index():
......
backend/areas/__init__.py
View file @ 9d43b6f4
from flask import Blueprint
from flask import Blueprint, jsonify
import yaml
from config import *
import helpers.kubernetes as k8s
import requests
api_v1 = Blueprint("api_v1", __name__, url_prefix="/api/v1")
......@@ -7,3 +12,80 @@ api_v1 = Blueprint("api_v1", __name__, url_prefix="/api/v1")
@api_v1.route("/health")
def api_index():
return "Stackspin API v1.0"
@api_v1.route("/environment")
def api_environment():
environment = {
"HYDRA_PUBLIC_URL": HYDRA_PUBLIC_URL,
"KRATOS_PUBLIC_URL": KRATOS_PUBLIC_URL,
"TELEPRESENCE": TELEPRESENCE,
}
return jsonify(environment)
# We want to know if
# 1. A release has happened recently and is already deployed on this cluster.
# 2. A release has happened recently but has not yet been deployed on this
# cluster -- that will then probably happen automatically during the next
# maintenance window.
#
# To get the last release, we get the contents of the `VERSION` file from
# the main branch. The `VERSION` file is only updated as part of the release
# process.
#
# To find out how long ago the currently running version was deployed, we look
# at the `lastUpdateTime` of the stackspin `GitRepo` object on the cluster.
@api_v1.route("/info")
def api_info():
# Get static info from configmap on cluster.
static_info = k8s.get_kubernetes_config_map_data(
"stackspin-static-info",
"flux-system",
)
results = static_info
# Get app versions from apps configmaps on cluster.
results['appVersions'] = {}
apps = k8s.get_kubernetes_config_map_data(
"stackspin-apps",
"flux-system",
)
for app, app_data in apps.items():
data = yaml.safe_load(app_data)
if 'version' in data:
results['appVersions'][app] = data['version']
apps_custom = k8s.get_kubernetes_config_map_data(
"stackspin-apps-custom",
"flux-system",
)
if apps_custom is not None:
for app, app_data in apps_custom.items():
data = yaml.safe_load(app_data)
if 'version' in data:
results['appVersions'][app] = data['version']
# Get last update time of stackspin GitRepo object on the cluster; that
# tells us when flux last updated the cluster based on changes in the
# stackspin git repo.
stackspin_repo = k8s.get_gitrepo('stackspin')
results['lastUpdated'] = stackspin_repo['status']['artifact']['lastUpdateTime']
# This is the branch (or other git ref, like tag or commit) that this
# cluster follows.
flux_ref = stackspin_repo['spec']['ref']
# `flux_ref` is a structured object, though as far as we've seen always a
# dict with a single entry. The key can be `branch` or `tag` or `commit`.
# We reduce this to a single string git ref for simplicity in the
# front-end.
ref = next(iter(flux_ref.values()))
results['followingGit'] = ref
# Get latest released version from gitlab. Whether it's considered
# "released" depends on which branch we're following, but usually that's
# the `vX` "production" branch.
git_release = 'Unknown'
result = requests.get(f"https://open.greenhost.net/stackspin/stackspin/-/raw/{ref}/VERSION", timeout=5)
if result.status_code == 200:
git_release = result.text.rstrip()
results['lastRelease'] = git_release
return jsonify(results)
backend/areas/apps/apps.py
View file @ 9d43b6f4
......@@ -29,6 +29,7 @@ def get_apps():
@api_v1.route('/apps/<string:slug>', methods=['GET'])
@jwt_required()
@cross_origin()
def get_app(slug):
"""Return data about a single app"""
app = AppsService.get_app(slug)
......
backend/areas/apps/apps_service.py
View file @ 9d43b6f4
import threading
from flask import current_app
from flask_jwt_extended import get_jwt
import ory_kratos_client
from ory_kratos_client.api import identity_api
from .models import App, AppRole
from areas.roles.models import Role
from areas.users.models import User
from config import *
from database import db
from helpers.access_control import user_has_access
from helpers.kratos_user import KratosUser
import helpers.kubernetes as k8s
from helpers.threads import request_provision
class AppsService:
@staticmethod
......@@ -18,7 +25,7 @@ class AppsService:
def get_accessible_apps():
apps = App.query.all()
kratos_admin_api_configuration = ory_kratos_client.Configuration(host=KRATOS_ADMIN_URL, discard_unknown_keys=True)
kratos_admin_api_configuration = ory_kratos_client.Configuration(host=KRATOS_ADMIN_URL)
with ory_kratos_client.ApiClient(kratos_admin_api_configuration) as kratos_admin_client:
kratos_identity_api = identity_api.IdentityApi(kratos_admin_client)
......@@ -42,3 +49,39 @@ class AppsService:
def get_app_roles():
app_roles = AppRole.query.all()
return [{"user_id": app_role.user_id, "app_id": app_role.app_id, "role_id": app_role.role_id} for app_role in app_roles]
@classmethod
def install_app(cls, app):
app.install()
# Create app roles for the new app for all admins, and reprovision. We
# do this asynchronously, because we first need to wait for the app
# installation to be finished -- otherwise the SCIM config for user
# provisioning is not ready yet.
current_app.logger.info("Starting thread for creating app roles.")
# We need to pass the app context to the thread, because it needs that
# for database operations.
ca = current_app._get_current_object()
threading.Thread(target=cls.create_admin_app_roles, args=(ca, app,)).start()
@staticmethod
def create_admin_app_roles(ca, app):
"""Create AppRole objects for the given app for all admins."""
with ca.app_context():
ca.logger.info("Waiting for kustomizations to be ready.")
k8s.wait_kustomization_ready(app)
for user in User.get_all():
if not user['stackspin_data']['stackspin_admin']:
# We are only dealing with admin users here.
continue
existing_app_role = AppRole.query.filter_by(app_id=app.id, user_id=user['id']).first()
if existing_app_role is None:
ca.logger.info(f"Creating app role for app {app.slug} for admin user {user['id']}")
app_role = AppRole(
user_id=user['id'],
app_id=app.id,
role_id=Role.ADMIN_ROLE_ID
)
db.session.add(app_role)
db.session.commit()
ca.logger.info("Requesting user provisioning.")
request_provision()
backend/areas/apps/models.py
View file @ 9d43b6f4
"""Everything to do with Apps"""
import os
import base64
import enum
import os
from sqlalchemy import ForeignKey, Integer, String, Boolean
from sqlalchemy import Boolean, DateTime, Enum, ForeignKey, Integer, String, Unicode
from sqlalchemy.orm import relationship
from database import db
......@@ -29,8 +30,18 @@ class App(db.Model):
# The URL is only stored in the DB for external applications; otherwise the
# URL is stored in a configmap (see get_url)
url = db.Column(String(length=128), unique=False)
def __repr__(self):
scim_url = db.Column(String(length=1024), nullable=True)
scim_token = db.Column(String(length=1024), nullable=True)
scim_group_support = db.Column(Boolean, nullable=False, server_default='0')
oauthclients = relationship("OAuthClientApp", back_populates="app")
def __init__(self, slug, name, external=False, url=None):
self.slug = slug
self.name = name
self.external = external
self.url = url
def __str__(self):
return f"{self.id} <{self.name}>"
def get_url(self):
......@@ -80,20 +91,17 @@ class App(db.Model):
def install(self):
"""Creates a Kustomization in the Kubernetes cluster that installs this application"""
# Generate the necessary passwords, etc. from a template
self.__generate_secrets()
# Create add-<app> kustomization
self.__create_kustomization()
def uninstall(self):
"""
Delete the app kustomization.
Delete the `add-$app` kustomization.
In our case, this triggers a deletion of the app's PVCs (so deletes all
data), as well as any other Kustomizations and HelmReleases related to
the app. It also triggers a deletion of the OAuth2Client object, but
does not delete the secrets generated by the `install` command. It also
does not remove the TLS secret generated by cert-manager.
This triggers a deletion of the app's PVCs (so deletes all data), as
well as any other Kustomizations and HelmReleases related to the app.
It also triggers a deletion of the OAuth2Client object. It does not
remove the TLS secret generated by cert-manager.
"""
self.__delete_kustomization()
......@@ -107,25 +115,15 @@ class App(db.Model):
# Delete all roles first
for role in self.roles:
db.session.delete(role)
# Delete all related oauthclients
for auth in self.oauthclients:
db.session.delete(auth)
db.session.commit()
db.session.delete(self)
return db.session.commit()
def __generate_secrets(self):
"""Generates passwords for app installation"""
# Create app variables secret
if self.variables_template_filepath:
k8s.create_variables_secret(self.slug, self.variables_template_filepath)
k8s.create_variables_secret(
self.slug,
os.path.join(
self.__get_templates_dir(),
"stackspin-oauth-variables.yaml.jinja"
)
)
def __create_kustomization(self):
"""Creates the `add-{app_slug}` kustomization in the Kubernetes cluster"""
kustomization_template_filepath = \
......@@ -137,16 +135,6 @@ class App(db.Model):
"""Deletes kustomization for this app"""
k8s.delete_kustomization(f"add-{self.slug}")
@property
def variables_template_filepath(self):
"""Path to the variables template used to generate secrets the app needs"""
variables_template_filepath = os.path.join(self.__get_templates_dir(),
f"stackspin-{self.slug}-variables.yaml.jinja")
if os.path.exists(variables_template_filepath):
return variables_template_filepath
return None
@property
def namespace(self):
"""
......@@ -154,7 +142,7 @@ class App(db.Model):
FIXME: This should probably become a database field.
"""
if self.slug in ['nextcloud', 'wordpress', 'wekan', 'zulip']:
if self.slug in ['nextcloud', 'wordpress', 'wekan', 'zulip', 'hedgedoc']:
return 'stackspin-apps'
return 'stackspin'
......@@ -192,10 +180,28 @@ class App(db.Model):
@staticmethod
def __get_templates_dir():
"""Returns directory that contains the Jinja templates used to create app secrets."""
"""Returns directory that contains the Jinja templates for kubernetes manifests."""
return os.path.join(os.path.dirname(os.path.realpath(__file__)), "templates")
class ProvisionStatus(enum.Enum):
SyncNeeded = "SyncNeeded"
Provisioned = "Provisioned"
# Provisioning is not necessary for this user/role, for
# example because the user has no access to this app.
NotProvisioned = "NotProvisioned"
# SCIM Provisioning is not supported for this particular app.
NotSupported = "NotSupported"
# This user needs to be deleted from this app.
ToDelete = "ToDelete"
# This app role entry corresponds to a Stackspin user that no longer
# exists.
Orphaned = "Orphaned"
# Something went wrong; more information can be found in the
# `last_provision_message`.
Error = "Error"
class AppRole(db.Model): # pylint: disable=too-few-public-methods
"""
The AppRole object, stores the roles Users have on Apps
......@@ -204,10 +210,25 @@ class AppRole(db.Model): # pylint: disable=too-few-public-methods
user_id = db.Column(String(length=64), primary_key=True)
app_id = db.Column(Integer, ForeignKey("app.id"), primary_key=True)
role_id = db.Column(Integer, ForeignKey("role.id"))
provision_status = db.Column(
Enum(
ProvisionStatus,
native_enum=False,
length=32,
values_callable=lambda _: [str(member.value) for member in ProvisionStatus]
),
nullable=False,
default=ProvisionStatus.SyncNeeded,
server_default=ProvisionStatus.SyncNeeded.value
)
last_provision_attempt = db.Column(DateTime, nullable=True)
last_provision_message = db.Column(Unicode(length=256), nullable=True)
scim_id = db.Column(Unicode(length=256), nullable=True)
role = relationship("Role")
app = relationship("App")
def __repr__(self):
def __str__(self):
return (f"role_id: {self.role_id}, user_id: {self.user_id},"
f" app_id: {self.app_id}, role: {self.role}")
......@@ -237,7 +258,7 @@ class AppStatus(): # pylint: disable=too-few-public-methods
kustomization = app.kustomization
if kustomization is not None and "status" in kustomization:
ks_ready, ks_message = AppStatus.check_condition(kustomization['status'])
ks_ready, ks_message = k8s.check_condition(kustomization['status'])
self.installed = True
if ks_ready:
self.ready = ks_ready
......@@ -254,7 +275,7 @@ class AppStatus(): # pylint: disable=too-few-public-methods
helmreleases = app.helmreleases
for helmrelease in helmreleases:
hr_status = helmrelease['status']
hr_ready, hr_message = AppStatus.check_condition(hr_status)
hr_ready, hr_message = k8s.check_condition(hr_status)
# For now, only show the message of the first HR that isn't ready
if not hr_ready:
......@@ -266,29 +287,9 @@ class AppStatus(): # pylint: disable=too-few-public-methods
self.ready = ks_ready
self.message = f"App Kustomization status: {ks_message}"
def __repr__(self):
def __str__(self):
return f"Installed: {self.installed}\tReady: {self.ready}\tMessage: {self.message}"
@staticmethod
def check_condition(status):
"""
Returns a tuple that has true/false for readiness and a message
Ready, in this case means that the condition's type == "Ready" and its
status == "True". If the condition type "Ready" does not occur, the
status is interpreted as not ready.
The message that is returned is the message that comes with the
condition with type "Ready"
:param status: Kubernetes resource's "status" object.
:type status: dict
"""
for condition in status["conditions"]:
if condition["type"] == "Ready":
return condition["status"] == "True", condition["message"]
return False, "Condition with type 'Ready' not found"
def to_dict(self):
"""Represents this app status as a dict"""
return {
......@@ -303,14 +304,29 @@ class OAuthClientApp(db.Model): # pylint: disable=too-few-public-methods
This mapping exists so that
* you can have a different name for the OAuth client than for the app, and
* you can have multiple OAuth clients that belong to the same app.
Also, some apps might have no OAuth client at all.
"""
__tablename__ = "oauthclient_app"
oauthclient_id = db.Column(String(length=64), primary_key=True)
app_id = db.Column(Integer, ForeignKey("app.id"))
app = relationship("App")
app = relationship("App", back_populates="oauthclients")
def __repr__(self):
def __str__(self):
return (f"oauthclient_id: {self.oauthclient_id}, app_id: {self.app_id},"
f" app: {self.app}")
class ScimAttribute(db.Model): # pylint: disable=too-few-public-methods
"""
The ScimAttribute object records that a certain user attribute needs to be
set in a certain app via SCIM.
"""
user_id = db.Column(String(length=64), primary_key=True)
app_id = db.Column(Integer, ForeignKey("app.id"), primary_key=True)
attribute = db.Column(String(length=64), primary_key=True)
def __str__(self):
return (f"attribute: {self.attribute}, user_id: {self.user_id},"
f" app_id: {self.app_id}")
backend/areas/apps/templates/stackspin-nextcloud-variables.yaml.jinja deleted 100644 → 0
View file @ 72207ecf
---
apiVersion: v1
kind: Secret
metadata:
name: stackspin-nextcloud-variables
data:
nextcloud_password: "{{ 32 | generate_password | b64encode }}"
nextcloud_mariadb_password: "{{ 32 | generate_password | b64encode }}"
nextcloud_mariadb_root_password: "{{ 32 | generate_password | b64encode }}"
nextcloud_redis_password: "{{ 32 | generate_password | b64encode }}"
onlyoffice_database_password: "{{ 32 | generate_password | b64encode }}"
onlyoffice_jwt_secret: "{{ 32 | generate_password | b64encode }}"
onlyoffice_rabbitmq_password: "{{ 32 | generate_password | b64encode }}"
backend/areas/apps/templates/stackspin-oauth-variables.yaml.jinja deleted 100644 → 0
View file @ 72207ecf
---
apiVersion: v1
kind: Secret
metadata:
name: stackspin-{{ app }}-oauth-variables
data:
client_id: "{{ app | b64encode }}"
client_secret: "{{ 32 | generate_password | b64encode }}"
backend/areas/apps/templates/stackspin-wekan-variables.yaml.jinja deleted 100644 → 0
View file @ 72207ecf
apiVersion: v1
kind: Secret
metadata:
name: stackspin-wekan-variables
data:
mongodb_password: "{{ 32 | generate_password | b64encode }}"
mongodb_root_password: "{{ 32 | generate_password | b64encode }}"
backend/areas/apps/templates/stackspin-wordpress-variables.yaml.jinja deleted 100644 → 0
View file @ 72207ecf
---
apiVersion: v1
kind: Secret
metadata:
name: stackspin-wordpress-variables
data:
wordpress_admin_password: "{{ 32 | generate_password | b64encode }}"
wordpress_mariadb_password: "{{ 32 | generate_password | b64encode }}"
wordpress_mariadb_root_password: "{{ 32 | generate_password | b64encode }}"
backend/areas/apps/templates/stackspin-zulip-variables.yaml.jinja deleted 100644 → 0
View file @ 72207ecf
apiVersion: v1
kind: Secret
metadata:
name: stackspin-zulip-variables
data:
admin_password: "{{ 32 | generate_password | b64encode }}"
memcached_password: "{{ 32 | generate_password | b64encode }}"
rabbitmq_password: "{{ 32 | generate_password | b64encode }}"
rabbitmq_erlang_cookie: "{{ 32 | generate_password | b64encode }}"
redis_password: "{{ 32 | generate_password | b64encode }}"
postgresql_password: "{{ 32 | generate_password | b64encode }}"
zulip_password: "{{ 32 | generate_password | b64encode }}"
backend/areas/auth/auth.py
View file @ 9d43b6f4
from flask import jsonify, request
from flask import current_app, jsonify, request
from flask_jwt_extended import create_access_token
from flask_cors import cross_origin
from datetime import timedelta
from areas import api_v1
from areas.apps import App, AppRole
from areas.apps.models import App, AppRole
from config import *
from helpers import HydraOauth, BadRequest, KratosApi
......@@ -28,23 +28,24 @@ def hydra_callback():
raise BadRequest("Missing code query param")
token = HydraOauth.get_token(state, code)
token_id = token["access_token"]
user_info = HydraOauth.get_user_info()
# Match Kratos identity with Hydra
identities = KratosApi.get("/identities")
identity = None
for i in identities.json():
if i["traits"]["email"] == user_info["email"]:
identity = i
kratos_id = user_info["sub"]
access_token = create_access_token(
identity=token, expires_delta=timedelta(days=365), additional_claims={"user_id": identity["id"]}
)
# TODO: add a check to see if this a valid ID/active account
try:
access_token = create_access_token(
identity=token_id, expires_delta=timedelta(hours=1), additional_claims={"user_id": kratos_id}
)
except Exception as e:
raise BadRequest("Error with creating auth token between backend and frontend")
apps = App.query.all()
app_roles = []
for app in apps:
tmp_app_role = AppRole.query.filter_by(
user_id=identity["id"], app_id=app.id
user_id=kratos_id, app_id=app.id
).first()
app_roles.append(
{
......@@ -57,7 +58,7 @@ def hydra_callback():
{
"accessToken": access_token,
"userInfo": {
"id": identity["id"],
"id": kratos_id,
"email": user_info["email"],
"name": user_info["name"],
"preferredUsername": user_info["preferred_username"],
......
backend/areas/resources/__init__.py 0 → 100644
View file @ 9d43b6f4
from .resources import *
from .resources_service import *
backend/areas/resources/resources.py 0 → 100644
View file @ 9d43b6f4
from flask import jsonify, request
from flask_cors import cross_origin
from flask_expects_json import expects_json
from flask_jwt_extended import get_jwt, jwt_required
from areas import api_v1
from helpers.auth_guard import admin_required
from .resources_service import ResourcesService
@api_v1.route("/resources", methods=["GET"])
@jwt_required()
@cross_origin()
@admin_required()
def get_resources():
res = ResourcesService.get_resources()
return jsonify(res)