stackspin issueshttps://open.greenhost.net/stackspin/stackspin/-/issues2024-02-16T14:20:10Zhttps://open.greenhost.net/stackspin/stackspin/-/issues/1667Upgrade flux to v2.22024-02-16T14:20:10ZArie PetersonUpgrade flux to v2.2Flux has released version `v2.2`, promoting some more APIs to stable `v1`. We need to upgrade by following the [documented procedure](https://github.com/fluxcd/flux2/releases/tag/v2.2.0). This needs to be done in two steps. To make thing...Flux has released version `v2.2`, promoting some more APIs to stable `v1`. We need to upgrade by following the [documented procedure](https://github.com/fluxcd/flux2/releases/tag/v2.2.0). This needs to be done in two steps. To make things easy, we'll do this in two different Stackspin releases.
1. [ ] Upgrade flux CRDs and controllers.
2. [ ] In a next release, upgrade pertinent flux resources (HelmReleases) to the new API versions.2.13https://open.greenhost.net/stackspin/stackspin/-/issues/1665Perform velero restic maintenance in the night2024-03-19T11:17:43ZArie PetersonPerform velero restic maintenance in the nightVelero does regular maintenance on the restic repo containing the backups. Velero by default chooses the schedule for that, I'm guessing based on the time velero was started. We'd like to avoid doing that during daytime, because restic p...Velero does regular maintenance on the restic repo containing the backups. Velero by default chooses the schedule for that, I'm guessing based on the time velero was started. We'd like to avoid doing that during daytime, because restic pruning can be very resource intensive. Let's see if we can instruct velero to do that during the night, or perhaps disable it and perform maintenance through a Job manually.https://open.greenhost.net/stackspin/stackspin/-/issues/1664Auto-create app kustomization variables2024-02-12T10:55:42ZJanekAuto-create app kustomization variablesHere a little prototype to modularize the `create-kustomization-variables-configmaps.sh` script:
```sh
for app in $({ kubectl get cm -n flux-system stackspin-apps -o jsonpath-as-json="{.data}"; kubectl get cm -n flux-system stackspin-app...Here a little prototype to modularize the `create-kustomization-variables-configmaps.sh` script:
```sh
for app in $({ kubectl get cm -n flux-system stackspin-apps -o jsonpath-as-json="{.data}"; kubectl get cm -n flux-system stackspin-apps-custom -o jsonpath-as-json="{.data}"; } 2>/dev/null | grep '"' | cut -d'"' -f2 | grep -v velero)
do kubectl get cm stackspin-${app}-kustomization-variables || kubectl create cm stackspin-${app}-kustomization-variables --from-literal=${app}_domain=${app}.${domain}
done
```
A few issues to be solved though:
- the default subdomain should be part of the stackspin-apps(-custom) configmap to set it appropriately
- it would need to respond to changes in either of the configmaps
But actually, should the creation of this configmap not simply be part of the app-installation routine so we also do not create excess unused configmaps? Unsure how I would best handle that in my extension repo then...https://open.greenhost.net/stackspin/stackspin/-/issues/1663Thoughts on subchart databases?2024-03-19T11:03:13ZJanekThoughts on subchart databases?You also seem to use the typical (bitnami) sub-chart databases, which I thought is not a good idea long-term because of database version upgrades. Are you planning on doing something about it or is that settled somehow?You also seem to use the typical (bitnami) sub-chart databases, which I thought is not a good idea long-term because of database version upgrades. Are you planning on doing something about it or is that settled somehow?JanekJanekhttps://open.greenhost.net/stackspin/stackspin/-/issues/1658Company name & slogan in flux variables?2024-01-23T19:02:40ZJanekCompany name & slogan in flux variables?Services like Nextcloud and Forgejo allow specifying the name of the service and a short slogan to be displayed, I think it would make sense to autofill that from `.flux.env`?Services like Nextcloud and Forgejo allow specifying the name of the service and a short slogan to be displayed, I think it would make sense to autofill that from `.flux.env`?https://open.greenhost.net/stackspin/stackspin/-/issues/1657Contributing Forgejo as new app2024-01-23T18:58:04ZJanekContributing Forgejo as new app# Integrate a new app
I am moving from Gitea to Forgejo and think this is a worthwhile app to integrate into Stackspin proper:
- I propose `code` or `forge` as default subdomain
- OIDC login is working and registration restricted to it
...# Integrate a new app
I am moving from Gitea to Forgejo and think this is a worthwhile app to integrate into Stackspin proper:
- I propose `code` or `forge` as default subdomain
- OIDC login is working and registration restricted to it
- I moved SSH access to the worker to Port 2222 (remember adjusting nftables.conf and sshd.conf) and opened port 22 for convenient git pushing - would you want to handle it like that as well?
- forgejo does not allow to name the default admin user "admin", so I named it "forgejo" for now, how would you like to handle that?
Follow-up to https://open.greenhost.net/stackspin/stackspin/-/issues/1310 and https://open.greenhost.net/stackspin/stackspin/-/issues/1364
## Flux
### Cluster config
* [ ] Flux kustomization: `flux2/cluster/optional/APP/APP.yaml` (Example: `flux2/cluster/optional/wekan/wekan.yaml`)
### Source (helmRepository / gitRepository)
* [ ] Create new `helmRepository` if needed in `flux2/core/base/sources/APP-helmrepository.yaml` (Example: `flux2/core/base/sources/wekan-helmrepository.yaml`)
* [ ] Include `APP-helmrepository.yaml` in `flux2/core/base/sources/kustomization.yaml`
### App config
* [ ] Add app secrets template (in `dashboard` repo): `backend/areas/apps/templates/stackspin-APP-variables.yaml.jinja` (Example: `backend/areas/apps/templates/stackspin-wekan-variables.yaml.jinja`)
* [ ] If the app is storing state to disk, add PVCs in `flux2/apps/APP/pvc.yaml` (Example: `flux2/apps/wekan/pvc.yaml`)
* [ ] Helm chart values configmap: `flux2/apps/APP/APP-values-configmap.yaml` (Example: `flux2/apps/wekan/wekan-values-configmap.yaml`)
* [ ] `HelmRelease` in `flux2/apps/APP/release.yaml` (Example: `flux2/apps/wekan/release.yaml`)
* [ ] Create a kustomization config map for your app that sets the app subdomain by adding an entry to `flux2/core/base/migration-scripts/create-kustomization-variables-configmaps-script-configmap.yaml`
* [ ] Add your app in the right places in the dashboard source, in particular including an app icon.
#### Single sign-on
* Integrate the new app into the single sign-on system
* [ ] Oauth client resource: `flux2/apps/APP/APP-oauth-client.yaml` (Example: `flux2/apps/wekan/wekan-oauth-client.yaml`)
* [ ] Configure app OIDC settings, probably via its helm values: `flux2/apps/APP/APP-values-configmap.yaml` (Example: `flux2/apps/wekan/wekan-values-configmap.yaml`)
* [ ] Add the app to the list of dashboard apps and oauthclients:
- in `flux2/core/base/dashboard/dashboard-apps-configmap.yaml` if this app will be part of the default Stackspin app set.
- If you have a self-managed cluster please create new configmaps `stackspin-apps-custom` and `stackspin-oauthclients-custom` with the same structure as the official ones.
* [ ] Disable user/pw login if possible (#881)
* Test SSO:
* [ ] Admin login should grant admin privileges
* [ ] Non-admin login should not grant admin privileges
#### Backup/restore
This applies if the app has any persistent storage that needs to be part of
backups.
* [ ] Add the label `stackspin.net/backupSet: "APP"` to some kubernetes
resources. This label is used by Velero when instructed to restore a single
app. Typically you should add it to:
* [ ] the pvc(s) in `flux2/apps/APP/pvc*.yaml` (Example: `flux2/apps/wekan/pvc.yaml`)
* [ ] any pod(s) that use those pvc(s); this would go in the chart's helm
values configmap, with the value typically called `podLabels`, or if it doesn't have
that maybe `commonLabels`: `flux2/apps/APP/APP-values-configmap.yaml` (Example: `flux2/apps/wekan/wekan-values-configmap.yaml`)
* [ ] the kubernetes objects controlling those pods, typically a deployment
(`deploymentLabels` or `commonLabels`) or statefulset (`statefulSetLabels`
or `commonLabels`).
* [ ] To the same pods, i.e., the ones that use the pvcs that need to be
backed up, add an annotation `backup.velero.io/backup-volumes: "volume-name"`,
where `volume-name` is the name of the volume internal to the pod kubernetes
object, as shown for example in `kubectl describe pod` output.
* [ ] Add app-specific backup/restore instructions to `docs/maintenance.rst` if
necessary.
### Etc
* [ ] Add app to `dump_secrets()` in `stackspin/cluster.py`
If you made it until here you have completed all necessary steps for adding a
custom app to your cluster.
If you intend to contribute to Stackspin by a new app merge request please
continue and follow the rest of the steps below.
## Tests
* [ ] Make sure testing app resources work (`test/pytest/test_resources.py`)
* [ ] Make sure testing app cert works (`test/pytest/test_certs.py`)
* [ ] Add automatic tests for your app. We use the cypress framework for this.
See `test/cypress/e2e` for the existing tests.
## CI
* Add the following elements:
* [ ] `.APP-rules` partial to `.gitlab/ci_templates/stackspin_common.yml`
* [ ] `enable-APP` job to `.gitlab/ci_pipelines/install_stackspin.yml`
* [ ] `APP-app-ready` job to `.gitlab/ci_pipelines/default.yml`
* [ ] `upgraded-APP-app-ready` job to `.gitlab/ci_pipelines/upgrade_test.yml`
* [ ] an entry to `app_subdomains` in `test/pytest/test_certs.py`
## Renovatebot
* [ ] Make sure that latest [renovate pipeline](https://open.greenhost.net/stackspin/renovate/-/pipelines)
checks for app updates **after the new app is merged into the main branch**
## Documentation
* Add app to:
* [ ] `Step 3: Install additional applications` in `docs/installation/install_stackspin.rst`
* [ ] `docs/installation/testing.rst`
* [ ] `docs/system_administration/migrating.rst`
* [ ] `docs/usage/applications.rst`
## Demo instance
* [ ] Install the app on `demo.stackspin.net` so it can be previewed by anyone.
* [ ] Update the nightly reset script for the demo instance, on `cli.stackspin.net`,
in `/srv/stackspin/clusters/demo.stackspin.net/custom-scripts/vars.sh`.
## Follow-up issues
Create follow-up issue with:
* [ ] Limit settings (<https://blog.kubecost.com/blog/requests-and-limits/#our-solution>)
* [ ] Set CPU request
* CPU limits are unset.
* Set CPU request to average CPU usage.
* [ ] Set memory requests and limits
* Set memory request to:
```
1.5 * avg(quantile_over_time(.99,container_memory_working_set_bytes{container_name!="POD",container_name!=""}[7d])) by (container_name,pod_name,namespace)`
```
* Limit set to `2 * request`https://open.greenhost.net/stackspin/stackspin/-/issues/1656Flask CLI should exit non-zero on error2024-02-01T10:29:22ZJanekFlask CLI should exit non-zero on error```sh
root@dashboard-664677d446-txzpk:/app# flask cli user create test@example.com
[2024-01-23 12:20:22,509] INFO in app: WERKZEUG_RUN_MAIN: unset
[2024-01-23 12:20:22,509] INFO in app: Not running initialization code (dev or cli mode).
...```sh
root@dashboard-664677d446-txzpk:/app# flask cli user create test@example.com
[2024-01-23 12:20:22,509] INFO in app: WERKZEUG_RUN_MAIN: unset
[2024-01-23 12:20:22,509] INFO in app: Not running initialization code (dev or cli mode).
[2024-01-23 12:20:22,520] INFO in cli: Creating user with email: (test@example.com)
root@dashboard-664677d446-txzpk:/app# echo $?
0
root@dashboard-664677d446-txzpk:/app# flask cli user create test@example.com
[2024-01-23 12:20:26,556] INFO in app: WERKZEUG_RUN_MAIN: unset
[2024-01-23 12:20:26,557] INFO in app: Not running initialization code (dev or cli mode).
[2024-01-23 12:20:26,566] INFO in cli: Creating user with email: (test@example.com)
[2024-01-23 12:20:26,592] INFO in cli: User already exists. Not recreating
root@dashboard-664677d446-txzpk:/app# echo $?
0
```
would expect a non-zero exit code for the second case, ideally different from 1 though so it can be distinguished from an unexpected errorhttps://open.greenhost.net/stackspin/stackspin/-/issues/1654Allow self-registration / account requests for SSO2024-02-14T10:10:09ZJanekAllow self-registration / account requests for SSOCurrently, to provide access to new users, we need to collect their email address and full name, have an admin create their account, and then instruct them on how to set their password.
If new users could simply visit the login page and ...Currently, to provide access to new users, we need to collect their email address and full name, have an admin create their account, and then instruct them on how to set their password.
If new users could simply visit the login page and there was an option so they can register for an account which then needs to be confirmed before they can use it, the whole process could be compressed and would need a lot less communication.
I think this needs two main config options:
- allow self-registration
- notify admin on new self-registration
New users register with E-mail address, password and name and then once their account is confirmed receive an e-mail notification and can get going immediately.
This could optionally be restricted to a specific mail-domain, and along that line it could be useful to have an option that prevents users from changing their account e-mail address as our SSO should be company-managed like with classic LDAP.https://open.greenhost.net/stackspin/stackspin/-/issues/1653Dashboard briefly shows "React App" as title when loading2024-02-01T10:20:20ZJanekDashboard briefly shows "React App" as title when loadingdunno if this can be fixed, thought I'd mention itdunno if this can be fixed, thought I'd mention itBackloghttps://open.greenhost.net/stackspin/stackspin/-/issues/1652Renew session when showing bulk useradd form2024-02-01T10:18:45ZJanekRenew session when showing bulk useradd formI just now clicked on "Add new users" in the Dashboard, diligently filled in the data and set permissions, only to be greeted with a 301 upon confirmation - meaning I had to log back in and do it all over again. I guess either save the f...I just now clicked on "Add new users" in the Dashboard, diligently filled in the data and set permissions, only to be greeted with a 301 upon confirmation - meaning I had to log back in and do it all over again. I guess either save the form state or verify the session upon showing the form, with the latter seeming more feasible.https://open.greenhost.net/stackspin/stackspin/-/issues/1651ErrorMessage when user is trying to log into app where they have No Access co...2024-02-01T10:17:56ZJanekErrorMessage when user is trying to log into app where they have No Access configuredI have configured a Testuser to have no access, and it sort of works fine:
The user can log in to Zulip and not to Gitea and HedgeDoc. But when the user tries to login there, they don't get a proper acess denied but a nondescript error m...I have configured a Testuser to have no access, and it sort of works fine:
The user can log in to Zulip and not to Gitea and HedgeDoc. But when the user tries to login there, they don't get a proper acess denied but a nondescript error message:
![image](/uploads/497f7b933f6a224cec7dbca19e8c7cff/image.png)https://open.greenhost.net/stackspin/stackspin/-/issues/1650Network Security2024-02-01T10:16:55ZJanekNetwork SecurityThrough something like Istio Ambient Mesh, Cilium or NeuVector so pods can only talk to those they need to talk to.
Especially important when going into multi-node setups: https://open.greenhost.net/stackspin/stackspin/-/issues/1366
Fol...Through something like Istio Ambient Mesh, Cilium or NeuVector so pods can only talk to those they need to talk to.
Especially important when going into multi-node setups: https://open.greenhost.net/stackspin/stackspin/-/issues/1366
Follow-up to https://open.greenhost.net/stackspin/stackspin/-/issues/1264Backloghttps://open.greenhost.net/stackspin/stackspin/-/issues/16492FA with Hardware Tokens2024-02-13T15:34:14ZJanek2FA with Hardware TokensIs this something that is simply not yet implemented on Stackspin's side, or would support for hardware tokens need upstream developments first?Is this something that is simply not yet implemented on Stackspin's side, or would support for hardware tokens need upstream developments first?2.12Arie PetersonArie Petersonhttps://open.greenhost.net/stackspin/stackspin/-/issues/1647Website page load is delayed by fetching unreachable script2024-01-09T14:14:52ZRemon HuijtsWebsite page load is delayed by fetching unreachable scriptWhen I visit the public Stackspin website, the homepage tries to fetch a script from https://analytics.greenhost.net/js/plausible.outbound-links.js?ver=1.3.0 but fails to connect. It blocks page loading for a few seconds.When I visit the public Stackspin website, the homepage tries to fetch a script from https://analytics.greenhost.net/js/plausible.outbound-links.js?ver=1.3.0 but fails to connect. It blocks page loading for a few seconds.https://open.greenhost.net/stackspin/stackspin/-/issues/1645Restoring onto new kubernetes cluster from velero backup2024-02-20T09:46:46ZJanekRestoring onto new kubernetes cluster from velero backupHere I am documenting my process and issues I am encountering while "cold" restoring a whole cluster from velero backup.
I have adapted the script from https://docs.stackspin.net/en/v2/system_administration/maintenance.html#restore:
...Here I am documenting my process and issues I am encountering while "cold" restoring a whole cluster from velero backup.
I have adapted the script from https://docs.stackspin.net/en/v2/system_administration/maintenance.html#restore:
backup='231028.13'; app=dashboard
namespace=stackspin-apps
restore=${backup}-$app-$(date +%s)
if test $app = dashboard
then kust=single-sign-on
hr="$kust-database"
namespace=stackspin
else hr="$namespace $app"
fi
flux suspend kustomization ${kust:-$app}
flux suspend helmrelease -n $namespace $hr
kubectl delete all -n $namespace -l stackspin.net/backupSet=$app
kubectl delete secret -n $namespace -l stackspin.net/backupSet=$app
kubectl delete configmap -n $namespace -l stackspin.net/backupSet=$app
kubectl delete pvc -n $namespace -l stackspin.net/backupSet=$app
velero restore create $restore --from-backup=$backup -l stackspin.net/backupSet=$app
echo "Waiting a few seconds for backup to restore..."
sleep 10
velero restore describe $restore
echo "Press enter if backup is ready:"
read
flux resume helmrelease -n $namespace $hr
flux resume kustomization ${kust:-$app}https://open.greenhost.net/stackspin/stackspin/-/issues/1644Use upstream local-path-provisioner again2024-01-09T10:52:41ZVaracUse upstream local-path-provisioner againWe were using our own custom local-path-provisioner because Velero needs `local-path` PVs, and until recently, the upstream Rancher local-path-provisioner could only create `hostPath` PVs.
I noticed that [`local-path` support was added ...We were using our own custom local-path-provisioner because Velero needs `local-path` PVs, and until recently, the upstream Rancher local-path-provisioner could only create `hostPath` PVs.
I noticed that [`local-path` support was added recently](https://github.com/rancher/local-path-provisioner#volume-types) finally, so I think it would make sense to use the upstream provisioner againin the long run, and stop maintaining a fork with all extra work that comes with.https://open.greenhost.net/stackspin/stackspin/-/issues/1642Adapt to new flux metrics2023-12-19T13:15:56ZArie PetersonAdapt to new flux metricsFlux changed up their prometheus metrics in `v2.1`. We fixed the immediate problem in !2146 to at least let grafana start, but we need to do some work to make sure our flux grafana dashboard actually works and shows flux metrics.Flux changed up their prometheus metrics in `v2.1`. We fixed the immediate problem in !2146 to at least let grafana start, but we need to do some work to make sure our flux grafana dashboard actually works and shows flux metrics.https://open.greenhost.net/stackspin/stackspin/-/issues/1634Mention collabora on the website2023-11-07T09:34:32ZArie PetersonMention collabora on the websiteWe should mention on the website that we offer Collabora as a possible alternative to Onlyoffice.We should mention on the website that we offer Collabora as a possible alternative to Onlyoffice.https://open.greenhost.net/stackspin/stackspin/-/issues/1624Update documentation2024-02-16T14:20:34ZArie PetersonUpdate documentationSeveral parts of our documentation are severely outdated. For example, we still warn admins to only create users for people they fully trust, because all users supposedly can manage other users, while that was changed quite some time ago...Several parts of our documentation are severely outdated. For example, we still warn admins to only create users for people they fully trust, because all users supposedly can manage other users, while that was changed quite some time ago. Ideally we'd go over the whole thing, not rewriting everything but at least do a sanity check and remove or update parts that not longer reflect the current state of affairs.https://open.greenhost.net/stackspin/stackspin/-/issues/1622Take URL parameters for recovery page2023-10-27T14:27:46ZJanekTake URL parameters for recovery pageWhen creating an account for a user, I usually asynchronously share the info with them, so I cannot directly share the link which expires within an hour. For that it would be handy to be able to point them to a prefilled recovery page wh...When creating an account for a user, I usually asynchronously share the info with them, so I cannot directly share the link which expires within an hour. For that it would be handy to be able to point them to a prefilled recovery page where the only thing they do is submit the form:
https://dashboard.DOMAIN/web/recovery?email=user@example.com