Feature #1542
openOpenAtlas on Kubernetes (ongoing)
Description
We will try to adapt and implement OpenAtlas with Kubernetes at the ACDH-CH.
Demo Version: https://openatlas-demo.acdh-ch-dev.oeaw.ac.at
(no daily recreation of the database)
To do (random order and non-exhaustive)¶
- Migrate projects
- Import files
- Backup Services
- daily, weekly, quarterly database (maybe it is better to write python script)
- IIIF
- Deployment on Rancher
- Workable demo version
- Test with multiple versions
- Daily recreation of the database
- Apache access/error log as stdout
- IIIF
- additional storage for converted files
Updated by Alexander Watzinger over 2 years ago
Today we have a Kubernetes meeting
Updated by Alexander Watzinger over 2 years ago
Today I did some cleanup at the ACDH-CH GitLab repository. I deleted the main and develop branch and uploaded them again because, like we discussed, Kubernetes specific changes should only be made in the feature_kubernetes branch. Once everything is working with Kubernetes we may look into merging it to main but have to do this carefully to be sure to not put our productive systems at risk, e.g. now it seems like passwords may be leaked, more about that further down.
Next I wanted to merge our current develop branch to feature_kubernetes. Originally it was a copy of main but I think it would be better to be a branch of develop to have the latest changes already merged. However, when pulling the latest feature_kubernetes form GitLab I noticed a few problematic changes made there:- A SQL dump was added to the Git repository by Berni (install/demo-dev-dump.sql) which shouldn't be part of the OpenAtlas repository so please clean this up (e.g. delete and put in .gitignore).
- Dalibor removed the instance/prodution.py from .gitignore, presumably to change configuration. This is not how it is supposed to work, e.g. passwords are saved in instance/production.py. If persistent changes are needed in the repository they should be made in config/default.py or, if they are Kubernetes specific we should add a config/kubernetes.py.
- fixing .gitignore to ignore instance/ again except specified files (see .gitignore in main branch for how it should be)
- removing database dump from repository
- merge develop to feature_kubernetes to be in sync with current development
However, it's great to see that you managed to get already something running at https://demo-acdh-ch.openatlas.eu/.
Updated by Dalibor Pancic over 2 years ago
Dalibor removed the instance/prodution.py from .gitignore, presumably to change configuration. This is not how it is supposed to work, e.g. passwords are saved in instance/production.py. If persistent changes are needed in the repository they should be made in config/default.py or, if they are Kubernetes specific we should add a config/kubernetes.py.
The openatlas app must be reconfigured that it can check if variables introduced over instance/prodution.py in feature_kubernetes are present as env variables.
They have to be introduced as Gitlab CI/CD variables for feature_kubernetes branch. By default, Openatlas expects that variables are hard-coded in the instance/prodution.py.
We can return instance/prodution.py to .gitignore after we adjust Openatlas that it can take variables from the "os.environ"
DATABASE_NAME = os.environ.get('POSTGRES_DB') DATABASE_USER = os.environ.get('POSTGRES_USER') DATABASE_PASS = os.environ.get('POSTGRES_PASSWORD') DATABASE_HOST = os.environ.get('POSTGRES_HOST') DATABASE_PORT = os.environ.get('POSTGRES_PORT') DATABASE_URL = os.environ.get('DATABASE_URL') SECRET_KEY = os.environ.get('SECRET_KEY')
Updated by Dalibor Pancic over 2 years ago
New domain https://discovery-demo-acdh-ch.openatlas.eu is created, pointed to the cluster and can be used for deployment.
Updated by Dalibor Pancic over 2 years ago
The problem with the production.py is solved and production.py is returned to the .gitignore.
All Kubernetes related configuration is introduced over requirements.txt, Procfile, start.sh and Gitlab CI/CD evironment variables.
Only these three new files are added to the default Openatlas repo, and they are located in the root of the feature_kubernetes branch.
In order to merge the develop with the feature_kubernetes branch, we just need to remove install/demo-dev*.sql dumps.
Updated by Alexander Watzinger over 2 years ago
Thanks a lot Dalibor for solving this. I really hope we see you at our OpenAtlas Summer Meeting next week. Would be nice to see in person again and your first drinks will be on us for all your help :)
Updated by Alexander Watzinger about 2 years ago
- Target version changed from Wishlist to 252
Moved this issue to the INDIGO roadmap version because we will need it for that cooperation first.
Updated by Alexander Watzinger about 2 years ago
- Target version changed from 252 to Wishlist
Moving to wishlist again while resolving the INDIGO version.
Updated by Bernhard Koschiček-Krombholz almost 2 years ago
- Status changed from In Progress to Acknowledged
- Assignee deleted (
Bernhard Koschiček-Krombholz)
Updated by Bernhard Koschiček-Krombholz 11 months ago
- Description updated (diff)
- Status changed from Acknowledged to Assigned
- Assignee set to Bernhard Koschiček-Krombholz
- Target version changed from Wishlist to 7.15.0
GitHub Action are working, but some tests fail.
We deployed a first version on ranger and it looks good. Now we can work on the harder issues like persistence, migration and update.
Updated by Bernhard Koschiček-Krombholz 11 months ago
- Related to Feature #1582: GitHub test actions added
Updated by Bernhard Koschiček-Krombholz 11 months ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz 10 months ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz 10 months ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz 10 months ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz 10 months ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz 10 months ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz 10 months ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz 9 months ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz 9 months ago
- Description updated (diff)
Updated by Omar Siam 9 months ago
Helm chart now should support external database as well as a database installed next to OpenAtlas. The docker-k8s feature branch at the moment deploys to a separate postgis database deployed on Rancher.
There is a pgadmin4 instance (credentials can be found in Rancher) to maintain database on this "external" PostgreSQL server.
We now need to test if it deploys correctly with the "embedded" PostgreSQL server. I am quite sure it should.
That should finish the Helm Chart tasks
I think we have a Workable demo version
as well as a version that can be combined with OpenAtlas-Discovery.
Daily security updates:
This can be achieved using github actions cron jobs. We probably should make sure a new version of the container image is not published if there is no updates.
Backup:
We need to create K8s cron jobs for this an add them to the helm chart. That should be a short task.
Updated by Bernhard Koschiček-Krombholz 9 months ago
Thank you, Omar!
Working on first demo version, have some problems with URL. (Going for https://openatlas-demo.acdh-dev.arz.oeaw.ac.at/)
- GitHub schedule: Make a new GitHub action file to pull build, check if there are updates available. If updates are available, rebuild everything using workflow_dispatch: {}
- Create cron job to save backups (files and database dumps) to the backup samba share
Updated by Omar Siam 9 months ago
Your options are
- https://openatlas-demo.acdh-ch-dev.oeaw.ac.at (publicly accessible)
- https://openatlas-demo.acdh-cluster-2.arz.oeaw.ac.at (private)
Updated by Bernhard Koschiček-Krombholz 9 months ago
- Description updated (diff)
Thank you, I put it now on https://openatlas-demo.acdh-ch-dev.oeaw.ac.at
Updated by Alexander Watzinger 9 months ago
- Description updated (diff)
- Status changed from Assigned to In Progress
Updated by Bernhard Koschiček-Krombholz 6 months ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz 6 months ago
- Related to Feature #2025: Daily Debian security updates for GitHub containers added
Updated by Alexander Watzinger 5 months ago
- Target version changed from 7.15.0 to 7.18.0
Assigning to a later version because of vacations and to not to interfere with our releases. It can of course be moved again once finished.
Updated by Alexander Watzinger 5 months ago
Just because Omar asked about the status the other day: we still need to test it in combination with a presentation site.
Once implemented please also add an entry with a link to the ACDH-CH demo presentation site to the "online sites list".
Updated by Alexander Watzinger 5 months ago
- Description updated (diff)
I'm not sure if already covered so I added NPM package update to the description at "to do".
This would be the case for e.g. 7.15.0 release with chart.js for type stats display.
Updated by Alexander Watzinger about 1 month ago
- Subject changed from OpenAtlas on Kubernetes to OpenAtlas on Kubernetes (ongoing)
- Target version deleted (
7.18.0)
Updated by Bernhard Koschiček-Krombholz 13 days ago
Apache access log example JSON:
"remote-address": tokens["remote-addr"](req, res), time: tokens["date"](req, res, "iso"), method: tokens["method"](req, res), url: tokens["url"](req, res), "http-version": tokens["http-version"](req, res), "status-code": tokens["status"](req, res), "content-length": tokens["res"](req, res, "content-length"), referrer: tokens["referrer"](req, res), "user-agent": tokens["user-agent"](req, res)