OpenAtlas

Today we have a Kubernetes meeting

Today I did some cleanup at the ACDH-CH GitLab repository. I deleted the main and develop branch and uploaded them again because, like we discussed, Kubernetes specific changes should only be made in the feature_kubernetes branch. Once everything is working with Kubernetes we may look into merging it to main but have to do this carefully to be sure to not put our productive systems at risk, e.g. now it seems like passwords may be leaked, more about that further down.

• A SQL dump was added to the Git repository by Berni (install/demo-dev-dump.sql) which shouldn't be part of the OpenAtlas repository so please clean this up (e.g. delete and put in .gitignore).
• Dalibor removed the instance/prodution.py from .gitignore, presumably to change configuration. This is not how it is supposed to work, e.g. passwords are saved in instance/production.py. If persistent changes are needed in the repository they should be made in config/default.py or, if they are Kubernetes specific we should add a config/kubernetes.py.

• fixing .gitignore to ignore instance/ again except specified files (see .gitignore in main branch for how it should be)
• removing database dump from repository
• merge develop to feature_kubernetes to be in sync with current development

However, it's great to see that you managed to get already something running at https://demo-acdh-ch.openatlas.eu/.

Dalibor removed the instance/prodution.py from .gitignore, presumably to change configuration. This is not how it is supposed to work, e.g. passwords are saved in instance/production.py. If persistent changes are needed in the repository they should be made in config/default.py or, if they are Kubernetes specific we should add a config/kubernetes.py.

The openatlas app must be reconfigured that it can check if variables introduced over instance/prodution.py in feature_kubernetes are present as env variables.
They have to be introduced as Gitlab CI/CD variables for feature_kubernetes branch. By default, Openatlas expects that variables are hard-coded in the instance/prodution.py.
We can return instance/prodution.py to .gitignore after we adjust Openatlas that it can take variables from the "os.environ"

DATABASE_NAME = os.environ.get('POSTGRES_DB')              
DATABASE_USER = os.environ.get('POSTGRES_USER') 
DATABASE_PASS = os.environ.get('POSTGRES_PASSWORD') 
DATABASE_HOST = os.environ.get('POSTGRES_HOST') 
DATABASE_PORT = os.environ.get('POSTGRES_PORT')
DATABASE_URL = os.environ.get('DATABASE_URL')
SECRET_KEY = os.environ.get('SECRET_KEY')

New domain https://discovery-demo-acdh-ch.openatlas.eu is created, pointed to the cluster and can be used for deployment.

The problem with the production.py is solved and production.py is returned to the .gitignore.
All Kubernetes related configuration is introduced over requirements.txt, Procfile, start.sh and Gitlab CI/CD evironment variables.
Only these three new files are added to the default Openatlas repo, and they are located in the root of the feature_kubernetes branch.

In order to merge the develop with the feature_kubernetes branch, we just need to remove install/demo-dev*.sql dumps.

Thanks a lot Dalibor for solving this. I really hope we see you at our OpenAtlas Summer Meeting next week. Would be nice to see in person again and your first drinks will be on us for all your help :)

Helm chart now should support external database as well as a database installed next to OpenAtlas. The docker-k8s feature branch at the moment deploys to a separate postgis database deployed on Rancher.
There is a pgadmin4 instance (credentials can be found in Rancher) to maintain database on this "external" PostgreSQL server.
We now need to test if it deploys correctly with the "embedded" PostgreSQL server. I am quite sure it should.
That should finish the Helm Chart tasks
I think we have a Workable demo version as well as a version that can be combined with OpenAtlas-Discovery.

Daily security updates:
This can be achieved using github actions cron jobs. We probably should make sure a new version of the container image is not published if there is no updates.

Backup:
We need to create K8s cron jobs for this an add them to the helm chart. That should be a short task.

There is a (new) permission setting in the actions yaml which has to state that a job wants to write to the container registry.

When forking actions are usually disabled. So no testing is done at all in a forked repository at first.

Deployment is now guarded with an if in acdh-oeaw clause.
The rest of the actions can run in other orgs or personal gh namespaces.

Thank you, Omar!
Working on first demo version, have some problems with URL. (Going for https://openatlas-demo.acdh-dev.arz.oeaw.ac.at/)

• GitHub schedule: Make a new GitHub action file to pull build, check if there are updates available. If updates are available, rebuild everything using workflow_dispatch: {}
• Create cron job to save backups (files and database dumps) to the backup samba share

• https://openatlas-demo.acdh-ch-dev.oeaw.ac.at (publicly accessible)
• https://openatlas-demo.acdh-cluster-2.arz.oeaw.ac.at (private)

file management on k8s probably with WebDAV

Just because Omar asked about the status the other day: we still need to test it in combination with a presentation site.
Once implemented please also add an entry with a link to the ACDH-CH demo presentation site to the "online sites list".

Apache access log example JSON:

    "remote-address": tokens["remote-addr"](req, res),
    time: tokens["date"](req, res, "iso"),
    method: tokens["method"](req, res),
    url: tokens["url"](req, res),
    "http-version": tokens["http-version"](req, res),
    "status-code": tokens["status"](req, res),
    "content-length": tokens["res"](req, res, "content-length"),
    referrer: tokens["referrer"](req, res),
    "user-agent": tokens["user-agent"](req, res)