- Table of contents
- Current issues
- Tasks to automate
- Update with new OpenAtlas releases
- Add Berni to the ACDH GitHub list
- Test stage
- Uploading a 14 MB image fails with 413 Request Entity Too Large although limit is set to 20 MB (in OpenAtlas) 2021-10-04
- Adapt backup procedures to daily, weekly, quarterly and test it
- Presentation site with OpenAtlas discovery. For the ACDH-CH demo we will use https://discovery-demo-acdh-ch.openatlas.eu
- INDIGO specific: connect to ARCHE file folders (has to be cleared with ARCHE team before)
Once it runs properly there are adaptions we want implemented before using it on productive systems that are maintained by the OpenAtlas team.
Debian packages instead installation over pip¶Although pip offers some advantages (e.g more current packages, usable for different installation environment) we prefer packages from the Debian repository.
Main reasons concern security, reliability and stability. Only exception are frontend libraries because they don't really affect server security.
- There should be automated daily security updates for Debian packages in place
Remove the need for extra branches for each instance¶
One reason for the extra branches seems to be setting of environment variables (e.g. password for database). We should find a different solution because using identical copies of branches disturbs our development workflow.
Another reason is to be able to control which instances are updated, we should keep this but find a different solution for it.
Tasks to automate¶
Once we got it all running we can look into automating tasks.
Debian package updates¶
Implement a mechanism for software updates with new/different packages
Database updates¶Most new versions also include database changes. Here are some thoughts about automating them:
- Update SQL scripts are already provided in install/upgrade named after the specific version e.g. 6.0.0.sql. Upgrades of former major versions are stored in install/upgrade/archive.
- We would need a mechanism that has to work outside/before the main application because changes may interfere with the initialization process in openatlas/__init__.py e.g. all types are loaded in the before_request() function -> Since every update will make new setup, I don't think, this will be a problem.
- In case that the database update, which runs in a transaction, fails the software shouldn't be updated but we need the software update to get the upgrade file. Maybe Kubernetes can help there, e.g. abort the update all together. -> If the transaction fails, the pipeline should fail and therefore not be deployed.
- The version update process should be "aware" if an update SQL is needed. The application code "knows" it's version, it's tracked as VERSION in config/default.py. To make the database aware of it's version we could add a value in the web/settings table but we still need functionality that checks if, which and in what order upgrade scripts are needed and can deal with failed update SQLs. -> We need to make the database aware of it's version in order properly check if a database exists and needs updates.
Packages¶We use Pipfile for now with Python and Heroku. Heroku will take care of security updates.
To add a new package:
pipenv install <package==version>
To update pipfile.lock:
- More on Pipfile: https://docs.pipenv.org/ https://github.com/pypa/pipfile
- How Heroku works: https://devcenter.heroku.com/articles/how-heroku-works
- What is Heroku: https://stackoverflow.com/tags/heroku/info
- Python Dependencies via Pip: https://devcenter.heroku.com/articles/python-pip
The folder openatlas/uploads, openatlas/export and openatlas/image_processing will contain files which users uploaded or generated. Therefore, additional volumes has to be created with the mount point /app/openatlas/<folder> (/app/openatlas/uploads) and read-only false.
Update with new OpenAtlas releases¶
- For Python packages Heroku/Kubernetes will use the Pipfile
- Update of npm packages is done via start.sh
- Database updates are done manually
Backups are made every day at 03:20 to a separate volume, which can be mounted to the container. At the moment only the last 10 Backups are kept.