Project

General

Profile

Actions

Issue #1542

Current issues

  • Add Berni to the ACDH GitHub list
  • Test stage
  • Uploading a 14 MB image fails with 413 Request Entity Too Large although limit is set to 20 MB (in OpenAtlas) 2021-10-04
  • Interface got messed up because JavaScript and CSS couldn't be loaded anymore (500). Redeploy on Rancher didn't help. A new build with gitlab pipeline wiped the database clean, although the code in start.sh should have prevented that. 2021-10-04
  • Adapt backup procedures to daily, weekly, quarterly and test it
  • Presentation site with OpenAtlas discovery. For the ACDH-CH demo we will use https://discovery-demo-acdh-ch.openatlas.eu
  • INDIGO specific: connect to ARCHE file folders (has to be cleared with ARCHE team before)

Issues

Once it runs properly there are adaptions we want implemented before using it on productive systems that are maintained by the OpenAtlas team.

Debian packages instead installation over pip

Although pip offers some advantages (e.g more current packages, usable for different installation environment) we prefer packages from the Debian repository.
Main reasons concern security, reliability and stability. Only exception are frontend libraries because they don't really affect server security.
  • There should be automated daily security updates for Debian packages in place

Remove the need for extra branches for each instance

One reason for the extra branches seems to be setting of environment variables (e.g. password for database). We should find a different solution because using identical copies of branches disturbs our development workflow.
Another reason is to be able to control which instances are updated, we should keep this but find a different solution for it.

Tasks to automate

Once we got it all running we can look into automating tasks.

Debian package updates

Implement a mechanism for software updates with new/different packages

Database updates

Most new versions also include database changes. Here are some thoughts about automating them:
  • Update SQL scripts are already provided in install/upgrade named after the specific version e.g. 6.0.0.sql. Upgrades of former major versions are stored in install/upgrade/archive.
  • We would need a mechanism that has to work outside/before the main application because changes may interfere with the initialization process in openatlas/__init__.py e.g. all types are loaded in the before_request() function -> Since every update will make new setup, I don't think, this will be a problem.
  • In case that the database update, which runs in a transaction, fails the software shouldn't be updated but we need the software update to get the upgrade file. Maybe Kubernetes can help there, e.g. abort the update all together. -> If the transaction fails, the pipeline should fail and therefore not be deployed.
  • The version update process should be "aware" if an update SQL is needed. The application code "knows" it's version, it's tracked as VERSION in config/default.py. To make the database aware of it's version we could add a value in the web/settings table but we still need functionality that checks if, which and in what order upgrade scripts are needed and can deal with failed update SQLs. -> We need to make the database aware of it's version in order properly check if a database exists and needs updates.

Installation

Packages

We use Pipfile for now with Python and Heroku. Heroku will take care of security updates.
To add a new package:
pipenv install <package==version>

To update pipfile.lock:
pipenv update

File folders

The folder openatlas/uploads, openatlas/export and openatlas/image_processing will contain files which users uploaded or generated. Therefore, additional volumes has to be created with the mount point /app/openatlas/<folder> (/app/openatlas/uploads) and read-only false.

Update with new OpenAtlas releases

  • For Python packages Heroku/Kubernetes will use the Pipfile
  • Update of npm packages is done via start.sh
  • Database updates are done manually

Backup

Backups are made every day at 03:20 to a separate volume, which can be mounted to the container. At the moment only the last 10 Backups are kept.

Updated by Bernhard Koschiček-Krombholz 17 days ago · 36 revisions

Also available in: PDF HTML TXT