Project

General

Profile

Actions

Feature #2343

open

Update OpenAtlas software to Debian 13 (Trixie)

Added by Alexander Watzinger over 1 year ago. Updated about 22 hours ago.

Status:
In Progress
Priority:
Normal
Category:
Backend
Target version:
Start date:
2024-09-03
Estimated time:

Description

Debian, the standard installation system for OpenAtlas, released the new version 13 trixie.
This issue provides information about the update and the new software versions being used with OpenAtlas, see comments for more details.
Branch: feature/trixie Compact instructions for upgrading Debian 12 to 13 (trixie), use at your own risk, see notes below for more details.
  • Backup everything, and specific for OpenAtlas:
    • Your databases, e.g. with pg_dumpall for all PostgreSQL databases
    • The folders at files/uploads for every instance
  • Update current system
    sudo apt update && sudo apt full-upgrade -y
    
  • Change all occurrences of bookworm to trixie in /etc/apt/sources
    sudo apt edit-sources
    
  • Upgrade - when prompted to upgrade PostgreSQL, choose No – the database will be upgraded manually later.
    sudo apt update && sudo apt upgrade --without-new-pkgs
    sudo apt update && sudo apt upgrade --with-new-pkgs
    sudo apt update && sudo apt dist-upgrade
    
  • Clean up
    sudo apt --purge autoremove -y
    sudo apt autoclean
    
  • Reboot the system
  • PostgreSQL cluster upgrade
    sudo apt install postgresql-postgis
    sudo systemctl stop postgresql
    sudo pg_dropcluster 17 main
    sudo pg_upgradecluster 15 main
    sudo pg_dropcluster 15 main
    sudo systemctl start postgresql
    
    As postgres user:
    reindexdb --all

Related issues 2 (2 open0 closed)

Related to OpenAtlas - Feature #2464: OpenAtlas major version 9.0.0In ProgressAlexander Watzinger2025-01-24Actions
Precedes OpenAtlas - Feature #2093: API: New and faster object serializationAcknowledged2024-09-04Actions
Actions #2

Updated by Alexander Watzinger 6 months ago

  • Precedes Feature #2093: API: New and faster object serialization added
Actions #3

Updated by Bernhard Koschiček-Krombholz 2 months ago

Work in Progress Upgrade List

Major package upgrades:
  • flask (*)
  • flask-babel (*)
  • pandas
  • flask-cors
  • rdflib (*)
  • email-validator
  • shapley (*)
  • pylint

(*) will definitively break OpenAtlas 8.x.x instances

Debian 12 → 13 (Bookworm → Trixie) Upgrade Notes

Previous upgrade issue: #2038


Steps Performed

Update current system

sudo apt update && sudo apt full-upgrade -y

Adjust APT sources

sudo apt edit-sources

Change all occurrences of bookworm to trixie.

Refresh and perform safe upgrade

sudo apt update && sudo apt upgrade --without-new-pkgs

When prompted by rkhunter, choose N (keep installed settings).

Continue upgrade

sudo apt update && sudo apt upgrade --with-new-pkgs

When prompted to upgrade PostgreSQL, choose No – the database will be upgraded manually later.

Clean up unused packages

sudo apt --purge autoremove -y
sudo apt autoclean

Reboot system after completion

sudo reboot

PostgreSQL cluster upgrade

sudo apt install postgresql-postgis
sudo systemctl stop postgresql
sudo pg_dropcluster 17 main
sudo pg_upgradecluster 15 main
sudo pg_dropcluster 15 main
sudo systemctl start postgresql

Get postgres user and reindex each database:

psql <database>
REINDEX DATABASE;

Or re-index all databases at once:

reindexdb --all

In case of collation warnings you could also call this statement:

ALTER DATABASE <database> REFRESH COLLATION VERSION;


Package Version Map (Debian 12 → Debian 13)

Package Version in Debian 12 (bookworm) Version in Debian 13 (trixie)
python3 3.11.2 3.13.5
python3-bcrypt 3.2.2 4.2.0
python3-dateutil 2.8.2 2.9.0
python3-psycopg2 2.9.5 2.9.10
python3-fuzzywuzzy 0.18.0 0.18.0
python3-flask 2.2.2 3.1.1
python3-flask-babel 2.0.0 4.1.0
python3-flask-login 0.6.2 0.6.3
python3-flaskext.wtf 1.1.1 1.2.2
python3-markdown 3.4.1 3.7
python3-numpy 1:1.24.2 1:2.2.4
python3-pandas 1.5.3 2.2.3
python3-jinja2 3.1.2 3.1.6
python3-flask-cors 3.0.10 6.0.1
python3-flask-restful 0.3.9 0.3.10
p7zip-full 16.02 16.02
python3-wand 0.6.11 0.6.13
python3-rdflib 6.1.1 7.1.1
python3-dicttoxml 1.7.15 1.7.16
python3-rdflib-jsonld 0.6.1 now included in python3-rdflib
python3-flasgger 0.9.5 0.9.7.2
python3-requests 2.28.1 2.32.3
exiftran 2.10 2.14
python3-email-validator 1.3.0 2.2.0
python3-svgwrite 1.4.3 1.4.3
python3-shapely 1.8.5 2.1.0
python3-xmltodict 0.13.0 0.13.0
python3-validators 0.20.0 0.20.0
python3-jwt 2.6.0 2.10.1
python3-python-flask-jwt-extended 4.4.4 4.7.1
python3-bs4 4.11.2 4.13.4
apache2 2.4.65 2.4.65
libapache2-mod-wsgi-py3 4.9.4 5.0.2
gettext 0.21 0.23.1
npm 9.2.0 9.2.0
postgresql 15.14 17.6
postgresql-15/17-postgis-3 3.3.2 3.5.2
postgresql-15/17-postgis-3-scripts 3.3.2 3.5.2
iipimage-server 1.1 1.1
libvips-tools 8.14.1 8.16.1
libapache2-mod-fcgid 1:2.3.9-4 1:2.3.9-4
python3-coverage 6.5.0 7.6.0
python3-pytest 6.5.0 8.3.5
mypy 1.0.1 1.15.0
pylint 2.16.2 3.3.4

Reference

Upgrade guide: https://fullmetalbrackets.com/blog/upgrade-debian-12-bookworm-debian-13-trixie/

Actions #4

Updated by Alexander Watzinger 2 months ago

  • Status changed from Acknowledged to In Progress
  • Assignee set to Alexander Watzinger

Thanks a lot @Bernhard Koschiček-Krombholz for testing and providing detailed information.
Like discussed this will be also part of the major 9.0.0 release.

Actions #5

Updated by Alexander Watzinger 2 months ago

Actions #6

Updated by Bernhard Koschiček-Krombholz about 1 month ago

You need to REINDEX each database with:

REINDEX DATABASE <database>;

And I just removed the @babel.localeselector from init.py line 39, and added locale_selector to Babel (Babel(app, locale_selector=get_locale))

There are still issues, but to get it running and basically working, just this was needed.

A added a branch: feature/trixie

Actions #7

Updated by Bernhard Koschiček-Krombholz about 1 month ago

  • Description updated (diff)

I added LazyString (babel) for type checking to some functions, and tests are now at 100% coverage. (one test is not 100% correct, I left a Todo there)

There are a lot of mypy issues with the forms, because they expect a str and not LazyString. It can be handled with str(), but maybe there is a sophisticated method to do this.

Actions #8

Updated by Alexander Watzinger about 1 month ago

Thanks a lot for this. You can leave the Mypy stuff to me. Except for very API specific stuff which I may not dare to touch.
Anyway, it would be ok to leave some Mypy issues for the 9.1.0 release, although a nice to have, there are more important things.

Actions #9

Updated by Alexander Watzinger 4 days ago

  • Description updated (diff)
Actions #10

Updated by Alexander Watzinger 4 days ago

Thanks to @Bernhard Koschiček-Krombholz for preparing the transition. I upgraded my ACDH machine today and it worked good so far. Only issue I had was with the IIIF server which needed:
cp -p /usr/lib/iipimage-server/iipsrv.fcgi /var/www/iipsrv/

Not sure if this was only at this machine or is needed for the upgrade in general (and should be added to your install instruction note). Also:
  • I merged feature/new_classes to feature/trixie which will now be the main develop branch for the next version
  • There were multiple merge conflicts but I hope I solved all of them correctly
  • I added an SQL command to re-index all databases and a collation SQL fix to your install note
  • I managed to solve all Mypy issues and will look into the Pylint issues next
Actions #11

Updated by Alexander Watzinger 3 days ago

  • Description updated (diff)
  • Assignee changed from Alexander Watzinger to Bernhard Koschiček-Krombholz

@Bernhard Koschiček-Krombholz:
I think I'm finished with this issue and assigned it to you. You did the most work anyway and I also spotted TODOs in your notes so feel free to close it anytime.

I put a compact server upgrade instruction in the description to make it more accessible. In case we find more we should update it there.

About Pylint:
I solved all errors and warnings and was surprised that afterwards only 28 refactoring issues remained (they were about 80 before).
So I dug into the changelogs and it seems they solved a lot of false positives in the last 2 years.
Although it doesn't change anything in functionality, it definitely feels a lot better now :)

Actions #12

Updated by Alexander Watzinger 2 days ago

  • Description updated (diff)
Actions #13

Updated by Alexander Watzinger 2 days ago

  • Description updated (diff)
Actions #14

Updated by Alexander Watzinger 2 days ago

  • Description updated (diff)
Actions #15

Updated by Alexander Watzinger 2 days ago

  • Description updated (diff)
Actions #16

Updated by Alexander Watzinger 2 days ago

  • Subject changed from Update Debian to 13 (Trixie) to Update Debian to 13 (Trixie)
  • Description updated (diff)
Actions #17

Updated by Alexander Watzinger about 22 hours ago

  • Subject changed from Update Debian to 13 (Trixie) to Update OpenAtlas software to Debian 13 (Trixie)
Actions

Also available in: Atom PDF