Actions

Copy link

Feature #2343

open

Update OpenAtlas software to Debian 13 (Trixie)

Added by Alexander Watzinger over 1 year ago. Updated 1 day ago.

Status:

In Progress

Priority:

Normal

Assignee:

Bernhard Koschiček-Krombholz

Category:

Backend

Target version:

9.0.0

Start date:

2024-09-03

Estimated time:

Description

Debian, the standard installation system for OpenAtlas, released the new version 13 trixie.
This issue provides information about the update and the new software versions being used with OpenAtlas, see comments for more details.
Branch: feature/trixie

Python from 3.11.2 to 3.13.5 (changelog 3.12, 3.13)
Flask from 2.2.2 to 3.1.1 (changelogs)
PostgreSQL from 15 to 17 (changelogs)
Jinja2 from 3.1.2 to 3.1.6 (changelogs)

Compact instructions for upgrading Debian 12 to 13 (trixie), use at your own risk, see notes below for more details.

Backup everything, and specific for OpenAtlas:
- Your databases, e.g. with pg_dumpall for all PostgreSQL databases
- The folders at files/uploads for every instance

Update current system

sudo apt update && sudo apt full-upgrade -y

Change all occurrences of bookworm to trixie in /etc/apt/sources
```
sudo apt edit-sources
```

Upgrade - when prompted to upgrade PostgreSQL, choose No – the database will be upgraded manually later.

sudo apt update && sudo apt upgrade --without-new-pkgs
sudo apt update && sudo apt upgrade --with-new-pkgs
sudo apt update && sudo apt dist-upgrade

Clean up

sudo apt --purge autoremove -y
sudo apt autoclean

Reboot the system

PostgreSQL cluster upgrade

sudo apt install postgresql-postgis
sudo systemctl stop postgresql
sudo pg_dropcluster 17 main
sudo pg_upgradecluster 15 main
sudo pg_dropcluster 15 main
sudo systemctl start postgresql

As postgres user:

reindexdb --all

Related issues 2 (2 open — 0 closed)

	Related to OpenAtlas - Feature #2464: OpenAtlas major version 9.0.0	In Progress	Alexander Watzinger	2025-01-24			Actions
	Precedes OpenAtlas - Feature #2093: API: New and faster object serialization	Acknowledged		2024-09-04			Actions

Issue # Delay: days Cancel

History
Notes
Property changes

Actions

Copy link

Updated by Alexander Watzinger 6 months ago

Precedes Feature #2093: API: New and faster object serialization added

Actions

Copy link

Updated by Bernhard Koschiček-Krombholz 2 months ago

Work in Progress Upgrade List¶

Major package upgrades:

flask (*)
flask-babel (*)
pandas
flask-cors
rdflib (*)
email-validator
shapley (*)
pylint

(*) will definitively break OpenAtlas 8.x.x instances

Debian 12 → 13 (Bookworm → Trixie) Upgrade Notes¶

Previous upgrade issue: #2038

Steps Performed¶

Update current system¶

sudo apt update && sudo apt full-upgrade -y

Adjust APT sources¶

sudo apt edit-sources

Change all occurrences of bookworm to trixie.

Refresh and perform safe upgrade¶

sudo apt update && sudo apt upgrade --without-new-pkgs

When prompted by rkhunter, choose N (keep installed settings).

Continue upgrade¶

sudo apt update && sudo apt upgrade --with-new-pkgs

When prompted to upgrade PostgreSQL, choose No – the database will be upgraded manually later.

Clean up unused packages¶

sudo apt --purge autoremove -y
sudo apt autoclean

Reboot system after completion¶

sudo reboot

PostgreSQL cluster upgrade¶

sudo apt install postgresql-postgis
sudo systemctl stop postgresql
sudo pg_dropcluster 17 main
sudo pg_upgradecluster 15 main
sudo pg_dropcluster 15 main
sudo systemctl start postgresql

Get postgres user and reindex each database:

psql <database>
REINDEX DATABASE;

Or re-index all databases at once:

reindexdb --all

In case of collation warnings you could also call this statement:

ALTER DATABASE <database> REFRESH COLLATION VERSION;

Package Version Map (Debian 12 → Debian 13)¶

Package	Version in Debian 12 (bookworm)	Version in Debian 13 (trixie)
python3	3.11.2	3.13.5
python3-bcrypt	3.2.2	4.2.0
python3-dateutil	2.8.2	2.9.0
python3-psycopg2	2.9.5	2.9.10
python3-fuzzywuzzy	0.18.0	0.18.0
python3-flask	2.2.2	3.1.1
python3-flask-babel	2.0.0	4.1.0
python3-flask-login	0.6.2	0.6.3
python3-flaskext.wtf	1.1.1	1.2.2
python3-markdown	3.4.1	3.7
python3-numpy	1:1.24.2	1:2.2.4
python3-pandas	1.5.3	2.2.3
python3-jinja2	3.1.2	3.1.6
python3-flask-cors	3.0.10	6.0.1
python3-flask-restful	0.3.9	0.3.10
p7zip-full	16.02	16.02
python3-wand	0.6.11	0.6.13
python3-rdflib	6.1.1	7.1.1
python3-dicttoxml	1.7.15	1.7.16
python3-rdflib-jsonld	0.6.1	now included in python3-rdflib
python3-flasgger	0.9.5	0.9.7.2
python3-requests	2.28.1	2.32.3
exiftran	2.10	2.14
python3-email-validator	1.3.0	2.2.0
python3-svgwrite	1.4.3	1.4.3
python3-shapely	1.8.5	2.1.0
python3-xmltodict	0.13.0	0.13.0
python3-validators	0.20.0	0.20.0
python3-jwt	2.6.0	2.10.1
python3-python-flask-jwt-extended	4.4.4	4.7.1
python3-bs4	4.11.2	4.13.4
apache2	2.4.65	2.4.65
libapache2-mod-wsgi-py3	4.9.4	5.0.2
gettext	0.21	0.23.1
npm	9.2.0	9.2.0
postgresql	15.14	17.6
postgresql-15/17-postgis-3	3.3.2	3.5.2
postgresql-15/17-postgis-3-scripts	3.3.2	3.5.2
iipimage-server	1.1	1.1
libvips-tools	8.14.1	8.16.1
libapache2-mod-fcgid	1:2.3.9-4	1:2.3.9-4
python3-coverage	6.5.0	7.6.0
python3-pytest	6.5.0	8.3.5
mypy	1.0.1	1.15.0
pylint	2.16.2	3.3.4

Reference¶

Upgrade guide: https://fullmetalbrackets.com/blog/upgrade-debian-12-bookworm-debian-13-trixie/

Actions

Copy link

Updated by Alexander Watzinger 2 months ago

Status changed from Acknowledged to In Progress
Assignee set to Alexander Watzinger

Thanks a lot @Bernhard Koschiček-Krombholz for testing and providing detailed information.
Like discussed this will be also part of the major 9.0.0 release.

Actions

Copy link

Updated by Alexander Watzinger 2 months ago

Related to Feature #2464: OpenAtlas major version 9.0.0 added

Actions

Copy link

Updated by Bernhard Koschiček-Krombholz about 1 month ago

You need to REINDEX each database with:

REINDEX DATABASE <database>;

And I just removed the @babel.localeselector from init.py line 39, and added locale_selector to Babel (Babel(app, locale_selector=get_locale))

There are still issues, but to get it running and basically working, just this was needed.

A added a branch: feature/trixie

Actions

Copy link

Updated by Bernhard Koschiček-Krombholz about 1 month ago

Description updated (diff)

I added LazyString (babel) for type checking to some functions, and tests are now at 100% coverage. (one test is not 100% correct, I left a Todo there)

There are a lot of mypy issues with the forms, because they expect a str and not LazyString. It can be handled with str(), but maybe there is a sophisticated method to do this.

Actions

Copy link

Updated by Alexander Watzinger about 1 month ago

Thanks a lot for this. You can leave the Mypy stuff to me. Except for very API specific stuff which I may not dare to touch.
Anyway, it would be ok to leave some Mypy issues for the 9.1.0 release, although a nice to have, there are more important things.

Actions

Copy link

Updated by Alexander Watzinger 4 days ago

Description updated (diff)

Actions

Copy link

#10

Updated by Alexander Watzinger 4 days ago

Thanks to @Bernhard Koschiček-Krombholz for preparing the transition. I upgraded my ACDH machine today and it worked good so far. Only issue I had was with the IIIF server which needed:

cp -p /usr/lib/iipimage-server/iipsrv.fcgi /var/www/iipsrv/

Not sure if this was only at this machine or is needed for the upgrade in general (and should be added to your install instruction note). Also:

I merged feature/new_classes to feature/trixie which will now be the main develop branch for the next version
There were multiple merge conflicts but I hope I solved all of them correctly
I added an SQL command to re-index all databases and a collation SQL fix to your install note
I managed to solve all Mypy issues and will look into the Pylint issues next

Actions

Copy link

#11

Updated by Alexander Watzinger 3 days ago

Description updated (diff)
Assignee changed from Alexander Watzinger to Bernhard Koschiček-Krombholz

@Bernhard Koschiček-Krombholz:
I think I'm finished with this issue and assigned it to you. You did the most work anyway and I also spotted TODOs in your notes so feel free to close it anytime.

I put a compact server upgrade instruction in the description to make it more accessible. In case we find more we should update it there.

About Pylint:
I solved all errors and warnings and was surprised that afterwards only 28 refactoring issues remained (they were about 80 before).
So I dug into the changelogs and it seems they solved a lot of false positives in the last 2 years.
Although it doesn't change anything in functionality, it definitely feels a lot better now :)

Actions

Copy link

#12