Project

General

Profile

Actions

Feature #2464

open

OpenAtlas major version 9.0.0

Added by Alexander Watzinger 11 months ago. Updated 3 days ago.

Status:
In Progress
Priority:
Normal
Category:
Backend
Target version:
Start date:
2025-01-24
Estimated time:
80.00 h

Description

In this new major version the underlying class functionality was completely rewritten to use model config files which contain all the information to build views, forms and similar in a generic way. Advantages are:
  • Easier model adaptions at one place
  • More consistent functionality and layout, e.g. table headers, form fields order, ...
  • Possibility to test for "allowed" relations
  • Less error prone and fewer tests are needed
  • See below for more detailed information

Update 22. December 2025

The code base has been rewritten to use the config files approach.
There is still a lot to do e.g. solving more complex class relations, rewriting the documentation and of course a lot of testing but it looks like it can be released around New Year.
Current development is done in the feature/trixie branch which is already based on Debian 13, see #2343

New features

  • Human remains can now also be used with Modification and Move
  • All possible model relations and found invalid relations are listed at the model view

UI improvements

The new system offers a lot of UI improvements all over, some of them are listed here.
  • Forms
    • Profile images (if available) are also shown at update forms
  • Actor
    • A map is shown at the actor detail view if e.g. a georeferenced residence is available
    • Relations can be created from every direction (was difficult for groups before)
  • Event
    • A preceding event can be inserted and added dynamically at an event form
    • Sub events can also be linked/created via the subs tab of an event detail view
  • File
    • File previews (if available) are now always shown in file tables. The option to toggle it was removed because it was often overlooked by users.
  • Reference system
    • Linked entities can be removed in related tabs of the reference system detail view
  • Source
    • The source index now also shows source translations
    • Features and stratigraphic units can be inserted/linked directly via the place tab at the source detail view too
    • The source of a source translation can be changed when updating a translation
    • Source translations annotations can be linked to entities of the source at insert too (before it was only possible at update)
  • Type
    • Types are linked in table views
    • Sub types can also be added at the type detail view
    • Types can also be removed in the entities tab at the type detail view (if not set required)
  • Admin
    • Easier data integrity checks with joining all link checks

Minor fixes

  • Tabs without content aren't shown anymore
  • Removed insert and continue buttons when inserting and linking with additional fields (which weren't working anyway)
  • Fixed subs at administrative units detail view also showing up in the entity tab
  • Fixed non selectable types not selectable as super at dynamic type add in forms
  • Check similar names: fixed already found entities showing up multiple times

Refactor

  • Use of Python data classes

Related issues 9 (8 open1 closed)

Related to OpenAtlas - Feature #2343: Update OpenAtlas software to Debian 13 (Trixie)ClosedBernhard Koschiček-Krombholz2024-09-03Actions
Precedes OpenAtlas - Feature #2448: Manual: documentation for developersAcknowledged2025-01-27Actions
Precedes OpenAtlas - Feature #2505: Make admin. units/historical places a custom typeAcknowledged2025-01-27Actions
Precedes OpenAtlas - Feature #2465: Check for required types at dynamic entity addAcknowledged2025-01-27Actions
Precedes OpenAtlas - Feature #2436: Structuring placesAcknowledged2025-01-27Actions
Precedes OpenAtlas - Feature #2600: Add external references to referencesAcknowledged2025-01-27Actions
Precedes OpenAtlas - Feature #2300: Load form tables dynamicallyIn ProgressKatharina Wünsche2025-01-27Actions
Precedes OpenAtlas - Feature #2642: CSV importer - selectable, inverse and references to vocabulariesAcknowledged2025-01-27Actions
Precedes OpenAtlas - Feature #2676: JSON importAcknowledged2025-01-27Actions
Actions #2

Updated by Alexander Watzinger 8 months ago

  • Target version changed from 9.0.0 to 8.12.0
Actions #3

Updated by Alexander Watzinger 8 months ago

  • Status changed from Acknowledged to In Progress
  • Assignee set to Alexander Watzinger
Actions #4

Updated by Alexander Watzinger 8 months ago

  • Precedes Feature #2448: Manual: documentation for developers added
Actions #5

Updated by Alexander Watzinger 7 months ago

  • Target version changed from 8.12.0 to 8.15.0
Actions #6

Updated by Alexander Watzinger 7 months ago

  • Description updated (diff)
Actions #8

Updated by Alexander Watzinger 7 months ago

  • Estimated time set to 80.00 h
Actions #9

Updated by Alexander Watzinger 6 months ago

  • Description updated (diff)
Actions #10

Updated by Alexander Watzinger 6 months ago

  • Description updated (diff)
Actions #11

Updated by Alexander Watzinger 6 months ago

  • Description updated (diff)
Actions #12

Updated by Alexander Watzinger 6 months ago

  • Precedes Feature #2505: Make admin. units/historical places a custom type added
Actions #13

Updated by Alexander Watzinger 6 months ago

  • Precedes Feature #2465: Check for required types at dynamic entity add added
Actions #14

Updated by Alexander Watzinger 6 months ago

Actions #15

Updated by Alexander Watzinger 6 months ago

  • Description updated (diff)

This feature will take a long time (some weeks) depending on my availability and other factors.
Because other features depend on it, I will try to give a sense about the status of this issue.

  • Using the feature/new_classes branch I added config files in /config which are meant to control all aspects of the display and form functions
  • I began with Source: the detail view already works and the insert form is shown correctly
Next steps
  • Insert a source
  • Update a source
  • Do this with all other classes - this will take the most time because I have to generalize all the special cases
  • A lot of testing and clean up

Once the tests pass it will be merged into develop. Although it won't be release ready at this point we decided to do it anyway e.g. to prevent massive merge conflicts with other features.
Additionally the development of other features depending on it can be started, which will also help to further test the changes.

Actions #16

Updated by Alexander Watzinger 5 months ago

  • Description updated (diff)

I finished with the Source class for view, insert, update.
Because generalizing the new system for all classes will take the most time, I added a status per class section in the description as a progress overview.

Actions #17

Updated by Alexander Watzinger 5 months ago

  • Description updated (diff)
Actions #18

Updated by Alexander Watzinger 5 months ago

  • Description updated (diff)
Actions #19

Updated by Alexander Watzinger 5 months ago

  • Description updated (diff)
Actions #20

Updated by Alexander Watzinger 5 months ago

  • Description updated (diff)
Actions #21

Updated by Alexander Watzinger 5 months ago

  • Description updated (diff)
Actions #22

Updated by Alexander Watzinger 5 months ago

  • Description updated (diff)
Actions #23

Updated by Alexander Watzinger 5 months ago

  • Description updated (diff)
Actions #24

Updated by Alexander Watzinger 4 months ago

  • Target version changed from 8.15.0 to 9.2.0
Actions #25

Updated by Alexander Watzinger 4 months ago

  • Precedes Feature #2600: Add external references to references added
Actions #26

Updated by Alexander Watzinger 4 months ago

  • Description updated (diff)
Actions #27

Updated by Alexander Watzinger 4 months ago

  • Description updated (diff)
Actions #28

Updated by Alexander Watzinger 4 months ago

  • Description updated (diff)
Actions #29

Updated by Alexander Watzinger 4 months ago

  • Description updated (diff)
Actions #30

Updated by Alexander Watzinger 4 months ago

  • Description updated (diff)
Actions #31

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
Actions #32

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
Actions #33

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
Actions #34

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
Actions #35

Updated by Alexander Watzinger 3 months ago

Actions #36

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
Actions #37

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
Actions #38

Updated by Bernhard Koschiček-Krombholz 3 months ago

  • Description updated (diff)
Actions #39

Updated by Bernhard Koschiček-Krombholz 3 months ago

  • Description updated (diff)
Actions #40

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
  • Target version changed from 9.2.0 to 9.0.0
Actions #41

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
Actions #42

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
Actions #43

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
Actions #44

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
Actions #45

Updated by Alexander Watzinger 3 months ago

  • Precedes Feature #2642: CSV importer - selectable, inverse and references to vocabularies added
Actions #46

Updated by Alexander Watzinger 3 months ago

We had the great opportunity to show colleagues the new system, especially the configuration part, at an ACDH Python User Group meeting.
Feedback was positive, we received a lot of valuable suggestions and hints.
For the configuration we will most likely go into the direction of Python classes. This will replace the current dictionary approach which was kind of a placeholder anyway.

Actions #47

Updated by Alexander Watzinger 2 months ago

  • Description updated (diff)
Actions #48

Updated by Alexander Watzinger 2 months ago

  • Description updated (diff)
Actions #49

Updated by Alexander Watzinger 2 months ago

  • Subject changed from Refactor OpenAtlas classes to OpenAtlas major version 9.0.0
  • Description updated (diff)
Actions #50

Updated by Alexander Watzinger 2 months ago

  • Related to Feature #2343: Update OpenAtlas software to Debian 13 (Trixie) added
Actions #51

Updated by Alexander Watzinger 2 months ago

  • Description updated (diff)
Actions #52

Updated by Alexander Watzinger 2 months ago

  • Description updated (diff)
Actions #53

Updated by Alexander Watzinger 2 months ago

A minor update:

I finished with all classes, what's missing are type hierarchy functions, some more complex relations between classes and some special functions like displaying root places at entity tabs of type detail views.

So I thought it's time to reactivate the tests. I got the base functionality working, went through them and added a return # Todo: continue tests where more adaptions are needed.
With that I got a coverage of 64.47%, so it's still a way to go but at least we now have passing tests again to help catch major issues in current development. And they are much faster now ;)

Actions #54

Updated by Alexander Watzinger 2 months ago

  • Description updated (diff)
Actions #55

Updated by Alexander Watzinger 2 months ago

Type hierarchies are implemented and test coverage is at 85% (also thanks to @Bernhard Koschiček-Krombholz).

Actions #56

Updated by Alexander Watzinger 2 months ago

  • Description updated (diff)
Actions #57

Updated by Alexander Watzinger 2 months ago

Development is progressing well:
  • Copy and dynamic entity add (in forms) are working again
  • Removed obsolete code like former managers and unused functions
  • Continued with tests
    • We are now at 91% coverage
    • I added comments about reasons where they are still turned off
Actions #58

Updated by Alexander Watzinger about 2 months ago

  • Description updated (diff)
Actions #59

Updated by Alexander Watzinger about 2 months ago

Minor update: most of the type and admin checks are working again and we are at 92.81% coverage.

Actions #60

Updated by Bernhard Koschiček-Krombholz about 2 months ago

Files, image and import tests are working: 96.81%.

Actions #61

Updated by Alexander Watzinger about 2 months ago

  • Description updated (diff)
Actions #62

Updated by Alexander Watzinger about 2 months ago

  • Description updated (diff)
Actions #63

Updated by Alexander Watzinger about 2 months ago

  • Description updated (diff)

I got all tests working, code coverage is at 99,24% (most remaining missing at api/external/arche.py)

Next steps for me will be:
  • Refactor configuration to a more class based system
  • Moving some database class definition parameters to the config files for more flexibility
  • Begin working through the refactoring tasks that accumulated while rewriting the code base
Actions #64

Updated by Alexander Watzinger about 2 months ago

  • Description updated (diff)
Update:
  • Icon and layout class options were moved from the database to the config files
  • Install and upgrade SQLs are adapted. This version won't work without a database upgrade but it can now be done via the database upgrade script.
  • I updated demo-dev to 9.0.0 to test it out -> seems to work and the presentation site seems fine too (but I had to remove a class called 'bone' - hope it's not dramatic and we can discuss it next week)
  • OpenAtlas class overview is now working -> code coverage is at 99.31%
Actions #65

Updated by Alexander Watzinger about 1 month ago

  • Description updated (diff)
Actions #66

Updated by Alexander Watzinger about 1 month ago

Actions #67

Updated by Alexander Watzinger about 1 month ago

Update:
  • 100% code coverage again. 315 of 12,573 statements are excluded with nocover comment, so about 97.50% real coverage.
  • Most Mypy issues resolved (26 remaining)
  • I think I managed to solve Actor relations, feel free to test it at the development demo
Only remaining issues I'm aware of are with more complex relations:
  • Display at tabs, e.g. showing actors connected via events in place detail views
  • Populating forms connected via location to object_location (or the other way round) when linking entities
Actions #68

Updated by Alexander Watzinger about 1 month ago

I managed to solve the last remaining Mypy issues.
Beside being one step closer to the major release it also makes it easier to spot new Mypy issues in current development.

Actions #69

Updated by Alexander Watzinger about 1 month ago

  • Description updated (diff)
Actions #70

Updated by Alexander Watzinger about 1 month ago

  • Description updated (diff)
Actions #71

Updated by Alexander Watzinger 29 days ago

  • Description updated (diff)
Update
  • All possible model relations and found invalid relations are now listed at the model view (see: demo-dev)
    I tested it with MAMEMS and demo-dev data which contained no invalid relations (which was quite reassuring). But it is planned to also test other projects just to be sure about their data and to further test the function itself.
  • I checked for hard coded class conditions (e.g. if entity.class_.name == 'file') and found about:
    • 20 in the API (which I can't comment on)
    • 20 in the rest of the application. They mostly concern files, types and object locations which are quite special and I wouldn't know how to handle them differently.
      All in all I would say we were quite successful in making the application much more generic.
  • The issue for end dates in table columns is fixed
  • Pylint checks are now only reporting refactor issues (83 currently) which should make detection of new issues (especially errors, warnings, ...) in current development much easier.
Actions #72

Updated by Alexander Watzinger 29 days ago

  • Description updated (diff)
Actions #73

Updated by Alexander Watzinger 28 days ago

  • Description updated (diff)
Actions #74

Updated by Nina Richards 15 days ago

As already visible in the demo-dev version (https://demo-dev.openatlas.eu/entity/50625), tabs that do not yet contain any entries are grouped under the “+ Relation” button and can be populated from there. However, it is unclear whether “+ Relation” is the most appropriate label, as it may not be immediately intuitive for users. A possible alternative would be to rename this button “+ Add”.

Actions #75

Updated by Alexander Watzinger 8 days ago

  • Description updated (diff)
Actions #76

Updated by Alexander Watzinger 8 days ago

Next OpenAtlas major version is now developed in feature/trixie which is already based on Debian 13.

Actions #77

Updated by Alexander Watzinger 4 days ago · Edited

I looked into more detail into gettext functions which are used for our user interface translations.

gettext vs lazy_gettext

lazy_gettext is only needed when used in class definitions (e.g. form classes) that are not inside a function.
For everything else we should use gettext because of various minor issues with lazy_gettext otherwise, e.g. it seems PyCharm gets confused and gives a lot of type warnings (Mypy works just fine).
I already re-written everything accordingly and put it online on demo + demo_dev. Please report if you notice anything (keeping in mind that newest translations of 9.0.0 are still missing in general).

Performance

I did some testing with gettext about performance while at it: translating 200,000 one word translations takes about 1 second at my local machine.
So we don't have to worry about translation performance in general. Only exception being column labels in tables, which I will take a look at.

Actions #78

Updated by Alexander Watzinger 3 days ago

Update about translations:
  • I finished translating new terms, so if you notice anything at demo or demo_dev in English or German please report
  • With the newest version of Poedit it is now possible to use translations in f-string, e.g. like:
    label = f'What a wonderful {_('world')}'
    
Actions

Also available in: Atom PDF