Feature #2676
open
JSON import
Description
Feature
A data import option of data that has a more complex structure than a csv would be a great addition, maybe something JSON?
Use case
We will have to import data that come with a lot of 1:n relations, e.g. a site with multiple objects, objects with multiple samples... While items can have more than one subordninate item, there won't be the other way round (an item with more than one super-ordinate items). Achieving this with a csv is very difficult. In contrast to csv, a JSON-(like) format would allow to include the relations between the items rather easily in a structured way.
Alternative
We thought about adding POST endpoints to the API to directly write to the database. However, we are aware that this would require authentification for the API and probably additional efforts to make the databse safe from malicious attemps through the API.
Updated by Alexander Watzinger about 1 month ago
- Status changed from New to Acknowledged
- Target version set to Wishlist
Updated by Alexander Watzinger about 1 month ago
- Start date changed from 2025-11-14 to 2025-01-27
- Follows Feature #2464: OpenAtlas major version 9.0.0 added
Updated by Bernhard Koschiček-Krombholz about 1 month ago
Currently, for our own import workflows, we write custom import scripts, which can be found at install/import_script/. These scripts use our internal functions to insert/update/link. It is kind of a local API approach.
I'm completely with you, CSV is very flat, and we tried to add more deepness to it, but it is very limited. And still, CSV is more convenient for most of our users.
JSON import
A JSON import would be the way to go. I put already a lot of thoughts into it, but for this, we absolutely need the new class system (#2464). With the new class system, we can use the config map variables also to define relations in the JSON.
API approach
I also put a lot of thoughts into it too already. We have the possibility to generate JWT tokens to access the API, even if it is not public (#1233). So the first steps are already done. But it is not "battle proofed", because all our APIs are open, and we have no production environment who uses this.
And we're also discussing of implementing basic POST functionality to insert, update and link data. But since currently, no resources are free on our side.
And again, it is also blocked by (#2464), because this whole insert/update/link things are completely rewritten.
I completely understand the need of a more sophisticated import (also @Enric Rodellas is in dire need of this). After the 9.0.0 release, we can look into it.