Project

General

Profile

Actions

Feature #1567

closed

Enhanced Import

Added by Stefan Eichert about 3 years ago. Updated 5 months ago.

Status:
Closed
Priority:
Normal
Category:
Data import/export
Target version:
Start date:
2021-08-25
Estimated time:

Description

Idea:
create a generic import functionality that imports all kind of data with their relation from discipline specific CSV templates into openatlas

Prerequisite:
discuss structure of generic import functionality

Stefan's approach:
1. Fill in thanados specific template like this one
https://docs.google.com/spreadsheets/d/1IIPGKvSRNi6Qeir6070VS4u7uvHdlpYenEhRkCaGJcI/edit?usp=sharing
2. Get the data as CSV to OpenAtlas
3. Map the CSV to the generic import format (Name, Alias, Geometry, Types, Dates, Reference Systems etc.)
4. Witchcraft and Magic -> all is imported to OpenAtlas

To-do:
  • Alias* ✅
  • Geometry as WKT* ✅
  • Existing External Reference System with ID including identifier* ✅
  • Value types* ✅
  • Administrative Unit* ✅
  • Historical Place* ✅
  • Existing References with ID including page number -> #2244
  • OpenAtlas Class name to know which level the place is -> #2244
  • Parent import ID -> #2244
    (* These will be added to all import options)
  • Better warnings (showing where warning appears)✅
  • Remove no cover by manipulating CSV in tests ✅
  • What if duplicate reference systems are given ✅
  • Handle error if multiple type ids from the same single type ✅
  • Translations ✅
  • Manual entry ✅

Related issues 3 (0 open3 closed)

Related to OpenAtlas - Feature #1047: Enter WKT geometriesClosedBernhard Koschiček-Krombholz2019-07-31Actions
Related to OpenAtlas - Feature #2193: Import artifactsClosedBernhard Koschiček-Krombholz2024-02-21Actions
Precedes OpenAtlas - Feature #2244: Import of place hierarchyClosedBernhard Koschiček-Krombholz2021-08-26Actions
Actions #1

Updated by Alexander Watzinger about 3 years ago

  • Category changed from Backend to Data import/export
  • Status changed from New to Acknowledged
Actions #2

Updated by Bernhard Koschiček-Krombholz about 3 years ago

Actions #3

Updated by Bernhard Koschiček-Krombholz 9 months ago

  • Status changed from Acknowledged to Assigned
  • Assignee set to Bernhard Koschiček-Krombholz
  • Target version changed from Wishlist to 8.7.0
Actions #4

Updated by Bernhard Koschiček-Krombholz 9 months ago

  • Target version changed from 8.7.0 to 8.4.0
Actions #5

Updated by Bernhard Koschiček-Krombholz 9 months ago

Actions #6

Updated by Bernhard Koschiček-Krombholz 9 months ago

  • Description updated (diff)

By comparing the example and the current import, following things needs to be added:

  • Alias*
  • Geometry as WKT (Should multiple geometries be possible)*
  • Existing References with ID including page number
  • Existing External Reference System with ID including identifier*
  • Value types*
  • OpenAtlas Class name to know which level the place is
  • Parent import ID

(* These can also be added to the current import)

For this, I will make a new button like "Place hierarchy".

First steps will be to develop the things, which can be added to the current import actions.

Actions #7

Updated by Bernhard Koschiček-Krombholz 9 months ago

  • Description updated (diff)
Actions #8

Updated by Bernhard Koschiček-Krombholz 9 months ago

  • Description updated (diff)
Actions #9

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)

Took some time, but I replaced northing/easting with WKT and therefore also support for Polygons and Linestrings.

Actions #10

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)
Actions #11

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)
Actions #12

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)

Added external reference system with id;identifier;match
Not sure if ; is the optimal delimiter, but it is a start and can be easily be changed.

external_reference_system wikidata geonames
156149;vienna;exact_match 6666;stadt-wien;close_match Q152419;close_match 2761369;exact_match
Actions #13

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)
Actions #14

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)
Included import of value types:
value_type_ids
666;12.34 777;987

Again delimiter is ; for type_id;value and space for each pair.

Actions #15

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)
Actions #16

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)
Added administrative unit and historical place:
administrative_unit historical_place
87 102
Actions #17

Updated by Bernhard Koschiček-Krombholz 8 months ago

Alex pointed out, at the developer meeting, that the implementation of external reference systems doesn't look nice. So I refactored it and each reference system has a own row with reference_system_* where * is the full name of the external reference system (spaces has to be transformed to _ and upper/lower case doesn't matter).
First value is the identifier and second value is the match.

reference_system_getty_aat reference_system_wikidata reference_system_geonames
300251645;exact_match Q152419;close_match 2761369;exact_match
Actions #18

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)
Actions #19

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)
Actions #20

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)
Actions #21

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)
Actions #22

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)
Actions #23

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)
  • Target version changed from 8.4.0 to 8.3.0
Actions #24

Updated by Bernhard Koschiček-Krombholz 8 months ago

Actions #25

Updated by Bernhard Koschiček-Krombholz 8 months ago

  • Description updated (diff)
  • Status changed from Assigned to Closed

Closing this issue and move place hierarchy related tasks to #2244.

Actions

Also available in: Atom PDF