Project

General

Profile

Actions
Copy link

Feature #1802

closed

Move CSV export functions to API

Added by Alexander Watzinger about 2 years ago. Updated about 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Bernhard Koschiček-Krombholz
Category:
Data import/export
Target version:
7.7.0
Start date:
2022-08-30
Estimated time:

Description

At the moment CSV are generated in the backend. New implementation will use API functions and also files won't be saved on the server anymore but can be directly downloaded instead.

Actions
Copy link
 #1

Updated by Alexander Watzinger about 2 years ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Alexander Watzinger about 2 years ago

  • Description updated (diff)
Actions
Copy link
 #3

Updated by Alexander Watzinger about 2 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by Bernhard Koschiček-Krombholz about 2 years ago

API To Do
The idea is to user the export=csvNetwork parameter and system_class/all.

Currently, this function take too long to load. Problems are:

  • Each entity get a geometry added in a loop (csv_export.py:36). Solution is to make an own geometry table. This will take much less time.
  • get_entities_grouped_by_class (csv_export.py:151) take way too long, because for every entity linked entities are collected. This take too much time for all entities.
  • link.csv takes too long, since it is dynamically generated for each entity. It would be way faster to just take the whole link table.
Actions
Copy link
 #5

Updated by Bernhard Koschiček-Krombholz about 2 years ago

Since the csvNetwork has performance issues with the test data (Thanados - over 300000 entities and links), a new API endpoint (api/export_database/<format>) was created, which directly collect all entries from the database without model functions (e.g. Link() and Entity())

Actions
Copy link
 #6

Updated by Bernhard Koschiček-Krombholz about 2 years ago

  • Status changed from Assigned to In Progress
Actions
Copy link
 #7

Updated by Bernhard Koschiček-Krombholz about 2 years ago

  • Description updated (diff)
  • Status changed from In Progress to Resolved

The former export function is now replaced within an API endpoint(/api/export_database/[csv, json, xml]). I have added links to admin "data transfer".
Alex, maybe you can review the feature (feature_export_csv) and give me the OK to merge it into develop. After that, I can change the manual.

Actions
Copy link
 #8

Updated by Alexander Watzinger about 2 years ago

  • Description updated (diff)
I tested it, it looks great and I merged it to develop (and removed feature branch online). I also removed the implementation details from the description:
  • I already updated the install notes and changelog
  • The "original CSV folder remove" for release notes I'm tracking elsewhere
  • Keep in mind that once released this may also affect the Kubernetes install procedure
  • As soon as the manual changes are done (in develop) you can close this issue

Thanks a lot Bernhard, moving this functionality to the API is very helpful on many levels.

Actions
Copy link
 #9

Updated by Bernhard Koschiček-Krombholz about 2 years ago

  • Status changed from Resolved to Closed

I wrote the manual entry.

Actions
Copy link

Also available in: Atom PDF