Project

General

Profile

Feature #1802

Move CSV export functions to API

Added by Alexander Watzinger 3 months ago. Updated about 2 months ago.

Status:
Closed
Priority:
Normal
Category:
Data import/export
Target version:
Start date:
2022-08-30
Estimated time:

Description

At the moment CSV are generated in the backend. New implementation will use API functions and also files won't be saved on the server anymore but can be directly downloaded instead.

History

#1

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
#2

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
#3

Updated by Alexander Watzinger 3 months ago

  • Description updated (diff)
#4

Updated by Bernhard Koschiček-Krombholz 3 months ago

API To Do
The idea is to user the export=csvNetwork parameter and system_class/all.

Currently, this function take too long to load. Problems are:

  • Each entity get a geometry added in a loop (csv_export.py:36). Solution is to make an own geometry table. This will take much less time.
  • get_entities_grouped_by_class (csv_export.py:151) take way too long, because for every entity linked entities are collected. This take too much time for all entities.
  • link.csv takes too long, since it is dynamically generated for each entity. It would be way faster to just take the whole link table.
#5

Updated by Bernhard Koschiček-Krombholz 3 months ago

Since the csvNetwork has performance issues with the test data (Thanados - over 300000 entities and links), a new API endpoint (api/export_database/<format>) was created, which directly collect all entries from the database without model functions (e.g. Link() and Entity())

#6

Updated by Bernhard Koschiček-Krombholz 2 months ago

  • Status changed from Assigned to In Progress
#7

Updated by Bernhard Koschiček-Krombholz 2 months ago

  • Description updated (diff)
  • Status changed from In Progress to Resolved

The former export function is now replaced within an API endpoint(/api/export_database/[csv, json, xml]). I have added links to admin "data transfer".
Alex, maybe you can review the feature (feature_export_csv) and give me the OK to merge it into develop. After that, I can change the manual.

#8

Updated by Alexander Watzinger 2 months ago

  • Description updated (diff)
I tested it, it looks great and I merged it to develop (and removed feature branch online). I also removed the implementation details from the description:
  • I already updated the install notes and changelog
  • The "original CSV folder remove" for release notes I'm tracking elsewhere
  • Keep in mind that once released this may also affect the Kubernetes install procedure
  • As soon as the manual changes are done (in develop) you can close this issue

Thanks a lot Bernhard, moving this functionality to the API is very helpful on many levels.

#9

Updated by Bernhard Koschiček-Krombholz about 2 months ago

  • Status changed from Resolved to Closed

I wrote the manual entry.

Also available in: Atom PDF