Feature #1802
closed
Move CSV export functions to API
Added by Alexander Watzinger about 2 years ago.
Updated almost 2 years ago.
Category:
Data import/export
Description
At the moment CSV are generated in the backend. New implementation will use API functions and also files won't be saved on the server anymore but can be directly downloaded instead.
- Description updated (diff)
- Description updated (diff)
- Description updated (diff)
API To Do
The idea is to user the export=csvNetwork parameter and system_class/all.
Currently, this function take too long to load. Problems are:
- Each entity get a geometry added in a loop (csv_export.py:36). Solution is to make an own geometry table. This will take much less time.
- get_entities_grouped_by_class (csv_export.py:151) take way too long, because for every entity linked entities are collected. This take too much time for all entities.
- link.csv takes too long, since it is dynamically generated for each entity. It would be way faster to just take the whole link table.
Since the csvNetwork has performance issues with the test data (Thanados - over 300000 entities and links), a new API endpoint (api/export_database/<format>) was created, which directly collect all entries from the database without model functions (e.g. Link() and Entity())
- Status changed from Assigned to In Progress
- Description updated (diff)
- Status changed from In Progress to Resolved
The former export function is now replaced within an API endpoint(/api/export_database/[csv, json, xml]). I have added links to admin "data transfer".
Alex, maybe you can review the feature (feature_export_csv) and give me the OK to merge it into develop. After that, I can change the manual.
- Description updated (diff)
I tested it, it looks great and I merged it to develop (and removed feature branch online). I also removed the implementation details from the description:
- I already updated the install notes and changelog
- The "original CSV folder remove" for release notes I'm tracking elsewhere
- Keep in mind that once released this may also affect the Kubernetes install procedure
- As soon as the manual changes are done (in develop) you can close this issue
Thanks a lot Bernhard, moving this functionality to the API is very helpful on many levels.
- Status changed from Resolved to Closed
I wrote the manual entry.
Also available in: Atom
PDF