Project

General

Profile

Actions

Feature #2045

closed

Performance

Added by Bernhard Koschiček-Krombholz 8 months ago. Updated 6 months ago.

Status:
Closed
Priority:
Normal
Category:
Backend
Target version:
Start date:
2023-08-09
Estimated time:

Description

Improved pre request time
  • Optimized Type.get_all() SQL (improvement for every call)
  • Optimized Type.get_all() to only fetch counts where needed (improvement for many calls)
  • Moved file stat functions from __init__.py to where it is needed (improvement for many calls)
Actions #1

Updated by Bernhard Koschiček-Krombholz 7 months ago

I tested some performance with Thanados data. Besides some operations are being slow due high linkage, I found another performance issue.
Views and API calls in Thanados always take at a minimum of 3 seconds. In fact, it takes 3 seconds that the view/API code is even reached. So I looked into the __init__.py and found the two baddies:

g.file_stats = get_file_stats()
~ 1.5 seconds (Thanados has over 17.000 files)
g.types = Type.get_all()
~ 1.15 seconds (Thanados has 3853 types)

These two are responsible for the delay.

Actions #2

Updated by Bernhard Koschiček-Krombholz 7 months ago

  • Description updated (diff)
Actions #3

Updated by Bernhard Koschiček-Krombholz 7 months ago

  • Description updated (diff)
Actions #4

Updated by Alexander Watzinger 7 months ago

Can be tested with a bigger project data set when e.g. calling:
http://localhost:5000/api/0.3/query/?system_classes=all&format=lp&column=name&sort=asc&limit=0

Actions #5

Updated by Bernhard Koschiček-Krombholz 7 months ago

  • Description updated (diff)
  • Status changed from Acknowledged to In Progress
  • Target version changed from 8.0.0 to 7.17.0
Actions #6

Updated by Bernhard Koschiček-Krombholz 6 months ago

  • Description updated (diff)
Actions #7

Updated by Alexander Watzinger 6 months ago

Today we looked into Type.get_all() at __init__.py and it looks like most of the cost comes from determining the count for entities using that type.
Next session we will try to make a switch at Type.count_all() to only get counts when needed (e.g. at type views).

Actions #8

Updated by Bernhard Koschiček-Krombholz 6 months ago

  • Subject changed from API: Performance to Performance
Actions #9

Updated by Alexander Watzinger 6 months ago

  • Description updated (diff)
Actions #10

Updated by Bernhard Koschiček-Krombholz 6 months ago

  • Description updated (diff)
  • Category changed from API to Backend
  • Status changed from In Progress to Closed
Actions

Also available in: Atom PDF