Feature #1275
closedAPI: parameters
Description
As discussed at the API meeting, the API need more parameters, so a frontend can used it.
Branch name: feature_api_pagination
First wave parameters:- Pagination -> Seek Method (done)
- Filtering (=, !=, <, <=, >, >=, LIKE, IN, AND, OR, AND NOT, OR NOT)(done)
- Sorting (done)
Real queries:show subunits for classes and types #1206Give me the entity and all linked entitiesGive me the entity and all subunits of this entityGive me the entity and all super types of this entityShow only specific entries of entities -> selective query (e.g.: Show only name and geo data) (done)-Give me a place and all actors linked to that place through events (could be also more specific if needed)
Please change the parameters to your needs.
Example of path.
/query/:
get:
tags:
- Users
summary: retrieves a list of geojson based on multiple query parameters
operationId: retrieveQuery
security:
- cookieAuth: []
parameters:
- in: query
name: entities[]
description: get specific entities by id
schema:
type: number
- in: query
name: classes[]
description: get specific classes by cidoc class code e.g. E18
schema:
type: string
- in: query
name: items[]
description: get specific menu items e.g. actor, place
schema:
type: string
- in: query
name: limit
description: number of geojson representations to be returned
schema:
type: number
- in: query
name: column
description: the column wich should be sorted and filtered
schema:
type: string
enum:
- id
- class_code
- name
- description
- created
- modified
- system_type
- begin_from
- begin_to
- end_from
- end_to
- in: query
name: sort
description: sort direction asc/desc
schema:
type: string
enum:
- asc
- desc
- in: query
name: filter
description: filter operator to be applied
schema:
type: string
- in: query
name: first
description: first id of a page
schema:
type: number
- in: query
name: last
description: last id of a page
schema:
type: number
- in: query
name: show
description: select which key should be shown e.g. when, types, relations, names, links, geometry, depictions, not
schema:
type: string
enum:
- when
- types
- relations
- names
- links
- geometry
- depictions
- not
responses:
'200':
description: a geojson representation of the specified entities
content:
application/ld+json:
schema:
$ref: '#/components/schemas/featureCollectionGeoJSON'
'404':
description: not found
Updated by Bernhard Koschiček-Krombholz over 4 years ago
- Description updated (diff)
Updated by Alexander Watzinger over 4 years ago
- Status changed from Acknowledged to Assigned
Updated by Bernhard Koschiček-Krombholz over 4 years ago
For pagination the Seek/Keysearch method will be used. This will be quite time-consuming to develop, but is needed, because we don't use an ORM.
Informational websites:- https://nordicapis.com/everything-you-need-to-know-about-api-pagination/
- https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/
- https://use-the-index-luke.com/no-offset
- https://use-the-index-luke.com/sql/partial-results/fetch-next-page
- https://leopard.in.ua/2014/10/11/postgresql-paginattion#.Xv7tgCgzaCg
- https://blog.jooq.org/2013/10/26/faster-sql-paging-with-jooq-using-the-seek-method/
- https://blog.jooq.org/2013/11/18/faster-sql-pagination-with-keysets-continued/
And a new folder in models will be created for all the helpers and new SQLs we need.
Updated by Bernhard Koschiček-Krombholz over 4 years ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz over 4 years ago
Filtering and sorting has to be on the server site. We need to write new SQL statements for this.
Updated by Bernhard Koschiček-Krombholz over 4 years ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz over 4 years ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz over 4 years ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz over 4 years ago
- Description updated (diff)
Updated by Christoph Hoffmann over 4 years ago
thanks for all your work! I'm just about to adapt my shenanigans to the proposed API, and have a couple of points:
- routing needs to accomodate for a trailing slash behind the baseURL (as it's actually the spec), it's ommited for must website urls nowadays but most API clients (and also mine :) ) parse URLs like that
- /api/0.1/query?entities[]=115&entities[]=116 needs to resolve just like /api/0.1/query/?entities[]=115&entities[]=116 (note the slash before the ?)
- the classes[] parameter does not work for me (working in branch feature_api_pagination) the API throws and error when I pass an invalid class name but times out with any valid class I pass
let me know If I should create proper tickets for this, and also thank you very much for updating the swagger so thoroughly!
Updated by Christoph Hoffmann over 4 years ago
this might be helpful? :) https://stackoverflow.com/questions/33241050/trailing-slash-triggers-404-in-flask-path-rule
Updated by Bernhard Koschiček-Krombholz over 4 years ago
I didn't know that trailing slashes are so important. This should take no time to implement.
I will see into the other bug. Should also be fixed in no time, but I fear it will be next week.
Thank you for the feedback!!
Updated by Bernhard Koschiček-Krombholz over 4 years ago
- trailing slashes are fixed. Thank you for the link, this spared me many new lines. (I merged it in develop)
- classes[] works, but not with real data, like with Thanados. It will time out, because of the sheer number of items to display. You can try it with something like /api/0.1/query?classes[]=E84. This should work. Maybe I need to redo my algorithm for catching the data. If this doesn't help, then pagination is our only hope.
And I added the parameter functions (sort, column, filter and limit) to the /query path ( e.g. /api/0.1/query?classes[]=E18&items[]=actor&sort=desc&limit=50).
Updated by Bernhard Koschiček-Krombholz over 4 years ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz over 4 years ago
As I go through the pagination feature, I saw a huge performance flaw in my code. For the purpose of error catching the function is executed twice: e.g. if all E18 are called, I first gather all E18 data, including all subdata, to see if everything is all right, and then do the same function again. This takes a lot of time. I will redo this.
But for now, pagination works (at this moment only for /api/0.1/class/ and /api/0.1/code/). You will get a result limited by the ?limit= parameter (20 by default) and a list of page number and with which ID the page starts. Then you can enter, for example /api/0.1/class/E18/?first=42523 and you get entities, limited by limit (or 20) including the id 42523 as the first entry. Alternately, you can enter /api/0.1/class/E18/?last=42523 and get entities, limited by limit (or 20) excluding the id 42523.
I didn't tested it on large dataset, but in theory this should work quite well.
The index json looks like that:
{ "entities": 31, "entity_per_page": 10, "index": [ { "page": 1, "start_id": 202 }, { "page": 2, "start_id": 122 }, { "page": 3, "start_id": 176 }, { "page": 4, "start_id": 218 } ], "total_pages": 4 }
Updated by Bernhard Koschiček-Krombholz over 4 years ago
Everything works with something like /api/0.1/query?classes[]=E18&items[]=actor&sort=desc&limit=10&entities[]=215&entities[]=202&classes[]=E33&items[]=place. (classes are E18 and E33, items are actor and place, entities are 215 and 202, parameters are sort desc and limit 10)
For each category a new dict appears with the pair code: "place" or class: "E18" and a result key with a list of the entites. The filter parameters are applied on every category. So limit=1 will give you something like that with the above query:
[
{
"entities": [
{
"@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
"features": [
{
"@id": "http://127.0.0.1:5000/entity/215",
"crmClass": "crm:E53_Place",
"geometry": {
"geometries": [],
"type": "GeometryCollection"
},
"properties": {
"title": "Location of ac3332er23rtt2"
},
"relations": [
{
"label": "ac3332er23rtt2",
"relationTo": "http://127.0.0.1:5000/api/0.1/entity/214",
"relationType": "crm:P53i_has_former_or_current_location"
}
],
"type": "Feature"
}
],
"type": "FeatureCollection"
},
{
"entities": 2,
"entity_per_page": 1,
"index": [
{
"page": 1,
"start_id": 215
},
{
"page": 2,
"start_id": 202
}
],
"total_pages": 2
}
]
},
{
"code": "actor",
"result": [
{
"@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
"features": [
{
"@id": "http://127.0.0.1:5000/entity/222",
"crmClass": "crm:E21_Person",
"geometry": {
"geometries": [],
"type": "GeometryCollection"
},
"properties": {
"title": "Yusuif"
},
"type": "Feature"
}
],
"type": "FeatureCollection"
},
{
"entities": 4,
"entity_per_page": 1,
"index": [
{
"page": 1,
"start_id": 222
},
{
"page": 2,
"start_id": 117
},
{
"page": 3,
"start_id": 118
},
{
"page": 4,
"start_id": 109
}
],
"total_pages": 4
}
]
},
{
"code": "place",
"result": [
{
"@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
"features": [
{
"@id": "http://127.0.0.1:5000/entity/218",
"crmClass": "crm:E18_Physical_Thing",
"geometry": {
"geometries": [],
"type": "GeometryCollection"
},
"properties": {
"title": "yyyyyyyyyyyyyyy"
},
"relations": [
{
"label": "Location of yyyyyyyyyyyyyyy",
"relationTo": "http://127.0.0.1:5000/api/0.1/entity/219",
"relationType": "crm:P53_has_former_or_current_location"
}
],
"type": "Feature"
}
],
"type": "FeatureCollection"
},
{
"entities": 31,
"entity_per_page": 1,
"index": [
{
"page": 1,
"start_id": 218
},
{
"page": 2,
"start_id": 216
},
{
"page": 3,
"start_id": 138
},
.....
],
"total_pages": 31
}
]
},
{
"class": "E18",
"result": [
{
"@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
"features": [
{
"@id": "http://127.0.0.1:5000/entity/218",
"crmClass": "crm:E18_Physical_Thing",
"geometry": {
"geometries": [],
"type": "GeometryCollection"
},
"properties": {
"title": "yyyyyyyyyyyyyyy"
},
"relations": [
{
"label": "Location of yyyyyyyyyyyyyyy",
"relationTo": "http://127.0.0.1:5000/api/0.1/entity/219",
"relationType": "crm:P53_has_former_or_current_location"
}
],
"type": "Feature"
}
],
"type": "FeatureCollection"
},
{
"entities": 29,
"entity_per_page": 1,
"index": [
{
"page": 1,
"start_id": 218
},
{
"page": 2,
"start_id": 216
},
{
"page": 3,
"start_id": 104
},
.....
],
"total_pages": 29
}
]
},
{
"class": "E33",
"result": [
{
"@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
"features": [
{
"@id": "http://127.0.0.1:5000/entity/115",
"crmClass": "crm:E33_Linguistic_Object",
"geometry": {
"geometries": [],
"type": "GeometryCollection"
},
"properties": {
"title": "test"
},
"relations": [
{
"label": "Contract",
"relationTo": "http://127.0.0.1:5000/api/0.1/entity/44",
"relationType": "crm:P2_has_type"
},
{
"label": "swdwfwfw",
"relationTo": "http://127.0.0.1:5000/api/0.1/entity/112",
"relationType": "crm:P128i_carries"
}
],
"type": "Feature",
"types": [
{
"hierarchy": "Source",
"identifier": "http://127.0.0.1:5000/api/0.1/entity/44",
"label": "Contract"
}
]
}
],
"type": "FeatureCollection"
},
{
"entities": 13,
"entity_per_page": 1,
"index": [
{
"page": 1,
"start_id": 115
},
{
"page": 2,
"start_id": 111
},
{
"page": 3,
"start_id": 185
},
.......
],
"total_pages": 13
}
]
}
]
So, if you want to get the first 10 of all menu items with pagination and asc: /api/0.1/query/?items[]=actor&items[]=place&items[]=source&items[]=event&items[]=object&items[]=reference&limit=10&sort=asc
Updated by Bernhard Koschiček-Krombholz over 4 years ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz over 4 years ago
For E18:
Limit | Time |
10 | 13,53 sec |
20 | 24,07 sec |
100 | 2 min |
Limit | Time |
10 | 14,08 sec |
20 | 26 sec |
100 | 1,7 min |
I think the problem is, that for each entity a lot of data is gathered. So my next focus will be, that we can select which information will be provided.
Edit:
Implemented a feature named show, where key can be excluded/included in the request.For places:
Limit | Time |
10 | 8,23 sec |
20 | 13,85 sec |
100 | 53,05 sec |
It is better, but still not good.
Updated by Bernhard Koschiček-Krombholz over 4 years ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz over 4 years ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz over 4 years ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz about 4 years ago
- Description updated (diff)
Updated by Bernhard Koschiček-Krombholz about 4 years ago
- Description updated (diff)
- Status changed from Assigned to Closed
Updated by Alexander Watzinger about 4 years ago
- Subject changed from API parameters to API: parameters
Updated by Bernhard Koschiček-Krombholz about 4 years ago
- Target version changed from 208 to 5.4.0