Project

General

Profile

Feature #1275

API parameters

Added by Bernhard Koschicek about 2 months ago. Updated 3 days ago.

Status:
Assigned
Priority:
Low
Category:
API
Target version:
Start date:
2020-08-04
Estimated time:
(Total: 0.00 h)

Description

As discussed at the API meeting, the API need more parameters, so a frontend can used it.

Branch name: feature_api_pagination

First wave parameters:
  • Pagination -> Seek Method (done)
  • Filtering (=, !=, <, <=, >, >=, LIKE, IN, AND, OR, AND NOT, OR NOT)(done)
  • Sorting (done)
Second wave parameters:
  • Real queries:
    • show subunits for classes and types #1206
    • Give me the entity and all linked entities
    • Give me the entity and all subunits of this entity
    • Give me the entity and all super types of this entity
    • Show only specific entries of entities --> selective query (e.g.: Show only name and geo data) (done)

Please change the parameters to your needs.

Example of path.

  /query/:
    get:
      tags:
      - Users
      summary: retrieves a list of geojson based on multiple query parameters
      operationId: retrieveQuery
      security: 
      - cookieAuth: []
      parameters: 
      - in: query
        name: entities[]
        description: get specific entities by id
        schema:
          type: number 
      - in: query
        name: classes[]
        description: get specific classes by cidoc class code e.g. E18
        schema:
          type: string
      - in: query
        name: items[]
        description: get specific menu items e.g. actor, place
        schema:
          type: string          
      - in: query
        name: limit
        description: number of geojson representations to be returned
        schema:
          type: number
      - in: query
        name: column
        description: the column wich should be sorted and filtered
        schema:
          type: string
          enum:
            - id
            -  class_code
            -  name
            -  description
            -  created
            -  modified
            -  system_type
            -  begin_from
            -  begin_to
            -  end_from
            -  end_to
      - in: query
        name: sort
        description: sort direction asc/desc
        schema:
          type: string
          enum:
            - asc
            - desc
      - in: query
        name: filter
        description: filter operator to be applied
        schema:
          type: string    
      - in: query
        name: first
        description: first id of a page
        schema:
          type: number
      - in: query
        name: last
        description: last id of a page
        schema:
          type: number
      - in: query
        name: show
        description: select which key should be shown e.g. when, types, relations, names, links, geometry, depictions, not
        schema:
          type: string
          enum:
            - when
            - types
            - relations
            - names
            - links
            - geometry
            - depictions
            - not
      responses:
        '200':
          description: a geojson representation of the specified entities
          content:
              application/ld+json:
                schema:
                  $ref: '#/components/schemas/featureCollectionGeoJSON'
        '404':
          description: not found


Subtasks

Feature #1318: Selective API ResolvedBernhard KoschicekActions

History

#1

Updated by Bernhard Koschicek about 2 months ago

  • Description updated (diff)
#2

Updated by Alexander Watzinger about 1 month ago

  • Target version set to API
#3

Updated by Alexander Watzinger about 1 month ago

  • Status changed from Acknowledged to Assigned
#5

Updated by Bernhard Koschicek 26 days ago

  • Description updated (diff)
#6

Updated by Bernhard Koschicek 26 days ago

Filtering and sorting has to be on the server site. We need to write new SQL statements for this.

#7

Updated by Bernhard Koschicek 26 days ago

  • Description updated (diff)
#8

Updated by Bernhard Koschicek 24 days ago

  • Description updated (diff)
#9

Updated by Bernhard Koschicek 22 days ago

  • Description updated (diff)
#10

Updated by Bernhard Koschicek 22 days ago

  • Description updated (diff)
#11

Updated by Christoph Hoffmann 17 days ago

thanks for all your work! I'm just about to adapt my shenanigans to the proposed API, and have a couple of points:

  • routing needs to accomodate for a trailing slash behind the baseURL (as it's actually the spec), it's ommited for must website urls nowadays but most API clients (and also mine :) ) parse URLs like that
    • /api/0.1/query?entities[]=115&entities[]=116 needs to resolve just like /api/0.1/query/?entities[]=115&entities[]=116 (note the slash before the ?)
  • the classes[] parameter does not work for me (working in branch feature_api_pagination) the API throws and error when I pass an invalid class name but times out with any valid class I pass

let me know If I should create proper tickets for this, and also thank you very much for updating the swagger so thoroughly!

#13

Updated by Bernhard Koschicek 17 days ago

I didn't know that trailing slashes are so important. This should take no time to implement.

I will see into the other bug. Should also be fixed in no time, but I fear it will be next week.

Thank you for the feedback!!

#14

Updated by Bernhard Koschicek 12 days ago

Christoph Hoffmann:
  • trailing slashes are fixed. Thank you for the link, this spared me many new lines. (I merged it in develop)
  • classes[] works, but not with real data, like with Thanados. It will time out, because of the sheer number of items to display. You can try it with something like /api/0.1/query?classes[]=E84. This should work. Maybe I need to redo my algorithm for catching the data. If this doesn't help, then pagination is our only hope.

And I added the parameter functions (sort, column, filter and limit) to the /query path ( e.g. /api/0.1/query?classes[]=E18&items[]=actor&sort=desc&limit=50).

#15

Updated by Bernhard Koschicek 10 days ago

  • Description updated (diff)
#16

Updated by Bernhard Koschicek 10 days ago

As I go through the pagination feature, I saw a huge performance flaw in my code. For the purpose of error catching the function is executed twice: e.g. if all E18 are called, I first gather all E18 data, including all subdata, to see if everything is all right, and then do the same function again. This takes a lot of time. I will redo this.
But for now, pagination works (at this moment only for /api/0.1/class/ and /api/0.1/code/). You will get a result limited by the ?limit= parameter (20 by default) and a list of page number and with which ID the page starts. Then you can enter, for example /api/0.1/class/E18/?first=42523 and you get entities, limited by limit (or 20) including the id 42523 as the first entry. Alternately, you can enter /api/0.1/class/E18/?last=42523 and get entities, limited by limit (or 20) excluding the id 42523.
I didn't tested it on large dataset, but in theory this should work quite well.

The index json looks like that:

  {
    "entities": 31,
    "entity_per_page": 10,
    "index": [
      {
        "page": 1,
        "start_id": 202
      },
      {
        "page": 2,
        "start_id": 122
      },
      {
        "page": 3,
        "start_id": 176
      },
      {
        "page": 4,
        "start_id": 218
      }
    ],
    "total_pages": 4
  }

#17

Updated by Bernhard Koschicek 6 days ago

Everything works with something like /api/0.1/query?classes[]=E18&items[]=actor&sort=desc&limit=10&entities[]=215&entities[]=202&classes[]=E33&items[]=place. (classes are E18 and E33, items are actor and place, entities are 215 and 202, parameters are sort desc and limit 10)
For each category a new dict appears with the pair code: "place" or class: "E18" and a result key with a list of the entites. The filter parameters are applied on every category. So limit=1 will give you something like that with the above query:

[
  {
    "entities": [
      {
        "@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
        "features": [
          {
            "@id": "http://127.0.0.1:5000/entity/215",
            "crmClass": "crm:E53_Place",
            "geometry": {
              "geometries": [],
              "type": "GeometryCollection" 
            },
            "properties": {
              "title": "Location of ac3332er23rtt2" 
            },
            "relations": [
              {
                "label": "ac3332er23rtt2",
                "relationTo": "http://127.0.0.1:5000/api/0.1/entity/214",
                "relationType": "crm:P53i_has_former_or_current_location" 
              }
            ],
            "type": "Feature" 
          }
        ],
        "type": "FeatureCollection" 
      },
      {
        "entities": 2,
        "entity_per_page": 1,
        "index": [
          {
            "page": 1,
            "start_id": 215
          },
          {
            "page": 2,
            "start_id": 202
          }
        ],
        "total_pages": 2
      }
    ]
  },
  {
    "code": "actor",
    "result": [
      {
        "@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
        "features": [
          {
            "@id": "http://127.0.0.1:5000/entity/222",
            "crmClass": "crm:E21_Person",
            "geometry": {
              "geometries": [],
              "type": "GeometryCollection" 
            },
            "properties": {
              "title": "Yusuif" 
            },
            "type": "Feature" 
          }
        ],
        "type": "FeatureCollection" 
      },
      {
        "entities": 4,
        "entity_per_page": 1,
        "index": [
          {
            "page": 1,
            "start_id": 222
          },
          {
            "page": 2,
            "start_id": 117
          },
          {
            "page": 3,
            "start_id": 118
          },
          {
            "page": 4,
            "start_id": 109
          }
        ],
        "total_pages": 4
      }
    ]
  },
  {
    "code": "place",
    "result": [
      {
        "@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
        "features": [
          {
            "@id": "http://127.0.0.1:5000/entity/218",
            "crmClass": "crm:E18_Physical_Thing",
            "geometry": {
              "geometries": [],
              "type": "GeometryCollection" 
            },
            "properties": {
              "title": "yyyyyyyyyyyyyyy" 
            },
            "relations": [
              {
                "label": "Location of yyyyyyyyyyyyyyy",
                "relationTo": "http://127.0.0.1:5000/api/0.1/entity/219",
                "relationType": "crm:P53_has_former_or_current_location" 
              }
            ],
            "type": "Feature" 
          }
        ],
        "type": "FeatureCollection" 
      },
      {
        "entities": 31,
        "entity_per_page": 1,
        "index": [
          {
            "page": 1,
            "start_id": 218
          },
          {
            "page": 2,
            "start_id": 216
          },
          {
            "page": 3,
            "start_id": 138
          },
          .....
        ],
        "total_pages": 31
      }
    ]
  },
  {
    "class": "E18",
    "result": [
      {
        "@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
        "features": [
          {
            "@id": "http://127.0.0.1:5000/entity/218",
            "crmClass": "crm:E18_Physical_Thing",
            "geometry": {
              "geometries": [],
              "type": "GeometryCollection" 
            },
            "properties": {
              "title": "yyyyyyyyyyyyyyy" 
            },
            "relations": [
              {
                "label": "Location of yyyyyyyyyyyyyyy",
                "relationTo": "http://127.0.0.1:5000/api/0.1/entity/219",
                "relationType": "crm:P53_has_former_or_current_location" 
              }
            ],
            "type": "Feature" 
          }
        ],
        "type": "FeatureCollection" 
      },
      {
        "entities": 29,
        "entity_per_page": 1,
        "index": [
          {
            "page": 1,
            "start_id": 218
          },
          {
            "page": 2,
            "start_id": 216
          },
          {
            "page": 3,
            "start_id": 104
          },
          .....
        ],
        "total_pages": 29
      }
    ]
  },
  {
    "class": "E33",
    "result": [
      {
        "@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
        "features": [
          {
            "@id": "http://127.0.0.1:5000/entity/115",
            "crmClass": "crm:E33_Linguistic_Object",
            "geometry": {
              "geometries": [],
              "type": "GeometryCollection" 
            },
            "properties": {
              "title": "test" 
            },
            "relations": [
              {
                "label": "Contract",
                "relationTo": "http://127.0.0.1:5000/api/0.1/entity/44",
                "relationType": "crm:P2_has_type" 
              },
              {
                "label": "swdwfwfw",
                "relationTo": "http://127.0.0.1:5000/api/0.1/entity/112",
                "relationType": "crm:P128i_carries" 
              }
            ],
            "type": "Feature",
            "types": [
              {
                "hierarchy": "Source",
                "identifier": "http://127.0.0.1:5000/api/0.1/entity/44",
                "label": "Contract" 
              }
            ]
          }
        ],
        "type": "FeatureCollection" 
      },
      {
        "entities": 13,
        "entity_per_page": 1,
        "index": [
          {
            "page": 1,
            "start_id": 115
          },
          {
            "page": 2,
            "start_id": 111
          },
          {
            "page": 3,
            "start_id": 185
          },
          .......
        ],
        "total_pages": 13
      }
    ]
  }
]

So, if you want to get the first 10 of all menu items with pagination and asc: /api/0.1/query/?items[]=actor&items[]=place&items[]=source&items[]=event&items[]=object&items[]=reference&limit=10&sort=asc

#18

Updated by Bernhard Koschicek 5 days ago

  • Description updated (diff)
#19

Updated by Bernhard Koschicek 4 days ago

Testes some queries on Thanatos data. It was ok fast.
For E18:
Limit Time
10 13,53 sec
20 24,07 sec
100 2 min
For places:
Limit Time
10 14,08 sec
20 26 sec
100 1,7 min

I think the problem is, that for each entity a lot of data is gathered. So my next focus will be, that we can select which information will be provided.

Edit:

Implemented a feature named show, where key can be excluded/included in the request.
For places:
Limit Time
10 8,23 sec
20 13,85 sec
100 53,05 sec

It is better, but still not good.

#20

Updated by Bernhard Koschicek 3 days ago

  • Description updated (diff)
#21

Updated by Bernhard Koschicek 3 days ago

  • Description updated (diff)
#22

Updated by Bernhard Koschicek 3 days ago

  • Description updated (diff)

Also available in: Atom PDF