OpenAtlas

For pagination the Seek/Keysearch method will be used. This will be quite time-consuming to develop, but is needed, because we don't use an ORM.

• https://nordicapis.com/everything-you-need-to-know-about-api-pagination/
• https://www.citusdata.com/blog/2016/03/30/five-ways-to-paginate/
• https://use-the-index-luke.com/no-offset
• https://use-the-index-luke.com/sql/partial-results/fetch-next-page
• https://leopard.in.ua/2014/10/11/postgresql-paginattion#.Xv7tgCgzaCg
• https://blog.jooq.org/2013/10/26/faster-sql-paging-with-jooq-using-the-seek-method/
• https://blog.jooq.org/2013/11/18/faster-sql-pagination-with-keysets-continued/

And a new folder in models will be created for all the helpers and new SQLs we need.

Filtering and sorting has to be on the server site. We need to write new SQL statements for this.

thanks for all your work! I'm just about to adapt my shenanigans to the proposed API, and have a couple of points:

• routing needs to accomodate for a trailing slash behind the baseURL (as it's actually the spec), it's ommited for must website urls nowadays but most API clients (and also mine :) ) parse URLs like that
  • /api/0.1/query?entities[]=115&entities[]=116 needs to resolve just like /api/0.1/query/?entities[]=115&entities[]=116 (note the slash before the ?)
• the classes[] parameter does not work for me (working in branch feature_api_pagination) the API throws and error when I pass an invalid class name but times out with any valid class I pass

let me know If I should create proper tickets for this, and also thank you very much for updating the swagger so thoroughly!

this might be helpful? :) https://stackoverflow.com/questions/33241050/trailing-slash-triggers-404-in-flask-path-rule

I didn't know that trailing slashes are so important. This should take no time to implement.

I will see into the other bug. Should also be fixed in no time, but I fear it will be next week.

Thank you for the feedback!!

• trailing slashes are fixed. Thank you for the link, this spared me many new lines. (I merged it in develop)
• classes[] works, but not with real data, like with Thanados. It will time out, because of the sheer number of items to display. You can try it with something like /api/0.1/query?classes[]=E84. This should work. Maybe I need to redo my algorithm for catching the data. If this doesn't help, then pagination is our only hope.

And I added the parameter functions (sort, column, filter and limit) to the /query path ( e.g. /api/0.1/query?classes[]=E18&items[]=actor&sort=desc&limit=50).

As I go through the pagination feature, I saw a huge performance flaw in my code. For the purpose of error catching the function is executed twice: e.g. if all E18 are called, I first gather all E18 data, including all subdata, to see if everything is all right, and then do the same function again. This takes a lot of time. I will redo this.
But for now, pagination works (at this moment only for /api/0.1/class/ and /api/0.1/code/). You will get a result limited by the ?limit= parameter (20 by default) and a list of page number and with which ID the page starts. Then you can enter, for example /api/0.1/class/E18/?first=42523 and you get entities, limited by limit (or 20) including the id 42523 as the first entry. Alternately, you can enter /api/0.1/class/E18/?last=42523 and get entities, limited by limit (or 20) excluding the id 42523.
I didn't tested it on large dataset, but in theory this should work quite well.

The index json looks like that:

  {
    "entities": 31,
    "entity_per_page": 10,
    "index": [
      {
        "page": 1,
        "start_id": 202
      },
      {
        "page": 2,
        "start_id": 122
      },
      {
        "page": 3,
        "start_id": 176
      },
      {
        "page": 4,
        "start_id": 218
      }
    ],
    "total_pages": 4
  }

Everything works with something like /api/0.1/query?classes[]=E18&items[]=actor&sort=desc&limit=10&entities[]=215&entities[]=202&classes[]=E33&items[]=place. (classes are E18 and E33, items are actor and place, entities are 215 and 202, parameters are sort desc and limit 10)
For each category a new dict appears with the pair code: "place" or class: "E18" and a result key with a list of the entites. The filter parameters are applied on every category. So limit=1 will give you something like that with the above query:

[
  {
    "entities": [
      {
        "@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
        "features": [
          {
            "@id": "http://127.0.0.1:5000/entity/215",
            "crmClass": "crm:E53_Place",
            "geometry": {
              "geometries": [],
              "type": "GeometryCollection" 
            },
            "properties": {
              "title": "Location of ac3332er23rtt2" 
            },
            "relations": [
              {
                "label": "ac3332er23rtt2",
                "relationTo": "http://127.0.0.1:5000/api/0.1/entity/214",
                "relationType": "crm:P53i_has_former_or_current_location" 
              }
            ],
            "type": "Feature" 
          }
        ],
        "type": "FeatureCollection" 
      },
      {
        "entities": 2,
        "entity_per_page": 1,
        "index": [
          {
            "page": 1,
            "start_id": 215
          },
          {
            "page": 2,
            "start_id": 202
          }
        ],
        "total_pages": 2
      }
    ]
  },
  {
    "code": "actor",
    "result": [
      {
        "@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
        "features": [
          {
            "@id": "http://127.0.0.1:5000/entity/222",
            "crmClass": "crm:E21_Person",
            "geometry": {
              "geometries": [],
              "type": "GeometryCollection" 
            },
            "properties": {
              "title": "Yusuif" 
            },
            "type": "Feature" 
          }
        ],
        "type": "FeatureCollection" 
      },
      {
        "entities": 4,
        "entity_per_page": 1,
        "index": [
          {
            "page": 1,
            "start_id": 222
          },
          {
            "page": 2,
            "start_id": 117
          },
          {
            "page": 3,
            "start_id": 118
          },
          {
            "page": 4,
            "start_id": 109
          }
        ],
        "total_pages": 4
      }
    ]
  },
  {
    "code": "place",
    "result": [
      {
        "@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
        "features": [
          {
            "@id": "http://127.0.0.1:5000/entity/218",
            "crmClass": "crm:E18_Physical_Thing",
            "geometry": {
              "geometries": [],
              "type": "GeometryCollection" 
            },
            "properties": {
              "title": "yyyyyyyyyyyyyyy" 
            },
            "relations": [
              {
                "label": "Location of yyyyyyyyyyyyyyy",
                "relationTo": "http://127.0.0.1:5000/api/0.1/entity/219",
                "relationType": "crm:P53_has_former_or_current_location" 
              }
            ],
            "type": "Feature" 
          }
        ],
        "type": "FeatureCollection" 
      },
      {
        "entities": 31,
        "entity_per_page": 1,
        "index": [
          {
            "page": 1,
            "start_id": 218
          },
          {
            "page": 2,
            "start_id": 216
          },
          {
            "page": 3,
            "start_id": 138
          },
          .....
        ],
        "total_pages": 31
      }
    ]
  },
  {
    "class": "E18",
    "result": [
      {
        "@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
        "features": [
          {
            "@id": "http://127.0.0.1:5000/entity/218",
            "crmClass": "crm:E18_Physical_Thing",
            "geometry": {
              "geometries": [],
              "type": "GeometryCollection" 
            },
            "properties": {
              "title": "yyyyyyyyyyyyyyy" 
            },
            "relations": [
              {
                "label": "Location of yyyyyyyyyyyyyyy",
                "relationTo": "http://127.0.0.1:5000/api/0.1/entity/219",
                "relationType": "crm:P53_has_former_or_current_location" 
              }
            ],
            "type": "Feature" 
          }
        ],
        "type": "FeatureCollection" 
      },
      {
        "entities": 29,
        "entity_per_page": 1,
        "index": [
          {
            "page": 1,
            "start_id": 218
          },
          {
            "page": 2,
            "start_id": 216
          },
          {
            "page": 3,
            "start_id": 104
          },
          .....
        ],
        "total_pages": 29
      }
    ]
  },
  {
    "class": "E33",
    "result": [
      {
        "@context": "https://raw.githubusercontent.com/LinkedPasts/linked-places/master/linkedplaces-context-v1.jsonld",
        "features": [
          {
            "@id": "http://127.0.0.1:5000/entity/115",
            "crmClass": "crm:E33_Linguistic_Object",
            "geometry": {
              "geometries": [],
              "type": "GeometryCollection" 
            },
            "properties": {
              "title": "test" 
            },
            "relations": [
              {
                "label": "Contract",
                "relationTo": "http://127.0.0.1:5000/api/0.1/entity/44",
                "relationType": "crm:P2_has_type" 
              },
              {
                "label": "swdwfwfw",
                "relationTo": "http://127.0.0.1:5000/api/0.1/entity/112",
                "relationType": "crm:P128i_carries" 
              }
            ],
            "type": "Feature",
            "types": [
              {
                "hierarchy": "Source",
                "identifier": "http://127.0.0.1:5000/api/0.1/entity/44",
                "label": "Contract" 
              }
            ]
          }
        ],
        "type": "FeatureCollection" 
      },
      {
        "entities": 13,
        "entity_per_page": 1,
        "index": [
          {
            "page": 1,
            "start_id": 115
          },
          {
            "page": 2,
            "start_id": 111
          },
          {
            "page": 3,
            "start_id": 185
          },
          .......
        ],
        "total_pages": 13
      }
    ]
  }
]

So, if you want to get the first 10 of all menu items with pagination and asc: /api/0.1/query/?items[]=actor&items[]=place&items[]=source&items[]=event&items[]=object&items[]=reference&limit=10&sort=asc

I think the problem is, that for each entity a lot of data is gathered. So my next focus will be, that we can select which information will be provided.

Edit:

It is better, but still not good.