Pagination in Action Rivers

[utamhank]

Getting data from specific pages from your Rest API GET response

In this post we will detail the multiple options available for pagination in an Action River REST Action. As you may have already experienced, RestAPI’s often return multiple pages of responses that correspond to a single GET request, but you may only be interested in data on a specific page or set of pages in the response. Rivery offers multiple options to deal with multi-page responses to help you extract the required data.

Where to find the Pagination options.

1. First, click on the “Create River” button and select Action River. You should see the following screen:
   

   

2. select REST Action.

3. Follow these instructions to set up your API connection. Ensure that the Rest API you are trying to use does return multiple pages of responses. Check your API docs for pagination details.

4. Click on the “Results” tab and ensure that the response type is set to “Data”.
   

   

5. Click on “Pagination details”, and select the drop down arrow. You will now have four options:

   

What do these 4 options mean?

1. No Pagination - This option is exactly what it sounds like. The request will return the only initial API response, without going through the multiple pages. Depending on the size of the response, this could potentially slow down your River/workflow. For an API call returning less than one page of information, this is an acceptable option.

2. Paginate by Offset - This is the most simple kind of Pagination. In the GET request, it may look like:

   
   
   Paginate by Offset uses the limit and offset variables in the GET call to specify the pages to return. In the above example, the response will return pages 100 through 120. In Rivery, selecting this option will bring up a page like this:
   

   • “Offset Start” is the page number at which to start the offset, if you want to only return pages 30-40 of a 100 page API request, for example, your Offset Start would be 30.

   • “Offset Key” is the JSON name of the returned data that you want to offset by. It is often called just “offset” but in certain cases, might be named something else like “start_page” or “begin_page” depending on the response.

   • “Stop according to” is the criteria at which the Action River will stop paginating and thus pulling data from the API. This box gives you two options:

     • “Page Size” - This option allows the user to specify the number of rows of data being returned by a typical page of the response, and to stop the pagination on a page that has less than the full number of entries (the last page). This option is especially useful when we have a variable number of pages or an unknown number of pages.

     • “Key in Response” - This option allows you to limit the number of pages to when a specific Key reaches a threshold. Going back to our example returning pages 30-40 of a 100 page API request, for example, your “Key in Response” would be something like “page_number” and you would set it to be 30.

3. “Paginate by Page” - This option allows you to iterate through all of the pages without an offset.

   • “Page Key” allows you to specify the key in the response that corresponds to the page number of the JSON response. It may be called “page_number” or “page_num” or just “page”.

   • “Page Value” - allows you to update the value of the next page during each iteration by using a preconfigured variable named “page_value”. To enable this feature, create a “page_value” variable in the Variables tab, select the “Paginate By Page” option under Results → Data, and check the “Update the value of the next page into a page_value variable in every page iteration” box:

     a.

     

     b.

     

• “Stop according to” - You can choose one of these two options:

  “Page Size” - This option allows the user to specify the number of rows of data being returned by a typical page of the response, and to stop the pagination on a page that has less than the full number of entries (the last page). This option is especially useful when we have a variable number of pages or an unknown number of pages.

For example, in this API you can create in this action river a variable called page size.

Then you can add it to the request params here:
And use it as a stop condition:

• “Key in Response” - This option allows you to limit the number of pages to when a specific Key reaches a threshold. Going back to our example returning pages 30-40 of a 100 page API request, your “Key in Response” would be something like “page_number” and you would set it to be 30.

4. “Next Page In Response” - This option is for the situation where the pagination does not follow a typical cadence set by a variable in the GET call, but instead relies on information derived from a previous GET response. For example, say you had an example API that took an “id” as a parameter input and returned customer information like name, age, height, weight, income, and a value called ‘id_behind’ corresponding to the id of the person behind you who is waiting in line. In order to get the information for the next customer in line you have to make another API call setting the id parameter to that of the “id_behind” of the previous customer! Rivery allows you to do this with three options:

   • “Static Url With Param From Response” - This option allows you to specify a new Url for the next page of the response as well as map a variable from the previous response to a variable you will pass into the GET call for the next page of the response.

     • “Next Page Key” - a key which contains the parameter that you will pass in to get the next page of the response. In our above example, this key would be “id_behind”.

     • “Send next page in key:” - This allows us to map the “Next Page Key” to the Key in the GET request for the next page. In the example above, this would be set to “id”.

     • “Send next page using a new url” - This allows us to change the Url call of the next page’s GET request to one of our choosing. Say we wanted to access customers in another database (called via a different REST Url) and pass in the “id_behind” parameter from our previous Url from our previous GET request. This box would allow us to do so.

     • “Stop according to” is the criteria at which the Action River will stop paginating and thus pulling data from the API. This box gives you two options:

       • “Page Size” - This option allows the user to specify the number of rows of data being returned by a typical page of the response, and to stop the pagination on a page that has less than the full number of entries (the last page). This option is especially useful when we have a variable number of pages or an unknown number of pages in the response.

       • “Key in Response” - This option allows you to limit the number of pages to when a specific Key reaches a threshold.

     • “Sleep After Each Page” - This is a universal option allowing you to specify a time delay in seconds between when an API paginates.

   • “Url In Response” - This option is for the case when the next page’s entire Url is stored within a key in the previous response.

     • “Next page key” - Here, all we have to do is specify the key in the previous response that holds the full Url for the next page.

   • “Next Page in Body” - This option is used for when the information needed is stored in a two layer next page call (the ‘next to next page’) where the information for the next page (call this, page ‘b’) is stored in the previous response (call this, page ‘a’), but the page with the information we want to derive (call this, page ‘c’) can only be accessed via a key in page ‘b’.

     • Page a → contains information to access → Page b → contains information to access-> Page c

     • “Next page key” - This allows us to specify the key in the previous response (page ‘’a’) where the page we want to navigate to is located in (page ‘b’).

     • “Send next page in key” - This allows us to specify the key in the page ‘b’ api call in order to access the page that we want (page ‘c’).

Dynamic changing of parameters

In many of these settings where the user passes key parameters and page numbers, you can pass variables as well, using the {your_variable_name} notation and by creating a variable by clicking the button on the top right. You can then insert these variables into the parameter boxes for these options to dynamically change the pagination of the API call as part of your logic river workflow.