Hello
I am working on a data pipeline in Rivery that pulls data from an external API; but I keep running into API rate limits; causing some data pulls to fail. The API enforces strict request limits per minute, and when exceeded, Rivery returns errors / incomplete data loads. I’ve tried adjusting scheduling intervals but I still hit the limit during high-frequency data pulls.
Is there a way to implement automatic retries, request throttling / batching within Rivery to handle API rate limits more gracefully? Would using different River configurations; such as incremental data loading / conditional logic, help optimize API requests?
I found this guide https://docs.rivery.io/docs/api-reference-Minitab related to this and found it quite informative but I’d love to hear best practices from others who have dealt with similar rate-limiting challenges.
Thank you !!