Best practices

Resilience

Why resilience matters

In our commitment to providing robust and reliable API integration services, it's essential to acknowledge the potential for disruptions. These can occur despite flawless execution from both ends due to:

Network fluctuations
Occasional service interruptions within the cloud environment
Other factors that can impact the seamless flow of data

Building resilient integrations

To mitigate these challenges and ensure uninterrupted operations, it's crucial for customers to incorporate resilience strategies into their integration architecture. Experience has taught us that designing for failure from the very beginning yields far fewer operational interruptions.

Below are key recommendations to bolster the resilience of your API integrations:

Retry Mechanisms: Implement retry logic within your systems to automatically retry failed API calls. This feature helps in handling transient issues and ensures that integration processes can recover from temporary disruptions without manual intervention. Examples are infrequent HTTP status 503 or 504 responses from load balancers.
Queue-based Requests with Dead-Letter Queues (DLQs): Utilize Queues/DLQs to capture and store messages or events that couldn't be processed initially. DLQs act as a safety net for preserving data integrity and allow for convenient investigation and resolution of failed messages to replay them when issues have been resolved.
Comprehensive Logging: Maintain detailed logs of all API interactions, including requests, responses, and relevant metadata. Logging facilitates effective troubleshooting by providing insights into system behavior and aiding in the identification of potential issues.
Unique Identifiers: Each API call is assigned a unique identifier on our side and presented in the response header: Trackingid . These identifiers enhance traceability and enable efficient support by allowing our team to quickly locate and address specific issues as they arise.

By incorporating these resilience measures into your integration workflows, you can minimize the impact of disruptions and ensure smooth access to our cloud services. Additionally, these practices streamline the support process, enabling our team to assist you promptly and effectively whenever needed.

Should you require further assistance or guidance on implementing these measures, please don't hesitate to reach out to our dedicated support team.

Poll vs Push

Our approach

Currently we strive to provide straightforward REST APIs. There is no inherent standard within these types of APIs to provide a push mechanism. Polling is therefore the preferred approach.

Benefits of polling

If one poll fails, chances are that it will be fixed on subsequent polls
No need to worry about missed push messages and resending them

Efficient polling strategies

To facilitate polling and to reduce the amount of data that needs to be processed:

API endpoints that are often polled support timestamps as query parameters for since functionality
This will only return data that's relevant (updated, created) since that timestamp

Need help?

If you need to poll one of the API endpoints and a since mechanism is not provided, feel free to reach out to us.

Single upserts vs bulk operations

Our approach

We prefer singular upserts over bulk operations. While some retrieval endpoints support paged results (see Cursor-based pagination), our creation and update endpoints do not support bulk operations.

Benefits of singular upserts

Atomicity
Clear error handling
Safe retry behaviour

Independent validation and processing ensure that failures for one entity do not impact other operations. Integrations can target specific items that failed and identify the cause more easily. This improves resilience, reduces the blast radius of integration or data mistakes, and gives both you and us better traceability through per-entity tracking IDs.

Trade-off

While the benefits of this approach are clear, it may require more HTTP requests for large datasets. If throughput is a concern, we advise parallelization of these requests, which still provides the same benefits while improving performance.

When parallelizing requests, limit concurrent operations to 5 to avoid overloading the backend systems and running into rate limits.