Overview / Introduction
Modern applications communicate over networks that are not perfectly reliable. A customer may click a Pay button twice, a mobile application may retry after a timeout, or an API gateway may resend a request because it never received a response. If the backend treats every repeated request as new, one business action can accidentally become two payments, two orders, two refunds, or two account transfers.
Idempotency is an API design technique that makes repeated execution of the same logical request safe. For operations where duplication can cause financial or operational damage, the client sends a unique idempotency key. The server uses that key to recognize retries and returns the result of the original operation instead of performing the operation again.
This blog explains idempotency from a backend engineering perspective, with payment and order examples. It covers request flow, data storage, concurrency, database constraints, expiration, benefits, drawbacks, and a practical Node.js-style implementation.
Why Duplicate Requests Happen
• A user double-clicks a payment or order button.
• The client times out even though the server completed the operation.
• A mobile network disconnects after the server commits data but before the response reaches the device.
• Automatic retry logic in a client, gateway, queue, or service sends the same command again.
• Two application instances receive nearly simultaneous copies of the same business request.
A Simple Failure Scenario
Assume a customer submits an order for ₹5,000. The backend successfully creates the order, but the response is lost. The client cannot know whether the request failed before or after processing, so it retries. Without protection, the second request can create another order and possibly trigger another payment. The technical failure is not the retry itself; retries are often necessary. The problem is a backend that cannot recognize that both requests represent the same intended action.
What Is Idempotency?
An operation is idempotent when performing the same logical operation multiple times has the same intended effect as performing it once. In an API, this does not necessarily mean that every response is byte-for-byte identical or that no logs are added. It means the protected business side effect is not repeated.
| Operation | Typical Behavior | Idempotency Concern |
|---|---|---|
| GET /orders/123 | Reads an existing resource | Naturally safe when it only reads data |
| PUT /profile/123 | Sets a resource to a requested state | Repeating the same update normally keeps the same state |
| DELETE /cart/123 | Removes a resource | Repeated calls should leave the resource deleted |
| POST /payments | Creates a new financial action | Dangerous if a retry creates another payment |
| POST /orders | Creates a new order | Dangerous if a retry creates another order |
What Is an Idempotency Key?
An idempotency key is a unique value that identifies one logical operation. The client generates the key before sending a sensitive request and reuses exactly the same key when retrying that request. A different business operation must use a different key.
POST /api/payments HTTP/1.1
Content-Type: application/json
Idempotency-Key: 8f6d2b6e-7b34-4a66-a8f8-0b70f20f6a11
{
"orderId": "ORD-10452",
"amount": 5000,
"currency": "INR"
}Recommended Key Properties
• Unique enough to avoid accidental collisions; UUID-style values are a common choice.
• Generated once for the logical action and retained for retries.
• Scoped appropriately, for example by account or API operation when required.
• Not derived only from timestamps, which can collide or be regenerated on retry.
• Treated as an identifier, not as authentication or authorization.
The Core Rule
The server must associate the key with the request and its outcome. When the same key arrives again, the backend should not repeat the protected side effect. It should return the stored outcome, or a clear in-progress/conflict response if the first request has not finished yet.
How the Backend Flow Works
The following original flow illustrates a common implementation. The exact storage technology can vary, but the decision point must happen before the protected operation is repeated.
Figure 1. Idempotent request flow for a payment or order API.
Step-by-Step Request Lifecycle
• 1. The client creates an idempotency key and sends it with the request.
• 2. The API authenticates the caller and validates the payload.
• 3. The backend checks its idempotency store for the key.
• 4. If a completed record exists, the backend returns the previously stored result.
• 5. If no record exists, the backend atomically reserves the key or creates an in-progress record.
• 6. The business operation executes once, ideally inside an appropriate transaction.
• 7. The backend stores the final status and response against the key.
• 8. Any later retry with the same key receives the stored result instead of triggering the operation again.
Suggested Idempotency Record
| Field | Purpose |
|---|---|
| key | Unique identifier supplied by the client |
| requestHash | Fingerprint used to detect reuse of the same key with a different payload |
| status | IN_PROGRESS, COMPLETED, or FAILED according to the chosen policy |
| responseCode | HTTP status returned by the original operation |
| responseBody | Stored response or enough information to reconstruct it |
| resourceId | Created payment/order identifier |
| createdAt / expiresAt | Retention and cleanup timestamps |
Practical Backend Example
The example below is intentionally framework-friendly rather than tied to a specific production database. The important behavior is the sequence: validate the key, reserve it atomically, perform the operation once, then persist the result.
app.post('/payments', async (req, res) => {
const key = req.get('Idempotency-Key');
if (!key) return res.status(400).json({ message: 'Idempotency key required' });
const requestHash = hashRequest(req.body);
const existing = await idempotencyStore.find(key);
if (existing) {
if (existing.requestHash !== requestHash) {
return res.status(409).json({ message: 'Key reused with different request' });
}
if (existing.status === 'COMPLETED') {
return res.status(existing.responseCode).json(existing.responseBody);
}
return res.status(409).json({ message: 'Request is already being processed' });
}
await idempotencyStore.reserveUnique(key, requestHash);
const payment = await paymentService.create(req.body);
const body = { id: payment.id, status: payment.status };
await idempotencyStore.complete(key, 201, body);
return res.status(201).json(body);
});Why a Simple “Check Then Insert” Is Not Enough
Two requests with the same key can arrive at almost the same time. Both may check the database before either has inserted the key. If both see “not found,” both can continue and create duplicate side effects. This is a race condition. The reservation step therefore needs an atomic guarantee, commonly a unique database constraint or another compare-and-set style operation.
CREATE TABLE api_idempotency (
idempotency_key VARCHAR(100) PRIMARY KEY,
request_hash VARCHAR(128) NOT NULL,
status VARCHAR(20) NOT NULL,
response_code INT,
response_body TEXT,
created_at TIMESTAMP NOT NULL,
expires_at TIMESTAMP NOT NULL
);Idempotency and Database Transactions
Transactions and idempotency solve related but different problems. A transaction protects consistency inside one execution: either a set of database changes commits together or it rolls back. Idempotency protects the system across repeated executions of the same logical request. A robust payment or order workflow often needs both.
Important Design Decisions
1. Same Key, Different Payload
A client should not be allowed to reuse one idempotency key for a different operation. Storing a normalized request hash makes this detectable. If the key matches but the payload hash differs, the API can reject the request with a conflict response rather than silently returning an unrelated result.
2. What Should Happen While the First Request Is Running?
When a duplicate arrives while the first request is still in progress, the API can reject the duplicate as “already processing,” wait for the original operation for a bounded period, or return a status that tells the client to retry later. The chosen behavior should be documented and consistent.
3. How Long Should Keys Be Stored?
Idempotency records should not necessarily live forever. Retention depends on the maximum realistic retry window and business requirements. A short TTL reduces storage but may allow a very late retry to be treated as new. A long TTL provides a larger safety window but increases storage and cleanup requirements.
Where to Use Idempotency
| Use Case | Risk Without Idempotency | Typical Protected Action |
|---|---|---|
| Payment creation | Customer can be charged twice | Create payment/charge |
| Order creation | Duplicate orders and inventory reservations | Create order |
| Refund | Customer may receive multiple refunds | Create refund |
| Money transfer | Balance may be moved more than once | Initiate transfer |
| Subscription creation | Multiple subscriptions can be created | Create subscription |
| External lender/provider submission | Same application may be submitted repeatedly | Submit enrollment/application |
When Idempotency May Be Unnecessary
Not every endpoint needs an idempotency store. Read-only endpoints normally do not create duplicate side effects. Some state-setting operations are already naturally idempotent. Adding key storage to every endpoint can create unnecessary complexity, so the mechanism is most valuable where repeated side effects are expensive, irreversible, or difficult to reconcile.
Security and Operational Considerations
• Authenticate and authorize the request normally; an idempotency key never replaces security checks.
• Avoid storing sensitive payment or personal data unnecessarily in the idempotency response record.
• Scope keys so one tenant or user cannot accidentally collide with another when the system design requires isolation.
• Monitor duplicate-hit rates, conflicts, expired keys, and stuck IN_PROGRESS records.
• Define retry behavior for downstream providers because your API can be idempotent while a downstream integration is not.
Benefits and Drawbacks
| Benefits | Drawbacks / Trade-offs |
|---|---|
| Prevents duplicate business side effects during retries. | Requires additional storage and cleanup logic. |
| Makes client retries safer after timeouts or network failures. | Concurrency must be handled correctly; naive check-then-insert logic can still race. |
| Improves reliability for payment, order, refund, and transfer APIs. | Storing responses or metadata increases operational complexity. |
| Provides a traceable identifier for one logical operation. | Key expiration policies must balance safety window and storage. |
| Can improve user trust by avoiding accidental duplicate actions. | Incorrect key reuse by clients must be detected and handled. |
Best Practices
• Require idempotency keys on high-risk create operations rather than treating them as an optional convention.
• Use a database unique constraint or equivalent atomic reservation to handle concurrent duplicates.
• Bind the key to the request payload using a deterministic hash.
• Store the result needed to reproduce the original response or reliably reconstruct it.
• Define behavior for failures: decide which errors are retryable and whether failed attempts release or retain the key.
• Use TTL/expiration and a cleanup process appropriate to the business retry window.
• Keep idempotency logic close to the API boundary, but keep business transactions and downstream guarantees explicit.
• Test double-clicks, client timeouts, concurrent requests, service restarts, and delayed retries – not only the happy path.
Idempotency vs. Related Techniques
| Technique | Primary Purpose | Does It Prevent Duplicate API Effects by Itself? |
|---|---|---|
| Idempotency key | Recognize repeated logical requests | Yes, when implemented correctly |
| Database transaction | Keep one execution atomic and consistent | No |
| Unique business constraint | Prevent duplicate values/resources | Sometimes, but not a complete API retry strategy |
| Rate limiting | Control request frequency | No |
| Message deduplication | Suppress repeated message processing | Yes for the protected message flow, with its own scope and retention |
Conclusion
Retries are a normal part of distributed systems, but duplicate payments and orders should not be. Idempotency gives a backend a reliable way to distinguish a retry from a new business action. The core pattern is straightforward: identify the logical request with a stable key, reserve that key atomically, execute the side effect once, and reuse the stored result for duplicates.
For fintech and commerce systems, the strongest implementation combines idempotency with database constraints, transactions, careful downstream integration, request hashing, expiration policies, and concurrency testing. The result is not merely a cleaner API; it is a safer user experience when networks, clients, and services behave unpredictably.