High-Volume Document Ingestion: How to Control Backpressure
Control the flow from upload to usable record with stage limits, safe retries, and measurable recovery
Published
Uploads can arrive faster than scanning, OCR, indexing, AI, or an ERP can handle them. A queue absorbs the burst, but it does not create downstream capacity. Without controls, the backlog grows while users see throttling, repeated work, and unpredictable completion times.
Backpressure means limiting how quickly work moves into a busy stage, and slowing or pausing intake when the system cannot safely absorb more. Design for completed, usable documents rather than the largest possible upload count.
1. Separate receipt from readiness
Define a durable acceptance point: source bytes are stored, their integrity is checked, and enough metadata and processing intent exist to resume the item. Reconcile accepted items that have not progressed, including cases where storage succeeds but the expected processing does not start.
Give the caller a stable ingestion reference and expose stage status. If security checks must precede access, enforce that rule in the application and access policy; a staging location or pending status alone is not quarantine. “Uploaded” should not imply that a document is safe to download, searchable, enriched, or delivered to a business system.
| Stage | Completion evidence and next-step rule |
|---|---|
| Receive | Stored source, checksum, source reference, and durable processing intent. The item remains restricted pending required checks. |
| Validate and scan | Metadata and security results. Rejected or suspicious files go to a controlled exception path. |
| Extract or transform | Output tied to the exact source version, processing configuration, and provider job reference where applicable. |
| Index | Searchable output and verified access filtering. A completed OCR job alone does not establish search readiness. |
| Deliver | Confirmed downstream outcome, or a visible pending state if delivery is unresolved. |
2. Give every item and stage a recoverable identity
For bulk imports, use a manifest listing expected source items, versions, checksums, sizes, metadata, and required processing. For continuous intake, retain equivalent information per item. Define a stable item key within the customer and source system. Identical bytes can legitimately belong to different cases, so a content hash alone is not always the business identity.
Record stage attempts and outcomes against that identity. A repeated message should resume missing work or return the existing result. Queue messages can carry protected object references and processing instructions; keep document bytes and unnecessary personal information out of the message payload.
S3 notifications can be duplicated or arrive out of order. Tie results to the source version and prevent a late result from overwriting the processing state of a newer version.
For external services, persist the request intent and returned job or transaction ID. Use the provider’s idempotency mechanism where supported, with a stable key for that operation. If the response is lost, reconcile the uncertain operation before creating another. Local duplicate detection cannot guarantee that a remote service was called only once or never charged twice.
3. Control concurrency and request rate separately
Concurrency limits the work running at once. Rate limits control requests over time. Ten workers making one call every 0.2 seconds could generate about 50 calls per second; that exceeds a downstream allowance of 20 calls per second even though concurrency looks modest. Multiple calls per document and retries add further demand.
Choose controls for the actual constraint: requests per second, pages per minute, active OCR jobs, model tokens, database writes, or a daily cost budget. Use a shared rate limiter where workers compete for one allowance. Account for all callers using that allowance, including retries and other workloads in the AWS account and Region.
For asynchronous OCR, worker count is only part of the picture. A worker can submit a job and finish while Textract keeps processing. Textract has separate request-rate and concurrent-job quotas. Track outstanding jobs until completion; successful submission is not completed OCR. AWS recommends completion notifications instead of repeatedly polling each job.
Apply the controls at the correct AWS boundary
For Lambda–SQS in standard polling mode, maximum concurrency limits one event source mapping. Reserved concurrency applies to the function. If you set both, reserved concurrency must cover the sum of the mappings’ maxima, with capacity for other triggers. Neither setting enforces a downstream request-rate limit.
AWS also provides a provisioned polling mode with its own controls. It cannot be combined with the event source mapping’s maximum-concurrency setting. Confirm the selected mode and current quotas rather than copying a single concurrency number into every deployment.
Review queue and event source configuration together. AWS recommends a visibility timeout of at least six times the function timeout, plus any batching window, and a maxReceiveCount of at least five. Validate the deployed settings and recovery window against the workload.
Record tuning in supported deployment configuration or infrastructure code so upgrades preserve it. Load-test the deployed processing paths and configuration before increasing concurrency or batch size.
For example, a nightly invoice import should slow down before it consumes the OCR capacity needed by interactive uploads. Give those workloads explicit capacity budgets. Separate queues can help when backed by scheduling and consumer limits. Where workloads share a processing path, pace the bulk producer and verify that interactive work still progresses. Pause intake at agreed backlog-age, retention, or cost thresholds, then resume below a lower threshold to avoid repeatedly stopping and starting.
4. Retry failed work without repeating successful effects
AWS documents at-least-once processing for Lambda–SQS event source mappings. Make repeated execution safe at each stage, including notifications, generated files, and business-system delivery.
Distinguish retryable outages and throttling from invalid metadata, unsupported files, and security failures. Use bounded retries with backoff and jitter, accounting for retries already performed by the SDK or platform. Give exhausted work a reviewable exception record with its item reference, stage, reason, and owner.
Monitor processing outcomes as well as queue failures. An application can record an unsuccessful operation while its message handler returns successfully. Check item and stage status, overdue work, Lambda errors, and queue alarms together. Correct the cause and retry only the affected operation.
Use partial batch responses for custom consumers that process batches
If nine messages succeed and one fails, enable ReportBatchItemFailures and implement the matching partial batch response so only the failed message is retried. An uncaught exception still fails the whole batch. With FIFO, stop after the first failure and return failed and unprocessed messages to preserve order.
Partial batch reporting also changes polling backoff: Lambda does not reduce polling in response to reported failures. Keep explicit rate controls and outage handling so repeated failures do not overwhelm a dependency.
A dead-letter queue needs alerts, an owner, and controlled replay. For standard SQS queues, AWS recommends longer retention on the dead-letter queue than on the source queue. Ensure source files and temporary processing objects also remain available for the intended recovery window. Replay at a controlled rate after the cause is fixed.
5. Measure completion and the time to recover
Track arrival and completion rates, oldest-item age, stage latency, throttling, retries, failed items, and cost. Break these down by priority and document class. A stable total queue depth can hide one class that never completes.
Use SQS metrics for operational alarms, but retain acceptance timestamps and item outcomes for completion reporting. Queue counts are approximate, and oldest-message age can exclude repeatedly failing messages. An empty queue can also mean work has moved into an asynchronous OCR job.
Estimate backlog recovery from spare capacity, not total capacity. In an illustrative stage completing 20 items per second while 12 new items arrive each second, only 8 items per second reduce the backlog. A backlog of 48,000 items would take about 100 minutes to clear under those steady assumptions, before additional retries or interruptions. These figures illustrate the calculation; they are not measured throughput results.
Reconcile each manifest item into one current outcome, such as completed, failed, excluded, or pending. Count stages separately so retries do not inflate document totals. Close a batch only when its remaining exceptions are resolved or explicitly accepted under the approved completion rule.
Run a realistic failure exercise
Load representative work
Mix real document formats, page counts, metadata sizes, and priorities. Measure interactive response times while the bulk load is running.
Interrupt a downstream stage
Throttle OCR or pause delivery to the ERP. Confirm bounded retries, a visible backlog, and intake controls before storage or cost limits are exhausted.
Lose a response and replay work
Simulate a successful external operation whose response is lost, a duplicate or out-of-order event, and an action failure with no Lambda error. Confirm that monitoring finds the failure and recovery avoids duplicate documents or transactions.
Restore service and reconcile
Measure recovery time, verify that urgent work still receives capacity, and reconcile the manifest with stored documents and final downstream outcomes.
Technical guidance reviewed: November 10, 2026. Confirm service quotas, polling mode, processing costs, and processing capabilities for the deployed configuration.
Scale the work through verified completion
Track each item from receipt through the processing stages and business outcome that define readiness. Bound concurrency and request rates, preserve progress across retries, and reconcile the results after recovery so a successful upload cannot conceal unfinished downstream work.