How Should an SFTP Document Ingestion Gateway Be Designed?

SFTP is a familiar way for banks, insurers, manufacturers, public bodies, and legacy applications to deliver documents. A successful transfer does not establish that the delivery is complete, safe, routed to the right customer, or accepted by the business process. Those outcomes need their own controls and evidence.

Treat SFTP as the transport boundary of an ingestion workflow. Define how the gateway recognizes a complete delivery, validates it, creates or updates the document record, handles retries, and tells the sender what actually happened.

FormKiQ’s point of view: connect the file drop to a governed record

The useful unit of work is a traceable delivery with an outcome, not a file appearing in a folder. Connect the partner identity, delivery identifier, received bytes, document record, workflow state, and acknowledgment so operations can explain both successful intake and exceptions.

FormKiQ’s Secure File Transfer Gateway, available with Advanced and Enterprise, supports ongoing SFTP and managed file transfer into its governed document layer. Configure the gateway’s source connection, metadata mapping, and routing for the partner contract. Agree and verify the completion protocol, scanning, duplicate behavior, acknowledgment format, and reconciliation requirements for the deployment.

Separate receipt from business acceptance

A transport receipt can be returned before OCR or business review finishes, provided the sender understands its meaning. Never report business acceptance merely because the SFTP connection closed successfully.

Give each sender an isolated identity and route

Use distinct accounts and credentials for partners and environments. Limit each identity to the required directories and operations, disable interactive shell access where applicable, and have clients verify the server host key. Document key rotation, expiry, and emergency revocation without relying on shared credentials.

Map the authenticated sender to an approved FormKiQ site, document class, and workflow through controlled configuration. Treat a site identifier or routing instruction inside a filename, manifest, or document as untrusted input until validated against that mapping. A partner must not route a delivery into another customer’s site by changing a field.

Restrict the worker identity as well as the SFTP account. Scope its landing-area access, decryption permissions, document API access, and acknowledgment destination to the work it performs. Network allowlists can supplement these boundaries, but do not establish which document destinations the sender is entitled to use.

Agree how a complete delivery is committed

Choose a completion contract that the actual client, server, and storage backend support. Temporary filenames followed by rename can be useful, but do not assume filesystem-style atomic rename on an object-storage-backed service. A manifest or completion marker is another option: send it after the payloads, then verify every declared item rather than treating the marker alone as acceptance.

The AWS Transfer Family client guidance notes that interrupted transfers can leave partial objects in S3. If Transfer Family is the chosen transport, use its applicable completion and partial-upload signals alongside the agreed validation rules. A generic object-created event should not, by itself, establish that the intended delivery is complete.

A manifest should identify the partner-scoped batch, schema version, expected items, byte sizes, checksums, and required metadata. Set a deadline for missing items and a rule for unexpected files. Decide whether the batch must be accepted as a unit or whether valid items may proceed independently with explicit per-item outcomes.

Capture the exact storage version or a protected staging copy used for validation. Prevent later sender overwrites from replacing the bytes between checksum verification, scanning, and ingestion. Completion detection is insufficient if the accepted path can change underneath the worker.

Quarantine before exposing documents to processing

Keep new arrivals in a restricted landing area outside normal search, previews, and business workflows. Validate allowed file types against actual content, names and paths, size limits, and manifest checksums. Require a known clean scan result before promotion; scan errors and unavailable scanners need an exception path rather than permissive fallback.

Follow the principles in the OWASP File Upload Cheat Sheet. For archives, constrain expanded size, item count, nesting, and processing time, and reject traversal paths. Encrypted or unsupported files should enter an agreed review path. Do not send confidential payloads to an external scanning service without approving that data flow.

Set partner quotas and processing budgets as well as per-file limits. A large delivery can exhaust storage, queue capacity, or OCR spend even when every individual file is within its limit. Retain rejected evidence only for the approved investigation and retention purpose, with narrower access than ordinary documents.

Make retries safe across the persistence boundary

Use a stable delivery key, such as partner, batch ID, and item ID, and bind it to the captured payload checksum. The same key and payload should return the recorded outcome. The same key with different bytes is a conflict to resolve, not a reason to overwrite a previously accepted document. A filename or content hash alone is not always a sufficient business identity.

Persist an ingestion ledger with conditional state transitions so concurrent workers cannot both claim a new item. Record the destination document and version identifiers and the downstream work still required. Handle the failure window where document creation succeeds but recording its result fails: use a supported idempotent creation path or a durable correlation and lookup mechanism before retrying creation. A ledger flag alone cannot make two separate systems commit atomically.

S3 event notifications use at-least-once delivery, and AWS’s ordering and duplicate-event guidance explains that notifications can arrive out of order. Build replay-safe transitions and downstream actions around the delivery identity. Do not promise exactly-once business effects just because a queue or worker suppresses a duplicate event.

Route and process with durable provenance

Keep the authenticated sender, source location, receipt time, payload checksum, delivery key, manifest version, routing-map version, and resulting FormKiQ document or version together. Restrict sensitive source names in general logs. Map external codes to controlled metadata values and quarantine unknown or unauthorized destinations.

After ingestion, schedule the required OCR, classification, extraction, review, or business-system update. Make the handoff durable and recoverable: if event publication fails after document persistence, reconciliation must find and reschedule the missing work. A retry should resume the unfinished stage rather than repeat every completed action. Separate input and output locations or event filters so processing artifacts cannot trigger recursive ingestion.

Set explicit rules for corrections and new versions. Identical bytes can legitimately belong to separate business records, while a corrected invoice can share a business reference with an earlier delivery. Let the partner contract and workflow determine the result instead of silently deduplicating either case.

Acknowledge outcomes and reconcile missing work

Return machine-readable receipts with batch and item identifiers, the acknowledged stage, timestamps, outcomes, reason codes, and a safe correlation reference. Store the receipt before delivery, retry failed acknowledgment delivery independently, and let the sender retrieve the current result without uploading the payload again.

Reconcile expected schedules and manifests against landing objects, the ingestion ledger, FormKiQ records, workflow results, and acknowledgments. Include a missing-delivery alert: no upload event means no event-driven worker will notice that the partner failed to send a scheduled batch. Route stuck or dead-lettered items to an owner with controlled replay tools.

Measure oldest pending item, missing files, scan failures, processing latency, duplicates, conflicts, and undelivered acknowledgments by partner. Define retention and cleanup for landing, quarantine, and accepted source copies; do not delete the only recoverable evidence before durable ingestion and the agreed recovery window are established.

Worked example: retry after a lost acknowledgment

1. 1. Receive a three-file batch

   Partner P-17 submits batch B-204 with three declared items. Only two complete files arrive. Record receipt, keep the batch pending, and identify the missing item. Do not announce complete batch acceptance.

2. 2. Validate and ingest the completed delivery

   The missing file arrives before the deadline. Capture and validate all three payloads, ingest the approved items, and record their FormKiQ document identifiers. Each item then has an independent processing outcome.

3. 3. Recover from an acknowledgment failure

   The receipt is stored but its delivery fails. The partner retries the same batch. Matching delivery keys and checksums return the existing outcomes and retry the acknowledgment without creating new documents or repeating completed business actions.

4. 4. Detect a changed payload

   A later retry reuses one item identifier with different bytes. Mark the conflict for review and retain its provenance. Process an authorized correction through the agreed replacement or versioning workflow.

Demonstrate failures before onboarding partners

• Interrupt an upload and send a completion marker early. Show that partial or missing items cannot enter ordinary processing.
• Overwrite a staged file after validation and attempt cross-partner routing. Show that the captured payload and permitted destination remain enforced.
• Replay concurrent events and stop a worker after document creation but before ledger completion. Show recovery without duplicate records.
• Fail scanning, downstream event publication, and acknowledgment delivery. Show separate exception handling, reconciliation, and safe replay.