Published

A document ingestion pipeline imports or synchronises files from other systems. It should recognise repeated operations without confusing them with new revisions or separate business records. Use scoped source identities, recorded operation outcomes, and reconciliation together; filenames and matching file contents answer only part of the question.
Consider a supplier certificate copied from SharePoint into two case files. A delivery is retried after a timeout, and the source file is later renamed. The retry should recover its earlier result, the rename should update the right mapping, and the two legitimate case records should remain distinct.
The same source change is delivered or processed again. Idempotent handling recovers the recorded outcome instead of independently creating the same destination effect.
Two inputs contain matching bytes. A checksum helps compare content, but the files may belong to different cases, owners, permissions, or retention policies.
A new source revision is a third case. It may belong to the same mapped document while requiring a new content version or metadata update. Choose the business-record boundary before deciding whether to reuse a document, add a version, or create a separate record.
Scope a source reference to the system, tenant or account, repository, and object identifier. Store the destination reference and the accepted source state with it. Define how copies, cross-repository moves, deletion, and restoration affect that relationship.
| Signal | What to use it for |
|---|---|
| Scoped source object ID | Resolve the mapped source object. Confirm the provider’s identity rules for rename, move, copy, and restore. |
| Source revision or change token | Recognise the observed source state. Distinguish content changes from metadata changes using the provider’s actual semantics. |
| Operation reference | Associate a requested effect with its claim, attempts, destination result, and recovery state. |
| Checksum | Compare captured file contents. Keep business-record identity and access decisions separate from a content match. |
| Path and business metadata | Update names, organisation, classification, routing, and access mappings as required by the source change. |
For example, Microsoft documents that OneDrive item IDs persist through rename and move operations. Keep the drive/repository context in the mapping and agree how operations across repositories are handled; a path change alone should not be your record-creation signal.
Microsoft Graph also distinguishes a file’s eTag for content and metadata from its content-focused cTag. The latter is not returned for folders in the documented business/SharePoint case. Treat tokens according to the source API, without assuming that every changed tag means new file bytes or that opaque tokens can be sorted as revision numbers.
Choose an operation reference that identifies the intended effect, such as one source revision entering one destination mapping under a defined transformation. An event ID can identify a delivery, but separate notifications may describe the same source state. Record the relationship between deliveries and the operation they trigger.
Claim the operation with concurrency control. Keep its source state, destination mapping, and processing status durable.
Use a supported idempotent write or a recoverable destination reference. Record which document or version the operation produced.
Return a completed result on retry. For interrupted work, reconcile the destination before repeating a write whose outcome is unknown.
A claim written before the destination call shows that work started; it does not show that it completed. If a worker stops after creating a document but before recording the result, recovery must identify that effect or raise an exception. Repeating a blind create can produce duplicates, while treating every existing claim as complete can lose work.
Define worker ownership, expiry, and recovery so an earlier worker cannot overwrite a newer accepted result. If the source API does not provide an ordering you can rely on, retrieve the current state and apply the configured conflict rule instead of comparing opaque version strings.
In this illustrative mapping, source object SP-991 in a particular library maps to document DOC-1042 in one destination site. The integration has defined a recoverable destination operation reference for each accepted source revision.
The ingestion process claims the operation for revision A and writes its destination version. The response is lost, leaving the recorded outcome uncertain.
The retry finds the unfinished operation and checks the destination using its operation reference. It records the already-created result and returns that outcome. If the result cannot be resolved unambiguously, it holds the operation for repair instead of independently creating another version.
A later source update changes the name while the content remains the same. Using the provider’s identity and content-change signals, the ingestion process updates the mapped name and any affected routing or access information according to policy.
Another case receives an identical certificate. The matching checksum is useful evidence, but that case keeps its own business relationship and access policy. It is not silently merged with DOC-1042.
Compare the expected source inventory with destination mappings and outcomes: missing objects, unmatched destination records, stale revisions, failed operations, metadata differences, and reported deletions. Record the source scope, permissions, checkpoint, and comparison time, then assign each exception an owner and repair action. Counts alone will miss different records that happen to total the same number.
A change feed also has its own contract. Microsoft’s driveItem delta documentation says it presents the latest state rather than every intermediate change, can repeat an item, and can require resynchronisation when a token is no longer usable. Use the documented pagination and occurrence rules, and preserve a recoverable checkpoint as local outcomes are committed. A current-state sync does not establish a complete revision history.
Handle source deletion and lost visibility separately: an object missing from a scan may require an access or source check before any destination action. Keep a reported deletion linked to the mapping and apply the destination’s retention, hold, access, and disposition rules. Reconciliation should surface uncertainty rather than convert it into an automatic purge.
A reliable ingestion process can explain which source object and state it accepted, which destination effect it recorded, and what remains unresolved. Deduplication handles repeated work within that scope; reconciliation finds the gaps and drives repair as the two systems change.
The open-source foundation — API-first, deployable into your own AWS account, and free to use. Right for architecture validation and early implementation.
Production-ready editions for departments and complex workflows. Start with a Proof-of-Value deployment or go straight to production.
For governance-heavy environments with residency, sovereignty, assurance, and multi-jurisdiction requirements. Talk to us about the right deployment model.