How Do You Prove a 100-Million-Document Migration Is Complete?
Five steps to account for every record, verify the results, and approve a safe cutover
A migration is complete when every source record has an explained outcome and users can safely work with the migrated records. A progress bar showing “100 million files processed” cannot prove either condition.
Start with five steps. They apply whether you are moving a single repository or planning a migration of 100 million documents.
1. Define what must move
Create a source inventory before transfer begins. Agree what is included, what is excluded, and which versions must survive. Count documents and versions separately.
Turn that inventory into a migration manifest: a protected list connecting each source item to its intended destination and verification result. It should cover:
- Identity and content: source identifiers, versions, locations, file sizes, and checksums where available.
- Business context: metadata, historical dates, attachments, and links to cases or other records.
- Controls: access rules, retention requirements, and legal holds.
- Decisions: destination mappings, transformation rules, and approved exclusions.
Keep the manifest and mapping rules under change control. Record the source snapshot or cutoff they describe, so later changes cannot silently alter an accepted batch.
If the source stays live during migration
Track changes after the baseline: new files, updates, moves, permission changes, holds, and deletions. Use the source’s supported change sequence or checkpoint where available; timestamps alone can miss late changes or clock differences.
Reconcile those changes through an agreed final cutoff. If reliable change tracking is unavailable, use a controlled rescan or write freeze. An export taken while records change is not automatically a consistent snapshot.
2. Move records with safe retries
Give each source item a stable migration identifier and retain its destination document and version identifiers. A repeated request should find the existing result rather than create an uncontrolled duplicate.
Track transfer, import, verification, and acceptance separately. A single “migrated” flag cannot explain that content arrived but its metadata failed.
Before bulk import, agree how normal workflows will behave. Replaying a batch must not send duplicate correspondence, create duplicate transactions, or enable deletion before a legal hold is enforced.
The retry failure to test
Stop a worker after it creates the destination record but before it saves the result. On restart, it must look up that record using a durable identity before attempting another creation. Coordinate workers so they cannot both claim the same unfinished item.
Test content, version, metadata, and relationship updates separately. A safe document-record retry does not establish that every other operation behaves the same way. Changed content under an existing identifier needs an explicit correction decision.
3. Verify the results in the destination
Read back the migrated records and test these five completion requirements. A transfer tool’s success report may cover only some of them.
| Completion requirement | How to verify it |
|---|---|
| Every source item is accounted for | Match each source item and version to its destination record or an approved exclusion. |
| File content is complete and readable | Compare checksums for unchanged files. Open files and check required pages and attachments. |
| Metadata, versions, and relationships are correct | Read back mapped values, historical dates, version order, and links to the correct case or customer. |
| Access, retention, and legal-hold rules are enforced | Test allowed and denied users, retention behavior, and whether holds actually prevent deletion. |
| Users can find and use migrated records | Check indexing and expected search results, then test routine tasks with the intended users. |
Automate checks across all records where practical. Use documented, risk-based samples for usability and specialist review. Sampling supports the checks performed; it does not establish that every record passed.
Keep failed OCR, incomplete indexing, damaged files, and unsupported formats visible. Storage success does not make those records usable, and a legal-hold label does not prove deletion is blocked.
Checksum and transfer-tool details
Compare checksums produced with the same algorithm over the same complete bytes. For an S3 destination, AWS’s integrity documentation explains why an ETag is not always a full-file MD5 checksum, including multipart uploads.
If conversion changes the bytes, preserve the original where required and verify the converted version separately. Check readability and the content that must survive the transformation.
Tools such as AWS DataSync offer verification with a defined scope. Use their results alongside checks of metadata, relationships, permissions, and business use.
4. Explain every difference
At the agreed cutoff, assign every source item one of three outcomes:
- Verified: its destination mapping and required checks passed.
- Approved exclusion: an authorized decision explains why it was not migrated.
- Unresolved: it is missing, failed a check, or still needs a decision.
Match identifiers before comparing totals. One missing item and one duplicate can cancel out in a headline count. When records are consolidated or split, keep the source mappings that explain the difference.
Example: the progress bar says complete, but five versions are missing
A batch contains 1,000 source versions. Twenty are approved exclusions, leaving 980 to verify. The transfer tool reports 980 operations, but reconciliation finds only 975 verified source-version mappings.
Retries inflated the operation count. The batch still has five unresolved versions.
Replay those five using their existing migration identifiers and repeat the required checks. The batch reconciles when 980 verified versions + 20 approved exclusions = 1,000 source versions.
Report differences by batch, repository, record type, or business unit so owners can resolve them. Any acceptance waiver should name the affected records, consequence, owner, and remediation requirement. Keep the waiver separate from a successful verification result.
5. Approve cutover, then plan source retirement
Cutover is the point when users begin working in the destination system. Before approving it:
- Finish the final source changes through the agreed cutoff and record any remaining exceptions.
- Obtain acceptance from the business, records, security, and application owners for their checks.
- Define rollback triggers and how new destination writes would be recovered if users return to the source.
- Agree the source’s read-only period and the evidence required before decommissioning.
Retiring the source is a separate decision covering retained records, recovery, obligations, and dependencies such as encryption keys. Keep manifests, mappings, rules, verification results, exclusions, and approvals under the appropriate access and retention controls.
Four failures to demonstrate before cutover
- Substitute a duplicate for a missing item. Show that identifier checks detect both despite matching totals.
- Stop a worker after record creation. Show that a retry preserves the existing record and its version mapping.
- Change permissions or place a hold during migration. Show that the final update and verification capture it.
- Fail indexing and add a new destination write after cutover. Show how acceptance and rollback handle each case.
How FormKiQ supports verifiable migrations
FormKiQ is one option for implementing this process. Its Legacy System Archive and Access solution supports governed access to legacy documents, and Integration Frameworks connect records to business systems.
The FormKiQ CLI provides import and verification tools, including CSV comparison with --verify. These support the wider plan; source completeness, relationships, access, and business acceptance still need their own evidence.
Talk to FormKiQ about migration mappings, verification coverage, and cutover evidence.