Document Migration and Vendor Lock-In: A Nightmare Scenario We've All Experienced

One of the key concerns with storing and managing documents is the lock-in factor. By having individual employees saving files on their hard drives, or throwing all of the files onto a file server with never enough structure defined, or by using a document management SaaS, there are major technical and process debts that build up from the moment that first file is stored.

Stakeholders who have experience with document handling know that files are way less fun and easy coming out than they were going in. But the pain of leaving them where they are, whether that’s security and compliance risks, documents that are impossible to find, or hardware or software that is falling over in one way or another, will eventually get bad enough that the dreaded migration needs to be started.

The Three Main Components of a Document Migration Project

So what will it take?

Well, it involves three parts, at least:

1. Find a new destination: for most organizations, this new destination is “a cloud”, whether that’s SharePoint, something simple like Google Drive, or a SaaS that promises to both store and manage your documents. Another option is just looking at a cloud provider itself, such as Amazon Web Services.
2. Move the documents: this could be copying files over the internet or a private network, or even services like the AWS Snow Family, where a device can be delivered to your server location, files copied over, and the data driven to its new home at AWS and copied into Amazon S3. But this is only part of the complexity, since you also need to determine HOW to get those files out of the current system, which can be as simple as dragging and dropping, or more complicated than the moon landing.
3. Move and rework the metadata: this piece can be nearly as important as moving over the files themselves. Any applications that use the data are likely to rely on metadata that’s external to the file objects themselves, whether that’s a folder location, tags, or information specific to a subset of documents, like a “Total Amount” field for financial documents like invoices and sales receipts. If documents are copied over without their metadata, restoring and reconnecting that metadata might take more time and effort than the file transfer itself. As with the files themselves, gaining access to the metadata can be a challenge, but often an even larger one than those file objects.

There are other parts in most cases, as well, such as the initial onboarding and customization of the new system, as well as the vendor and cost management of gradually ramping down one system while ramping up another, where there may be months or even several years where both systems are in use.

While it’s not possible to undo decisions that created the challenges within the source system, it is possible to reduce the lock-in risk when choosing and implementing the destination system. This isn’t just for the present, of course, but for that future state when there may be a need to migrate your documents and metadata all over again.

Key Items to Consider when Planning Your Next Document Migration

So what are the things to consider when choosing a new system for storing and managing documents and metadata, in order to reduce migration friction at the outset and in the long run?

Well, I’ve come up with a list for just such an occasion.

1. Open Source, Open Standards, and Open Formats

   Ideally, the destination system will make use of an open source codebase, open standards for metadata, such as Content Management Interoperability Services (CMIS), and open formats for files, such as archiving as PDF or other open file formats.

   In our case, FormKiQ has the ability to make all source code available to license holders (including our free Open Core product) with some restrictions, and we are able to provide interoperability with CMIS. For file formats, we have conversion functionality available to create PDFs from closed file formats such as MS Word.

2. API Access and Integration

   Robust API access allows for more seamless integration with other systems and easier extraction of data.

   In our case, FormKiQ’s API capabilities enable automation of document and metadata migration processes, significantly reducing manual effort and potential errors.

3. Flexible Deployment Options

   The ability to control your own documents and metadata is made easier with solutions that deploy directly into a cloud account under your control. This also allows for adherence to regional data sovereignty laws and optimization of latency for end-users, and can make a SOC 2 audit or compliance questionnaires far easier to complete.

   In our case, FormKiQ not only runs in AWS regions worldwide, deploying into your own AWS account, it’s also available to run locally or on other platforms, and is also available within AWS GovCloud US West.

4. Modular Architecture

   A modular system design allows for parts of the document management solution to be replaced or upgraded without impacting the whole. This can be crucial for adopting new technologies or making system improvements with minimal disruption.

   In our case, while FormKiQ is a complete solution for document and enterprise content management, the modular architecture and available source code (with some restrictions) allows for better upgrades or even replacements of entire functionality with other options.

5. Data Portability Services

   Tools and services can be utilized that facilitate data portability. As mentioned above, AWS’s Snow Family for physical data transfer is a great example. Similarly, services that can export data in a standardized, easily ingestible format can simplify migrations.

   In our case, FormKiQ provides multiple ways to import and export data, including our API, our File CLI, and various AWS products and services. As our documents are stored in Amazon S3 and the metadata in Amazon DynamoDB, the data is about as portable as possible.

6. Comprehensive Documentation and Support

   Detailed documentation and responsive support can drastically reduce the complexity of migrating documents and metadata. Documentation should cover common migration paths, data export/import procedures, and troubleshooting.

   In our case, FormKiQ provides onboarding and operational support, including being available via Slack, email, or videoconference to help plan, execute, and troubleshoot migration and other tasks.

7. Scalable and/or Usage-based Pricing Model

   One of the drawbacks of a SaaS product hosted by an external vendor is the pricing models that are often employed, where growth in users can result in per-user surcharges, or where pricing is based on the maximum usage possible, rather than actual usage.

   In our case, usage of FormKiQ, whether compute or storage, is calculated and charged through your AWS account, removing any service margins that are experienced with external vendor infrastructure.

8. Regular Data Backups and Exports

   Implementing a routine for regular data backups and exports not only serves as a good practice for disaster recovery but also ensures that you always have an up-to-date copy of your data in a format that can be migrated to another system.

   In our case, FormKiQ works within the existing AWS products and services to ensure reliable backup, restore, and disaster recovery, which are also available to help with migration, if needed.

9. Vendor Neutrality

   Solutions and integrations that are vendor-neutral ensure that your system can interoperate with a wide array of other systems and services, reducing the risk of vendor lock-in.

   In our case, FormKiQ’s usage of AWS products, including Amazon S3 and Amazon DynamoDB, within a cloud account under your organization’s control, combined with our source code often made available (with some restrictions), ensures that it’s possible to detach our participation as a vendor completely without losing access to the system or its functionality.

10. Customization and Extensibility

    A system that allows for easy and workable migration generally also allows for customization and extensions without proprietary restrictions. In addition to the benefits for migration in or out, being able to tailor the system to your specific needs without altering its core ensures that you can adapt to new requirements without being forced to migrate again to another new system.

    In our case, FormKiQ is the most flexible and customizable document and enterprise content management platform available, owing to its source code availability (with some restrictions), modular architecture, and its ability to leverage the 200+ products available within your AWS account.

Finding the Right Migration Partner(s)

Navigating the complexities of document management and migration demands a strategic approach that emphasizes flexibility, control, and future-readiness. We have worked hard at FormKiQ to embody these principles by offering a source-available, cloud-accessible, and AWS-powered platform designed to mitigate the risks of vendor lock-in and adapt to the evolving needs of organizations.

We are committed to open standards, robust API integration, and flexible deployment options to ensure unparalleled autonomy and efficiency when managing your enterprise information.

By choosing FormKiQ, organizations can not only ensure a seamless migration process but also secure a scalable, customizable solution that can evolve alongside their growth and changing requirements.

For more information on the tools and solutions we provide to migrate your documents and metadata, please contact us or schedule a consultation call.