Metadata Search vs. Full-Text Search: Which Document Queries Need Which Architecture?

Full-text search is powerful, but it is not the default answer to every document query. Many high-value business questions use known identifiers, dates, classifications, relationships, and status. FormKiQ’s attribute search and configured composite keys can support known lookup patterns without requiring content search. The right choice depends on query flexibility, freshness, authorization, and the workload you actually measure.

Start with a query inventory: what users ask, which fields or content they search, which results they may see, and how complete and current the answer must be. Treat the patterns below as design choices to validate in your deployment.

FormKiQ’s point of view: define what a correct answer means

We see operational retrieval and content discovery as complementary parts of a document platform. “Find this contract” and “find clauses about assignment” need different evidence that the search worked. A ranked list should not silently become the authoritative population for a business action.

Choose the simplest search path that meets the query’s requirements, then prove its coverage, freshness, permissions, and cost. Keep stable document and business identifiers across the paths so a discovery result can lead back to the governed record. Adding a search engine should expand useful queries without weakening control of the underlying documents.

Separate retrieval from discovery

Retrieval starts with a known fact: invoice number, contract ID, employee ID, case number, vendor, date range, status, or classification. The user expects exact and complete results.

Discovery starts with language: documents discussing a topic, clauses similar to an example, references to a concept, or content containing a phrase. The user expects ranked relevance and may refine the query.

The two modes have different correctness models. An operational report must account for every qualifying record within its stated scope and cutoff; a relevance-ranked first page is insufficient. Topic discovery can use curated topic metadata or a separate retrieval system that matches meaning. Keyword search alone does not establish conceptual similarity.

Use metadata for known business dimensions

Metadata search works well for:

• documents for one customer, vendor, employee, or case;
• records in a defined status or workflow queue;
• contracts expiring within a date range;
• documents with a sensitivity or retention classification;
• invoices by company, currency, year, and payment state;
• all artifacts related through a stable entity ID.

FormKiQ lets you configure composite keys in a site or classification schema for supported multi-attribute searches. This configures attribute combinations within FormKiQ’s existing DynamoDB-backed search model; it does not mean creating custom database indexes. Check that the configured combinations support the queries you need, and validate supported range, sorting, and pagination behavior. Existing documents may need reindexing after composite-key changes.

Use OpenSearch for ad hoc queries, aggregations, and analytics

Predictable lookup patterns are a strong fit for configured composite keys. When users build changing combinations of metadata filters, Boolean conditions, sort orders, and aggregations, OpenSearch may be the better fit even if they never search document text. Its value includes flexible structured queries, alongside linguistic discovery.

For example, an analyst may combine vendor, currency, amount range, approval status, and receipt date, then change the grouping for the next question. Rather than preparing a composite key for each anticipated combination, evaluate the OpenSearch path over the indexed fields. In FormKiQ, advanced queries use /queryFulltext; supported multi-attribute criteria are also available through /searchFulltext. Despite their endpoint names, these paths can query metadata.

Ad hoc queries still depend on populated fields, appropriate mappings, index coverage, and authorization. New fields or changed field types may require configuration and reindexing. Evaluate that operational work and indexing delay against the flexibility users need.

Aggregation questions ask for a summary of the matching population: how many invoices are pending per vendor, what amount is outstanding in each currency, how intake volume changes by month, or how document amounts are distributed. OpenSearch supports bucket and metric aggregations for grouping records and calculating measures such as sums and averages. These can support configured operational dashboards and exploratory analysis without returning every matching document to the browser.

Define the unit being counted: indexed documents, current agreements, invoices, or distinct cases. Versions and attachments can inflate a document count. Use suitable numeric, date, and exact-value fields, state the time zone and data cutoff, and keep different currencies separate unless a defined conversion is applied. Check bucket limits and any approximate results before describing an aggregation as exhaustive. Permission filtering must govern the aggregate population as well as individual hits.

OpenSearch analytics describe the data available in its index. Historical workflow trends require retained events or snapshots; a current status field cannot reconstruct past transitions. For reconciled financial reporting, complex joins across systems, or long-term analytical models, assess a reporting store or warehouse alongside the search layer.

Use full text for language inside the document

Full-text search is useful when users need language-aware matching, phrase queries, fuzzy matching, relevance ranking, highlighting, or queries across extracted text. Available behavior depends on the engine, analyzers, indexed fields, and query configuration. Test representative languages and phrases rather than assuming the same behavior across backends.

For both ad hoc metadata queries and content search, evaluate index cost, refresh delay, mapping changes, reindexing, storage duplication, regional placement, and authorization filtering alongside the query flexibility.

Use the KnowledgeBase module for meaning-based retrieval

FormKiQ’s KnowledgeBase module supports retrieval by meaning and natural-language questions over a selected document collection. The implementation can use Amazon Bedrock Knowledge Bases or Amazon S3 Vectors, depending on the retrieval and application requirements.

Bedrock Knowledge Bases provides a managed ingestion and retrieval workflow, with generated answers when configured. An S3 Vectors-based implementation stores and searches embeddings, with document preparation, embedding generation, source references, and any answer generation connected through the retrieval workflow. These options can also be combined: Bedrock Knowledge Bases can use S3 Vectors as its vector store.

Define the collection, document versions, chunking, embedding model, retrieval permissions, and relevance tests for the module’s implementation. Similarity results are candidates for review, not proof that every relevant document was found. Inspect the source passage before using it in a business decision, and validate generated answers against their source references.

Specify freshness and completeness

A metadata query is only as complete as its populated attributes, index coverage, and traversal of results. Define how missing values, duplicates, deleted records, and changed classifications are handled. For reports, record the query scope and cutoff and reconcile against the authoritative population. Pagination over changing data may need a snapshot or reconciliation strategy.

At the database layer, DynamoDB Query applies filters after reading a page. An empty page can still have a continuation key, and global secondary indexes offer eventual consistency. Filtering alone does not make an unsupported attribute combination into an efficient lookup. In an application, follow the search API’s continuation contract and test its consistency behavior.

For content search, track extraction and indexing states separately from document receipt. A new document, corrected OCR result, or deleted record can take time to appear or disappear in a secondary index. Show pending or failed processing to operators, define a freshness target, and reconcile the index after retries or rebuilds. “No matches” should not be presented as proof of absence when coverage is incomplete.

Authorization must apply before results are returned

Search indexes often duplicate document content and metadata. A user should not receive a restricted title, snippet, count, or facet merely because the underlying file download would later be denied.

Enforce the applicable site and document access policy before exposing results, including snippets, facets, counts, exports, and cached responses. Test how permission revocation, group changes, deletion, and reclassification affect each path. Recheck access when the document is opened; a previously permitted search result should not grant continuing access.

If indexed permission attributes can lag behind current policy, define a safe handling strategy for that interval. Removing unauthorized hits from one ranked page is insufficient if aggregates still reveal them or the application claims a complete result set. Any final authorization checks must also be reflected in pagination and the information returned to the user. A retention hold concerns preservation; it does not automatically define who may search the record.

Treat OCR quality as search quality

A full-text engine cannot find text that OCR failed to extract. Measure OCR coverage by document type, language, scan quality, handwriting, and layout. Record the source document version, extraction service/configuration, and processing outcome. Assess whether reprocessing improves coverage before replacing an indexed text rendition, and keep the source-to-index relationship traceable.

For critical identifiers, validate and store structured attributes rather than relying on OCR text alone.

Measure relevance and completeness separately

Test metadata reports against a known qualifying population, including all pages, stable ordering, missing fields, and changes during traversal. Test content retrieval with representative queries, human-reviewed relevant documents, missed matches, ranking quality, zero-result cases, and permission boundaries. Keep correctness criteria separate from speed.

Track tail latency, read/write or service costs, index size, extraction cost, refresh delay, and rebuild duration under realistic concurrency. Include index maintenance and backfill work in the comparison. A small proof of value should demonstrate the query model; a production estimate also needs representative volumes and operational costs.

Use a layered search strategy

A practical interface can combine exact filters with full-text discovery. Users first limit site, document type, entity, date, status, or sensitivity, then search language within the authorized subset.

Selective filters can improve relevance and narrow the search, but their cost and effect depend on the backend and query plan. Distinguish an exhaustive filtered report from ranked discovery in the interface. A ranked top-k list should be labelled and evaluated as such, even when it includes exact metadata filters.

Worked example: renewals and clause discovery

Illustrative configuration exercise, not a performance benchmark. A contracts team needs both a complete renewal worklist and a way to discover assignment clauses.

1. 1. Define the operational population

   For active agreements expiring within 90 days, define accepted status values, the date boundary and time zone, and the treatment of missing expiry dates. Use a supported structured query plan and reconcile every result page against the expected population.

2. 2. Search the contract language

   For “change of control,” run a tested phrase query over the authorized, indexed contract versions. Show extraction or indexing failures to the operator. For meaning-based retrieval or natural-language questions, use the KnowledgeBase module with the selected Bedrock Knowledge Bases or S3 Vectors implementation and inspect the source passages.

3. 3. Return to the governed record

   Each hit links to its document ID and the version used for extraction. Recheck current permissions and inspect the source clause before accepting it as evidence for a renewal decision.

4. 4. Test a change

   Change an expiry date, revoke a reviewer’s access, and replace a scanned contract. Verify the freshness target, permission enforcement, and linkage to the current version across both search paths.

How FormKiQ supports both approaches

FormKiQ’s search documentation distinguishes DynamoDB-backed metadata retrieval from configured Typesense and OpenSearch text search. Multi-attribute DynamoDB queries require suitable composite keys. Depending on the installed backend, text search uses /search with Typesense or /searchFulltext with OpenSearch; /queryFulltext exposes advanced OpenSearch queries. The Enhanced Full-Text Search add-on supplies the OpenSearch path. Confirm edition, modules, configuration, and supported query shapes in the target deployment.

Match predictable lookups to supported attribute queries and composite keys. Evaluate OpenSearch for changing metadata combinations as well as content search, and prove each path against its acceptance criteria. Use the KnowledgeBase module for meaning-based retrieval and knowledge-backed questions, with Bedrock Knowledge Bases or S3 Vectors selected for the implementation. Scope collection access, source references, and any answer generation explicitly. The module complements the metadata and full-text query paths described above.

A search-architecture drill buyers can run

• Traverse a multi-page report. Confirm all qualifying records, including a page with no visible matches but further results available.
• Add a record with missing metadata. Check that the operational exception is visible rather than silently absent from a completeness claim.
• Delay or fail text extraction. Confirm that operators can identify the coverage gap and retry it.
• Revoke access after indexing. Check hits, snippets, counts, facets, cached results, exports, and document retrieval.
• Change a schema or replace a file. Verify backfill or reindexing and correct document-version references.
• Reconcile an aggregate. Compare counts and totals with a known population, checking versions, missing fields, bucket limits, permissions, and indexing delay.
• Run realistic concurrent queries. Measure tail latency, freshness, relevance, completeness, and total operating cost separately.