How to Extract Critical Data from Complex Documents with IDP Forge: A Step-by-Step Guide

There Are Two Ways to Extract Data with IDP Forge

Before you build anything, it's worth knowing you have two starting points, and they're built for two different jobs.

Create New Pipeline

This is the full builder, the seven-stage pipeline (Source, Initialization, Parse, Split/Classify, Extract, Post-processing, Destination) configured stage by stage and published as a versioned pipeline you can run repeatedly.

Use this when:

• You're setting up a real, recurring workflow (an inbox that needs to be watched, a storage bucket that needs to be polled, an API integration that needs to fire on every job)
• You need a specific schema bound to a document class, validation rules, or a specific destination
• You want the run to be reproducible; every job records the exact pipeline version it ran on, so you can trace a result back later.

Once published, the pipeline stays live: new documents keep flowing through it via whatever source you configured, and you can edit it later, which creates a new draft version without disturbing jobs already in flight.

Quick Run

Quick Run skips the builder entirely. Drop in a document (or a small batch) and get extraction results back immediately, without creating, naming, or publishing a pipeline first.

Use this when:

• You're testing what IDP Forge does with a document type before deciding how to configure a real pipeline
• You want a one-off extraction and don't need it to run again
• You're evaluating accuracy or output shape before committing to a schema or destination

Creating Your Data Extraction Pipeline Using IDP Forge: Step-by-Step Guide 

Once you've decided a real pipeline is what you need, going stage by stage in the right order, and knowing what each one is actually for, saves you from having to circle back later. Here's what to have figured out before and during each step.

Step 1: Choose Where Your Documents Come From

A pipeline can pull from multiple sources at once, and results are combined automatically. Your options are:

• API: documents submitted directly via POST /v1/jobs
• Email: a generated inbox address, so documents can just be emailed in
• Storage: S3, Azure Blob, GCS, or SFTP, checked either on a schedule or triggered by new files
• Manual upload: drag-and-drop from inside the app

Each source stamps a tag onto every job it creates, so later on you can filter and analyze results by where they came from. Set up your connector credentials in Admin first, not inside the pipeline itself, so that if a password or API key changes, you update it once and every pipeline using that connection updates automatically. Deciding your source upfront also tells you what volume and format of documents to expect, which makes every later step easier to configure correctly the first time.

Step 2: Decide What You'll Accept

Before any document is actually processed, Initialization sets the ground rules: which file types are allowed, the maximum size and page count, and what happens if a file doesn't meet those rules. It also handles early cleanup like deskewing a crooked scan and normalizing the file for OCR. Antivirus scanning always runs here and can't be turned off, so nothing unsafe makes it further into the pipeline. Getting these limits right early avoids documents silently failing later for reasons that have nothing to do with extraction quality.

Step 3: Let Parse Do the Reading

This is where IDP Forge actually looks at the document for the first time. Parse figures out the layout (where the text blocks, tables, and images are), detects the language, and runs OCR to capture the actual text. Everything after this stage works from what Parse produces, so it's worth getting its settings right from the start:

• Output format: Markdown keeps structure like headings and tables intact; Text is a flatter, simpler transcription. Whichever you choose is what Split, Classify, and Extract will read.
• Bounding boxes: the exact pixel location of everything detected on the page. Turn this on if you plan to have humans visually review extracted values, since it's what lets a reviewer see the exact spot on the page a value came from.
• Confidence scores: a per-element score that later stages use to decide what needs a second look.
• Chunking: for very large documents that a model can't read in one pass, Parse can break the document into chunks instead of processing it whole.

Step 4: Split Multi-Document Files and Classify Each One

This is the stage built specifically for messy, real-world submissions. A single uploaded file is often not one document; it might be an email with three attachments, or a scanned batch of unrelated forms. Split finds the boundaries between documents inside that one file, and turns each piece into its own document that gets processed on its own from here forward.

Classify then looks at each of those pieces and assigns it a type from your taxonomy (your organization's list of document categories, like "Invoice" or "Bank Statement"). Split and Classify can run together or independently, so you can split without classifying, classify without splitting, or do both, whatever fits your documents.

A few things worth setting up correctly from the start:

• Classification decides what schema is used to extract data. If a document doesn't match any category in your taxonomy, it isn't treated as an error; it just falls through to a generic extraction that returns the raw data it found without validating it against a specific structure.
• Low confidence on a split (the system isn't fully sure where one document ends and the next begins) never stops a job. It gets logged for someone to double-check later, but processing keeps moving. Classify behaves a bit more cautiously; a low-confidence classification can hold a document for review, since getting the type wrong affects everything downstream.
• Before you can publish a taxonomy where categories route to specific schemas, every category needs a schema assigned to it. Build your taxonomy with this in mind so you're not blocked at publish time.

Step 5: Define What Data You Actually Want (Schemas)

A schema is simply the list of fields you want pulled out of a given document type, what each one is called, what type of value it holds, and the rules that decide whether it's valid. Field types cover the obvious cases (text, numbers, dates, true/false) as well as more complex structures:

• Tables: for repeating data like invoice line items, extracted row by row
• Groups: for related fields that belong together, like the parts of an address

Every field also carries a confidence threshold (80% by default). Anything extracted below that threshold gets flagged for a person to check, but it doesn't fail the job; it's just a signal, not a stop sign.

If you don't want to build a schema by hand, you can hand IDP Forge a few sample documents, and it will propose one, detecting fields and tables and suggesting a type and confidence for each. Starting from an inferred draft instead of a blank one is usually the faster path.

Schemas are versioned too, just like pipelines. Editing a published schema opens a new draft without touching the version currently running in production, so you can improve your schema over time without disrupting documents that are actively being processed.

Step 6: Extract the Data

With a schema in place, Extract knows exactly which fields to look for and pulls each one out, returning both the value and a confidence score. Behind the scenes, it splits the work across four specialized processes depending on what kind of content it's reading: plain field values, tables, images/figures like charts or signatures, and checkbox or radio-button selections.

If no schema is bound (say, the document didn't match anything in your taxonomy), Extract still returns everything it can find as free-form data, rather than failing outright. Either way, you get one consistent output format back, so your application only has to handle one shape of result no matter which mode produced it.

This is also where you decide whether low-confidence results get automatically routed to a human reviewer, or delivered as-is with the confidence signal attached for you to handle yourself. Deciding this now, rather than after documents are already flowing through, means you won't have uncertain results slipping out unnoticed while you figure it out.

Step 7: Validate the Data (Post-Processing)

Once fields are extracted, Post-processing checks whether they actually make sense: does the format match what's expected, do the line items on an invoice actually add up to the stated total, does a value match something in your own records? You can also configure auto-fill, which populates a field from a lookup in your data instead of relying on what's printed on the page. If a rule fails here, it raises a clear, specific exception rather than silently letting bad data through.

Step 8: Send It Somewhere

Finally, Destination decides where the finished, structured result goes, whether that's a storage bucket, a folder, a webhook, or directly into another system of record. You can send results to more than one destination at once, and each one can have its own output format. Confirming this before you publish means results start landing exactly where your downstream systems expect them, from the very first job.

Confidence Scores: What Happens When the System Isn't Sure

The single most important idea in IDP Forge is that low confidence isn't a failure; it's a signal. Anywhere a value looks uncertain, whether that's a low-confidence field, a failed validation rule, or an unclear document boundary, that signal can be routed into a review queue instead of being silently accepted or silently rejected.

When something is flagged, it lands in a work queue, essentially a live, filtered view of documents that need attention. From there, a person can:

• Claim a document from the queue
• Correct only the specific fields that were flagged, not the whole document
• Submit it, either delivering it immediately or sending it to a second reviewer if your coverage rules require extra scrutiny (for example, every transaction over $50,000, or every document from a brand-new reviewer's first 30 days)
• Or park it if it needs an answer from somewhere else, or skip it to someone else's queue.

Every correction is logged with the before-and-after value, which does double duty: it fixes today's document, and it becomes a training signal for improving accuracy over time. You can also track how the whole review process is performing: how long documents wait before someone picks them up, how long they take to fix, how many corrections a document needs on average, and, most importantly, what percentage of documents make it through with no human involvement at all. That last number is really the whole point, and it should climb steadily as your schema and taxonomy mature.