> ## Documentation Index
> Fetch the complete documentation index at: https://docs.raydocs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Extraction Schema

> Complete guide to the extraction JSON schema format for AI-powered data extraction

The extraction JSON schema defines what data to extract from documents using AI. It uses a group-first approach where extractions are organized into logical groups that can be processed in parallel based on their dependencies.

## Schema Structure

The schema consists of three main sections:

```json theme={null}
{
  "config": {
    // Global configuration
  },
  "groups": {
    // All extraction groups
  },
  "definitions": {
    // Reusable schema definitions
  }
}
```

<Note>
  Documents are automatically processed using VLM (Vision Language Model) parsing with per-page chunking for optimal extraction quality across all document types.
</Note>

## Global Configuration

The `config` section defines default settings that apply to all groups unless overridden.

```json theme={null}
{
  "config": {
    "system_message": "Default extraction behavior",
    "reasoning_enabled": false,
    "extraction_title_prompt": "Generate a concise title summarizing the main subject matter (3-5 words)"
  }
}
```

### Configuration Options

| Option                    | Type    | Default | Description                                    |
| ------------------------- | ------- | ------- | ---------------------------------------------- |
| `system_message`          | string  | `null`  | Custom system instructions for the AI          |
| `reasoning_enabled`       | boolean | `false` | Enable reasoning mode for all extractions      |
| `extraction_title_prompt` | string  | `null`  | Custom prompt for generating extraction titles |

<Warning>
  `reasoning_enabled` is template-level only and cannot be overridden at the group level. It applies to all groups uniformly.
</Warning>

## Groups

Groups are the primary organizing principle for extractions. Each group contains its own fields and can override template-level configuration.

<Tip>
  All fields within a group are extracted together in a single LLM call, sharing the same document chunks. This makes groups ideal for **semantically related fields** — information that tends to appear together in the same sections of your documents.
</Tip>

```json theme={null}
{
  "groups": {
    "company_info": {
      "config": {
        "iterates_on": "companies.list"
      },
      "search_query": "company overview, legal name, industry sector",
      "extraction_prompt": "Extract core company details",
      "fields": {
        "name": {
          "type": "string",
          "extraction_prompt": "Extract the full legal company name"
        },
        "sector": {
          "type": "string",
          "extraction_prompt": "Extract the industry classification"
        }
      }
    },
    "financial_metrics": {
      "search_query": "financial metrics, revenue, annual figures",
      "fields": {
        "revenue": {
          "type": "number",
          "extraction_prompt": "Extract the annual revenue figure",
          "references": ["@{company_info.name}"]
        }
      }
    }
  }
}
```

### Group-Level Configuration

Each group can have a `config` object with these options:

| Option           | Type                          | Description                                                                                                                             |
| ---------------- | ----------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| `system_message` | string                        | Override template system message for this group                                                                                         |
| `iterates_on`    | string                        | Path to an array field to iterate over (e.g., `"companies.list"`)                                                                       |
| `execution_mode` | `"automatic"` \| `"per_page"` | `"automatic"` (default) finds useful chunks for one group call. `"per_page"` runs the same group independently for every document page. |

### Group-Level Properties

Properties available directly at the group level (outside `config`):

| Property            | Type   | Description                                                 |
| ------------------- | ------ | ----------------------------------------------------------- |
| `search_query`      | string | Search instruction for RAG to find relevant document chunks |
| `extraction_prompt` | string | Extraction instruction for the AI                           |

### Page-by-Page Groups

Use `execution_mode: "per_page"` for a repeated table or list that must be
read consistently across a whole document, for example bank-statement
transactions. Raydocs runs the group's schema and prompt once per page. The
model receives only that page's parsed chunks, rather than retrieved chunks
from elsewhere in the document.

```json theme={null}
{
  "groups": {
    "transactions": {
      "config": {
        "execution_mode": "per_page"
      },
      "extraction_prompt": "Extract every transaction visible on this page. Return an empty array when there are none.",
      "fields": {
        "items": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "date": { "type": "string" },
              "label": { "type": "string" },
              "amount": { "type": "number" }
            }
          }
        }
      }
    }
  }
}
```

The group result stays flat and preserves page provenance:

```json theme={null}
{
  "transactions": [
    {
      "source": { "document_id": "doc_123", "page_number": 1 },
      "data": { "items": [] }
    }
  ]
}
```

Page records are ordered by the input document order, then by page. Their
`source.document_id` and `source.page_number` always identify the original
source page; the internal batch ordinal is not exposed. A source page with no
matching items is retained with the schema's empty value. An inserted blank
page in a composed document view has no source provenance and is skipped.
Page boundaries are best effort: a row split across pages is not reconstructed
in this mode.

`per_page` cannot be combined with `iterates_on`. A page-by-page group may
depend on a normal group, but normal groups and other page-by-page groups
cannot depend on a page-by-page group's list result.

## Definitions

The `definitions` section contains reusable schema components:

```json theme={null}
{
  "definitions": {
    "monetary_amount": {
      "type": "number",
      "extraction_prompt": "Extract and normalize monetary value to a number"
    },
    "date_field": {
      "type": "string",
      "extraction_prompt": "Extract date in YYYY-MM-DD format"
    }
  }
}
```

Reference definitions in your fields using `$ref`:

```json theme={null}
{
  "fields": {
    "revenue": {
      "$ref": "#/definitions/monetary_amount"
    },
    "founding_date": {
      "$ref": "#/definitions/date_field"
    }
  }
}
```

## Dependencies and Parallel Processing

Dependencies between groups are automatically computed based on:

* **Field references** using mentions (`@{group.field}`)
* **Iteration dependencies** (`iterates_on`)
* **Page-by-page group dependencies** (`execution_mode: "per_page"` may depend on a normal group)

Groups are processed in parallel when their dependencies are satisfied.

### Field References (Mentions)

Fields can reference values from other groups:

```json theme={null}
{
  "groups": {
    "calculations": {
      "fields": {
        "roi_percentage": {
          "type": "number",
          "extraction_prompt": "Calculate ROI using @{financials.investment} and @{financials.current_value}"
        }
      }
    }
  }
}
```

### Iteration

Groups can iterate over arrays using `iterates_on`:

```json theme={null}
{
  "groups": {
    "investments": {
      "fields": {
        "companies": {
          "type": "array",
          "items": { "type": "string" }
        }
      }
    },
    "company_details": {
      "config": {
        "iterates_on": "investments.companies"
      },
      "search_query": "investment amount, @{iterator}",
      "fields": {
        "amount": {
          "type": "number",
          "extraction_prompt": "Extract the investment amount"
        }
      }
    }
  }
}
```

## Reasoning Mode

When enabled, reasoning mode enhances extraction quality by wrapping fields with metadata that records reasoning and sources.

### Enabling Reasoning Mode

```json theme={null}
{
  "config": {
    "reasoning_enabled": true
  }
}
```

### Field Structure in Reasoning Mode

Eligible fields are wrapped with metadata:

```json theme={null}
{
  "field_name": {
    "metadata": {
      "sources": [{ "chunk_id": "...", "text": "...", "comment": "..." }],
      "reasoning": "Explanation of extraction logic..."
    },
    "value": "extracted value"
  }
}
```

## Atomic Fields

The `atomic` flag controls how fields are wrapped when reasoning mode is enabled.

### Default Atomicity Rules

<AccordionGroup>
  <Accordion title="Simple fields (string, number, boolean)">
    Wrapped by default unless `atomic: false`
  </Accordion>

  <Accordion title="Complex objects">
    Not wrapped by default unless `atomic: true`, but their simple fields are wrapped
  </Accordion>

  <Accordion title="Arrays">
    Not wrapped by default unless `atomic: true`, but their simple items are wrapped
  </Accordion>

  <Accordion title="References ($ref)">
    Follow the atomicity rule of their target types, unless overridden with `atomic` flag
  </Accordion>
</AccordionGroup>

### Examples

<Tabs>
  <Tab title="Simple Fields">
    ```json theme={null}
    {
      "name": {
        "type": "string"
        // No atomic flag, will be wrapped by default
      },
      "description": {
        "type": "string",
        "atomic": false
        // Explicitly not wrapped
      }
    }
    ```
  </Tab>

  <Tab title="Objects">
    ```json theme={null}
    {
      "address": {
        "type": "object",
        // Object itself NOT wrapped
        "properties": {
          "street": { "type": "string" },  // WILL be wrapped
          "city": { "type": "string", "atomic": false }  // NOT wrapped
        }
      },
      "person": {
        "type": "object",
        "atomic": true,  // Object WILL be wrapped as a whole
        "properties": {
          "name": { "type": "string" },
          "age": { "type": "number" }
        }
      }
    }
    ```
  </Tab>

  <Tab title="Arrays">
    ```json theme={null}
    {
      "tags": {
        "type": "array",
        // Array itself NOT wrapped
        "items": {
          "type": "string"  // Items ARE wrapped
        }
      },
      "investments": {
        "type": "array",
        "atomic": true,  // Array WILL be wrapped as a whole
        "items": {
          "type": "object",
          "properties": {
            "company": { "type": "string" },
            "amount": { "type": "number" }
          }
        }
      }
    }
    ```
  </Tab>
</Tabs>

### When to Use Atomicity Flags

<CardGroup cols={2}>
  <Card title="Use atomic: true when">
    * Treating a complex object or array as a single unit
    * Needing reasoning about the entire structure
    * The field represents a cohesive concept
  </Card>

  <Card title="Use atomic: false when">
    * You don't need reasoning metadata for a specific field
    * Optimizing output size
    * The field value is straightforward
  </Card>
</CardGroup>

## Writing Effective Search Queries

The `search_query` property is used by RAG to find relevant document chunks. Writing effective queries is critical for extraction quality.

<Warning>
  Write short, dense semantic phrases — NOT natural language sentences. Embedding models compute similarity based on meaning, and unnecessary grammar reduces signal-to-noise ratio.
</Warning>

### Best Practices

<Steps>
  <Step title="Use Concise Noun Phrases">
    Avoid imperative verbs like "Find", "Get", "Extract" and question phrasing.

    ```json theme={null}
    // ❌ Bad
    { "search_query": "Find the company's legal name and incorporation details" }

    // ✅ Good
    { "search_query": "company legal name, incorporation details" }
    ```
  </Step>

  <Step title="Remove Stopwords">
    Words like "the", "in", "for", "of" have negligible embedding value.

    ```json theme={null}
    // ❌ Bad
    { "search_query": "The name of the CEO of the company" }

    // ✅ Good
    { "search_query": "CEO name, company leadership" }
    ```
  </Step>

  <Step title="Add Domain Keywords">
    Include domain-specific terms for disambiguation.

    ```json theme={null}
    // ❌ Bad
    { "search_query": "In the context of EU privacy law, what are the obligations?" }

    // ✅ Good
    { "search_query": "GDPR, data portability obligations" }
    ```
  </Step>

  <Step title="Keep Queries Short (3-10 words)">
    Anything longer becomes noisy. Anything shorter lacks discriminative power.

    ```json theme={null}
    // ❌ Too short
    { "search_query": "revenue" }

    // ✅ Optimal
    { "search_query": "financial performance, revenue, profit margins" }
    ```
  </Step>
</Steps>

### Quick Reference

| ❌ Avoid                         | ✅ Use Instead                              |
| ------------------------------- | ------------------------------------------ |
| "Find the company's legal name" | "company legal name"                       |
| "What is the total revenue?"    | "total revenue, annual revenue"            |
| "In the context of GDPR..."     | "GDPR, {topic}"                            |
| Single word: "revenue"          | With context: "annual revenue, YoY growth" |

## Complex Schema Best Practices

When working with nested objects, arrays, or multiple `$ref` definitions:

### Split Complex Groups

```json theme={null}
// ❌ Problematic: One group with many complex fields
{
  "groups": {
    "company_financials": {
      "fields": {
        "valuation": { "$ref": "#/definitions/amount" },
        "revenue": { "$ref": "#/definitions/amount" },
        "funding_rounds": { "type": "array", "items": { "$ref": "#/definitions/funding_round" } },
        "key_metrics": { "type": "array", "items": { "$ref": "#/definitions/metric" } }
      }
    }
  }
}

// ✅ Better: Split into focused groups
{
  "groups": {
    "valuation_info": {
      "search_query": "company valuation, valuation date",
      "fields": {
        "valuation": { "$ref": "#/definitions/amount" }
      }
    },
    "funding_history": {
      "search_query": "funding rounds, investments, Series A B C",
      "fields": {
        "rounds": { "type": "array", "items": { "$ref": "#/definitions/funding_round" } }
      }
    }
  }
}
```

### Use atomic: true for Complex Definitions

```json theme={null}
{
  "fields": {
    "deal_value": {
      "$ref": "#/definitions/amount",
      "atomic": true  // Reasoning applies to the whole amount
    }
  }
}
```

## Complete Example

```json theme={null}
{
  "config": {
    "system_message": "Extract information with high accuracy",
    "reasoning_enabled": true,
    "extraction_title_prompt": "Create a brief title for this financial document"
  },
  "definitions": {
    "monetary_value": {
      "type": "number",
      "extraction_prompt": "Extract and normalize monetary value"
    },
    "date_field": {
      "type": "string",
      "extraction_prompt": "Extract date in YYYY-MM-DD format"
    }
  },
  "groups": {
    "company_info": {
      "search_query": "company overview, legal name, founding date",
      "fields": {
        "name": {
          "type": "string",
          "extraction_prompt": "Extract full legal name"
        },
        "founding_date": {
          "$ref": "#/definitions/date_field"
        }
      }
    },
    "financial_metrics": {
      "search_query": "financial metrics, revenue, @{company_info.name}",
      "fields": {
        "revenue": {
          "$ref": "#/definitions/monetary_value"
        }
      }
    },
    "investment_rounds": {
      "search_query": "investment rounds, funding history",
      "fields": {
        "rounds": {
          "type": "array",
          "items": { "type": "string" }
        }
      }
    },
    "round_details": {
      "config": {
        "iterates_on": "investment_rounds.rounds"
      },
      "search_query": "investment amount, @{iterator}",
      "fields": {
        "amount": {
          "$ref": "#/definitions/monetary_value"
        }
      }
    }
  }
}
```