Extract API JSON Output Format

Overview

The JSON output of the Extract API consists of two main components: the content and the annotations.

Content is what's in the document. These are the items you can find in the document, for example: paragraphs, headings, titles, figures, and miscellaneous text. Content consists of the type of content it is as well as its text.

Annotations provide structural and relational metadata. Table structure annotations represent the relationship between table cells, including positions so that you can "build" the table from its smallest parts. Relation annotations describe semantic relationships between content items, such as a title supporting a figure.

There are IDs linking the annotations back to their associated content.

Let's dive into the structure of the JSON.

Overall Structure

The specific dictionary captured in this JSON contains the two overall components we mentioned above, plus per-page dimensions. It has three keys: content_tree, annotations, and pdf_pages.

PDF Pages

pdf_pages is a list with one entry per page of the document, in page order. Each entry describes the page's dimensions and rotation:

height: float # page height in points
width: float # page width in points
required_ccw_rotation: int # counter-clockwise rotation (in degrees) applied to normalize the page

These dimensions are useful for converting the normalized bounding boxes returned in locations (see Location Bounding Boxes) into absolute coordinates on a rendered page.

Content Tree

The content tree contains the text content of the document, whether it's found directly as text or within a table or title. It has a tree structure, meaning each item may have "children", to represent hierarchical structure of the document. The highest level is a DOCUMENT, which contains everything as its children. Under the DOCUMENT, there can be other hierarchical structures, e.g. a header-level 2 H2 can be a child of a header-level 1 H1, and a paragraph and a miscellaneous text can be children of this H2. There can be more complex structures like a table has all its own table cells as its children.

With document_type=hierarchical_v2, text that is not part of a table is also organized hierarchically (for example, paragraphs nested under the headings they fall beneath), in addition to tables containing their own table cells as children.

The types are as follows:

content_tree: Dict[str, Any]

Keys:

children: List[Dict[str, Any]] # recursively contains more dicts with children
content: str | None # text or None if not applicable, such as for unparsed figures
type: str
uid: str # unique ID

type can be one of

    'DOCUMENT',
    'PARAGRAPH',
    'TEXT',
    'H1',
    'H2',
    'H3',
    'H4',
    'H5',
    'TABLE_CELL',
    'TABLE',
    'TABLE_OF_CONTENTS',
    'TABLE_OF_CONTENTS_TITLE',
    'TABLE_TITLE',
    'TABLE_CAPTION',
    'TABLE_LABEL',
    'TABLE_FOOTER',
    'FIGURE_TITLE',
    'FIGURE_CAPTION',
    'FIGURE_LABEL',
    'FIGURE_FOOTER',
    'IMAGE_TITLE',
    'IMAGE_CAPTION',
    'IMAGE_LABEL',
    'IMAGE_FOOTER',
    'PAGE_HEADER',
    'PAGE_FOOTER',
    'PAGE_FOOTNOTE',
    'FIGURE',
    'IMAGE'

Additionally, when figure_extraction="true" is passed as an API parameter, the document may contain a FIGURE (or a specific chart type such as BAR_CHART, LINE_PLOT, SCATTER_PLOT, or PIE_CHART) whose extracted data is represented with these types:

    'FIGURE_EXTRACTED_TABLE',
    'FIGURE_EXTRACTED_TABLE_CELL'

The set of type values is open. The lists above reflect the content types available today, but they are not a closed set. As we release new features, we may introduce new content types. If the behavior of an existing type changes, we will notify you in advance, but new types can be added at any time. When consuming Extract output, treat the set of types as extensible and tolerate types you do not recognize.

Annotations

Annotations are a list of dictionaries that provide structural and relational information about the content. There are three types of annotations:

• Table structure annotations describe the relationships between table cells, including their positions (row, column) and spans, so you can reconstruct the table from its individual cells.
• Figure-extracted table structure annotations are similar to table structure annotations but for tables extracted from figures.
• Relation annotations describe semantic relationships between content items. There are two types of relation annotations:
  • Support relations link a supporting element to its target, such as a title supporting a figure. Returned when include_relations is set to "true" in the API request.
  • Row key parent relations describe parent-child hierarchies between row header cells in a table. Returned when extract_table_row_header_hierarchy is set to "true" in the API request.

The types are as follows:

annotations: List[Dict[str, Any]]

Table Structure Annotations

Keys:

content_uids: List[str] # unique ID that maps to content uids
type: str # 'table_structure'
data: Dict[str, Any] # contains the position, span, and header flags of the cell

Keys for "data":

index: [int, int] # row and column in the table
span: [int, int] # how much the cell spans
is_column_header: bool # whether the cell is a column header
is_projected_row_header: bool # whether the cell is a projected (spanning) row header

is_column_header marks cells that head a column (typically the top row(s) of the table). is_projected_row_header marks cells that head a group of rows — a row header that "projects" down over the rows beneath it. These per-cell flags describe an individual cell's role; the parent-child hierarchy between row header cells is described separately by the row_key_parent relation annotation (see below), which is returned only when extract_table_row_header_hierarchy is set to "true".

An example table structure annotation:

{
  "content_uids": ["7"],
  "type": "table_structure",
  "data": {
    "index": [0, 0],
    "span": [1, 1],
    "is_column_header": true,
    "is_projected_row_header": false
  }
}

Figure-Extracted Table Structure Annotations

Keys:

content_uids: List[str] # unique ID that maps to content uids
type: str # 'figure_extracted_table_structure'
data: Dict[str, Any] # contains indices, span, and value of the cell

Keys for "data":

index: [int, int] # row and column in the table
span: [int, int] # how much the cell spans
value: str # the value of the cell, predicted by the model or from a text box

Relation Annotations

Keys:

type: str # 'relation'
data: Dict[str, str] # contains the relation type and the source and target content uids

Keys for "data":

relation_type: str # the type of the relation, e.g. 'support' or 'row_key_parent'
source_content_uid: str # uid of the source content item
target_content_uid: str # uid of the target content item

The available relation types are:

• support — describes a semantic support relationship between content items, such as a title supporting a figure. Returned when include_relations is set to "true" in the API request.
• row_key_parent — describes a parent-child hierarchy between row header cells within a table. The source is the parent row header cell and the target is the child row header cell. Returned when extract_table_row_header_hierarchy is set to "true" in the API request.

For example, a relation annotation linking a figure title to its figure:

{
  "data": {
    "relation_type": "support",
    "source_content_uid": "10",
    "target_content_uid": "12"
  },
  "type": "relation"
}

Here, the content with uid "10" (e.g. a FIGURE_TITLE) supports the content with uid "12" (e.g. a BAR_CHART).

A relation annotation linking a parent row header to a child row header in a table:

{
  "data": {
    "relation_type": "row_key_parent",
    "source_content_uid": "20",
    "target_content_uid": "25"
  },
  "type": "relation"
}

Here, the table cell with uid "20" (e.g. "Revenue") is the parent row header of the table cell with uid "25" (e.g. "Product Revenue"). This hierarchy is determined using heuristic rules based on cell features such as indentation, bold formatting, font size, and capitalization.

Connecting The Dots

Let's see an example that will help us make sense of how the UI output, annotations, and content tree all work together.

Here is an example image in the UI:

As you can see, there's a table. But what does that look like in the JSON output?

Let's start at the top level of the content tree:

>>> output['content_tree']['type']
'DOCUMENT'

This is the overall document. We must go down to its children to find the text, titles, and tables.

>>> output['content_tree']['children'][5]['type']
'TABLE'

Here we have a table. This table contains children - its cells:

>>> output['content_tree']['children'][5]['children']
[{'children': [], 'content': '2048', 'type': 'TABLE_CELL', 'uid': '133'}, {'children': [], 'content': '4096', 'type': 'TABLE_CELL', 'uid': '134'}, {'children': [], 'content': 'Methods', 'type': 'TABLE_CELL', 'uid': '135'}, {'children': [], 'content': 'Extrapolation', 'type': 'TABLE_CELL', 'uid': '136'}, {'children': [], 'content': 'ROPE', 'type': 'TABLE_CELL', 'uid': '137'}, {'children': [], 'content': '73.6', 'type': 'TABLE_CELL', 'uid': '138'}, {'children': [], 'content': '294.45', 'type': 'TABLE_CELL', 'uid': '139'}, {'children': [], 'content': 'ROPE + BCA', 'type': 'TABLE_CELL', 'uid': '140'}, {'children': [], 'content': '25.57', 'type': 'TABLE_CELL', 'uid': '141'}, {'children': [], 'content': '25.65', 'type': 'TABLE_CELL', 'uid': '142'}, {'children': [], 'content': 'Alibi', 'type': 'TABLE_CELL', 'uid': '143'}, {'children': [], 'content': '23.14', 'type': 'TABLE_CELL', 'uid': '144'}, {'children': [], 'content': '24.26', 'type': 'TABLE_CELL', 'uid': '145'}, {'children': [], 'content': 'Alibi + BCA', 'type': 'TABLE_CELL', 'uid': '146'}, {'children': [], 'content': '24.6', 'type': 'TABLE_CELL', 'uid': '147'}, {'children': [], 'content': '25.37', 'type': 'TABLE_CELL', 'uid': '148'}, {'children': [], 'content': 'XPOS (Ours)', 'type': 'TABLE_CELL', 'uid': '149'}, {'children': [], 'content': '22.56', 'type': 'TABLE_CELL', 'uid': '150'}, {'children': [], 'content': '28.43', 'type': 'TABLE_CELL', 'uid': '151'}, {'children': [], 'content': 'XPOS + BCA (Ours)', 'type': 'TABLE_CELL', 'uid': '152'}, {'children': [], 'content': '21.6', 'type': 'TABLE_CELL', 'uid': '153'}, {'children': [], 'content': '20.73', 'type': 'TABLE_CELL', 'uid': '154'}]

Nice! We have all the cells that comprise the table. Let's examine an example cell more closely:

>>> output['content_tree']['children'][5]['children'][0]
{'children': [], 'content': '2048', 'type': 'TABLE_CELL', 'uid': '133'}

The content indicates that the text in this cell is "2048", and it has no children (i.e. this is the smallest unit - a single table cell). To check if there are annotations associated with this cell - which would allow us to connect it to other cells - we must find a uid to content_uid match. Here it is:

>>> output['annotations'][126]
{'content_uids': ['133'], 'data': {'index': [0, 1], 'span': [1, 1], 'is_column_header': False, 'is_projected_row_header': False}, 'type': 'table_structure'}

Here, we have the annotation of that cell (row 0, column 1, one cell, and neither a column header nor a projected row header) and its corresponding content ID.

If we go through all the annotations, we will find one for each of the content_uids, which will tell us which row and column the cell is in. That way we have the text in each cell - in the content - as well as their relationship to each other - in the annotations.

See the Toolkit page for further information on rebuilding tables.

Location Bounding Boxes

By default, the output JSON returns bounding box information for the content and annotations.

This can be seen in the output "locations" key within annotations and contents. "locations" is a list of dictionaries with the following keys:

page_number: int # Page number where the content or annotation is found (0-indexed)
height: float # Normalized height of the bounding box
width: float # Normalized width of the bounding box
x: float # Normalized x0 of the bounding box (left)
y: float # Normalized y0 of the bounding box (top)

To remove "locations" from the output JSON use the optional output_format parameter as follows:

response = requests.get(response_url, params={"output_format": "structured_document"}, headers=headers)

"locations" is a list to allow for multiple locations associated with an annotation or content. "locations" will be None if it's associated with a higher-level object like a document page, but its children will each contain a locations entry.

These are page-relative locations. To convert to bounding box coordinates on your own rendered page, use (page_width*x, page_height*y, page_width*(x+width), page_height*(y+height)). The page_width and page_height for each page are available in the top-level pdf_pages list (see PDF Pages).

To include the bounding-box locations of images in the output, set the optional include_images parameter to "true" in the API request. When enabled, IMAGE content nodes carry locations describing where each image sits on the page.

Character Offsets

Some use cases, such as those involving visual rendering, require character locations. The output format structured_document_with_char_offsets includes this information:

response = requests.get(response_url, params={"output_format": "structured_document_with_char_offsets"}, headers=headers)

It will include locations as mentioned above as well as a new dictionary key text_node_data. text_node_data is a dictionary with the following keys and values:

texts: list[str] # List of all the strings in the bounding boxes that form the segment bounding box
text_locations: list[dict[str, float | int]] # List of locations in the same format as the overall segment's location, namely a dictionary with page_number, height, width, x, and y
character_offsets: list[list[float]] # List of relative locations of characters in the bounding box. These are normalized between 0 and 1 and correspond to the same indexed text box in `texts`/`text_locations`

To understand this output format better, it's worth highlighting that Extract's output contains segments of certain categories, such as tables and paragraphs. There are smaller bounding boxes of text that can make up a larger segment. The default output format only has the segment-level information. For specialized use cases, however, you can use this output format to have all the individual text boxes with the locations of each character in each box.

One important note: the character offsets are based on graphemes rather than the underlying unicode values. This is particularly important for non-Latin based characters such as those in Arabic, Mandarin, and Thai. In these languages, a single character like ต in Thai may have multiple unicode values, but it is treated as a single grapheme with a single character offset in Extract.

We determine the graphemes of a text — and therefore the character offsets — using the ugrapheme (opens in a new tab) library. Grapheme cluster boundaries follow the extended grapheme cluster rules defined in Unicode Standard Annex #29 (Unicode Text Segmentation) (opens in a new tab), so segmentation behaves consistently with the Unicode Standard's definition of a user-perceived character.