PDF.co Document Parser: Template Creation Guide

Table of Contents:

• PDF.co Document Parser: Template Creation Guide
  • What is Document Parser and How It Works
  • Template Editor
  • Parameters of the search-based (flexible) table powered by AI
  • APPENDIX 1: Macros
    • Special Functions
  • APPENDIX 2: Sample templates
    • Sample 1

What is Document Parser and How It Works

Document Parser is the versatile document parsing engine for accurate and easy data extraction from PDF and scanned documents. Create and maintain extraction templates without coding!

Extracts data from invoices, statements, reports, paystubs, tables, reciepts. Supports both native electronic and scanned PDF files, PNG, JPG, TIFF images. Supports English, German, French, Spanish and many other languages including dual language documents. Available via Web API, Zapier, Integromat, SalesForce and via self-hosted on-premise API server.

Visual Document Parser Template Editor

(Online version of PDF.co Document Parser templates editor is here. Create, test and maintain data extraction templates.

Template Objects

Template objects define objects to extract from input document. These can be:

1. Field mapped from Virtual Grid - extracts value from virtual grid generated by the engine for input document. If you have documents with the same layout, this object is useful. Basically, it is similar to converting document to a spreadsheet and then telling the engine to get value from a virtual cell at (row, column).
2. Field from Rectangle Selection - extracts data from a rectangle selection by coordinates. Use it when you have document with objects placed at the very same place all the time. You can optionally set expression (with macros) to additionally locate a matching expression (for example, a date or currency) inside a rectangle selection.
3. Field from auto Key-Value - runs a search for key-value pairs in the document and generates output objects as Key-value pairs. you should define expression with macros).
4. Table from Rectangle - reads a table from a rectangle.
5. Field based on Text Search - runs a text search on a whole page searching for a predefined macros (for example, you can find a date, currency, SSN, phone number) and returns it as a value/
6. Table based on Search - finds a table using AI or using search expressions defined by a JSON config containing a text search pattern for beginning and ending pattern for a table. With this approach you can extract multipage tables. Explore more details
7. Field with Static Value - returns a static value. Useful if you need to generate some predefined values, like a template name, or company name and return a predefined value along with other objects.

Expression parameter (for field mapped from a rectangle, text search based field, fied from auto key-value and others)

Expression parameter can contain:

1. Macros (see the list here )
2. Regular Expressions (don’t forget to enable regex checkbox)
3. Mixed macros and regular expressions (don’t forget to enable regex checkbox)
4. Special Functions for AI-powered data extraction of specific values like a company name. The list of available special functions is available here.

Special Markers (for use inside expressions)**

You can use special markers inside expression parameter. Marker helps to point a specific part of expression to become output value or a field name (otherwise a whole expression is used):

• ?<value> marker points to the regular expression group that must be used as a final value for the object. Example: Invoice ?<value>(\d+) will extract 12345 as final field value from Invoice 12345 string.
• ?<key> marker points to the regular expression group that must be used a field name for the object. Important: multiple matches for this expression will auto-generate multiple objects for the output. Example: (?<key>{{SentenceWithSingleSpaces}}): (?<value>{{SentenceWithSingleSpaces}}) will extract key1: value, key2: value as two separate objects named key1 and key2 accordingly.

Search-based Table Object

This object is defined by a JSON-based configuration.

Table objects return tabular data you need to extract. Table objects can be defined by

1. rectangle coordinates (use Table from Rectangle type of object)
2. AI-powered automated table detection (automatically finds tables on pages, using Table from Auto Detection)
3. finds table by the set of rules using text or regular expressions search that defines the table’s start, the end, and rows (using Table from Search)

You can also define multitple table types inside JSON based Tables section inside this object configuration.

Table parameters (tableProperties object):

• name - table name to distinguish different tables in the result.

• autoDetection object [optional] - defines auto detection mode to use AI based table detection. IMPORTANT: when this section is set and tableIndex is not -1 then other params like start, end, row are ignored because table is auto detected.
  • pageIndex [required] - sets index of the page to find table on (starts at 0 (zero)).
  • tableIndex [required] - -1 by default (means that auto detection is disabled by default. Set to 0 or higher index so it will detect a table on given page, starting from top to bottom. IMPORTANT: when this section is set and tableIndex is not -1 then other params like start, end, row are ignored because table is auto detected.
• start - group of parameters that define the start of the table:
  • expression - macro expression to find the start of the table, or
  • y - the top coordinate of the table. You can find PDF points coordinates in your PDF file using our Simplified PDF Viewer.
  • pageIndex - index of the page containing the y coordinate.
  • regex - indicates if the expression parameter contains regular expression.
• end object - group of parameters that define the end of the table:
  • expression - macro expression to find the end of the table, or
  • y - the bottom coordinate of the table. You can find PDF points coordinates in your PDF file using our Simplified PDF Viewer.
  • regex - indicates if the expression parameter contains regular expression.
• subItemStart object - [optional] parameters that define the start of the table sub-item. Sub-items are used for tables with complex multiline rows:
  • expression - macro expression to find the start of the sub-item.
  • regex - indicates if the expression parameter contains regular expression.
• subItemEnd object - [optional] parameters that define the end of the table sub-item:
  • expression - macro expression to find the end of the sub-item.
  • regex - indicates if the expression parameter contains regular expression.
• introduction object - Parameters to parse values from sub-headers. Values parsed from the introduction expression will be repeated in the beginning of every row.
  • expression - macro expression to parse introduction items.
  • regex - indicates if the expression parameter contains regular expression.
• row object - [optional] group of parameters that define table rows:
  • expression - the main macro expression to find a row. Named groups in this expression will go to the result table as columns. See example below.
  • regex - indicates if expression contains regular expression.
  • subExpression1, subExpression2, subExpression3, subExpression4, subExpression5 - additional expressions to parse some remaining parts of row data which the main expression cannot parse in one pass. Sub-expressions are executed after the main expression for the text chunks between matches of the main expression. Can be used to parse hanging rows (wrapped multiline cells).
• columns array - [optional] array that defines column properties. Names of columns should correspond to the names of the capturing groups of the row expression. Column properties:
  • name - defines column name.
  • x - [optional] X coordinate of the left column edge in PDF Points. You can find PDF points coordinates in your PDF file using our Simplified PDF Viewer.
  • type - [optional] defines column data type. Should be one of these values:
    • string
    • integer
    • date
    • decimal
    • for more see also the types descriptions in fields section.
  • dateFormat - [optional] See dateFormat description in fields section.
  • outputDateFormat - [optional] See outputDateFormat description in fields section.
  • coalesceWith - [optional] Name of column to merge the parsed value with.
• rowMergingRule string - [optional] For the fields of rectangle type and table data type. Defines the rule to merge multiline data in table cells. Supported values:
  • none - default, no rule.
  • byBorders - combine lines within a table cell framed by border lines.
  • hangingRows - join table row that contains only a single cell up to the previous row if there is no separating line between them. Useful for tables without borders between rows.
• multipage boolean - [optional] defines whether the table may continue on further pages.
• horizontalSeparationOffset - offset from the table start to the beginning of the first table row.
• horizontalSeparationStep - row height. These two parameters help the parser distinguish rows in tables without horizontal separators. This works only with tables with fixed row height.

Example of table parsing:

The table above, can be parsed with macro expressions or with explicitly defined column coordinates.

1. Extracting table using AI powered auto detector:

Full template:

autoDetectTableField value:

{
  "autoDetection": {
    "pageIndex": 0,
    "tableIndex": 0
  },
  "columns": [
    {
      "name": "description",
      "type": "string"
    },
    {
      "name": "interval",
      "type": "string"
    },
    {
      "name": "quantity",
      "type": "integer"
    },
    {
      "name": "amount",
      "type": "decimal"
    }
  ]
}

Full template:

{
  "templateVersion": 4,
  "templatePriority": 0,
  "culture": "en-US",
  "objects": [],
  "templateName": "",
  "options": {
    "ocrMode": "auto",
    "ocrLanguage": "eng"
  },
  "objects": [
    {
      "name": "AutoDetectTable",
      "objectType": "table",
      "tableProperties": {
        "autoDetection": {
          "pageIndex": 0,
          "tableIndex": 0
        },
        "columns": [
          {
            "name": "description",
            "type": "string"
          },
          {
            "name": "interval",
            "type": "string"
          },
          {
            "name": "quantity",
            "type": "integer"
          },
          {
            "name": "amount",
            "type": "decimal"
          }
        ]
      }
    }
  ]
}


2. Extracting table using markers defines by macros:

`searchBasedTable` object properties:

```JSON
{
  "start": {
    "expression": "Amount{{Space}}{{OpeningParenthesis}}{{Dollar}}{{ClosingParenthesis}}"
  },
  "end": {
    "expression": "Total in USD"
  },
  "row": {
    "expression": "{{LineStart}}{{Spaces}}(?<description>{{SentenceWithSingleSpaces}})(?<interval>{{3Letters}}{{Space}}{{Digits}}{{Space}}{{Minus}}{{Space}}{{3Letters}}{{Space}}{{Digits}}){{Spaces}}(?<quantity>{{Digits}}){{Spaces}}(?<amount>{{Number}})",
    "regex": true
  },
  "columns": [
    {
      "name": "description",
      "type": "string"
    },
    {
      "name": "interval",
      "type": "string"
    },
    {
      "name": "quantity",
      "type": "integer"
    },
    {
      "name": "amount",
      "type": "decimal"
    }
  ]
}

Full Template:

{
  "templateVersion": 4,
  "templatePriority": 0,
  "culture": "en-US",
  "objects": [],
  "templateName": "",
  "options": {
    "ocrMode": "auto",
    "ocrLanguage": "eng"
  },
  "objects": [
    {
      "name": "searchBasedTable1",
      "objectType": "table",
      "tableProperties": {
        "start": {
          "expression": "Amount{{Space}}{{OpeningParenthesis}}{{Dollar}}{{ClosingParenthesis}}"
        },
        "end": {
          "expression": "Total in USD"
        },
        "row": {
          "expression": "{{LineStart}}{{Spaces}}(?<description>{{SentenceWithSingleSpaces}})(?<interval>{{3Letters}}{{Space}}{{Digits}}{{Space}}{{Minus}}{{Space}}{{3Letters}}{{Space}}{{Digits}}){{Spaces}}(?<quantity>{{Digits}}){{Spaces}}(?<amount>{{Number}})",
          "regex": true
        },
        "columns": [
          {
            "name": "description",
            "type": "string"
          },
          {
            "name": "interval",
            "type": "string"
          },
          {
            "name": "quantity",
            "type": "integer"
          },
          {
            "name": "amount",
            "type": "decimal"
          }
        ]
      }
    }
  ]
}

APPENDIX 1: Macros

Built-in macros:

Special Functions

You can also insert so called special function which looks like this: $$functionName. Special fucntions are created for AI-powered value detection, like a company name, max number in a whole document, max date or even finding and decoding QR Code barcode value inside document.

All special functions are listed here

APPENDIX 2: Sample templates

Sample 1

Sample document text:

    DigitalOcean
    101 Avenue of the Americas, 10th Floor
    New York, NY 10013
                                                        Date Issued: February 1, 2016
                                                         Period: January 1 - 31, 2016
                                                              Invoice Number: 1234567

        Description                                 Hours     Start          End            USD
        Website-Dev (1GB)                           744       01-01 00:00    01-31 23:59    $10.00
        Website-Live (1GB)                          744       01-01 00:00    01-31 23:59    $10.00
        Database-Live (2GB)                         744       01-01 00:00    01-31 23:59    $20.00
        Tasks-Dev (1GB)                             744       01-01 00:00    01-31 23:59    $10.00
                                                                                     Total: $50.00

     Bill To:
     Samee Sikka <admin@meee.org>
     meee.org
     Gouran

        If you have a credit card on file it will be automatically charged within 24 hours.

Sample template (YAML):

{
  "templateVersion": 4,
  "templatePriority": 0,
  "templateName": "DigitalOcean Invoice",
  "objects": [
    {
      "name": "companyName",
      "objectType": "field",
      "fieldProperties": {
        "fieldType": "static",
        "expression": "DigitalOcean"
      }
    },
    {
      "name": "invoiceId",
      "objectType": "field",
      "fieldProperties": {
        "fieldType": "macros",
        "expression": "Invoice Number: ({{Digits}})",
        "regex": true
      }
    },
    {
      "name": "dateIssued",
      "objectType": "field",
      "fieldProperties": {
        "fieldType": "macros",
        "expression": "Date Issued: ({{SmartDate}})",
        "dataType": "date",
        "dateFormat": "auto-mdy"
      }
    },
    {
      "name": "total",
      "objectType": "field",
      "fieldProperties": {
        "fieldType": "macros",
        "expression": "Total: {{Dollar}}({{Number}})",
        "dataType": "decimal"
      }
    },
    {
      "name": "currency",
      "objectType": "field",
      "fieldProperties": {
        "fieldType": "static",
        "expression": "USD"
      }
    },
    {
      "name": "table1",
      "objectType": "table",
      "tableProperties": {
        "start": {
          "expression": "Description{{Spaces}}Hours"
        },
        "end": {
          "expression": "Total:"
        },
        "row": {
          "expression": "{{LineStart}}{{Spaces}}(?<description>{{SentenceWithSingleSpaces}}){{Spaces}}(?<hours>{{Digits}}){{Spaces}}(?<start>{{2Digits}}{{Minus}}{{2Digits}}{{Space}}{{2Digits}}{{Colon}}{{2Digits}}){{Spaces}}(?<end>{{2Digits}}{{Minus}}{{2Digits}}{{Space}}{{2Digits}}{{Colon}}{{2Digits}}){{Spaces}}{{Dollar}}(?<unitPrice>{{Number}})",
          "regex": true
        },
        "columns": [
          {
            "name": "hours",
            "type": "integer"
          },
          {
            "name": "unitPrice",
            "type": "decimal"
          }
        ]
      }
    }
  ]
}

Result (JSON):

{
  "templateName": "DigitalOcean Invoice",
  "templateVersion": "4",
  "objects": [
    {
      "name": "companyName",
      "objectType": "field",
      "value": "DigitalOcean"
    },
    {
      "name": "invoiceId",
      "objectType": "field",
      "value": "1234567",
      "pageIndex": 0,
    },
    {
      "name": "dateIssued",
      "objectType": "field",
      "value": "2016-02-01T00:00:00",
      "pageIndex": 0,
    },
    {
      "name": "total",
      "objectType": "field",
      "value": 50.00,
      "pageIndex": 0,
    },
    {
      "name": "currency",
      "objectType": "field",
      "value": "USD"
    },
    {
      "name": "table1",
      "objectType": "table",
      "rows": [
        {
          "description": {
            "value": "Website-Dev (1GB)",
            "pageIndex": 0
          },
          "hours": {
            "value": 744,
            "pageIndex": 0
          },
          "start": {
            "value": "01-01 00:00",
            "pageIndex": 0
          },
          "end": {
            "value": "01-31 23:59",
            "pageIndex": 0
          },
          "unitPrice": {
            "value": 10.00,
            "pageIndex": 0
          }
        },
        {
          "description": {
            "value": "Website-Live (1GB)",
            "pageIndex": 0
          },
          "hours": {
            "value": 744,
            "pageIndex": 0
          },
          "start": {
            "value": "01-01 00:00",
            "pageIndex": 0
          },
          "end": {
            "value": "01-31 23:59",
            "pageIndex": 0
          },
          "unitPrice": {
            "value": 10.00,
            "pageIndex": 0
          }
        },
        {
          "description": {
            "value": "Database-Live (2GB)",
            "pageIndex": 0
          },
          "hours": {
            "value": 744,
            "pageIndex": 0
          },
          "start": {
            "value": "01-01 00:00",
            "pageIndex": 0
          },
          "end": {
            "value": "01-31 23:59",
            "pageIndex": 0
          },
          "unitPrice": {
            "value": 20.00,
            "pageIndex": 0
          }
        },
        {
          "description": {
            "value": "Tasks-Dev (1GB)",
            "pageIndex": 0
          },
          "hours": {
            "value": 744,
            "pageIndex": 0
          },
          "start": {
            "value": "01-01 00:00",
            "pageIndex": 0
          },
          "end": {
            "value": "01-31 23:59",
            "pageIndex": 0
          },
          "unitPrice": {
            "value": 10.00,
            "pageIndex": 0
          }
        }
      ]
    }
  ]
}

Copyright (c) 2018-2022 ByteScout, Inc.

PDF.co (on-demand platform) with Document Parser

ByteScout (on-prem tools)