Data Extraction using Form Recognizer of Cognitive Service (v2.1)

Azure Form Recognizer is a cognitive service that lets you build an automated process of data extraction that is able to extract key-value pairs and table data from documents like PDF, JPG, or PNG. There are two possibilities of data extraction. The first key benefit of the service is fully managed and does not require any manual labeling and with the second approach, you can train custom models to extract structured data from your forms and documents.

You can call Form Recognizer models by using a REST API or client library SDKs to reduce complexity and integrate it into your application. We will start with the REST API integration.

Pre-Requisites

1. An Azure Subscription
2. Subscription to the Form Recognizer
3. An Azure Storage Account

Subscribe to Form Recognizer service

First of all, Go to the Azure portal and create a new Form Recognizer resource . In the Create pane, provide the following information:

Name: Give a descriptive name for your resource. Example demoformrecognizer.
Subscription: Select the Azure subscription from the drop-down which has been granted access.
Location: Whatever location you can choose from the list but will no impact on the runtime availability of your resource.
Pricing tier: The cost of your resource depends on the pricing tier you choose and your usage.
Resource group: The Azure resource group that will contain your resource. You can create a new group or add it to a pre-existing group.

Create Form Recognizer

Once you are done with all the details click on the Review + Create button. After reviewing the details click on the create button. Your deployment will take some time to create a new instance.
Now go back to home and click on the All Resources from the portal menu. Click on the form recognizer instance and open the overview. Now we need to add keys and an endpoint as shown in the below screen:
Note: Key1 and endpoint will be used to create the project in docker. We’ll have to replace the api_key with the key that we’ll copy from the key1. Similarly, we’ll be replacing the custom_models_endpoint with the endpoint URL in the python code later.

Form Recognizer Service Key and Endpoint

Azure Storage Account

We need a storage account to upload the sample files and pdf documents that we will use to train the model for data extraction.
1. Type the Storage account in the search bar and click on that.
2. Click on the Add button and add the below details and click on Review + Create Button.

Blob Storage Account

Once you are done with Storage Account click on the created account then click on the CORS tab from the left navigation and select the Allowed Methods from the highlighted box.

Blob Storage Account

Click on the Shared access signature tab from the left navigation and select the checkbox that is selected in the screenshot. Then click on Generate SAS Connection String. After that, you will see Connection String, SAS Token, and Blob service SAS URL.

This connection string will use to create the connection string from the docker screen.
https://<<NameOfStorage>>.blob.core.windows.net/<<PutNameOfContainer>>?sv=2019-12-12&ss=b&srt=sco&sp=rwdlacx&se=2021-07-01T11:43:42Z&st=2020-12-09T03:43:42Z&spr=https&sig=X%2B1%2B4X6zRQTN8uy0PxomsKixoYEkpnYGlTaCSU%2Fim%2Fo%3D

Generate Blob service SAS URL

Create a container for file uploading. It’s a place where we will upload the sample file for model training. We’ll upload at least five files.

Docker pull for the Form Recognizer container

In this quickstart, you’ll use the Form Recognizer REST API with the sample labeling tool to train a custom model.
Set up the sample labeling tool
To run the below commands docker should be configured on your local computer. Well, we do this configuration for the Windows operating system. Open the command prompt and run the below commands.

1. Get the sample labeling tool container with the docker pull command.
   docker pull mcr.microsoft.com/azure-cognitive-services/custom-form/labeltool:latest-preview
2. Now you’re ready to run the container with docker run.
   docker run -it -p 3000:80 mcr.microsoft.com/azure-cognitive-services/custom-form/labeltool:latest-preview eula=accept

Once you are done with the commands open http://localhost:3000/connections URL in the browser.

1. Create a connection using docker.
2. Create a project in docker using the docker connection.

After project creation select that project from the right panel and uploaded images on blob storage will appear for labeling. Labeling fields will be highlighted. Now you need to create the tags in the right panel then select the highlighted area from the document then click on the label. Which fields you want to capture from the document you can select and according to the fields, you can assign the label. One by one you need to set the labeling for all the documents.

Set labeling for document fields

Once you are done with the labeling of all the fields, click on the Train tab from the left navigation and give the name of the model then click on the Train button. Once the training is done with the process model id will appear on the same screen and that model ID we are passing as a parameter from angular code (replace <<ModelId>> with the model ID).

Train Model

Python Service Endpoint
Now we will create a python service to hit the Form Recognizer service endpoint and will hit the service endpoint from the angular.

Here is the python service code:

Here is the Angular code to call the python endpoint from UI.

Here is code of Python service endpoint and angular that call the python service for data extraction from documents.