![\"Screenshot](\"https://user-images.githubusercontent.com/9599/176274424-441aee18-8e8c-44bf-9748-f53e33e3fa76.png\")
\n\n## Limits\n\nRelevant [limits](https://docs.aws.amazon.com/textract/latest/dg/limits.html) for PDF files:\n\n> For asynchronous operations, JPEG and PNG files have a 10MB size limit. PDF and TIFF files have a 500MB limit. PDF and TIFF files have a limit of 3,000 pages.\n>\n> For PDFs: The maximum height and width is 40 inches and 2880 points. PDFs cannot be password protected. PDFs can contain JPEG 2000 formatted images.\n\n## Uploading to S3\n\nI used my [s3-credentials](https://github.com/simonw/s3-credentials/) tool to create an S3 bucket with credentials for uploading files to it:\n\n```\n~ % s3-credentials create sfms-history -c\nCreated bucket: sfms-history\nCreated user: 's3.read-write.sfms-history' with permissions boundary: 'arn:aws:iam::aws:policy/AmazonS3FullAccess'\nAttached policy s3.read-write.sfms-history to user s3.read-write.sfms-history\nCreated access key for user: s3.read-write.sfms-history\n{\n \"UserName\": \"s3.read-write.sfms-history\",\n \"AccessKeyId\": \"AKIAWXFXAIOZBOQM4XUH\",\n \"Status\": \"Active\",\n \"SecretAccessKey\": \"...\",\n \"CreateDate\": \"2022-06-28 17:55:10+00:00\"\n}\n```\nI stored the secret access key in 1Password, then used it in [Transmit](https://panic.com/transmit/) to upload the PDF files.\n\n## Starting a text detection job\n\nFor PDFs you need to run in async mode, where you get back a job ID and then poll for completion.\n\nYou can ask it to send you notifications via an SNS queue too, but this is optional. You can ignore SNS entirely, which is what I did.\n\nTo start the job, provide it with the bucket and the name of the file to process:\n\n```python\nimport boto3\n\ntextract = boto3.client(\"textract\")\nresponse = textract.start_document_text_detection(\n DocumentLocation={\n 'S3Object': {\n 'Bucket': \"sfms-history\",\n 'Name': \"Meetings and Minutes/Minutes/1946-1949/1946-10-04_SFMS_MeetingMinutes.pdf\"\n }\n }\n)\njob_id = response[\"JobId\"]\n```\n\n## Polling for completion\n\nYou can then use that `job_id` to poll for completion. The `textract.get_document_text_detection` call returns a `JobStatus` key of `IN_PROGRESS` if it is still processing.\n\nHere's a function I wrote to poll for completion:\n```python\nimport time\n\ndef poll_until_done(job_id):\n while True:\n response = textract.get_document_text_detection(JobId=job_id)\n status = response[\"JobStatus\"]\n if status != \"IN_PROGRESS\":\n return response\n print(\".\", end=\"\")\n time.sleep(10)\n\n# Usage, given a response from textract.start_document_text_detection:\ncompletion_response = poll_until_done(response[\"JobId\"])\n```\nThis can take a surprisingly long time - it took seven minutes for a 6 page typewritten PDF file for me, and ten minutes for a 56 page handwritten one.\n\nI was wondering how long you have to retrieve the results of a job. The [get_document_text_detection()](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/textract.html#Textract.Client.get_document_text_detection) documentation says:\n\n> A `JobId` value is only valid for 7 days.\n\n## Fetching the results\n\nThe response that you get back at the end is paginated. Here's a function to gather all of the \"blocks\" of text that it detected across multiple pages:\n\n```python\ndef get_all_blocks(job_id):\n blocks = []\n next_token = None\n first = True\n while first or next_token:\n first = False\n kwargs = {\"JobId\": job_id}\n if next_token:\n kwargs[\"NextToken\"] = next_token\n response = textract.get_document_text_detection(**kwargs)\n blocks.extend(response[\"Blocks\"])\n next_token = response.get(\"NextToken\")\n return blocks\n```\n(I could have used [this boto3 pagination trick](https://til.simonwillison.net/aws/helper-for-boto-aws-pagination) instead.)\n\nBlocks come in three types: `LINE`, `WORD`, and `PAGE`. The `PAGE` blocks do not contain any text, just indications of which lines and words were on the page. The `LINE` and `WORD` blocks duplicate each other - you probably just want the `LINE` blocks.\n\nHere's an example of a `LINE` block:\n\n```json\n{\n \"BlockType\": \"LINE\",\n \"Confidence\": 90.4699478149414,\n \"Text\": \"1\",\n \"Geometry\": {\n \"BoundingBox\": {\n \"Width\": 0.00758015550673008,\n \"Height\": 0.011477531865239143,\n \"Left\": 0.9904273152351379,\n \"Top\": 0.00909337680786848\n },\n \"Polygon\": [\n {\n \"X\": 0.9904273152351379,\n \"Y\": 0.00909337680786848\n },\n {\n \"X\": 0.9980074763298035,\n \"Y\": 0.00909337680786848\n },\n {\n \"X\": 0.9980074763298035,\n \"Y\": 0.0205709096044302\n },\n {\n \"X\": 0.9904273152351379,\n \"Y\": 0.0205709096044302\n }\n ]\n },\n \"Id\": \"6b04b8df-bec1-42d3-bfff-29f0edd38976\",\n \"Relationships\": [\n {\n \"Type\": \"CHILD\",\n \"Ids\": [\n \"58890ca7-5ed5-4b14-ad60-475e5d0dd79e\"\n ]\n }\n ],\n \"Page\": 1\n}\n```\nI found that joining together those lines on a `\\n` gave me the results I needed:\n```python\nprint(\"\\n\".join([block[\"Text\"] for block in blocks if block[\"BlockType\"] == \"LINE\"]))\n```\nTruncated output:\n```\n1\nORGANIZATION MEETING\nof the\nSAN FRANCISCO MICROSCOPICAL SOCIETY\nOctober 4, 1946\nThe meeting ws.s held at 8:00 P.M. on October 4, 1946, in the\nAuditorium of the San Francisco Department of Health, 101 Grove Street,\nSan Francisco.\nChairman George Herbert Needham called the audience of sixty-\nfive persons to order. He told of the high aims, ideals, and fine fellow-\nship enjoyed by the original society which was organized in 1870 and incor-\nporated in 1872, but which was dissolved following the San Francisco fire\nof 1906. He related his efforts to find a surviving member which finally\nresulted in a telegram of greeting from Dr. Kaspar Pischell of Ross, Cali-\nfornia, which read as follows:\n\"BEST WISHES AT THIS REUNION. I AM SORRY I CANNOT BE WITH YOU.\"\n```", "html": "Textract is the AWS OCR API. It's very good - I've fed it hand-written notes from the 1890s and it read them better than I could.
\nIt can be run directly against JPEG or PNG images up to 5MB, but if you want to run OCR against a PDF file you have to first upload it to an S3 bucket.
\nUpdate 30th June 2022: I used what I learned in this TIL to build s3-ocr, a command line utility for running OCR against PDFs in an S3 bucket.
\nYou don't need to use the API at all to try Textract out against a document: they offer a demo tool in the AWS console:
\nhttps://us-west-1.console.aws.amazon.com/textract/home?region=us-west-1#/demo
\n
Relevant limits for PDF files:
\n\n\nFor asynchronous operations, JPEG and PNG files have a 10MB size limit. PDF and TIFF files have a 500MB limit. PDF and TIFF files have a limit of 3,000 pages.
\nFor PDFs: The maximum height and width is 40 inches and 2880 points. PDFs cannot be password protected. PDFs can contain JPEG 2000 formatted images.
\n
I used my s3-credentials tool to create an S3 bucket with credentials for uploading files to it:
\n~ % s3-credentials create sfms-history -c\nCreated bucket: sfms-history\nCreated user: 's3.read-write.sfms-history' with permissions boundary: 'arn:aws:iam::aws:policy/AmazonS3FullAccess'\nAttached policy s3.read-write.sfms-history to user s3.read-write.sfms-history\nCreated access key for user: s3.read-write.sfms-history\n{\n \"UserName\": \"s3.read-write.sfms-history\",\n \"AccessKeyId\": \"AKIAWXFXAIOZBOQM4XUH\",\n \"Status\": \"Active\",\n \"SecretAccessKey\": \"...\",\n \"CreateDate\": \"2022-06-28 17:55:10+00:00\"\n}\nI stored the secret access key in 1Password, then used it in Transmit to upload the PDF files.
\nFor PDFs you need to run in async mode, where you get back a job ID and then poll for completion.
\nYou can ask it to send you notifications via an SNS queue too, but this is optional. You can ignore SNS entirely, which is what I did.
\nTo start the job, provide it with the bucket and the name of the file to process:
\nimport boto3\n\ntextract = boto3.client(\"textract\")\nresponse = textract.start_document_text_detection(\n DocumentLocation={\n 'S3Object': {\n 'Bucket': \"sfms-history\",\n 'Name': \"Meetings and Minutes/Minutes/1946-1949/1946-10-04_SFMS_MeetingMinutes.pdf\"\n }\n }\n)\njob_id = response[\"JobId\"]You can then use that job_id to poll for completion. The textract.get_document_text_detection call returns a JobStatus key of IN_PROGRESS if it is still processing.
Here's a function I wrote to poll for completion:
\nimport time\n\ndef poll_until_done(job_id):\n while True:\n response = textract.get_document_text_detection(JobId=job_id)\n status = response[\"JobStatus\"]\n if status != \"IN_PROGRESS\":\n return response\n print(\".\", end=\"\")\n time.sleep(10)\n\n# Usage, given a response from textract.start_document_text_detection:\ncompletion_response = poll_until_done(response[\"JobId\"])This can take a surprisingly long time - it took seven minutes for a 6 page typewritten PDF file for me, and ten minutes for a 56 page handwritten one.
\nI was wondering how long you have to retrieve the results of a job. The get_document_text_detection() documentation says:
\n\n\nA
\nJobIdvalue is only valid for 7 days.
The response that you get back at the end is paginated. Here's a function to gather all of the \"blocks\" of text that it detected across multiple pages:
\ndef get_all_blocks(job_id):\n blocks = []\n next_token = None\n first = True\n while first or next_token:\n first = False\n kwargs = {\"JobId\": job_id}\n if next_token:\n kwargs[\"NextToken\"] = next_token\n response = textract.get_document_text_detection(**kwargs)\n blocks.extend(response[\"Blocks\"])\n next_token = response.get(\"NextToken\")\n return blocks(I could have used this boto3 pagination trick instead.)
\nBlocks come in three types: LINE, WORD, and PAGE. The PAGE blocks do not contain any text, just indications of which lines and words were on the page. The LINE and WORD blocks duplicate each other - you probably just want the LINE blocks.
Here's an example of a LINE block:
{\n \"BlockType\": \"LINE\",\n \"Confidence\": 90.4699478149414,\n \"Text\": \"1\",\n \"Geometry\": {\n \"BoundingBox\": {\n \"Width\": 0.00758015550673008,\n \"Height\": 0.011477531865239143,\n \"Left\": 0.9904273152351379,\n \"Top\": 0.00909337680786848\n },\n \"Polygon\": [\n {\n \"X\": 0.9904273152351379,\n \"Y\": 0.00909337680786848\n },\n {\n \"X\": 0.9980074763298035,\n \"Y\": 0.00909337680786848\n },\n {\n \"X\": 0.9980074763298035,\n \"Y\": 0.0205709096044302\n },\n {\n \"X\": 0.9904273152351379,\n \"Y\": 0.0205709096044302\n }\n ]\n },\n \"Id\": \"6b04b8df-bec1-42d3-bfff-29f0edd38976\",\n \"Relationships\": [\n {\n \"Type\": \"CHILD\",\n \"Ids\": [\n \"58890ca7-5ed5-4b14-ad60-475e5d0dd79e\"\n ]\n }\n ],\n \"Page\": 1\n}I found that joining together those lines on a \\n gave me the results I needed:
print(\"\\n\".join([block[\"Text\"] for block in blocks if block[\"BlockType\"] == \"LINE\"]))Truncated output:
\n1\nORGANIZATION MEETING\nof the\nSAN FRANCISCO MICROSCOPICAL SOCIETY\nOctober 4, 1946\nThe meeting ws.s held at 8:00 P.M. on October 4, 1946, in the\nAuditorium of the San Francisco Department of Health, 101 Grove Street,\nSan Francisco.\nChairman George Herbert Needham called the audience of sixty-\nfive persons to order. He told of the high aims, ideals, and fine fellow-\nship enjoyed by the original society which was organized in 1870 and incor-\nporated in 1872, but which was dissolved following the San Francisco fire\nof 1906. He related his efforts to find a surviving member which finally\nresulted in a telegram of greeting from Dr. Kaspar Pischell of Ross, Cali-\nfornia, which read as follows:\n\"BEST WISHES AT THIS REUNION. I AM SORRY I CANNOT BE WITH YOU.\"\n