# til

path: aws_ocr-pdf-textract.md
topic: aws
title: Running OCR against a PDF file with AWS Textract
url: https://github.com/simonw/til/blob/main/aws/ocr-pdf-textract.md
created: 2022-06-28T12:32:43-07:00
created_utc: 2022-06-28T19:32:43+00:00
updated: 2022-06-30T15:48:16-07:00
updated_utc: 2022-06-30T22:48:16+00:00
shot_hash: 752af8acdf6d6838cef061e37bda9b59
slug: ocr-pdf-textract
body:
[Textract](https://aws.amazon.com/textract/) is the AWS OCR API. It's very good - I've fed it hand-written notes from the 1890s and it read them better than I could.

It can be run directly against JPEG or PNG images up to 5MB, but if you want to run OCR against a PDF file you have to first upload it to an S3 bucket.

**Update 30th June 2022**: I used what I learned in this TIL [to build s3-ocr](https://simonwillison.net/2022/Jun/30/s3-ocr/), a command line utility for running OCR against PDFs in an S3 bucket.

## Try it out first

You don't need to use the API at all to try Textract out against a document: they offer a demo tool in the AWS console:

https://us-west-1.console.aws.amazon.com/textract/home?region=us-west-1#/demo

<img alt="Screenshot of the demo interface showing uploaded image and resulting text" src="https://user-images.githubusercontent.com/9599/176274424-441aee18-8e8c-44bf-9748-f53e33e3fa76.png" width="600">

## Limits

Relevant [limits](https://docs.aws.amazon.com/textract/latest/dg/limits.html) for PDF files:

> For asynchronous operations, JPEG and PNG files have a 10MB size limit. PDF and TIFF files have a 500MB limit. PDF and TIFF files have a limit of 3,000 pages.
>
> For PDFs: The maximum height and width is 40 inches and 2880 points. PDFs cannot be password protected. PDFs can contain JPEG 2000 formatted images.

## Uploading to S3

I used my [s3-credentials](https://github.com/simonw/s3-credentials/) tool to create an S3 bucket with credentials for uploading files to it:

```
~ % s3-credentials create sfms-history -c
Created bucket: sfms-history
Created  user: 's3.read-write.sfms-history' with permissions boundary: 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
Attached policy s3.read-write.sfms-history to user s3.read-write.sfms-history
Created access key for user: s3.read-write.sfms-history
{
    "UserName": "s3.read-write.sfms-history",
    "AccessKeyId": "AKIAWXFXAIOZBOQM4XUH",
    "Status": "Active",
    "SecretAccessKey": "...",
    "CreateDate": "2022-06-28 17:55:10+00:00"
}
```
I stored the secret access key in 1Password, then used it in [Transmit](https://panic.com/transmit/) to upload the PDF files.

## Starting a text detection job

For PDFs you need to run in async mode, where you get back a job ID and then poll for completion.

You can ask it to send you notifications via an SNS queue too, but this is optional. You can ignore SNS entirely, which is what I did.

To start the job, provide it with the bucket and the name of the file to process:

```python
import boto3

textract = boto3.client("textract")
response = textract.start_document_text_detection(
    DocumentLocation={
        'S3Object': {
            'Bucket': "sfms-history",
            'Name': "Meetings and Minutes/Minutes/1946-1949/1946-10-04_SFMS_MeetingMinutes.pdf"
        }
    }
)
job_id = response["JobId"]
```

## Polling for completion

You can then use that `job_id` to poll for completion. The `textract.get_document_text_detection` call returns a `JobStatus` key of `IN_PROGRESS` if it is still processing.

Here's a function I wrote to poll for completion:
```python
import time

def poll_until_done(job_id):
    while True:
        response = textract.get_document_text_detection(JobId=job_id)
        status = response["JobStatus"]
        if status != "IN_PROGRESS":
            return response
        print(".", end="")
        time.sleep(10)

# Usage, given a response from textract.start_document_text_detection:
completion_response = poll_until_done(response["JobId"])
```
This can take a surprisingly long time - it took seven minutes for a 6 page typewritten PDF file for me, and ten minutes for a 56 page handwritten one.

I was wondering how long you have to retrieve the results of a job. The [get_document_text_detection()](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/textract.html#Textract.Client.get_document_text_detection) documentation says:

> A `JobId` value is only valid for 7 days.

## Fetching the results

The response that you get back at the end is paginated. Here's a function to gather all of the "blocks" of text that it detected across multiple pages:

```python
def get_all_blocks(job_id):
    blocks = []
    next_token = None
    first = True
    while first or next_token:
        first = False
        kwargs = {"JobId": job_id}
        if next_token:
            kwargs["NextToken"] = next_token
        response = textract.get_document_text_detection(**kwargs)
        blocks.extend(response["Blocks"])
        next_token = response.get("NextToken")
    return blocks
```
(I could have used [this boto3 pagination trick](https://til.simonwillison.net/aws/helper-for-boto-aws-pagination) instead.)

Blocks come in three types: `LINE`, `WORD`, and `PAGE`. The `PAGE` blocks do not contain any text, just indications of which lines and words were on the page. The `LINE` and `WORD` blocks duplicate each other - you probably just want the `LINE` blocks.

Here's an example of a `LINE` block:

```json
{
  "BlockType": "LINE",
  "Confidence": 90.4699478149414,
  "Text": "1",
  "Geometry": {
    "BoundingBox": {
      "Width": 0.00758015550673008,
      "Height": 0.011477531865239143,
      "Left": 0.9904273152351379,
      "Top": 0.00909337680786848
    },
    "Polygon": [
      {
        "X": 0.9904273152351379,
        "Y": 0.00909337680786848
      },
      {
        "X": 0.9980074763298035,
        "Y": 0.00909337680786848
      },
      {
        "X": 0.9980074763298035,
        "Y": 0.0205709096044302
      },
      {
        "X": 0.9904273152351379,
        "Y": 0.0205709096044302
      }
    ]
  },
  "Id": "6b04b8df-bec1-42d3-bfff-29f0edd38976",
  "Relationships": [
    {
      "Type": "CHILD",
      "Ids": [
        "58890ca7-5ed5-4b14-ad60-475e5d0dd79e"
      ]
    }
  ],
  "Page": 1
}
```
I found that joining together those lines on a `\n` gave me the results I needed:
```python
print("\n".join([block["Text"] for block in blocks if block["BlockType"] == "LINE"]))
```
Truncated output:
```
1
ORGANIZATION MEETING
of the
SAN FRANCISCO MICROSCOPICAL SOCIETY
October 4, 1946
The meeting ws.s held at 8:00 P.M. on October 4, 1946, in the
Auditorium of the San Francisco Department of Health, 101 Grove Street,
San Francisco.
Chairman George Herbert Needham called the audience of sixty-
five persons to order. He told of the high aims, ideals, and fine fellow-
ship enjoyed by the original society which was organized in 1870 and incor-
porated in 1872, but which was dissolved following the San Francisco fire
of 1906. He related his efforts to find a surviving member which finally
resulted in a telegram of greeting from Dr. Kaspar Pischell of Ross, Cali-
fornia, which read as follows:
"BEST WISHES AT THIS REUNION. I AM SORRY I CANNOT BE WITH YOU."
```

html:
<p><a href="https://aws.amazon.com/textract/" rel="nofollow">Textract</a> is the AWS OCR API. It's very good - I've fed it hand-written notes from the 1890s and it read them better than I could.</p>
<p>It can be run directly against JPEG or PNG images up to 5MB, but if you want to run OCR against a PDF file you have to first upload it to an S3 bucket.</p>
<p><strong>Update 30th June 2022</strong>: I used what I learned in this TIL <a href="https://simonwillison.net/2022/Jun/30/s3-ocr/" rel="nofollow">to build s3-ocr</a>, a command line utility for running OCR against PDFs in an S3 bucket.</p>
<h2>
<a id="user-content-try-it-out-first" class="anchor" href="#try-it-out-first" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Try it out first</h2>
<p>You don't need to use the API at all to try Textract out against a document: they offer a demo tool in the AWS console:</p>
<p><a href="https://us-west-1.console.aws.amazon.com/textract/home?region=us-west-1#/demo" rel="nofollow">https://us-west-1.console.aws.amazon.com/textract/home?region=us-west-1#/demo</a></p>
<p><a href="https://user-images.githubusercontent.com/9599/176274424-441aee18-8e8c-44bf-9748-f53e33e3fa76.png" target="_blank" rel="nofollow"><img alt="Screenshot of the demo interface showing uploaded image and resulting text" src="https://user-images.githubusercontent.com/9599/176274424-441aee18-8e8c-44bf-9748-f53e33e3fa76.png" width="600" style="max-width:100%;"></a></p>
<h2>
<a id="user-content-limits" class="anchor" href="#limits" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Limits</h2>
<p>Relevant <a href="https://docs.aws.amazon.com/textract/latest/dg/limits.html" rel="nofollow">limits</a> for PDF files:</p>
<blockquote>
<p>For asynchronous operations, JPEG and PNG files have a 10MB size limit. PDF and TIFF files have a 500MB limit. PDF and TIFF files have a limit of 3,000 pages.</p>
<p>For PDFs: The maximum height and width is 40 inches and 2880 points. PDFs cannot be password protected. PDFs can contain JPEG 2000 formatted images.</p>
</blockquote>
<h2>
<a id="user-content-uploading-to-s3" class="anchor" href="#uploading-to-s3" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Uploading to S3</h2>
<p>I used my <a href="https://github.com/simonw/s3-credentials/">s3-credentials</a> tool to create an S3 bucket with credentials for uploading files to it:</p>
<pre><code>~ % s3-credentials create sfms-history -c
Created bucket: sfms-history
Created  user: 's3.read-write.sfms-history' with permissions boundary: 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
Attached policy s3.read-write.sfms-history to user s3.read-write.sfms-history
Created access key for user: s3.read-write.sfms-history
{
    "UserName": "s3.read-write.sfms-history",
    "AccessKeyId": "AKIAWXFXAIOZBOQM4XUH",
    "Status": "Active",
    "SecretAccessKey": "...",
    "CreateDate": "2022-06-28 17:55:10+00:00"
}
</code></pre>
<p>I stored the secret access key in 1Password, then used it in <a href="https://panic.com/transmit/" rel="nofollow">Transmit</a> to upload the PDF files.</p>
<h2>
<a id="user-content-starting-a-text-detection-job" class="anchor" href="#starting-a-text-detection-job" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Starting a text detection job</h2>
<p>For PDFs you need to run in async mode, where you get back a job ID and then poll for completion.</p>
<p>You can ask it to send you notifications via an SNS queue too, but this is optional. You can ignore SNS entirely, which is what I did.</p>
<p>To start the job, provide it with the bucket and the name of the file to process:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">boto3</span>

<span class="pl-s1">textract</span> <span class="pl-c1">=</span> <span class="pl-s1">boto3</span>.<span class="pl-en">client</span>(<span class="pl-s">"textract"</span>)
<span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">textract</span>.<span class="pl-en">start_document_text_detection</span>(
    <span class="pl-v">DocumentLocation</span><span class="pl-c1">=</span>{
        <span class="pl-s">'S3Object'</span>: {
            <span class="pl-s">'Bucket'</span>: <span class="pl-s">"sfms-history"</span>,
            <span class="pl-s">'Name'</span>: <span class="pl-s">"Meetings and Minutes/Minutes/1946-1949/1946-10-04_SFMS_MeetingMinutes.pdf"</span>
        }
    }
)
<span class="pl-s1">job_id</span> <span class="pl-c1">=</span> <span class="pl-s1">response</span>[<span class="pl-s">"JobId"</span>]</pre></div>
<h2>
<a id="user-content-polling-for-completion" class="anchor" href="#polling-for-completion" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Polling for completion</h2>
<p>You can then use that <code>job_id</code> to poll for completion. The <code>textract.get_document_text_detection</code> call returns a <code>JobStatus</code> key of <code>IN_PROGRESS</code> if it is still processing.</p>
<p>Here's a function I wrote to poll for completion:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">time</span>

<span class="pl-k">def</span> <span class="pl-en">poll_until_done</span>(<span class="pl-s1">job_id</span>):
    <span class="pl-k">while</span> <span class="pl-c1">True</span>:
        <span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">textract</span>.<span class="pl-en">get_document_text_detection</span>(<span class="pl-v">JobId</span><span class="pl-c1">=</span><span class="pl-s1">job_id</span>)
        <span class="pl-s1">status</span> <span class="pl-c1">=</span> <span class="pl-s1">response</span>[<span class="pl-s">"JobStatus"</span>]
        <span class="pl-k">if</span> <span class="pl-s1">status</span> <span class="pl-c1">!=</span> <span class="pl-s">"IN_PROGRESS"</span>:
            <span class="pl-k">return</span> <span class="pl-s1">response</span>
        <span class="pl-en">print</span>(<span class="pl-s">"."</span>, <span class="pl-s1">end</span><span class="pl-c1">=</span><span class="pl-s">""</span>)
        <span class="pl-s1">time</span>.<span class="pl-en">sleep</span>(<span class="pl-c1">10</span>)

<span class="pl-c"># Usage, given a response from textract.start_document_text_detection:</span>
<span class="pl-s1">completion_response</span> <span class="pl-c1">=</span> <span class="pl-en">poll_until_done</span>(<span class="pl-s1">response</span>[<span class="pl-s">"JobId"</span>])</pre></div>
<p>This can take a surprisingly long time - it took seven minutes for a 6 page typewritten PDF file for me, and ten minutes for a 56 page handwritten one.</p>
<p>I was wondering how long you have to retrieve the results of a job. The <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/textract.html#Textract.Client.get_document_text_detection" rel="nofollow">get_document_text_detection()</a> documentation says:</p>
<blockquote>
<p>A <code>JobId</code> value is only valid for 7 days.</p>
</blockquote>
<h2>
<a id="user-content-fetching-the-results" class="anchor" href="#fetching-the-results" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Fetching the results</h2>
<p>The response that you get back at the end is paginated. Here's a function to gather all of the "blocks" of text that it detected across multiple pages:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">get_all_blocks</span>(<span class="pl-s1">job_id</span>):
    <span class="pl-s1">blocks</span> <span class="pl-c1">=</span> []
    <span class="pl-s1">next_token</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>
    <span class="pl-s1">first</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
    <span class="pl-k">while</span> <span class="pl-s1">first</span> <span class="pl-c1">or</span> <span class="pl-s1">next_token</span>:
        <span class="pl-s1">first</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>
        <span class="pl-s1">kwargs</span> <span class="pl-c1">=</span> {<span class="pl-s">"JobId"</span>: <span class="pl-s1">job_id</span>}
        <span class="pl-k">if</span> <span class="pl-s1">next_token</span>:
            <span class="pl-s1">kwargs</span>[<span class="pl-s">"NextToken"</span>] <span class="pl-c1">=</span> <span class="pl-s1">next_token</span>
        <span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">textract</span>.<span class="pl-en">get_document_text_detection</span>(<span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
        <span class="pl-s1">blocks</span>.<span class="pl-en">extend</span>(<span class="pl-s1">response</span>[<span class="pl-s">"Blocks"</span>])
        <span class="pl-s1">next_token</span> <span class="pl-c1">=</span> <span class="pl-s1">response</span>.<span class="pl-en">get</span>(<span class="pl-s">"NextToken"</span>)
    <span class="pl-k">return</span> <span class="pl-s1">blocks</span></pre></div>
<p>(I could have used <a href="https://til.simonwillison.net/aws/helper-for-boto-aws-pagination" rel="nofollow">this boto3 pagination trick</a> instead.)</p>
<p>Blocks come in three types: <code>LINE</code>, <code>WORD</code>, and <code>PAGE</code>. The <code>PAGE</code> blocks do not contain any text, just indications of which lines and words were on the page. The <code>LINE</code> and <code>WORD</code> blocks duplicate each other - you probably just want the <code>LINE</code> blocks.</p>
<p>Here's an example of a <code>LINE</code> block:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-ent">"BlockType"</span>: <span class="pl-s"><span class="pl-pds">"</span>LINE<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"Confidence"</span>: <span class="pl-c1">90.4699478149414</span>,
  <span class="pl-ent">"Text"</span>: <span class="pl-s"><span class="pl-pds">"</span>1<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"Geometry"</span>: {
    <span class="pl-ent">"BoundingBox"</span>: {
      <span class="pl-ent">"Width"</span>: <span class="pl-c1">0.00758015550673008</span>,
      <span class="pl-ent">"Height"</span>: <span class="pl-c1">0.011477531865239143</span>,
      <span class="pl-ent">"Left"</span>: <span class="pl-c1">0.9904273152351379</span>,
      <span class="pl-ent">"Top"</span>: <span class="pl-c1">0.00909337680786848</span>
    },
    <span class="pl-ent">"Polygon"</span>: [
      {
        <span class="pl-ent">"X"</span>: <span class="pl-c1">0.9904273152351379</span>,
        <span class="pl-ent">"Y"</span>: <span class="pl-c1">0.00909337680786848</span>
      },
      {
        <span class="pl-ent">"X"</span>: <span class="pl-c1">0.9980074763298035</span>,
        <span class="pl-ent">"Y"</span>: <span class="pl-c1">0.00909337680786848</span>
      },
      {
        <span class="pl-ent">"X"</span>: <span class="pl-c1">0.9980074763298035</span>,
        <span class="pl-ent">"Y"</span>: <span class="pl-c1">0.0205709096044302</span>
      },
      {
        <span class="pl-ent">"X"</span>: <span class="pl-c1">0.9904273152351379</span>,
        <span class="pl-ent">"Y"</span>: <span class="pl-c1">0.0205709096044302</span>
      }
    ]
  },
  <span class="pl-ent">"Id"</span>: <span class="pl-s"><span class="pl-pds">"</span>6b04b8df-bec1-42d3-bfff-29f0edd38976<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"Relationships"</span>: [
    {
      <span class="pl-ent">"Type"</span>: <span class="pl-s"><span class="pl-pds">"</span>CHILD<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"Ids"</span>: [
        <span class="pl-s"><span class="pl-pds">"</span>58890ca7-5ed5-4b14-ad60-475e5d0dd79e<span class="pl-pds">"</span></span>
      ]
    }
  ],
  <span class="pl-ent">"Page"</span>: <span class="pl-c1">1</span>
}</pre></div>
<p>I found that joining together those lines on a <code>\n</code> gave me the results I needed:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">print</span>(<span class="pl-s">"<span class="pl-cce">\n</span>"</span>.<span class="pl-en">join</span>([<span class="pl-s1">block</span>[<span class="pl-s">"Text"</span>] <span class="pl-k">for</span> <span class="pl-s1">block</span> <span class="pl-c1">in</span> <span class="pl-s1">blocks</span> <span class="pl-k">if</span> <span class="pl-s1">block</span>[<span class="pl-s">"BlockType"</span>] <span class="pl-c1">==</span> <span class="pl-s">"LINE"</span>]))</pre></div>
<p>Truncated output:</p>
<pre><code>1
ORGANIZATION MEETING
of the
SAN FRANCISCO MICROSCOPICAL SOCIETY
October 4, 1946
The meeting ws.s held at 8:00 P.M. on October 4, 1946, in the
Auditorium of the San Francisco Department of Health, 101 Grove Street,
San Francisco.
Chairman George Herbert Needham called the audience of sixty-
five persons to order. He told of the high aims, ideals, and fine fellow-
ship enjoyed by the original society which was organized in 1870 and incor-
porated in 1872, but which was dissolved following the San Francisco fire
of 1906. He related his efforts to find a surviving member which finally
resulted in a telegram of greeting from Dr. Kaspar Pischell of Ross, Cali-
fornia, which read as follows:
"BEST WISHES AT THIS REUNION. I AM SORRY I CANNOT BE WITH YOU."
</code></pre>


path: zsh_argument-heredoc.md
topic: zsh
title: Passing command arguments using heredoc syntax
url: https://github.com/simonw/til/blob/main/zsh/argument-heredoc.md
created: 2022-07-07T15:12:24-07:00
created_utc: 2022-07-07T22:12:24+00:00
updated: 2022-07-07T15:12:24-07:00
updated_utc: 2022-07-07T22:12:24+00:00
shot_hash: 5bc4afb5fe0d53b58aeb6d70ac6ec3d2
slug: argument-heredoc
body:
This trick works for both Bash and zsh.

I wanted to pass the following as an argument to the sqlite-utils CLI tool:

```
insert into documents select
  substr(s3_ocr_etag, 2, 8) as id,
  key as title,
  key as path,
  replace(s3_ocr_etag, '"', '') as etag
from
  index2.ocr_jobs;
```

Problem: this contains both single AND double quotes, which makes string escaping a tiny bit tricky.

Solution: use heredoc syntax:
```
sqlite-utils sfms.db --attach index2 index.db "$(cat <<EOF
insert into documents select
  substr(s3_ocr_etag, 2, 8) as id,
  key as title,
  key as path,
  replace(s3_ocr_etag, '"', '') as etag
from
  index2.ocr_jobs;
EOF
)"
```
Breaking that apart: the main trick here is to use `cat <<EOF ... EOF` to wrap the literal chunk of text:
```
$(cat <<EOF
insert into documents select
  substr(s3_ocr_etag, 2, 8) as id,
  key as title,
  key as path,
  replace(s3_ocr_etag, '"', '') as etag
from
  index2.ocr_jobs;
EOF
)
```
Then to pass it as an argument to the `sqlite-utils` command use `"$(cat ...)"` - the double quotes around that ensure that tokens in that input are not treated as separate arguments by zsh/bash.

html:
<p>This trick works for both Bash and zsh.</p>
<p>I wanted to pass the following as an argument to the sqlite-utils CLI tool:</p>
<pre><code>insert into documents select
  substr(s3_ocr_etag, 2, 8) as id,
  key as title,
  key as path,
  replace(s3_ocr_etag, '"', '') as etag
from
  index2.ocr_jobs;
</code></pre>
<p>Problem: this contains both single AND double quotes, which makes string escaping a tiny bit tricky.</p>
<p>Solution: use heredoc syntax:</p>
<pre><code>sqlite-utils sfms.db --attach index2 index.db "$(cat &lt;&lt;EOF
insert into documents select
  substr(s3_ocr_etag, 2, 8) as id,
  key as title,
  key as path,
  replace(s3_ocr_etag, '"', '') as etag
from
  index2.ocr_jobs;
EOF
)"
</code></pre>
<p>Breaking that apart: the main trick here is to use <code>cat &lt;&lt;EOF ... EOF</code> to wrap the literal chunk of text:</p>
<pre><code>$(cat &lt;&lt;EOF
insert into documents select
  substr(s3_ocr_etag, 2, 8) as id,
  key as title,
  key as path,
  replace(s3_ocr_etag, '"', '') as etag
from
  index2.ocr_jobs;
EOF
)
</code></pre>
<p>Then to pass it as an argument to the <code>sqlite-utils</code> command use <code>"$(cat ...)"</code> - the double quotes around that ensure that tokens in that input are not treated as separate arguments by zsh/bash.</p>


path: github_reporting-bugs.md
topic: github
title: Reporting bugs in GitHub to GitHub
url: https://github.com/simonw/til/blob/main/github/reporting-bugs.md
created: 2022-07-07T22:14:58-07:00
created_utc: 2022-07-08T05:14:58+00:00
updated: 2022-07-07T22:16:56-07:00
updated_utc: 2022-07-08T05:16:56+00:00
shot_hash: 2928232be1272a8682e2681ad0296ec5
slug: reporting-bugs
body:
I found out today (via [this post](https://github.com/github-community/community/discussions/19988)) about a dedicated interface for reporting bugs in GitHub to GitHub:

https://support.github.com/contact/bug-report

It includes full markdown support, which means you can include animated GIFs that illustrate the bug.

Once reported, you can track the status of your bug reports here:

https://support.github.com/tickets

html:
<p>I found out today (via <a href="https://github.com/github-community/community/discussions/19988">this post</a>) about a dedicated interface for reporting bugs in GitHub to GitHub:</p>
<p><a href="https://support.github.com/contact/bug-report">https://support.github.com/contact/bug-report</a></p>
<p>It includes full markdown support, which means you can include animated GIFs that illustrate the bug.</p>
<p>Once reported, you can track the status of your bug reports here:</p>
<p><a href="https://support.github.com/tickets">https://support.github.com/tickets</a></p>


path: github-actions_conditionally-run-a-second-job.md
topic: github-actions
title: Conditionally running a second job in a GitHub Actions workflow
url: https://github.com/simonw/til/blob/main/github-actions/conditionally-run-a-second-job.md
created: 2022-07-11T13:39:01-07:00
created_utc: 2022-07-11T20:39:01+00:00
updated: 2022-07-11T17:05:01-07:00
updated_utc: 2022-07-12T00:05:01+00:00
shot_hash: 2c87c48078adb1b230e8e2e14af183e9
slug: conditionally-run-a-second-job
body:
My [simonwillisonblog-backup workflow](https://github.com/simonw/simonwillisonblog-backup/blob/main/.github/workflows/backup.yml) periodically creates a JSON backup of my  blog's PostgreSQL database, using [db-to-sqlite](https://datasette.io/tools/db-to-sqlite) and [sqlite-diffable](https://datasette.io/tools/sqlite-diffable). It then commits any changes back to the repo using this pattern:

```yaml
    - name: Commit any changes
      run: |-
        git config user.name "Automated"
        git config user.email "actions@users.noreply.github.com"
        git add simonwillisonblog
        timestamp=$(date -u)
        git commit -m "Latest data: ${timestamp}" || exit 0
        git push
```

I decided to upgrade it to also build and deploy a SQLite database of the content to [datasette.simonwillison.net](https://datasette.simonwillison.net/) - but only if a change had been detected.

I figured out the following pattern for doing that.

First, I added a line to the above block that set a `change_detected` [output variable](https://docs.github.com/en/actions/using-jobs/defining-outputs-for-jobs) for that step if it made it past the `|| exit 0`. I also added an `id` to the step so I could reference it later on:

```yaml
    - name: Commit any changes
      id: commit_and_push
      run: |-
        git config user.name "Automated"
        git config user.email "actions@users.noreply.github.com"
        git add simonwillisonblog
        timestamp=$(date -u)
        git commit -m "Latest data: ${timestamp}" || exit 0
        git push
        echo "::set-output name=change_detected::1"
```
This next piece took me a while to figure out: I also had to declare that output variable at the top of the initial job, copying the result of the named step:
```yaml
jobs:
  backup:
    runs-on: ubuntu-latest
    outputs:
      change_detected: ${{ steps.commit_and_push.outputs.change_detected }}
```
Without this, the output is not visible to the second job.

My second job started like this:
```yaml
  build_and_deploy:
    runs-on: ubuntu-latest
    needs: backup
    if: ${{ inputs.force_deploy || needs.backup.outputs.change_detected }}
    steps:
```
This second job is called `build_and_deploy` and specify that it `needs: backup` - so it should run directly after that backup job completes.

That new job has an `if:` expression which looks at `needs.backup.outputs.change_detected` to read the variable that was set by my `echo "::set-output` line.

I'm also checking `inputs.force_deploy` here. That's a separate mechanism, which allows me to trigger the workflow with manually and specify that a deploy should happen even if no changes were detected - useful for when I alter the code that configures the deployed Datasette instance.

The `force_deploy` variable comes from this section at the start of the YAML:

```yaml
on:
  push:
    branches:
      - main
  workflow_dispatch:
    inputs:
      force_deploy:
        description: 'Deploy even if no changes detected'
        required: false 
        type: boolean
```

This configuration adds the following UI which I can use to manually trigger the workflow:

![Screenshot of the UI showing the force_deploy checkbox](https://user-images.githubusercontent.com/9599/178353732-77e58ddd-c78c-4032-aeea-cb388bac8ec6.jpg)

html:
<p>My <a href="https://github.com/simonw/simonwillisonblog-backup/blob/main/.github/workflows/backup.yml">simonwillisonblog-backup workflow</a> periodically creates a JSON backup of my  blog's PostgreSQL database, using <a href="https://datasette.io/tools/db-to-sqlite" rel="nofollow">db-to-sqlite</a> and <a href="https://datasette.io/tools/sqlite-diffable" rel="nofollow">sqlite-diffable</a>. It then commits any changes back to the repo using this pattern:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Commit any changes</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        git config user.name "Automated"</span>
<span class="pl-s">        git config user.email "actions@users.noreply.github.com"</span>
<span class="pl-s">        git add simonwillisonblog</span>
<span class="pl-s">        timestamp=$(date -u)</span>
<span class="pl-s">        git commit -m "Latest data: ${timestamp}" || exit 0</span>
<span class="pl-s">        git push</span></pre></div>
<p>I decided to upgrade it to also build and deploy a SQLite database of the content to <a href="https://datasette.simonwillison.net/" rel="nofollow">datasette.simonwillison.net</a> - but only if a change had been detected.</p>
<p>I figured out the following pattern for doing that.</p>
<p>First, I added a line to the above block that set a <code>change_detected</code> <a href="https://docs.github.com/en/actions/using-jobs/defining-outputs-for-jobs">output variable</a> for that step if it made it past the <code>|| exit 0</code>. I also added an <code>id</code> to the step so I could reference it later on:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Commit any changes</span>
      <span class="pl-ent">id</span>: <span class="pl-s">commit_and_push</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        git config user.name "Automated"</span>
<span class="pl-s">        git config user.email "actions@users.noreply.github.com"</span>
<span class="pl-s">        git add simonwillisonblog</span>
<span class="pl-s">        timestamp=$(date -u)</span>
<span class="pl-s">        git commit -m "Latest data: ${timestamp}" || exit 0</span>
<span class="pl-s">        git push</span>
<span class="pl-s">        echo "::set-output name=change_detected::1"</span></pre></div>
<p>This next piece took me a while to figure out: I also had to declare that output variable at the top of the initial job, copying the result of the named step:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">jobs</span>:
  <span class="pl-ent">backup</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">outputs</span>:
      <span class="pl-ent">change_detected</span>: <span class="pl-s">${{ steps.commit_and_push.outputs.change_detected }}</span></pre></div>
<p>Without this, the output is not visible to the second job.</p>
<p>My second job started like this:</p>
<div class="highlight highlight-source-yaml"><pre>  <span class="pl-ent">build_and_deploy</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">needs</span>: <span class="pl-s">backup</span>
    <span class="pl-ent">if</span>: <span class="pl-s">${{ inputs.force_deploy || needs.backup.outputs.change_detected }}</span>
    <span class="pl-ent">steps</span>:</pre></div>
<p>This second job is called <code>build_and_deploy</code> and specify that it <code>needs: backup</code> - so it should run directly after that backup job completes.</p>
<p>That new job has an <code>if:</code> expression which looks at <code>needs.backup.outputs.change_detected</code> to read the variable that was set by my <code>echo "::set-output</code> line.</p>
<p>I'm also checking <code>inputs.force_deploy</code> here. That's a separate mechanism, which allows me to trigger the workflow with manually and specify that a deploy should happen even if no changes were detected - useful for when I alter the code that configures the deployed Datasette instance.</p>
<p>The <code>force_deploy</code> variable comes from this section at the start of the YAML:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">on</span>:
  <span class="pl-ent">push</span>:
    <span class="pl-ent">branches</span>:
      - <span class="pl-s">main</span>
  <span class="pl-ent">workflow_dispatch</span>:
    <span class="pl-ent">inputs</span>:
      <span class="pl-ent">force_deploy</span>:
        <span class="pl-ent">description</span>: <span class="pl-s"><span class="pl-pds">'</span>Deploy even if no changes detected<span class="pl-pds">'</span></span>
        <span class="pl-ent">required</span>: <span class="pl-c1">false</span> 
        <span class="pl-ent">type</span>: <span class="pl-s">boolean</span></pre></div>
<p>This configuration adds the following UI which I can use to manually trigger the workflow:</p>
<p><a href="https://user-images.githubusercontent.com/9599/178353732-77e58ddd-c78c-4032-aeea-cb388bac8ec6.jpg" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/178353732-77e58ddd-c78c-4032-aeea-cb388bac8ec6.jpg" alt="Screenshot of the UI showing the force_deploy checkbox" style="max-width:100%;"></a></p>


path: npm_upgrading-packages.md
topic: npm
title: Upgrading packages with npm
url: https://github.com/simonw/til/blob/main/npm/upgrading-packages.md
created: 2022-07-13T08:04:16-07:00
created_utc: 2022-07-13T15:04:16+00:00
updated: 2022-07-13T08:04:16-07:00
updated_utc: 2022-07-13T15:04:16+00:00
shot_hash: 351d93c47aa1870a75ffc310816b9650
slug: upgrading-packages
body:
There's a new version of [Vite](https://vitejs.dev/) out (3.0) and I wanted to upgrade my [datasette-table](https://github.com/simonw/datasette-table) package to use it.

I mainly followed the guide on [Update all the Node.js dependencies to their latest version](https://nodejs.dev/learn/update-all-the-nodejs-dependencies-to-their-latest-version) to work out how to do this.

My `package.json` started out containing this:

```json
  "dependencies": {
    "lit": "^2.0.0"
  },
  "devDependencies": {
    "vite": "^2.6.4"
  }
```
The `^` syntax here pins to a major version - running `npm update` will update the `package-lock.json` file to point to the highest `2.x` version of the package but won't increase the major version to `3.x`.

`npm outdated` shows if there are any releases that go beyond my pinned packages. `npm help outdated` explains [how it works in detail](https://manpages.ubuntu.com/manpages/focal/man1/npm-outdated.1.html) (`npm outdated --help` shows a less useful summary).

Running it against my project shows:
```
datasette-table % npm outdated
Package  Current  Wanted  Latest  Location           Depended by
vite      2.9.14  2.9.14   3.0.0  node_modules/vite  datasette-table
```
OK, so there's a major version upgrade available.

The `npm` tool itself doesn't have a way of applying that automatically - you need to install an extra tool, [npm-check-updates](https://manpages.ubuntu.com/manpages/focal/man1/npm-outdated.1.html):
```
npm install -g npm-check-updates
```
Then run `npm-check-updates -u` to apply those upgrades directly to `package.json`:
```
datasette-table % npm-check-updates -u
Upgrading .../datasette-table/package.json
[====================] 2/2 100%

 lit   ^2.0.0  →  ^2.2.7     
 vite  ^2.6.4  →  ^3.0.0     

Run npm install to install new versions.
```
`git diff` shows the changes it made:
```
datasette-table % git diff
diff --git a/package.json b/package.json
index 7682f38..43bfa14 100644
--- a/package.json
+++ b/package.json
@@ -13,10 +13,10 @@
     "serve": "vite preview"
   },
   "dependencies": {
-    "lit": "^2.0.0"
+    "lit": "^2.2.7"
   },
   "devDependencies": {
-    "vite": "^2.6.4"
+    "vite": "^3.0.0"
   },
   "repository": {
     "type": "git",
```
Note that it upgraded `lit` as well - `npm-check-updates` "upgrades your `package.json` dependencies to the latest versions, ignoring specified versions".

Finally, run `npm install` to install the new versions:
```
datasette-table % npm install

changed 1 package, and audited 21 packages in 901ms

4 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
```

html:
<p>There's a new version of <a href="https://vitejs.dev/" rel="nofollow">Vite</a> out (3.0) and I wanted to upgrade my <a href="https://github.com/simonw/datasette-table">datasette-table</a> package to use it.</p>
<p>I mainly followed the guide on <a href="https://nodejs.dev/learn/update-all-the-nodejs-dependencies-to-their-latest-version" rel="nofollow">Update all the Node.js dependencies to their latest version</a> to work out how to do this.</p>
<p>My <code>package.json</code> started out containing this:</p>
<div class="highlight highlight-source-json"><pre>  <span class="pl-ent">"dependencies"</span>: {
    <span class="pl-ent">"lit"</span>: <span class="pl-s"><span class="pl-pds">"</span>^2.0.0<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"devDependencies"</span>: {
    <span class="pl-ent">"vite"</span>: <span class="pl-s"><span class="pl-pds">"</span>^2.6.4<span class="pl-pds">"</span></span>
  }</pre></div>
<p>The <code>^</code> syntax here pins to a major version - running <code>npm update</code> will update the <code>package-lock.json</code> file to point to the highest <code>2.x</code> version of the package but won't increase the major version to <code>3.x</code>.</p>
<p><code>npm outdated</code> shows if there are any releases that go beyond my pinned packages. <code>npm help outdated</code> explains <a href="https://manpages.ubuntu.com/manpages/focal/man1/npm-outdated.1.html" rel="nofollow">how it works in detail</a> (<code>npm outdated --help</code> shows a less useful summary).</p>
<p>Running it against my project shows:</p>
<pre><code>datasette-table % npm outdated
Package  Current  Wanted  Latest  Location           Depended by
vite      2.9.14  2.9.14   3.0.0  node_modules/vite  datasette-table
</code></pre>
<p>OK, so there's a major version upgrade available.</p>
<p>The <code>npm</code> tool itself doesn't have a way of applying that automatically - you need to install an extra tool, <a href="https://manpages.ubuntu.com/manpages/focal/man1/npm-outdated.1.html" rel="nofollow">npm-check-updates</a>:</p>
<pre><code>npm install -g npm-check-updates
</code></pre>
<p>Then run <code>npm-check-updates -u</code> to apply those upgrades directly to <code>package.json</code>:</p>
<pre><code>datasette-table % npm-check-updates -u
Upgrading .../datasette-table/package.json
[====================] 2/2 100%

 lit   ^2.0.0  →  ^2.2.7     
 vite  ^2.6.4  →  ^3.0.0     

Run npm install to install new versions.
</code></pre>
<p><code>git diff</code> shows the changes it made:</p>
<pre><code>datasette-table % git diff
diff --git a/package.json b/package.json
index 7682f38..43bfa14 100644
--- a/package.json
+++ b/package.json
@@ -13,10 +13,10 @@
     "serve": "vite preview"
   },
   "dependencies": {
-    "lit": "^2.0.0"
+    "lit": "^2.2.7"
   },
   "devDependencies": {
-    "vite": "^2.6.4"
+    "vite": "^3.0.0"
   },
   "repository": {
     "type": "git",
</code></pre>
<p>Note that it upgraded <code>lit</code> as well - <code>npm-check-updates</code> "upgrades your <code>package.json</code> dependencies to the latest versions, ignoring specified versions".</p>
<p>Finally, run <code>npm install</code> to install the new versions:</p>
<pre><code>datasette-table % npm install

changed 1 package, and audited 21 packages in 901ms

4 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities
</code></pre>


path: electron_testing-electron-playwright.md
topic: electron
title: Testing Electron apps with Playwright and GitHub Actions
url: https://github.com/simonw/til/blob/main/electron/testing-electron-playwright.md
created: 2022-07-13T15:29:19-07:00
created_utc: 2022-07-13T22:29:19+00:00
updated: 2022-07-13T15:29:19-07:00
updated_utc: 2022-07-13T22:29:19+00:00
shot_hash: b6eb2943ffaec25569035cc04383de7d
slug: testing-electron-playwright
body:
Yesterday [I figured out (issue 133)](https://github.com/simonw/datasette-app/issues/133) how to use Playwright to run tests against my Electron app, and then execute those tests in CI using GitHub Actions, for my [datasett-app](https://github.com/simonw/datasette-app) repo for my [Datasette Desktop](https://datasette.io/desktop) macOS application.

## Installing @playwright/test

You need to install the `@playwright/test` package. You can do that like so:

    npm i -D @playwright/test

This adds it to `devDependencies` in your `package.json`, something like this:

```
  "devDependencies": {
    "@playwright/test": "^1.23.2",
```

## Writing a test

I dropped the following into a `test/spec.mjs` file:

```javascript
import { test, expect } from '@playwright/test';
import { _electron } from 'playwright';

test('App launches and quits', async () => {
  const app = await _electron.launch({args: ['main.js']);
  const window = await app.firstWindow();
  await expect(await window.title()).toContain('Loading');
  await app.close();
});
```
The `.mjs` extension is necessary in order to use `import`, since it lets Node.js know that this file is a JavaScript module.

The test can be run using `playwright test`.

I later added it to my `package.json` section like this:
```json
  "scripts": {
    "test": "playwright test"
  }
```
Now I can run the Playwright tests using `npm test`.

## Recording video of the tests

Recording videos of the test runs turns out to be easy: change the `_electron.launch()` line to look like this:

```javascript
  const app = await _electron.launch({
    args: ['main.js'],
    recordVideo: {dir: 'test-videos'}
  });
```
This creates the videos as `.webm` files in the `test-videos` directory.

These videos can be opened in Chrome, or can be converted to `mp4` using `ffmpeg` (available on macOS via `brew install ffmpeg`):

    ffmpeg -i bc74c2a51bd91fe6f6cb815e6b99b6c7.webm bc74c2a51bd91fe6f6cb815e6b99b6c7.mp4

Converting to `.mp4` means you can drag and drop them onto a GitHub Issues thread and get an embedded video player. [Here's an example](https://github.com/simonw/datasette-app/issues/133#issuecomment-1182530789) I recorded.

## Custom timeouts

Playwright has a default 30s timeout on every action it takes. This turned out to be a bit too short for one of my tests, which installs a Python interpreter and a bunch of Python packages and can take 57s. Here's how I fixed that so the test could pass:

```javascript
test('App launches and quits', async () => {
  // This disables the global 30s timeout
  test.setTimeout(0);
  const app = await _electron.launch({
    args: ['main.js'],
    recordVideo: {dir: 'test-videos'}
  });
  const window = await app.firstWindow();
  // This sets a timeout of 90s for the page to load and the
  // element with id="run-sql-link" to appear in the DOM:
  await window.waitForSelector('#run-sql-link', {
    timeout: 90000
  });
  await app.close();
});
```

## Running it in GitHub Actions

I'm using the `macos-latest` image in my GitHub Actions workflow. The relevant configuration in my `.github/workflows/test.yml` file looks like this:

```yaml
name: Test

on: push

jobs:
  test:
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v3
      - name: Configure Node caching
        uses: actions/cache@v3
        env:
          cache-name: cache-node-modules
        with:
          path: ~/.npm
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-build-${{ env.cache-name }}-
            ${{ runner.os }}-build-
            ${{ runner.os }}-
      - name: Install Node dependencies
        run: npm install
      - name: Run tests
        run: npm test
        timeout-minutes: 5
      - name: Upload test videos
        uses: actions/upload-artifact@v3
        with:
          name: test-videos
          path: test-videos/
```
This workflow configures NPM caching to avoid downloading everything every time, installs the dependencies, runs the tests, and then uploads the videos at the end.

Those videos end up attached to the workflow run as an artifact that can be downloaded and viewed locally.

html:
<p>Yesterday <a href="https://github.com/simonw/datasette-app/issues/133">I figured out (issue 133)</a> how to use Playwright to run tests against my Electron app, and then execute those tests in CI using GitHub Actions, for my <a href="https://github.com/simonw/datasette-app">datasett-app</a> repo for my <a href="https://datasette.io/desktop" rel="nofollow">Datasette Desktop</a> macOS application.</p>
<h2>
<a id="user-content-installing-playwrighttest" class="anchor" href="#installing-playwrighttest" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing @playwright/test</h2>
<p>You need to install the <code>@playwright/test</code> package. You can do that like so:</p>
<pre><code>npm i -D @playwright/test
</code></pre>
<p>This adds it to <code>devDependencies</code> in your <code>package.json</code>, something like this:</p>
<pre><code>  "devDependencies": {
    "@playwright/test": "^1.23.2",
</code></pre>
<h2>
<a id="user-content-writing-a-test" class="anchor" href="#writing-a-test" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Writing a test</h2>
<p>I dropped the following into a <code>test/spec.mjs</code> file:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">import</span> <span class="pl-kos">{</span> <span class="pl-s1">test</span><span class="pl-kos">,</span> <span class="pl-s1">expect</span> <span class="pl-kos">}</span> <span class="pl-k">from</span> <span class="pl-s">'@playwright/test'</span><span class="pl-kos">;</span>
<span class="pl-k">import</span> <span class="pl-kos">{</span> <span class="pl-s1">_electron</span> <span class="pl-kos">}</span> <span class="pl-k">from</span> <span class="pl-s">'playwright'</span><span class="pl-kos">;</span>

<span class="pl-en">test</span><span class="pl-kos">(</span><span class="pl-s">'App launches and quits'</span><span class="pl-kos">,</span> <span class="pl-k">async</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
  <span class="pl-k">const</span> <span class="pl-s1">app</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">_electron</span><span class="pl-kos">.</span><span class="pl-en">launch</span><span class="pl-kos">(</span><span class="pl-kos">{</span><span class="pl-c1">args</span>: <span class="pl-kos">[</span><span class="pl-s">'main.js'</span><span class="pl-kos">]</span><span class="pl-kos"></span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">window</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">app</span><span class="pl-kos">.</span><span class="pl-en">firstWindow</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">await</span> <span class="pl-en">expect</span><span class="pl-kos">(</span><span class="pl-k">await</span> <span class="pl-s1">window</span><span class="pl-kos">.</span><span class="pl-en">title</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">toContain</span><span class="pl-kos">(</span><span class="pl-s">'Loading'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">await</span> <span class="pl-s1">app</span><span class="pl-kos">.</span><span class="pl-en">close</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<p>The <code>.mjs</code> extension is necessary in order to use <code>import</code>, since it lets Node.js know that this file is a JavaScript module.</p>
<p>The test can be run using <code>playwright test</code>.</p>
<p>I later added it to my <code>package.json</code> section like this:</p>
<div class="highlight highlight-source-json"><pre>  <span class="pl-ent">"scripts"</span>: {
    <span class="pl-ent">"test"</span>: <span class="pl-s"><span class="pl-pds">"</span>playwright test<span class="pl-pds">"</span></span>
  }</pre></div>
<p>Now I can run the Playwright tests using <code>npm test</code>.</p>
<h2>
<a id="user-content-recording-video-of-the-tests" class="anchor" href="#recording-video-of-the-tests" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Recording video of the tests</h2>
<p>Recording videos of the test runs turns out to be easy: change the <code>_electron.launch()</code> line to look like this:</p>
<div class="highlight highlight-source-js"><pre>  <span class="pl-k">const</span> <span class="pl-s1">app</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">_electron</span><span class="pl-kos">.</span><span class="pl-en">launch</span><span class="pl-kos">(</span><span class="pl-kos">{</span>
    <span class="pl-c1">args</span>: <span class="pl-kos">[</span><span class="pl-s">'main.js'</span><span class="pl-kos">]</span><span class="pl-kos">,</span>
    <span class="pl-c1">recordVideo</span>: <span class="pl-kos">{</span><span class="pl-c1">dir</span>: <span class="pl-s">'test-videos'</span><span class="pl-kos">}</span>
  <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<p>This creates the videos as <code>.webm</code> files in the <code>test-videos</code> directory.</p>
<p>These videos can be opened in Chrome, or can be converted to <code>mp4</code> using <code>ffmpeg</code> (available on macOS via <code>brew install ffmpeg</code>):</p>
<pre><code>ffmpeg -i bc74c2a51bd91fe6f6cb815e6b99b6c7.webm bc74c2a51bd91fe6f6cb815e6b99b6c7.mp4
</code></pre>
<p>Converting to <code>.mp4</code> means you can drag and drop them onto a GitHub Issues thread and get an embedded video player. <a href="https://github.com/simonw/datasette-app/issues/133#issuecomment-1182530789">Here's an example</a> I recorded.</p>
<h2>
<a id="user-content-custom-timeouts" class="anchor" href="#custom-timeouts" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Custom timeouts</h2>
<p>Playwright has a default 30s timeout on every action it takes. This turned out to be a bit too short for one of my tests, which installs a Python interpreter and a bunch of Python packages and can take 57s. Here's how I fixed that so the test could pass:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-en">test</span><span class="pl-kos">(</span><span class="pl-s">'App launches and quits'</span><span class="pl-kos">,</span> <span class="pl-k">async</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
  <span class="pl-c">// This disables the global 30s timeout</span>
  <span class="pl-s1">test</span><span class="pl-kos">.</span><span class="pl-en">setTimeout</span><span class="pl-kos">(</span><span class="pl-c1">0</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">app</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">_electron</span><span class="pl-kos">.</span><span class="pl-en">launch</span><span class="pl-kos">(</span><span class="pl-kos">{</span>
    <span class="pl-c1">args</span>: <span class="pl-kos">[</span><span class="pl-s">'main.js'</span><span class="pl-kos">]</span><span class="pl-kos">,</span>
    <span class="pl-c1">recordVideo</span>: <span class="pl-kos">{</span><span class="pl-c1">dir</span>: <span class="pl-s">'test-videos'</span><span class="pl-kos">}</span>
  <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">window</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">app</span><span class="pl-kos">.</span><span class="pl-en">firstWindow</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-c">// This sets a timeout of 90s for the page to load and the</span>
  <span class="pl-c">// element with id="run-sql-link" to appear in the DOM:</span>
  <span class="pl-k">await</span> <span class="pl-s1">window</span><span class="pl-kos">.</span><span class="pl-en">waitForSelector</span><span class="pl-kos">(</span><span class="pl-s">'#run-sql-link'</span><span class="pl-kos">,</span> <span class="pl-kos">{</span>
    <span class="pl-c1">timeout</span>: <span class="pl-c1">90000</span>
  <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">await</span> <span class="pl-s1">app</span><span class="pl-kos">.</span><span class="pl-en">close</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<h2>
<a id="user-content-running-it-in-github-actions" class="anchor" href="#running-it-in-github-actions" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running it in GitHub Actions</h2>
<p>I'm using the <code>macos-latest</code> image in my GitHub Actions workflow. The relevant configuration in my <code>.github/workflows/test.yml</code> file looks like this:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Test</span>

<span class="pl-ent">on</span>: <span class="pl-s">push</span>

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">test</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">macos-latest</span>
    <span class="pl-ent">steps</span>:
      - <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v3</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Configure Node caching</span>
        <span class="pl-ent">uses</span>: <span class="pl-s">actions/cache@v3</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">cache-name</span>: <span class="pl-s">cache-node-modules</span>
        <span class="pl-ent">with</span>:
          <span class="pl-ent">path</span>: <span class="pl-s">~/.npm</span>
          <span class="pl-ent">key</span>: <span class="pl-s">${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}</span>
          <span class="pl-ent">restore-keys</span>: <span class="pl-s">|</span>
<span class="pl-s">            ${{ runner.os }}-build-${{ env.cache-name }}-</span>
<span class="pl-s">            ${{ runner.os }}-build-</span>
<span class="pl-s">            ${{ runner.os }}-</span>
<span class="pl-s"></span>      - <span class="pl-ent">name</span>: <span class="pl-s">Install Node dependencies</span>
        <span class="pl-ent">run</span>: <span class="pl-s">npm install</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Run tests</span>
        <span class="pl-ent">run</span>: <span class="pl-s">npm test</span>
        <span class="pl-ent">timeout-minutes</span>: <span class="pl-c1">5</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Upload test videos</span>
        <span class="pl-ent">uses</span>: <span class="pl-s">actions/upload-artifact@v3</span>
        <span class="pl-ent">with</span>:
          <span class="pl-ent">name</span>: <span class="pl-s">test-videos</span>
          <span class="pl-ent">path</span>: <span class="pl-s">test-videos/</span></pre></div>
<p>This workflow configures NPM caching to avoid downloading everything every time, installs the dependencies, runs the tests, and then uploads the videos at the end.</p>
<p>Those videos end up attached to the workflow run as an artifact that can be downloaded and viewed locally.</p>


path: python_pip-tools.md
topic: python
title: Freezing requirements with pip-tools
url: https://github.com/simonw/til/blob/main/python/pip-tools.md
created: 2022-07-14T11:22:48-07:00
created_utc: 2022-07-14T18:22:48+00:00
updated: 2022-07-14T11:22:48-07:00
updated_utc: 2022-07-14T18:22:48+00:00
shot_hash: 409fd87ff5f49f0d4d8fcd482e6c8861
slug: pip-tools
body:
I tried [pip-tools](https://github.com/jazzband/pip-tools) for the first time today to pin the requirements for the [natbat/pillarpointstewards](https://github.com/natbat/pillarpointstewards) Django app.

The [pip-tools README](https://github.com/jazzband/pip-tools/blob/master/README.rst) is really good. Here's how I'm using it.

## Create a requirements.in

The `requirements.in` file contains your un-pinned requirements. Mine ended up looking like this:
```
Django~=4.0
gunicorn
whitenoise
psycopg2-binary
dj-database-url
pytest-django
django-extensions
django-htmx
pytz
bs4
httpx
sentry-sdk
pytest-httpx
ics==0.7
```
`Django~=4.0` means version Django 4.0 or higher, but less than 4.1. I pinned it like this because I saw a warning from `django-extensions`:

> `django.utils.deprecation.RemovedInDjango41Warning: 'django_extensions' defines default_app_config = 'django_extensions.apps.DjangoExtensionsConfig'.`

I'm pinning `ics` to the exact version 0.7 due to a broken test I experienced with more recent versions.

## Compiling requirements.txt from requirements.in

The `pip-compile` command runs against a `requirements.in` file and writes `requirements.txt` in the current directory with pinned versions of the packages.

You have to `pip install pip-tools` first to get that command.

I ran it like this:

    pip-compile --upgrade --generate-hashes requirements.in

The `--upgrade` option here causes it to check PyPI for the maximum version of each package that matches the lines in `requirements.in` - without this it may settle for packages that are currently in your local cache.

`--generate-hashes` adds hashes to the generated file, as a security measure.

I then committed both `requirements.in` and `requirements.txt` to the repo.

Here's the generated [requirements.txt](https://github.com/natbat/pillarpointstewards/blob/ff2a8aa14a41e548f37d61312dc3c6f0036aa73c/requirements.txt).

## Updating requirements locally with pip-sync

The other tool installed by `pip-tools` is `pip-sync`. It's useful for keeping your local development environment updated with the exact packages from `requirements.txt`.

Run it like this:

    pip-sync

It will uninstall and reinstall packages in your virtual environment until they exactly match `requirements.txt`.

html:
<p>I tried <a href="https://github.com/jazzband/pip-tools">pip-tools</a> for the first time today to pin the requirements for the <a href="https://github.com/natbat/pillarpointstewards">natbat/pillarpointstewards</a> Django app.</p>
<p>The <a href="https://github.com/jazzband/pip-tools/blob/master/README.rst">pip-tools README</a> is really good. Here's how I'm using it.</p>
<h2>
<a id="user-content-create-a-requirementsin" class="anchor" href="#create-a-requirementsin" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create a requirements.in</h2>
<p>The <code>requirements.in</code> file contains your un-pinned requirements. Mine ended up looking like this:</p>
<pre><code>Django~=4.0
gunicorn
whitenoise
psycopg2-binary
dj-database-url
pytest-django
django-extensions
django-htmx
pytz
bs4
httpx
sentry-sdk
pytest-httpx
ics==0.7
</code></pre>
<p><code>Django~=4.0</code> means version Django 4.0 or higher, but less than 4.1. I pinned it like this because I saw a warning from <code>django-extensions</code>:</p>
<blockquote>
<p><code>django.utils.deprecation.RemovedInDjango41Warning: 'django_extensions' defines default_app_config = 'django_extensions.apps.DjangoExtensionsConfig'.</code></p>
</blockquote>
<p>I'm pinning <code>ics</code> to the exact version 0.7 due to a broken test I experienced with more recent versions.</p>
<h2>
<a id="user-content-compiling-requirementstxt-from-requirementsin" class="anchor" href="#compiling-requirementstxt-from-requirementsin" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Compiling requirements.txt from requirements.in</h2>
<p>The <code>pip-compile</code> command runs against a <code>requirements.in</code> file and writes <code>requirements.txt</code> in the current directory with pinned versions of the packages.</p>
<p>You have to <code>pip install pip-tools</code> first to get that command.</p>
<p>I ran it like this:</p>
<pre><code>pip-compile --upgrade --generate-hashes requirements.in
</code></pre>
<p>The <code>--upgrade</code> option here causes it to check PyPI for the maximum version of each package that matches the lines in <code>requirements.in</code> - without this it may settle for packages that are currently in your local cache.</p>
<p><code>--generate-hashes</code> adds hashes to the generated file, as a security measure.</p>
<p>I then committed both <code>requirements.in</code> and <code>requirements.txt</code> to the repo.</p>
<p>Here's the generated <a href="https://github.com/natbat/pillarpointstewards/blob/ff2a8aa14a41e548f37d61312dc3c6f0036aa73c/requirements.txt">requirements.txt</a>.</p>
<h2>
<a id="user-content-updating-requirements-locally-with-pip-sync" class="anchor" href="#updating-requirements-locally-with-pip-sync" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Updating requirements locally with pip-sync</h2>
<p>The other tool installed by <code>pip-tools</code> is <code>pip-sync</code>. It's useful for keeping your local development environment updated with the exact packages from <code>requirements.txt</code>.</p>
<p>Run it like this:</p>
<pre><code>pip-sync
</code></pre>
<p>It will uninstall and reinstall packages in your virtual environment until they exactly match <code>requirements.txt</code>.</p>


path: pytest_playwright-pytest.md
topic: pytest
title: Using pytest and Playwright to test a JavaScript web application
url: https://github.com/simonw/til/blob/main/pytest/playwright-pytest.md
created: 2022-07-24T11:47:44-07:00
created_utc: 2022-07-24T18:47:44+00:00
updated: 2022-07-29T19:45:44-07:00
updated_utc: 2022-07-30T02:45:44+00:00
shot_hash: 9b85080adcfca5f340fa9755a5c6fdd0
slug: playwright-pytest
body:
I [decided to add](https://github.com/simonw/datasette-lite/issues/35) automated tests to my [Datasette Lite](https://simonwillison.net/2022/May/4/datasette-lite/) project. Datasette Lite bundles my Datasette Python web application as a client-side application running inside WebAssembly using Pyodide.

I wrote the tests using [playwright-pytest](https://github.com/microsoft/playwright-pytest), which lets you write tests in Python using Microsoft's [Playwright](https://playwright.dev/) browser automation library.

## Installing playwright-pytest

Two steps:

    pip install playwright-pytest

Then a second step to install the browsers using by Playwright itself:

    playwright install

I had those browsers installed already, but I still needed to run that command since the updated version of `playwright-pytest` needed more recent versions.

(I had limited internet while doing this, and discovered that you can trick Playwright into using an older browser version by renaming a folder in `~/Library/Caches/ms-playwright` to the one that shows up in the error message that says that the browsers cannot be found.)

## An initial test

The first test I wrote looked like this, saved in `test_app.py`:

```python
from playwright.sync_api import Page, expect

def test_initial_load(page: Page):
    page.goto("https://lite.datasette.io/")
    loading = page.locator("#loading-indicator")
    expect(loading).to_have_css("display", "block")
    # Give it up to 60s to finish loading
    expect(loading).to_have_css("display", "none", timeout=60 * 1000)

    # Should load faster the second time thanks to cache
    page.goto("https://lite.datasette.io/")
    expect(loading).to_have_css("display", "none", timeout=20 * 1000)
```

Then run the test by running this in the same directory as that file:

    pytest

`playwright-pytest` provides the `page` fixture - annotating it with `: Page` is optional but if you do that then VS Code knows what it is and can provide autocomplete in the editor.

`page.goto()` causes the browser to navigate to that URL.

`page.locator("#loading-indicator")` returns a wrapper "locator" object for the specified CSS selector.

This line is interesting:

```python
expect(loading).to_have_css("display", "block")
```
The `expect()` helper function encapsulates the concept of polling the page to wait for something to become true within a time limit. This is the key technique for avoiding "flaky" tests when working with Playwright.


The assertions [are listed here](https://playwright.dev/python/docs/test-assertions).

You don't actually need to use `expect()` though - that's useful if you don't know how long it will take for the page to load, but if you know the page is already loaded you can write assertions like this instead:

```python
assert [
    el.inner_text()
    for el in page.query_selector_all("h2")
] == ["fixtures", "content"]
```

## pytest options

The `playwright-pytest` package adds a bunch of new options to `pytest`. The most useful is `--headed`:

    pytest --headed

This runs the tests in "headed" mode - which means a visible browser window pops up during the tests so you can see what is happening.

`--browser firefox` runs them using Firefox instead of Chromium.

`--tracing on` is really interesting: it generates a trace ZIP file which you can then open using https://trace.playwright.dev/ to explore a detailed trace of the test as it executed.

`--video on` records a video (as a `.webm` file) of the test. I've not tried it yet, but `--video retain-on-failure` only keeps that video if the test fails.

Here's [documentation on all of the options](https://playwright.dev/python/docs/test-runners).

## Running a localhost static server during the tests

I wanted to run the tests against the most recent version of my code, which consists of an `index.html` file and a `webworker.js` file. Because these use web workers they need to be run from an actual localhost web server, so I needed to start one at the beginning of the tests and shut it down at the end.

I wrote about my solution for this in another TIL: [Start a server in a subprocess during a pytest session](https://til.simonwillison.net/pytest/subprocess-server).

## My test suite so far

Here's where [I've got to so far](https://github.com/simonw/datasette-lite/blob/daba69708c7a72adad20dce3c534b9a399ef11c8/tests/test_datasette_lite.py):

```python
from playwright.sync_api import Browser, Page, expect
from subprocess import Popen, PIPE
import pathlib
import pytest
import time
from http.client import HTTPConnection

root = pathlib.Path(__file__).parent.parent.absolute()


@pytest.fixture(scope="module")
def static_server():
    process = Popen(
        ["python", "-m", "http.server", "8123", "--directory", root], stdout=PIPE
    )
    retries = 5
    while retries > 0:
        conn = HTTPConnection("localhost:8123")
        try:
            conn.request("HEAD", "/")
            response = conn.getresponse()
            if response is not None:
                yield process
                break
        except ConnectionRefusedError:
            time.sleep(1)
            retries -= 1

    if not retries:
        raise RuntimeError("Failed to start http server")
    else:
        process.terminate()
        process.wait()


@pytest.fixture(scope="module")
def dslite(static_server, browser: Browser) -> Page:
    page = browser.new_page()
    page.goto("http://localhost:8123/")
    loading = page.locator("#loading-indicator")
    expect(loading).to_have_css("display", "block")
    # Give it up to 60s to finish loading
    expect(loading).to_have_css("display", "none", timeout=60 * 1000)
    return page


def test_initial_load(dslite: Page):
    expect(dslite.locator("#loading-indicator")).to_have_css("display", "none")


def test_has_two_databases(dslite: Page):
    assert [el.inner_text() for el in dslite.query_selector_all("h2")] == [
        "fixtures",
        "content",
    ]


def test_navigate_to_database(dslite: Page):
    h2 = dslite.query_selector("h2")
    assert h2.inner_text() == "fixtures"
    h2.query_selector("a").click()
    expect(dslite).to_have_title("fixtures")
    dslite.query_selector("textarea#sql-editor").fill(
        "SELECT * FROM no_primary_key limit 1"
    )
    dslite.query_selector("input[type=submit]").click()
    expect(dslite).to_have_title("fixtures: SELECT * FROM no_primary_key limit 1")
    table = dslite.query_selector("table.rows-and-columns")
    table_html = "".join(table.inner_html().split())
    assert table_html == (
        '<thead><tr><thclass="col-content"scope="col">content</th>'
        '<thclass="col-a"scope="col">a</th><thclass="col-b"scope="col">b</th>'
        '<thclass="col-c"scope="col">c</th></tr></thead><tbody><tr>'
        '<tdclass="col-content">1</td><tdclass="col-a">a1</td>'
        '<tdclass="col-b">b1</td><tdclass="col-c">c1</td></tr></tbody>'
    )
```

## Running it in GitHub Actions

Here's the [GitHub Actions workflow](https://github.com/simonw/datasette-lite/blob/main/.github/workflows/test.yml) I'm using to run the tests:

```yaml
name: Test

on:
  push:
  pull_request:

permissions:
  contents: read

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v3
    - name: Set up Python 3.10
      uses: actions/setup-python@v3
      with:
        python-version: "3.10"
        cache: 'pip'
        cache-dependency-path: '**/dev-requirements.txt'
    - name: Cache Playwright browsers
      uses: actions/cache@v3
      with:
        path: ~/.cache/ms-playwright/
        key: ${{ runner.os }}-browsers
    - name: Install dependencies
      run: |
        pip install -r dev-requirements.txt
        playwright install
    - name: Run test
      run: |
        pytest
```
[dev-requirements.txt](https://raw.githubusercontent.com/simonw/datasette-lite/main/dev-requirements.txt) contains this:
```
pytest-playwright==0.3.0
playwright==1.24.0
```

html:
<p>I <a href="https://github.com/simonw/datasette-lite/issues/35">decided to add</a> automated tests to my <a href="https://simonwillison.net/2022/May/4/datasette-lite/" rel="nofollow">Datasette Lite</a> project. Datasette Lite bundles my Datasette Python web application as a client-side application running inside WebAssembly using Pyodide.</p>
<p>I wrote the tests using <a href="https://github.com/microsoft/playwright-pytest">playwright-pytest</a>, which lets you write tests in Python using Microsoft's <a href="https://playwright.dev/" rel="nofollow">Playwright</a> browser automation library.</p>
<h2>
<a id="user-content-installing-playwright-pytest" class="anchor" href="#installing-playwright-pytest" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing playwright-pytest</h2>
<p>Two steps:</p>
<pre><code>pip install playwright-pytest
</code></pre>
<p>Then a second step to install the browsers using by Playwright itself:</p>
<pre><code>playwright install
</code></pre>
<p>I had those browsers installed already, but I still needed to run that command since the updated version of <code>playwright-pytest</code> needed more recent versions.</p>
<p>(I had limited internet while doing this, and discovered that you can trick Playwright into using an older browser version by renaming a folder in <code>~/Library/Caches/ms-playwright</code> to the one that shows up in the error message that says that the browsers cannot be found.)</p>
<h2>
<a id="user-content-an-initial-test" class="anchor" href="#an-initial-test" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>An initial test</h2>
<p>The first test I wrote looked like this, saved in <code>test_app.py</code>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">playwright</span>.<span class="pl-s1">sync_api</span> <span class="pl-k">import</span> <span class="pl-v">Page</span>, <span class="pl-s1">expect</span>

<span class="pl-k">def</span> <span class="pl-en">test_initial_load</span>(<span class="pl-s1">page</span>: <span class="pl-v">Page</span>):
    <span class="pl-s1">page</span>.<span class="pl-en">goto</span>(<span class="pl-s">"https://lite.datasette.io/"</span>)
    <span class="pl-s1">loading</span> <span class="pl-c1">=</span> <span class="pl-s1">page</span>.<span class="pl-en">locator</span>(<span class="pl-s">"#loading-indicator"</span>)
    <span class="pl-en">expect</span>(<span class="pl-s1">loading</span>).<span class="pl-en">to_have_css</span>(<span class="pl-s">"display"</span>, <span class="pl-s">"block"</span>)
    <span class="pl-c"># Give it up to 60s to finish loading</span>
    <span class="pl-en">expect</span>(<span class="pl-s1">loading</span>).<span class="pl-en">to_have_css</span>(<span class="pl-s">"display"</span>, <span class="pl-s">"none"</span>, <span class="pl-s1">timeout</span><span class="pl-c1">=</span><span class="pl-c1">60</span> <span class="pl-c1">*</span> <span class="pl-c1">1000</span>)

    <span class="pl-c"># Should load faster the second time thanks to cache</span>
    <span class="pl-s1">page</span>.<span class="pl-en">goto</span>(<span class="pl-s">"https://lite.datasette.io/"</span>)
    <span class="pl-en">expect</span>(<span class="pl-s1">loading</span>).<span class="pl-en">to_have_css</span>(<span class="pl-s">"display"</span>, <span class="pl-s">"none"</span>, <span class="pl-s1">timeout</span><span class="pl-c1">=</span><span class="pl-c1">20</span> <span class="pl-c1">*</span> <span class="pl-c1">1000</span>)</pre></div>
<p>Then run the test by running this in the same directory as that file:</p>
<pre><code>pytest
</code></pre>
<p><code>playwright-pytest</code> provides the <code>page</code> fixture - annotating it with <code>: Page</code> is optional but if you do that then VS Code knows what it is and can provide autocomplete in the editor.</p>
<p><code>page.goto()</code> causes the browser to navigate to that URL.</p>
<p><code>page.locator("#loading-indicator")</code> returns a wrapper "locator" object for the specified CSS selector.</p>
<p>This line is interesting:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">expect</span>(<span class="pl-s1">loading</span>).<span class="pl-en">to_have_css</span>(<span class="pl-s">"display"</span>, <span class="pl-s">"block"</span>)</pre></div>
<p>The <code>expect()</code> helper function encapsulates the concept of polling the page to wait for something to become true within a time limit. This is the key technique for avoiding "flaky" tests when working with Playwright.</p>
<p>The assertions <a href="https://playwright.dev/python/docs/test-assertions" rel="nofollow">are listed here</a>.</p>
<p>You don't actually need to use <code>expect()</code> though - that's useful if you don't know how long it will take for the page to load, but if you know the page is already loaded you can write assertions like this instead:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">assert</span> [
    <span class="pl-s1">el</span>.<span class="pl-en">inner_text</span>()
    <span class="pl-k">for</span> <span class="pl-s1">el</span> <span class="pl-c1">in</span> <span class="pl-s1">page</span>.<span class="pl-en">query_selector_all</span>(<span class="pl-s">"h2"</span>)
] <span class="pl-c1">==</span> [<span class="pl-s">"fixtures"</span>, <span class="pl-s">"content"</span>]</pre></div>
<h2>
<a id="user-content-pytest-options" class="anchor" href="#pytest-options" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>pytest options</h2>
<p>The <code>playwright-pytest</code> package adds a bunch of new options to <code>pytest</code>. The most useful is <code>--headed</code>:</p>
<pre><code>pytest --headed
</code></pre>
<p>This runs the tests in "headed" mode - which means a visible browser window pops up during the tests so you can see what is happening.</p>
<p><code>--browser firefox</code> runs them using Firefox instead of Chromium.</p>
<p><code>--tracing on</code> is really interesting: it generates a trace ZIP file which you can then open using <a href="https://trace.playwright.dev/" rel="nofollow">https://trace.playwright.dev/</a> to explore a detailed trace of the test as it executed.</p>
<p><code>--video on</code> records a video (as a <code>.webm</code> file) of the test. I've not tried it yet, but <code>--video retain-on-failure</code> only keeps that video if the test fails.</p>
<p>Here's <a href="https://playwright.dev/python/docs/test-runners" rel="nofollow">documentation on all of the options</a>.</p>
<h2>
<a id="user-content-running-a-localhost-static-server-during-the-tests" class="anchor" href="#running-a-localhost-static-server-during-the-tests" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running a localhost static server during the tests</h2>
<p>I wanted to run the tests against the most recent version of my code, which consists of an <code>index.html</code> file and a <code>webworker.js</code> file. Because these use web workers they need to be run from an actual localhost web server, so I needed to start one at the beginning of the tests and shut it down at the end.</p>
<p>I wrote about my solution for this in another TIL: <a href="https://til.simonwillison.net/pytest/subprocess-server" rel="nofollow">Start a server in a subprocess during a pytest session</a>.</p>
<h2>
<a id="user-content-my-test-suite-so-far" class="anchor" href="#my-test-suite-so-far" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>My test suite so far</h2>
<p>Here's where <a href="https://github.com/simonw/datasette-lite/blob/daba69708c7a72adad20dce3c534b9a399ef11c8/tests/test_datasette_lite.py">I've got to so far</a>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">playwright</span>.<span class="pl-s1">sync_api</span> <span class="pl-k">import</span> <span class="pl-v">Browser</span>, <span class="pl-v">Page</span>, <span class="pl-s1">expect</span>
<span class="pl-k">from</span> <span class="pl-s1">subprocess</span> <span class="pl-k">import</span> <span class="pl-v">Popen</span>, <span class="pl-v">PIPE</span>
<span class="pl-k">import</span> <span class="pl-s1">pathlib</span>
<span class="pl-k">import</span> <span class="pl-s1">pytest</span>
<span class="pl-k">import</span> <span class="pl-s1">time</span>
<span class="pl-k">from</span> <span class="pl-s1">http</span>.<span class="pl-s1">client</span> <span class="pl-k">import</span> <span class="pl-v">HTTPConnection</span>

<span class="pl-s1">root</span> <span class="pl-c1">=</span> <span class="pl-s1">pathlib</span>.<span class="pl-v">Path</span>(<span class="pl-s1">__file__</span>).<span class="pl-s1">parent</span>.<span class="pl-s1">parent</span>.<span class="pl-en">absolute</span>()


<span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-en">fixture</span>(<span class="pl-s1">scope</span><span class="pl-c1">=</span><span class="pl-s">"module"</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">static_server</span>():
    <span class="pl-s1">process</span> <span class="pl-c1">=</span> <span class="pl-v">Popen</span>(
        [<span class="pl-s">"python"</span>, <span class="pl-s">"-m"</span>, <span class="pl-s">"http.server"</span>, <span class="pl-s">"8123"</span>, <span class="pl-s">"--directory"</span>, <span class="pl-s1">root</span>], <span class="pl-s1">stdout</span><span class="pl-c1">=</span><span class="pl-v">PIPE</span>
    )
    <span class="pl-s1">retries</span> <span class="pl-c1">=</span> <span class="pl-c1">5</span>
    <span class="pl-k">while</span> <span class="pl-s1">retries</span> <span class="pl-c1">&gt;</span> <span class="pl-c1">0</span>:
        <span class="pl-s1">conn</span> <span class="pl-c1">=</span> <span class="pl-v">HTTPConnection</span>(<span class="pl-s">"localhost:8123"</span>)
        <span class="pl-k">try</span>:
            <span class="pl-s1">conn</span>.<span class="pl-en">request</span>(<span class="pl-s">"HEAD"</span>, <span class="pl-s">"/"</span>)
            <span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">conn</span>.<span class="pl-en">getresponse</span>()
            <span class="pl-k">if</span> <span class="pl-s1">response</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>:
                <span class="pl-k">yield</span> <span class="pl-s1">process</span>
                <span class="pl-k">break</span>
        <span class="pl-k">except</span> <span class="pl-v">ConnectionRefusedError</span>:
            <span class="pl-s1">time</span>.<span class="pl-en">sleep</span>(<span class="pl-c1">1</span>)
            <span class="pl-s1">retries</span> <span class="pl-c1">-=</span> <span class="pl-c1">1</span>

    <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">retries</span>:
        <span class="pl-k">raise</span> <span class="pl-v">RuntimeError</span>(<span class="pl-s">"Failed to start http server"</span>)
    <span class="pl-k">else</span>:
        <span class="pl-s1">process</span>.<span class="pl-en">terminate</span>()
        <span class="pl-s1">process</span>.<span class="pl-en">wait</span>()


<span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-en">fixture</span>(<span class="pl-s1">scope</span><span class="pl-c1">=</span><span class="pl-s">"module"</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">dslite</span>(<span class="pl-s1">static_server</span>, <span class="pl-s1">browser</span>: <span class="pl-v">Browser</span>) <span class="pl-c1">-&gt;</span> <span class="pl-v">Page</span>:
    <span class="pl-s1">page</span> <span class="pl-c1">=</span> <span class="pl-s1">browser</span>.<span class="pl-en">new_page</span>()
    <span class="pl-s1">page</span>.<span class="pl-en">goto</span>(<span class="pl-s">"http://localhost:8123/"</span>)
    <span class="pl-s1">loading</span> <span class="pl-c1">=</span> <span class="pl-s1">page</span>.<span class="pl-en">locator</span>(<span class="pl-s">"#loading-indicator"</span>)
    <span class="pl-en">expect</span>(<span class="pl-s1">loading</span>).<span class="pl-en">to_have_css</span>(<span class="pl-s">"display"</span>, <span class="pl-s">"block"</span>)
    <span class="pl-c"># Give it up to 60s to finish loading</span>
    <span class="pl-en">expect</span>(<span class="pl-s1">loading</span>).<span class="pl-en">to_have_css</span>(<span class="pl-s">"display"</span>, <span class="pl-s">"none"</span>, <span class="pl-s1">timeout</span><span class="pl-c1">=</span><span class="pl-c1">60</span> <span class="pl-c1">*</span> <span class="pl-c1">1000</span>)
    <span class="pl-k">return</span> <span class="pl-s1">page</span>


<span class="pl-k">def</span> <span class="pl-en">test_initial_load</span>(<span class="pl-s1">dslite</span>: <span class="pl-v">Page</span>):
    <span class="pl-en">expect</span>(<span class="pl-s1">dslite</span>.<span class="pl-en">locator</span>(<span class="pl-s">"#loading-indicator"</span>)).<span class="pl-en">to_have_css</span>(<span class="pl-s">"display"</span>, <span class="pl-s">"none"</span>)


<span class="pl-k">def</span> <span class="pl-en">test_has_two_databases</span>(<span class="pl-s1">dslite</span>: <span class="pl-v">Page</span>):
    <span class="pl-k">assert</span> [<span class="pl-s1">el</span>.<span class="pl-en">inner_text</span>() <span class="pl-k">for</span> <span class="pl-s1">el</span> <span class="pl-c1">in</span> <span class="pl-s1">dslite</span>.<span class="pl-en">query_selector_all</span>(<span class="pl-s">"h2"</span>)] <span class="pl-c1">==</span> [
        <span class="pl-s">"fixtures"</span>,
        <span class="pl-s">"content"</span>,
    ]


<span class="pl-k">def</span> <span class="pl-en">test_navigate_to_database</span>(<span class="pl-s1">dslite</span>: <span class="pl-v">Page</span>):
    <span class="pl-s1">h2</span> <span class="pl-c1">=</span> <span class="pl-s1">dslite</span>.<span class="pl-en">query_selector</span>(<span class="pl-s">"h2"</span>)
    <span class="pl-k">assert</span> <span class="pl-s1">h2</span>.<span class="pl-en">inner_text</span>() <span class="pl-c1">==</span> <span class="pl-s">"fixtures"</span>
    <span class="pl-s1">h2</span>.<span class="pl-en">query_selector</span>(<span class="pl-s">"a"</span>).<span class="pl-en">click</span>()
    <span class="pl-en">expect</span>(<span class="pl-s1">dslite</span>).<span class="pl-en">to_have_title</span>(<span class="pl-s">"fixtures"</span>)
    <span class="pl-s1">dslite</span>.<span class="pl-en">query_selector</span>(<span class="pl-s">"textarea#sql-editor"</span>).<span class="pl-en">fill</span>(
        <span class="pl-s">"SELECT * FROM no_primary_key limit 1"</span>
    )
    <span class="pl-s1">dslite</span>.<span class="pl-en">query_selector</span>(<span class="pl-s">"input[type=submit]"</span>).<span class="pl-en">click</span>()
    <span class="pl-en">expect</span>(<span class="pl-s1">dslite</span>).<span class="pl-en">to_have_title</span>(<span class="pl-s">"fixtures: SELECT * FROM no_primary_key limit 1"</span>)
    <span class="pl-s1">table</span> <span class="pl-c1">=</span> <span class="pl-s1">dslite</span>.<span class="pl-en">query_selector</span>(<span class="pl-s">"table.rows-and-columns"</span>)
    <span class="pl-s1">table_html</span> <span class="pl-c1">=</span> <span class="pl-s">""</span>.<span class="pl-en">join</span>(<span class="pl-s1">table</span>.<span class="pl-en">inner_html</span>().<span class="pl-en">split</span>())
    <span class="pl-k">assert</span> <span class="pl-s1">table_html</span> <span class="pl-c1">==</span> (
        <span class="pl-s">'&lt;thead&gt;&lt;tr&gt;&lt;thclass="col-content"scope="col"&gt;content&lt;/th&gt;'</span>
        <span class="pl-s">'&lt;thclass="col-a"scope="col"&gt;a&lt;/th&gt;&lt;thclass="col-b"scope="col"&gt;b&lt;/th&gt;'</span>
        <span class="pl-s">'&lt;thclass="col-c"scope="col"&gt;c&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;'</span>
        <span class="pl-s">'&lt;tdclass="col-content"&gt;1&lt;/td&gt;&lt;tdclass="col-a"&gt;a1&lt;/td&gt;'</span>
        <span class="pl-s">'&lt;tdclass="col-b"&gt;b1&lt;/td&gt;&lt;tdclass="col-c"&gt;c1&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;'</span>
    )</pre></div>
<h2>
<a id="user-content-running-it-in-github-actions" class="anchor" href="#running-it-in-github-actions" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running it in GitHub Actions</h2>
<p>Here's the <a href="https://github.com/simonw/datasette-lite/blob/main/.github/workflows/test.yml">GitHub Actions workflow</a> I'm using to run the tests:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Test</span>

<span class="pl-ent">on</span>:
  <span class="pl-ent">push</span>:
  <span class="pl-ent">pull_request</span>:

<span class="pl-ent">permissions</span>:
  <span class="pl-ent">contents</span>: <span class="pl-s">read</span>

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">test</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v3</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Set up Python 3.10</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">actions/setup-python@v3</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">python-version</span>: <span class="pl-s"><span class="pl-pds">"</span>3.10<span class="pl-pds">"</span></span>
        <span class="pl-ent">cache</span>: <span class="pl-s"><span class="pl-pds">'</span>pip<span class="pl-pds">'</span></span>
        <span class="pl-ent">cache-dependency-path</span>: <span class="pl-s"><span class="pl-pds">'</span>**/dev-requirements.txt<span class="pl-pds">'</span></span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Cache Playwright browsers</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">actions/cache@v3</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">path</span>: <span class="pl-s">~/.cache/ms-playwright/</span>
        <span class="pl-ent">key</span>: <span class="pl-s">${{ runner.os }}-browsers</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Install dependencies</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        pip install -r dev-requirements.txt</span>
<span class="pl-s">        playwright install</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Run test</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        pytest</span></pre></div>
<p><a href="https://raw.githubusercontent.com/simonw/datasette-lite/main/dev-requirements.txt" rel="nofollow">dev-requirements.txt</a> contains this:</p>
<pre><code>pytest-playwright==0.3.0
playwright==1.24.0
</code></pre>


path: fly_redbean-on-fly.md
topic: fly
title: Deploying a redbean app to Fly
url: https://github.com/simonw/til/blob/main/fly/redbean-on-fly.md
created: 2022-07-24T17:46:24-07:00
created_utc: 2022-07-25T00:46:24+00:00
updated: 2022-07-24T17:46:24-07:00
updated_utc: 2022-07-25T00:46:24+00:00
shot_hash: 89bed785f237c5f289ee7400beef1c18
slug: redbean-on-fly
body:
[redbean](https://redbean.dev/) is a fascinating project - it provides a web server in a self-contained executable which you can add assets (or dynamic Lua code) to just by zipping them into the same binary package.

I decided to try running it on [Fly](https://fly.io). Here's the recipe that worked for me.

## The Dockerfile

I copied this Dockerfile, unmodified, from https://github.com/kissgyorgy/redbean-docker/blob/master/Dockerfile-multistage by György Kiss:

```dockerfile
FROM alpine:latest as build

ARG DOWNLOAD_FILENAME=redbean-original-2.0.8.com

RUN apk add --update zip bash
RUN wget https://redbean.dev/${DOWNLOAD_FILENAME} -O redbean.com
RUN chmod +x redbean.com

# normalize the binary to ELF
RUN sh /redbean.com --assimilate

# Add your files here
COPY assets /assets
WORKDIR /assets
RUN zip -r /redbean.com *

# just for debugging purposes
RUN ls -la /redbean.com
RUN zip -sf /redbean.com


FROM scratch

COPY --from=build /redbean.com /
CMD ["/redbean.com", "-vv", "-p", "80"]
```

It uses a multi-stage build to download redbean, copy in the contents of your `assets/` folder, zip those back up and then create a TINY container from `scratch` that copies in just that executable.

I made an `assets/` folder with something fun in it (a copy of my [Datasette Lite](https://github.com/simonw/datasette-lite) app) like this:
```
mkdir assets
cd assets
wget https://lite.datasette.io/index.html
wget https://lite.datasette.io/webworker.js
```
## Deploying to Fly

First I needed to create a new application. I ran this:

    fly apps create redbean-on-fly

Then I needed a `fly.toml` file. I created this one (copied from a previous example, but I updated the internal server port and the name):

```toml
app = "redbean-on-fly"

kill_signal = "SIGINT"
kill_timeout = 5

[[services]]
  internal_port = 80
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["http"]
    port = "80"

  [[services.ports]]
    handlers = ["tls", "http"]
    port = "443"

  [[services.tcp_checks]]
    interval = 10000
    timeout = 2000
    grace_period = "10s"
```

Finally, I deployed the app by running this in the same directory as `fly.toml`:

    fly deploy

This uploaded the `Dockerfile` and `assets/` folder to Fly, ran the build there, then deployed the result.

html:
<p><a href="https://redbean.dev/" rel="nofollow">redbean</a> is a fascinating project - it provides a web server in a self-contained executable which you can add assets (or dynamic Lua code) to just by zipping them into the same binary package.</p>
<p>I decided to try running it on <a href="https://fly.io" rel="nofollow">Fly</a>. Here's the recipe that worked for me.</p>
<h2>
<a id="user-content-the-dockerfile" class="anchor" href="#the-dockerfile" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>The Dockerfile</h2>
<p>I copied this Dockerfile, unmodified, from <a href="https://github.com/kissgyorgy/redbean-docker/blob/master/Dockerfile-multistage">https://github.com/kissgyorgy/redbean-docker/blob/master/Dockerfile-multistage</a> by György Kiss:</p>
<div class="highlight highlight-source-dockerfile"><pre><span class="pl-k">FROM</span> alpine:latest as build

<span class="pl-k">ARG</span> DOWNLOAD_FILENAME=redbean-original-2.0.8.com

<span class="pl-k">RUN</span> apk add --update zip bash
<span class="pl-k">RUN</span> wget https://redbean.dev/${DOWNLOAD_FILENAME} -O redbean.com
<span class="pl-k">RUN</span> chmod +x redbean.com

<span class="pl-c"><span class="pl-c">#</span> normalize the binary to ELF</span>
<span class="pl-k">RUN</span> sh /redbean.com --assimilate

<span class="pl-c"><span class="pl-c">#</span> Add your files here</span>
<span class="pl-k">COPY</span> assets /assets
<span class="pl-k">WORKDIR</span> /assets
<span class="pl-k">RUN</span> zip -r /redbean.com *

<span class="pl-c"><span class="pl-c">#</span> just for debugging purposes</span>
<span class="pl-k">RUN</span> ls -la /redbean.com
<span class="pl-k">RUN</span> zip -sf /redbean.com


<span class="pl-k">FROM</span> scratch

<span class="pl-k">COPY</span> --from=build /redbean.com /
<span class="pl-k">CMD</span> [<span class="pl-s">"/redbean.com"</span>, <span class="pl-s">"-vv"</span>, <span class="pl-s">"-p"</span>, <span class="pl-s">"80"</span>]</pre></div>
<p>It uses a multi-stage build to download redbean, copy in the contents of your <code>assets/</code> folder, zip those back up and then create a TINY container from <code>scratch</code> that copies in just that executable.</p>
<p>I made an <code>assets/</code> folder with something fun in it (a copy of my <a href="https://github.com/simonw/datasette-lite">Datasette Lite</a> app) like this:</p>
<pre><code>mkdir assets
cd assets
wget https://lite.datasette.io/index.html
wget https://lite.datasette.io/webworker.js
</code></pre>
<h2>
<a id="user-content-deploying-to-fly" class="anchor" href="#deploying-to-fly" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Deploying to Fly</h2>
<p>First I needed to create a new application. I ran this:</p>
<pre><code>fly apps create redbean-on-fly
</code></pre>
<p>Then I needed a <code>fly.toml</code> file. I created this one (copied from a previous example, but I updated the internal server port and the name):</p>
<div class="highlight highlight-source-toml"><pre><span class="pl-smi">app</span> = <span class="pl-s"><span class="pl-pds">"</span>redbean-on-fly<span class="pl-pds">"</span></span>

<span class="pl-smi">kill_signal</span> = <span class="pl-s"><span class="pl-pds">"</span>SIGINT<span class="pl-pds">"</span></span>
<span class="pl-smi">kill_timeout</span> = <span class="pl-c1">5</span>

[[<span class="pl-en">services</span>]]
  <span class="pl-smi">internal_port</span> = <span class="pl-c1">80</span>
  <span class="pl-smi">protocol</span> = <span class="pl-s"><span class="pl-pds">"</span>tcp<span class="pl-pds">"</span></span>

  [<span class="pl-en">services</span>.<span class="pl-en">concurrency</span>]
    <span class="pl-smi">hard_limit</span> = <span class="pl-c1">25</span>
    <span class="pl-smi">soft_limit</span> = <span class="pl-c1">20</span>

  [[<span class="pl-en">services</span>.<span class="pl-en">ports</span>]]
    <span class="pl-smi">handlers</span> = [<span class="pl-s"><span class="pl-pds">"</span>http<span class="pl-pds">"</span></span>]
    <span class="pl-smi">port</span> = <span class="pl-s"><span class="pl-pds">"</span>80<span class="pl-pds">"</span></span>

  [[<span class="pl-en">services</span>.<span class="pl-en">ports</span>]]
    <span class="pl-smi">handlers</span> = [<span class="pl-s"><span class="pl-pds">"</span>tls<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>http<span class="pl-pds">"</span></span>]
    <span class="pl-smi">port</span> = <span class="pl-s"><span class="pl-pds">"</span>443<span class="pl-pds">"</span></span>

  [[<span class="pl-en">services</span>.<span class="pl-en">tcp_checks</span>]]
    <span class="pl-smi">interval</span> = <span class="pl-c1">10000</span>
    <span class="pl-smi">timeout</span> = <span class="pl-c1">2000</span>
    <span class="pl-smi">grace_period</span> = <span class="pl-s"><span class="pl-pds">"</span>10s<span class="pl-pds">"</span></span></pre></div>
<p>Finally, I deployed the app by running this in the same directory as <code>fly.toml</code>:</p>
<pre><code>fly deploy
</code></pre>
<p>This uploaded the <code>Dockerfile</code> and <code>assets/</code> folder to Fly, ran the build there, then deployed the result.</p>


path: docker_test-fedora-in-docker.md
topic: docker
title: Testing things in Fedora using Docker
url: https://github.com/simonw/til/blob/main/docker/test-fedora-in-docker.md
created: 2022-07-27T15:41:43-07:00
created_utc: 2022-07-27T22:41:43+00:00
updated: 2022-07-27T15:41:43-07:00
updated_utc: 2022-07-27T22:41:43+00:00
shot_hash: 5a2832241e5fe9b2ff76ad09099e73ae
slug: test-fedora-in-docker
body:
I got [a report](https://twitter.com/peterjanes/status/1552407491819884544) of a bug with my [s3-ocr tool](https://simonwillison.net/2022/Jun/30/s3-ocr/) running on Fedora.

I attempted to replicate the bug in a Fedora container using Docker, by running this command:

```
docker run -it fedora:latest /bin/bash
```
This downloaded [the official image](https://hub.docker.com/_/fedora) and dropped me into a Bash shell.

It turns out Fedora won't let you run `pip install` with its default Python 3 without first creating a virtual environment:

```
[root@d1146e0061d1 /]# python3 -m pip install s3-ocr
/usr/bin/python3: No module named pip
[root@d1146e0061d1 /]# python3 -m venv project_venv
[root@d1146e0061d1 /]# source project_venv/bin/activate
(project_venv) [root@d1146e0061d1 /]# python -m pip install s3-ocr
Collecting s3-ocr
  Downloading s3_ocr-0.5-py3-none-any.whl (14 kB)
Collecting sqlite-utils
  ...
```
Having done that I could test out my `s3-ocr` command like so:

```
(project_venv) [root@d1146e0061d1 /]# s3-ocr start --help
Usage: s3-ocr start [OPTIONS] BUCKET [KEYS]...

  Start OCR tasks for PDF files in an S3 bucket

      s3-ocr start name-of-bucket path/to/one.pdf path/to/two.pdf
  ...
```

html:
<p>I got <a href="https://twitter.com/peterjanes/status/1552407491819884544" rel="nofollow">a report</a> of a bug with my <a href="https://simonwillison.net/2022/Jun/30/s3-ocr/" rel="nofollow">s3-ocr tool</a> running on Fedora.</p>
<p>I attempted to replicate the bug in a Fedora container using Docker, by running this command:</p>
<pre><code>docker run -it fedora:latest /bin/bash
</code></pre>
<p>This downloaded <a href="https://hub.docker.com/_/fedora" rel="nofollow">the official image</a> and dropped me into a Bash shell.</p>
<p>It turns out Fedora won't let you run <code>pip install</code> with its default Python 3 without first creating a virtual environment:</p>
<pre><code>[root@d1146e0061d1 /]# python3 -m pip install s3-ocr
/usr/bin/python3: No module named pip
[root@d1146e0061d1 /]# python3 -m venv project_venv
[root@d1146e0061d1 /]# source project_venv/bin/activate
(project_venv) [root@d1146e0061d1 /]# python -m pip install s3-ocr
Collecting s3-ocr
  Downloading s3_ocr-0.5-py3-none-any.whl (14 kB)
Collecting sqlite-utils
  ...
</code></pre>
<p>Having done that I could test out my <code>s3-ocr</code> command like so:</p>
<pre><code>(project_venv) [root@d1146e0061d1 /]# s3-ocr start --help
Usage: s3-ocr start [OPTIONS] BUCKET [KEYS]...

  Start OCR tasks for PDF files in an S3 bucket

      s3-ocr start name-of-bucket path/to/one.pdf path/to/two.pdf
  ...
</code></pre>


path: python_struct-endianness.md
topic: python
title: struct endianness in Python
url: https://github.com/simonw/til/blob/main/python/struct-endianness.md
created: 2022-07-28T08:48:00-07:00
created_utc: 2022-07-28T15:48:00+00:00
updated: 2022-07-30T11:07:51-07:00
updated_utc: 2022-07-30T18:07:51+00:00
shot_hash: 476bf6166105d7cedf06d8715bdda098
slug: struct-endianness
body:
TIL the Python standard library [struct](https://docs.python.org/3/library/struct.html) module defaults to interpreting binary strings using the endianness of your machine.

Which means that this code:

```python
def decode_matchinfo(buf): 
    # buf is a bytestring of unsigned integers, each 4 bytes long 
    return struct.unpack("I" * (len(buf) // 4), buf) 
```
Behaves differently on big-endian v.s. little-endian systems.

I found this out thanks to [this bug report](https://github.com/simonw/sqlite-fts4/issues/6) against my sqlite-fts4 library.

My `decode_matchinfo()` function runs against a binary data structure returned by SQLite - more details on that in [Exploring search relevance algorithms with SQLite](https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/).

SQLite doesn't change the binary format depending on the endianness of the system, which means that my function here works correctly on little-endian but does the wrong thing on big-endian systems:

**Update:** I was entirely wrong about this. SQLite DOES change the format based on the endianness of the system. My bug fix was incorrect - see [this issue comment](https://github.com/simonw/sqlite-fts4/issues/6#issuecomment-1200053863) for details.

On little-endian systems:

```pycon
>>> buf = b'\x01\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00'
>>> decode_matchinfo(buf)
(1, 2, 2, 2)
```
But on big-endian systems:
```pycon
>>> buf = b'\x01\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00'
>>> decode_matchinfo(buf)
(16777216, 33554432, 33554432, 33554432)
```
The fix is to add a first character to that format string specifying the endianness that should be used, see [Byte Order, Size, and Alignment](https://docs.python.org/3/library/struct.html#struct-alignment) in the Python documentation.

```pycon
>>> struct.unpack("<IIII", buf)
(1, 2, 2, 2)
>>> struct.unpack(">IIII", buf)
(16777216, 33554432, 33554432, 33554432)
```
So [the fix](https://github.com/simonw/sqlite-fts4/commit/ed6ea76a727243e9b0bff4fe7cf7022fcd1ec834) for my bug was to rewrite the function to look like this:
```python
def decode_matchinfo(buf):
    # buf is a bytestring of unsigned integers, each 4 bytes long
    return struct.unpack("<" + ("I" * (len(buf) // 4)), buf)
```
## Bonus: How to tell which endianness your system has

Turns out Python can tell you if you are big-endian or little-endian like this:

```pycon
>>> from sys import byteorder
>>> byteorder
'little'
```

html:
<p>TIL the Python standard library <a href="https://docs.python.org/3/library/struct.html" rel="nofollow">struct</a> module defaults to interpreting binary strings using the endianness of your machine.</p>
<p>Which means that this code:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">decode_matchinfo</span>(<span class="pl-s1">buf</span>): 
    <span class="pl-c"># buf is a bytestring of unsigned integers, each 4 bytes long </span>
    <span class="pl-k">return</span> <span class="pl-s1">struct</span>.<span class="pl-en">unpack</span>(<span class="pl-s">"I"</span> <span class="pl-c1">*</span> (<span class="pl-en">len</span>(<span class="pl-s1">buf</span>) <span class="pl-c1">//</span> <span class="pl-c1">4</span>), <span class="pl-s1">buf</span>) </pre></div>
<p>Behaves differently on big-endian v.s. little-endian systems.</p>
<p>I found this out thanks to <a href="https://github.com/simonw/sqlite-fts4/issues/6">this bug report</a> against my sqlite-fts4 library.</p>
<p>My <code>decode_matchinfo()</code> function runs against a binary data structure returned by SQLite - more details on that in <a href="https://simonwillison.net/2019/Jan/7/exploring-search-relevance-algorithms-sqlite/" rel="nofollow">Exploring search relevance algorithms with SQLite</a>.</p>
<p>SQLite doesn't change the binary format depending on the endianness of the system, which means that my function here works correctly on little-endian but does the wrong thing on big-endian systems:</p>
<p><strong>Update:</strong> I was entirely wrong about this. SQLite DOES change the format based on the endianness of the system. My bug fix was incorrect - see <a href="https://github.com/simonw/sqlite-fts4/issues/6#issuecomment-1200053863">this issue comment</a> for details.</p>
<p>On little-endian systems:</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; buf <span class="pl-k">=</span> <span class="pl-s"><span class="pl-k">b</span><span class="pl-pds">'</span><span class="pl-cce">\x01\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00</span><span class="pl-pds">'</span></span>
&gt;&gt;&gt; decode_matchinfo(buf)
(1, 2, 2, 2)</pre></div>
<p>But on big-endian systems:</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; buf <span class="pl-k">=</span> <span class="pl-s"><span class="pl-k">b</span><span class="pl-pds">'</span><span class="pl-cce">\x01\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00</span><span class="pl-pds">'</span></span>
&gt;&gt;&gt; decode_matchinfo(buf)
(16777216, 33554432, 33554432, 33554432)</pre></div>
<p>The fix is to add a first character to that format string specifying the endianness that should be used, see <a href="https://docs.python.org/3/library/struct.html#struct-alignment" rel="nofollow">Byte Order, Size, and Alignment</a> in the Python documentation.</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; struct.unpack(<span class="pl-s"><span class="pl-pds">"</span>&lt;IIII<span class="pl-pds">"</span></span>, buf)
(1, 2, 2, 2)
&gt;&gt;&gt; struct.unpack(<span class="pl-s"><span class="pl-pds">"</span>&gt;IIII<span class="pl-pds">"</span></span>, buf)
(16777216, 33554432, 33554432, 33554432)</pre></div>
<p>So <a href="https://github.com/simonw/sqlite-fts4/commit/ed6ea76a727243e9b0bff4fe7cf7022fcd1ec834">the fix</a> for my bug was to rewrite the function to look like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">decode_matchinfo</span>(<span class="pl-s1">buf</span>):
    <span class="pl-c"># buf is a bytestring of unsigned integers, each 4 bytes long</span>
    <span class="pl-k">return</span> <span class="pl-s1">struct</span>.<span class="pl-en">unpack</span>(<span class="pl-s">"&lt;"</span> <span class="pl-c1">+</span> (<span class="pl-s">"I"</span> <span class="pl-c1">*</span> (<span class="pl-en">len</span>(<span class="pl-s1">buf</span>) <span class="pl-c1">//</span> <span class="pl-c1">4</span>)), <span class="pl-s1">buf</span>)</pre></div>
<h2>
<a id="user-content-bonus-how-to-tell-which-endianness-your-system-has" class="anchor" href="#bonus-how-to-tell-which-endianness-your-system-has" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Bonus: How to tell which endianness your system has</h2>
<p>Turns out Python can tell you if you are big-endian or little-endian like this:</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; <span class="pl-k">from</span> sys <span class="pl-k">import</span> byteorder
&gt;&gt;&gt; byteorder
'little'</pre></div>


path: github_migrate-github-wiki.md
topic: github
title: Migrating a GitHub wiki from one repository to another
url: https://github.com/simonw/til/blob/main/github/migrate-github-wiki.md
created: 2022-07-28T13:08:10-07:00
created_utc: 2022-07-28T20:08:10+00:00
updated: 2022-07-28T13:08:10-07:00
updated_utc: 2022-07-28T20:08:10+00:00
shot_hash: 29874b26af6a87a24af53657537870e1
slug: migrate-github-wiki
body:
I figured out how to migrate a [GitHub wiki](https://docs.github.com/en/communities/documenting-your-project-with-wikis/about-wikis) (public or private) from one repository to another while preserving all history.

The trick is that GitHub wikis are just Git repositories. Which means you can clone them, edit them and push them.

This means you can migrate them between their parent repos like so. `myorg/old-repo` is the repo you are moving from, and `myorg/new-repo` is the destination.

    git clone https://github.com/myorg/old-repo.wiki.git
    cd old-repo.wiki
    git remote remove origin
    git remote add origin https://github.com/myorg/new-repo.wiki.git
    git push --set-upstream origin master --force

This will entirely over-write the content and history of the wiki attached to the `new-repo` repository with the content and history from the wiki in `old-repo`.

html:
<p>I figured out how to migrate a <a href="https://docs.github.com/en/communities/documenting-your-project-with-wikis/about-wikis">GitHub wiki</a> (public or private) from one repository to another while preserving all history.</p>
<p>The trick is that GitHub wikis are just Git repositories. Which means you can clone them, edit them and push them.</p>
<p>This means you can migrate them between their parent repos like so. <code>myorg/old-repo</code> is the repo you are moving from, and <code>myorg/new-repo</code> is the destination.</p>
<pre><code>git clone https://github.com/myorg/old-repo.wiki.git
cd old-repo.wiki
git remote remove origin
git remote add origin https://github.com/myorg/new-repo.wiki.git
git push --set-upstream origin master --force
</code></pre>
<p>This will entirely over-write the content and history of the wiki attached to the <code>new-repo</code> repository with the content and history from the wiki in <code>old-repo</code>.</p>


path: docker_emulate-s390x-with-qemu.md
topic: docker
title: Emulating a big-endian s390x with QEMU
url: https://github.com/simonw/til/blob/main/docker/emulate-s390x-with-qemu.md
created: 2022-07-29T19:44:50-07:00
created_utc: 2022-07-30T02:44:50+00:00
updated: 2022-07-29T19:44:50-07:00
updated_utc: 2022-07-30T02:44:50+00:00
shot_hash: 6069203118a27e680e662ef1f051a50c
slug: emulate-s390x-with-qemu
body:
I got [a bug report](https://github.com/simonw/sqlite-fts4/issues/6) concerning my [sqlite-fts4](https://github.com/simonw/sqlite-fts4) project running on PPC64 and s390x architectures.

The s390x is an [IBM mainframe architecture](https://en.wikipedia.org/wiki/Linux_on_IBM_Z), which I found glamorous!

The bug related to those machines being big-endian v.s. my software being tested on little-endian machines. My first attempt at fixing it (see [this TIL](https://til.simonwillison.net/python/struct-endianness)) turned out not to be correct. I really needed a way to test agaist an emulated s390x machine with big-endian byte order.

I figured out how to do that using Docker for Mac and QEMU.

## multiarch/qemu-user-static:register

This is the first command to run. It does something magical to your Docker installation:

    docker run --rm --privileged multiarch/qemu-user-static:register --reset

The [qemu-user-static README](https://github.com/multiarch/qemu-user-static/blob/master/README.md) says:

> `multiarch/qemu-user-static` and `multiarch/qemu-user-static:register` images execute the register script that registers below kind of `/proc/sys/fs/binfmt_misc/qemu-$arch` files for all supported processors except the current one in it when running the container.

It continues:

> The `--reset` option is implemented at the register script that executes find `/proc/sys/fs/binfmt_misc -type f -name 'qemu-*' -exec sh -c 'echo -1 > {}' \;` to remove `binfmt_misc` entry files before register the entry.

I don't understand what this means. But running this command was essential for the next command to work.

## multiarch/ubuntu-core:s390x-focal

Having run that command, the following command drops you into a shell in an emulated s390x machine running Ubuntu Focal:

    docker run -it multiarch/ubuntu-core:s390x-focal /bin/bash

Using `-focal` gives you Python 3.8. I previously tried `s390x-bionic` but that gave me Python 3.6.

You don't actually get Python until you install it, like so:

    apt-get -y update && apt-get -y install python3

This will take a while. I think it's slower because the hardware is being emulated.

Now you can check the Python version and confirm that the byte order is big-endian like this:

```
root@ea63e288ce49:/# python3 --version
Python 3.8.10
root@ea63e288ce49:/# python3 -c 'import sys; print(sys.byteorder)'
big
```

## Doing this in GitHub Actions

I figured out the [following recipe](https://github.com/simonw/playing-with-actions-2/blob/4408f8136b8b37160685e8961742eb11589b3f66/.github/workflows/qemu.yml) for running this in GitHub Actions.

In this example I'm cloning my `sqlite-fts4` repo and running the tests in it as well:

```yaml
name: QEMU to run s390x-focal

on:
  push:
  workflow_dispatch:

jobs:
  one:
    runs-on: ubuntu-latest
    steps:
    - name: Setup multiarch/qemu-user-static
      run: |
        docker run --rm --privileged multiarch/qemu-user-static:register --reset
    - name: ubuntu-core:s390x-focal
      uses: docker://multiarch/ubuntu-core:s390x-focal
      with:
        args: >
          bash -c
          "uname -a &&
          lscpu | grep Endian &&
          apt-get -y update &&
          apt-get -y install python3 git python3.8-venv &&
          python3 --version &&
          python3 -c 'import sys; print(sys.byteorder)' &&
          git clone https://github.com/simonw/sqlite-fts4 &&
          cd sqlite-fts4 &&
          python3 -m venv venv &&
          source venv/bin/activate &&
          pip install -e '.[test]' &&
          pytest
          "
```

html:
<p>I got <a href="https://github.com/simonw/sqlite-fts4/issues/6">a bug report</a> concerning my <a href="https://github.com/simonw/sqlite-fts4">sqlite-fts4</a> project running on PPC64 and s390x architectures.</p>
<p>The s390x is an <a href="https://en.wikipedia.org/wiki/Linux_on_IBM_Z" rel="nofollow">IBM mainframe architecture</a>, which I found glamorous!</p>
<p>The bug related to those machines being big-endian v.s. my software being tested on little-endian machines. My first attempt at fixing it (see <a href="https://til.simonwillison.net/python/struct-endianness" rel="nofollow">this TIL</a>) turned out not to be correct. I really needed a way to test agaist an emulated s390x machine with big-endian byte order.</p>
<p>I figured out how to do that using Docker for Mac and QEMU.</p>
<h2>
<a id="user-content-multiarchqemu-user-staticregister" class="anchor" href="#multiarchqemu-user-staticregister" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>multiarch/qemu-user-static:register</h2>
<p>This is the first command to run. It does something magical to your Docker installation:</p>
<pre><code>docker run --rm --privileged multiarch/qemu-user-static:register --reset
</code></pre>
<p>The <a href="https://github.com/multiarch/qemu-user-static/blob/master/README.md">qemu-user-static README</a> says:</p>
<blockquote>
<p><code>multiarch/qemu-user-static</code> and <code>multiarch/qemu-user-static:register</code> images execute the register script that registers below kind of <code>/proc/sys/fs/binfmt_misc/qemu-$arch</code> files for all supported processors except the current one in it when running the container.</p>
</blockquote>
<p>It continues:</p>
<blockquote>
<p>The <code>--reset</code> option is implemented at the register script that executes find <code>/proc/sys/fs/binfmt_misc -type f -name 'qemu-*' -exec sh -c 'echo -1 &gt; {}' \;</code> to remove <code>binfmt_misc</code> entry files before register the entry.</p>
</blockquote>
<p>I don't understand what this means. But running this command was essential for the next command to work.</p>
<h2>
<a id="user-content-multiarchubuntu-cores390x-focal" class="anchor" href="#multiarchubuntu-cores390x-focal" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>multiarch/ubuntu-core:s390x-focal</h2>
<p>Having run that command, the following command drops you into a shell in an emulated s390x machine running Ubuntu Focal:</p>
<pre><code>docker run -it multiarch/ubuntu-core:s390x-focal /bin/bash
</code></pre>
<p>Using <code>-focal</code> gives you Python 3.8. I previously tried <code>s390x-bionic</code> but that gave me Python 3.6.</p>
<p>You don't actually get Python until you install it, like so:</p>
<pre><code>apt-get -y update &amp;&amp; apt-get -y install python3
</code></pre>
<p>This will take a while. I think it's slower because the hardware is being emulated.</p>
<p>Now you can check the Python version and confirm that the byte order is big-endian like this:</p>
<pre><code>root@ea63e288ce49:/# python3 --version
Python 3.8.10
root@ea63e288ce49:/# python3 -c 'import sys; print(sys.byteorder)'
big
</code></pre>
<h2>
<a id="user-content-doing-this-in-github-actions" class="anchor" href="#doing-this-in-github-actions" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Doing this in GitHub Actions</h2>
<p>I figured out the <a href="https://github.com/simonw/playing-with-actions-2/blob/4408f8136b8b37160685e8961742eb11589b3f66/.github/workflows/qemu.yml">following recipe</a> for running this in GitHub Actions.</p>
<p>In this example I'm cloning my <code>sqlite-fts4</code> repo and running the tests in it as well:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">QEMU to run s390x-focal</span>

<span class="pl-ent">on</span>:
  <span class="pl-ent">push</span>:
  <span class="pl-ent">workflow_dispatch</span>:

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">one</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">name</span>: <span class="pl-s">Setup multiarch/qemu-user-static</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        docker run --rm --privileged multiarch/qemu-user-static:register --reset</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">ubuntu-core:s390x-focal</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">docker://multiarch/ubuntu-core:s390x-focal</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">args</span>: <span class="pl-s">&gt;</span>
<span class="pl-s">          bash -c</span>
<span class="pl-s">          "uname -a &amp;&amp;</span>
<span class="pl-s">          lscpu | grep Endian &amp;&amp;</span>
<span class="pl-s">          apt-get -y update &amp;&amp;</span>
<span class="pl-s">          apt-get -y install python3 git python3.8-venv &amp;&amp;</span>
<span class="pl-s">          python3 --version &amp;&amp;</span>
<span class="pl-s">          python3 -c 'import sys; print(sys.byteorder)' &amp;&amp;</span>
<span class="pl-s">          git clone https://github.com/simonw/sqlite-fts4 &amp;&amp;</span>
<span class="pl-s">          cd sqlite-fts4 &amp;&amp;</span>
<span class="pl-s">          python3 -m venv venv &amp;&amp;</span>
<span class="pl-s">          source venv/bin/activate &amp;&amp;</span>
<span class="pl-s">          pip install -e '.[test]' &amp;&amp;</span>
<span class="pl-s">          pytest</span>
<span class="pl-s">          "</span></pre></div>


path: sqlite_related-content.md
topic: sqlite
title: Related content with SQLite FTS and a Datasette template function
url: https://github.com/simonw/til/blob/main/sqlite/related-content.md
created: 2022-07-31T17:53:23-07:00
created_utc: 2022-08-01T00:53:23+00:00
updated: 2022-07-31T19:05:19-07:00
updated_utc: 2022-08-01T02:05:19+00:00
shot_hash: 8e79f62047ef43ed821e41c7c35bfa5e
slug: related-content
body:
Today I added "related TILs" to this TIL website - so each TIL now shows five related TILs at the bottom of the page.

I'm generating the related content using a SQLite full-text search query.

## The related content query

I take the title and body text of an entry, strip out any non-alphanumeric characters, de-dupe the words, and then combine them to form a big OR search query.

I built an initial prototype of this using [an Observable notebook](https://observablehq.com/@simonw/turn-pasted-text-into-a-big-sqlite-fts-or-query) that generated these queries from pasted text. See also [my research issue](https://github.com/simonw/til/issues/50).

Here's a simple version of that query, using some example words pulled from [this entry](https://til.simonwillison.net/github-actions/postgresq-service-container). It executes against `til_fts` which is a SQLite FTS table [created](https://github.com/simonw/til/blob/cb62b8ab4c7b26ec7a895adea7f2d405b48686ba/build_database.py#L98-L100) using the `.enable_fts()` method of the [sqlite-utils Python library](https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-fts).
```sql
select title, rank from til_fts where til_fts match '
  i OR wanted OR to OR run OR some OR django OR tests OR using OR
  pytestdjango OR and OR with OR configured OR pick OR up OR the
  OR databaseurl OR environment OR variable OR via OR djdatabaseurl
  OR against OR a OR postgresql OR server OR running OR in OR
  github OR actions OR it OR took OR while OR figure OR out OR
  right OR pattern OR trick OR was OR define OR postgres OR service'
order by rank limit 5
```

Here are the results from [that query](https://til.simonwillison.net/tils?sql=select+title%2C+rank+from+til_fts+where+til_fts+match+%27%0D%0A++i+OR+wanted+OR+to+OR+run+OR+some+OR+django+OR+tests+OR+using+OR%0D%0A++pytestdjango+OR+and+OR+with+OR+configured+OR+pick+OR+up+OR+the%0D%0A++OR+databaseurl+OR+environment+OR+variable+OR+via+OR+djdatabaseurl%0D%0A++OR+against+OR+a+OR+postgresql+OR+server+OR+running+OR+in+OR%0D%0A++github+OR+actions+OR+it+OR+took+OR+while+OR+figure+OR+out+OR%0D%0A++right+OR+pattern+OR+trick+OR+was+OR+define+OR+postgres+OR+service%27%0D%0Aorder+by+rank+limit+5). Unsurprisingly the entry itself shows up first, but the other items look relevant enough to me:

title | rank
-- | --
Running tests against PostgreSQL in a service container | -61.04335068286244
Talking to a PostgreSQL service container from inside a Docker container | -37.54518907167069
Allowing a container in Docker Desktop for Mac to talk to a PostgreSQL server on the host machine | -29.712660785491842
Installing different PostgreSQL server versions in GitHub Actions | -27.738649522063493
Docker Compose for Django development | -25.027695483498224

The key trick here is the `order by rank` - SQLite FTS has a robust scoring mechanism built in, and it turns out it's good enough that simply feeding in all of the words from the entry and sorting by that rank provides good-enough related content results on its own.

## Improving the query

In order to display related content I need to do two additional things: I need to filter out the current entry, and I need to join against the `til` table in order to retrieve the information I want to display.

Here's the finished query:

```sql
select
  til.topic, til.slug, til.title, til.created
from
  til
  join til_fts on til.rowid = til_fts.rowid
where
  til_fts match :words
  and not (
    til.slug = :slug
    and til.topic = :topic
  )
order by
  til_fts.rank
limit
  5
```
And [an example of it running](https://til.simonwillison.net/tils?sql=select%0D%0A++til.topic%2C+til.slug%2C+til.title%2C+til.created%0D%0Afrom%0D%0A++til%0D%0A++join+til_fts+on+til.rowid+%3D+til_fts.rowid%0D%0Awhere%0D%0A++til_fts+match+%3Awords%0D%0A++and+not+(%0D%0A++++til.slug+%3D+%3Aslug%0D%0A++++and+til.topic+%3D+%3Atopic%0D%0A++)%0D%0Aorder+by%0D%0A++til_fts.rank%0D%0Alimit%0D%0A++5&words=i+OR+wanted+OR+to+OR+run+OR+some+OR+django+OR+tests+OR+using+OR+++pytestdjango+OR+and+OR+with+OR+configured+OR+pick+OR+up+OR+the+++OR+databaseurl+OR+environment+OR+variable+OR+via+OR+djdatabaseurl+++OR+against+OR+a+OR+postgresql+OR+server+OR+running+OR+in+OR+++github+OR+actions+OR+it+OR+took+OR+while+OR+figure+OR+out+OR+++right+OR+pattern+OR+trick+OR+was+OR+define+OR+postgres+OR+service&slug=postgresq-service-container&topic=github-actions), which returns the following:

topic | slug | title | created
-- | -- | -- | --
github-actions | service-containers-docker | Talking to a PostgreSQL service container from inside a Docker container | 2020-09-18T11:43:19-07:00
docker | docker-for-mac-container-to-postgresql-on-host | Allowing a container in Docker Desktop for Mac to talk to a PostgreSQL server on the host machine | 2022-03-31T22:48:17-07:00
github-actions | different-postgresql-versions | Installing different PostgreSQL server versions in GitHub Actions | 2021-07-05T17:43:13-07:00
docker | docker-compose-for-django-development | Docker Compose for Django development | 2021-05-24T22:08:23-07:00
django | just-with-django | Using just with Django | 2022-06-06T14:24:37-07:00

## Implementing this as a Datasette template function

I decided to implement this using a custom Jinja template function, added using a Datasette [one-off plugin](https://docs.datasette.io/en/stable/writing_plugins.html#writing-one-off-plugins).

The custom function is called `related_tils()`, and is called from the template [like this](https://github.com/simonw/til/blob/cb62b8ab4c7b26ec7a895adea7f2d405b48686ba/templates/pages/%7Btopic%7D/%7Bslug%7D.html#L37-L45):

```html+jinja
{% set related = related_tils(til) %}
{% if related %}
  <h3>Related</h3>
  <ul class="related">
    {% for til in related %}
    <li><span class="topic">{{ til.topic }}</span> <a href="/{{ til.topic }}/{{ til.slug }}">{{ til.title }}</a> - {{ til.created[:10] }}</li>
    {% endfor %}
  </ul>
{% endif %}
```
Here's the implementation of that custom template function. It's registered using the `extra_template_vars` Datasette plugin hook (described [here](https://docs.datasette.io/en/stable/plugin_hooks.html#extra-template-vars-template-database-table-columns-view-name-request-datasette)).

The function itself is an inner `async def` function, because it needs to be able to use `await` in order to execute its SQL query.

Jinja automatically awaits this kind of function when the template is rendered (see `enable_async=True` in [the Jinja documentation](https://jinja.palletsprojects.com/en/3.0.x/api/?highlight=enable_async#jinja2.Environment)).

So this function takes a TIL entry, creates a de-duped list of words from the title and body, uses that to construct and execute the SQL query and returns the results:

```python
from datasette import hookimpl
import re

non_alphanumeric = re.compile(r"[^a-zA-Z0-9\s]")
multi_spaces = re.compile(r"\s+")


@hookimpl
def extra_template_vars(request, datasette):
    async def related_tils(til):
        text = til["title"] + " " + til["body"]
        text = non_alphanumeric.sub(" ", text)
        text = multi_spaces.sub(" ", text)
        words = list(set(text.lower().strip().split()))
        sql = """
        select
          til.topic, til.slug, til.title, til.created
        from
          til
          join til_fts on til.rowid = til_fts.rowid
        where
          til_fts match :words
          and not (
            til.slug = :slug
            and til.topic = :topic
          )
        order by
          til_fts.rank
        limit
          5
        """
        result = await datasette.get_database().execute(
            sql,
            {"words": " OR ".join(words), "slug": til["slug"], "topic": til["topic"]},
        )
        return result.rows

    return {
        "related_tils": related_tils,
    }
```
Here's [the full code](https://github.com/simonw/til/blob/cb62b8ab4c7b26ec7a895adea7f2d405b48686ba/plugins/template_vars.py) for that plugin, which also registers some other template functions.

html:
<p>Today I added "related TILs" to this TIL website - so each TIL now shows five related TILs at the bottom of the page.</p>
<p>I'm generating the related content using a SQLite full-text search query.</p>
<h2>
<a id="user-content-the-related-content-query" class="anchor" href="#the-related-content-query" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>The related content query</h2>
<p>I take the title and body text of an entry, strip out any non-alphanumeric characters, de-dupe the words, and then combine them to form a big OR search query.</p>
<p>I built an initial prototype of this using <a href="https://observablehq.com/@simonw/turn-pasted-text-into-a-big-sqlite-fts-or-query" rel="nofollow">an Observable notebook</a> that generated these queries from pasted text. See also <a href="https://github.com/simonw/til/issues/50">my research issue</a>.</p>
<p>Here's a simple version of that query, using some example words pulled from <a href="https://til.simonwillison.net/github-actions/postgresq-service-container" rel="nofollow">this entry</a>. It executes against <code>til_fts</code> which is a SQLite FTS table <a href="https://github.com/simonw/til/blob/cb62b8ab4c7b26ec7a895adea7f2d405b48686ba/build_database.py#L98-L100">created</a> using the <code>.enable_fts()</code> method of the <a href="https://sqlite-utils.datasette.io/en/stable/python-api.html#python-api-fts" rel="nofollow">sqlite-utils Python library</a>.</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> title, rank <span class="pl-k">from</span> til_fts <span class="pl-k">where</span> til_fts match <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s">  i OR wanted OR to OR run OR some OR django OR tests OR using OR</span>
<span class="pl-s">  pytestdjango OR and OR with OR configured OR pick OR up OR the</span>
<span class="pl-s">  OR databaseurl OR environment OR variable OR via OR djdatabaseurl</span>
<span class="pl-s">  OR against OR a OR postgresql OR server OR running OR in OR</span>
<span class="pl-s">  github OR actions OR it OR took OR while OR figure OR out OR</span>
<span class="pl-s">  right OR pattern OR trick OR was OR define OR postgres OR service<span class="pl-pds">'</span></span>
<span class="pl-k">order by</span> rank <span class="pl-k">limit</span> <span class="pl-c1">5</span></pre></div>
<p>Here are the results from <a href="https://til.simonwillison.net/tils?sql=select+title%2C+rank+from+til_fts+where+til_fts+match+%27%0D%0A++i+OR+wanted+OR+to+OR+run+OR+some+OR+django+OR+tests+OR+using+OR%0D%0A++pytestdjango+OR+and+OR+with+OR+configured+OR+pick+OR+up+OR+the%0D%0A++OR+databaseurl+OR+environment+OR+variable+OR+via+OR+djdatabaseurl%0D%0A++OR+against+OR+a+OR+postgresql+OR+server+OR+running+OR+in+OR%0D%0A++github+OR+actions+OR+it+OR+took+OR+while+OR+figure+OR+out+OR%0D%0A++right+OR+pattern+OR+trick+OR+was+OR+define+OR+postgres+OR+service%27%0D%0Aorder+by+rank+limit+5" rel="nofollow">that query</a>. Unsurprisingly the entry itself shows up first, but the other items look relevant enough to me:</p>
<table>
<thead>
<tr>
<th>title</th>
<th>rank</th>
</tr>
</thead>
<tbody>
<tr>
<td>Running tests against PostgreSQL in a service container</td>
<td>-61.04335068286244</td>
</tr>
<tr>
<td>Talking to a PostgreSQL service container from inside a Docker container</td>
<td>-37.54518907167069</td>
</tr>
<tr>
<td>Allowing a container in Docker Desktop for Mac to talk to a PostgreSQL server on the host machine</td>
<td>-29.712660785491842</td>
</tr>
<tr>
<td>Installing different PostgreSQL server versions in GitHub Actions</td>
<td>-27.738649522063493</td>
</tr>
<tr>
<td>Docker Compose for Django development</td>
<td>-25.027695483498224</td>
</tr>
</tbody>
</table>
<p>The key trick here is the <code>order by rank</code> - SQLite FTS has a robust scoring mechanism built in, and it turns out it's good enough that simply feeding in all of the words from the entry and sorting by that rank provides good-enough related content results on its own.</p>
<h2>
<a id="user-content-improving-the-query" class="anchor" href="#improving-the-query" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Improving the query</h2>
<p>In order to display related content I need to do two additional things: I need to filter out the current entry, and I need to join against the <code>til</code> table in order to retrieve the information I want to display.</p>
<p>Here's the finished query:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  <span class="pl-c1">til</span>.<span class="pl-c1">topic</span>, <span class="pl-c1">til</span>.<span class="pl-c1">slug</span>, <span class="pl-c1">til</span>.<span class="pl-c1">title</span>, <span class="pl-c1">til</span>.<span class="pl-c1">created</span>
<span class="pl-k">from</span>
  til
  <span class="pl-k">join</span> til_fts <span class="pl-k">on</span> <span class="pl-c1">til</span>.<span class="pl-c1">rowid</span> <span class="pl-k">=</span> <span class="pl-c1">til_fts</span>.<span class="pl-c1">rowid</span>
<span class="pl-k">where</span>
  til_fts match :words
  <span class="pl-k">and</span> not (
    <span class="pl-c1">til</span>.<span class="pl-c1">slug</span> <span class="pl-k">=</span> :slug
    <span class="pl-k">and</span> <span class="pl-c1">til</span>.<span class="pl-c1">topic</span> <span class="pl-k">=</span> :topic
  )
<span class="pl-k">order by</span>
  <span class="pl-c1">til_fts</span>.<span class="pl-c1">rank</span>
<span class="pl-k">limit</span>
  <span class="pl-c1">5</span></pre></div>
<p>And <a href="https://til.simonwillison.net/tils?sql=select%0D%0A++til.topic%2C+til.slug%2C+til.title%2C+til.created%0D%0Afrom%0D%0A++til%0D%0A++join+til_fts+on+til.rowid+%3D+til_fts.rowid%0D%0Awhere%0D%0A++til_fts+match+%3Awords%0D%0A++and+not+(%0D%0A++++til.slug+%3D+%3Aslug%0D%0A++++and+til.topic+%3D+%3Atopic%0D%0A++)%0D%0Aorder+by%0D%0A++til_fts.rank%0D%0Alimit%0D%0A++5&amp;words=i+OR+wanted+OR+to+OR+run+OR+some+OR+django+OR+tests+OR+using+OR+++pytestdjango+OR+and+OR+with+OR+configured+OR+pick+OR+up+OR+the+++OR+databaseurl+OR+environment+OR+variable+OR+via+OR+djdatabaseurl+++OR+against+OR+a+OR+postgresql+OR+server+OR+running+OR+in+OR+++github+OR+actions+OR+it+OR+took+OR+while+OR+figure+OR+out+OR+++right+OR+pattern+OR+trick+OR+was+OR+define+OR+postgres+OR+service&amp;slug=postgresq-service-container&amp;topic=github-actions" rel="nofollow">an example of it running</a>, which returns the following:</p>
<table>
<thead>
<tr>
<th>topic</th>
<th>slug</th>
<th>title</th>
<th>created</th>
</tr>
</thead>
<tbody>
<tr>
<td>github-actions</td>
<td>service-containers-docker</td>
<td>Talking to a PostgreSQL service container from inside a Docker container</td>
<td>2020-09-18T11:43:19-07:00</td>
</tr>
<tr>
<td>docker</td>
<td>docker-for-mac-container-to-postgresql-on-host</td>
<td>Allowing a container in Docker Desktop for Mac to talk to a PostgreSQL server on the host machine</td>
<td>2022-03-31T22:48:17-07:00</td>
</tr>
<tr>
<td>github-actions</td>
<td>different-postgresql-versions</td>
<td>Installing different PostgreSQL server versions in GitHub Actions</td>
<td>2021-07-05T17:43:13-07:00</td>
</tr>
<tr>
<td>docker</td>
<td>docker-compose-for-django-development</td>
<td>Docker Compose for Django development</td>
<td>2021-05-24T22:08:23-07:00</td>
</tr>
<tr>
<td>django</td>
<td>just-with-django</td>
<td>Using just with Django</td>
<td>2022-06-06T14:24:37-07:00</td>
</tr>
</tbody>
</table>
<h2>
<a id="user-content-implementing-this-as-a-datasette-template-function" class="anchor" href="#implementing-this-as-a-datasette-template-function" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Implementing this as a Datasette template function</h2>
<p>I decided to implement this using a custom Jinja template function, added using a Datasette <a href="https://docs.datasette.io/en/stable/writing_plugins.html#writing-one-off-plugins" rel="nofollow">one-off plugin</a>.</p>
<p>The custom function is called <code>related_tils()</code>, and is called from the template <a href="https://github.com/simonw/til/blob/cb62b8ab4c7b26ec7a895adea7f2d405b48686ba/templates/pages/%7Btopic%7D/%7Bslug%7D.html#L37-L45">like this</a>:</p>
<div class="highlight highlight-text-html-django"><pre><span class="pl-e">{%</span> <span class="pl-s">set</span> <span class="pl-s">related</span> = <span class="pl-s">related_tils</span>(<span class="pl-s">til</span>) <span class="pl-e">%}</span>
<span class="pl-e">{%</span> <span class="pl-k">if</span> <span class="pl-s">related</span> <span class="pl-e">%}</span>
  &lt;<span class="pl-ent">h3</span>&gt;Related&lt;/<span class="pl-ent">h3</span>&gt;
  &lt;<span class="pl-ent">ul</span> <span class="pl-e">class</span>=<span class="pl-s"><span class="pl-pds">"</span>related<span class="pl-pds">"</span></span>&gt;
    <span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">til</span> <span class="pl-k">in</span> <span class="pl-s">related</span> <span class="pl-e">%}</span>
    &lt;<span class="pl-ent">li</span>&gt;&lt;<span class="pl-ent">span</span> <span class="pl-e">class</span>=<span class="pl-s"><span class="pl-pds">"</span>topic<span class="pl-pds">"</span></span>&gt;{{ til.topic }}&lt;/<span class="pl-ent">span</span>&gt; &lt;<span class="pl-ent">a</span> <span class="pl-e">href</span>=<span class="pl-s"><span class="pl-pds">"</span>/{{ til.topic }}/{{ til.slug }}<span class="pl-pds">"</span></span>&gt;{{ til.title }}&lt;/<span class="pl-ent">a</span>&gt; - {{ til.created[:1<span class="pl-smi">0]</span> }}&lt;/<span class="pl-ent">li</span>&gt;
    <span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span>
  &lt;/<span class="pl-ent">ul</span>&gt;
<span class="pl-e">{%</span> <span class="pl-k">endif</span> <span class="pl-e">%}</span></pre></div>
<p>Here's the implementation of that custom template function. It's registered using the <code>extra_template_vars</code> Datasette plugin hook (described <a href="https://docs.datasette.io/en/stable/plugin_hooks.html#extra-template-vars-template-database-table-columns-view-name-request-datasette" rel="nofollow">here</a>).</p>
<p>The function itself is an inner <code>async def</code> function, because it needs to be able to use <code>await</code> in order to execute its SQL query.</p>
<p>Jinja automatically awaits this kind of function when the template is rendered (see <code>enable_async=True</code> in <a href="https://jinja.palletsprojects.com/en/3.0.x/api/?highlight=enable_async#jinja2.Environment" rel="nofollow">the Jinja documentation</a>).</p>
<p>So this function takes a TIL entry, creates a de-duped list of words from the title and body, uses that to construct and execute the SQL query and returns the results:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">datasette</span> <span class="pl-k">import</span> <span class="pl-s1">hookimpl</span>
<span class="pl-k">import</span> <span class="pl-s1">re</span>

<span class="pl-s1">non_alphanumeric</span> <span class="pl-c1">=</span> <span class="pl-s1">re</span>.<span class="pl-en">compile</span>(<span class="pl-s">r"[^a-zA-Z0-9\s]"</span>)
<span class="pl-s1">multi_spaces</span> <span class="pl-c1">=</span> <span class="pl-s1">re</span>.<span class="pl-en">compile</span>(<span class="pl-s">r"\s+"</span>)


<span class="pl-en">@<span class="pl-s1">hookimpl</span></span>
<span class="pl-k">def</span> <span class="pl-en">extra_template_vars</span>(<span class="pl-s1">request</span>, <span class="pl-s1">datasette</span>):
    <span class="pl-k">async</span> <span class="pl-k">def</span> <span class="pl-en">related_tils</span>(<span class="pl-s1">til</span>):
        <span class="pl-s1">text</span> <span class="pl-c1">=</span> <span class="pl-s1">til</span>[<span class="pl-s">"title"</span>] <span class="pl-c1">+</span> <span class="pl-s">" "</span> <span class="pl-c1">+</span> <span class="pl-s1">til</span>[<span class="pl-s">"body"</span>]
        <span class="pl-s1">text</span> <span class="pl-c1">=</span> <span class="pl-s1">non_alphanumeric</span>.<span class="pl-en">sub</span>(<span class="pl-s">" "</span>, <span class="pl-s1">text</span>)
        <span class="pl-s1">text</span> <span class="pl-c1">=</span> <span class="pl-s1">multi_spaces</span>.<span class="pl-en">sub</span>(<span class="pl-s">" "</span>, <span class="pl-s1">text</span>)
        <span class="pl-s1">words</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-en">set</span>(<span class="pl-s1">text</span>.<span class="pl-en">lower</span>().<span class="pl-en">strip</span>().<span class="pl-en">split</span>()))
        <span class="pl-s1">sql</span> <span class="pl-c1">=</span> <span class="pl-s">"""</span>
<span class="pl-s">        select</span>
<span class="pl-s">          til.topic, til.slug, til.title, til.created</span>
<span class="pl-s">        from</span>
<span class="pl-s">          til</span>
<span class="pl-s">          join til_fts on til.rowid = til_fts.rowid</span>
<span class="pl-s">        where</span>
<span class="pl-s">          til_fts match :words</span>
<span class="pl-s">          and not (</span>
<span class="pl-s">            til.slug = :slug</span>
<span class="pl-s">            and til.topic = :topic</span>
<span class="pl-s">          )</span>
<span class="pl-s">        order by</span>
<span class="pl-s">          til_fts.rank</span>
<span class="pl-s">        limit</span>
<span class="pl-s">          5</span>
<span class="pl-s">        """</span>
        <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">datasette</span>.<span class="pl-en">get_database</span>().<span class="pl-en">execute</span>(
            <span class="pl-s1">sql</span>,
            {<span class="pl-s">"words"</span>: <span class="pl-s">" OR "</span>.<span class="pl-en">join</span>(<span class="pl-s1">words</span>), <span class="pl-s">"slug"</span>: <span class="pl-s1">til</span>[<span class="pl-s">"slug"</span>], <span class="pl-s">"topic"</span>: <span class="pl-s1">til</span>[<span class="pl-s">"topic"</span>]},
        )
        <span class="pl-k">return</span> <span class="pl-s1">result</span>.<span class="pl-s1">rows</span>

    <span class="pl-k">return</span> {
        <span class="pl-s">"related_tils"</span>: <span class="pl-s1">related_tils</span>,
    }</pre></div>
<p>Here's <a href="https://github.com/simonw/til/blob/cb62b8ab4c7b26ec7a895adea7f2d405b48686ba/plugins/template_vars.py">the full code</a> for that plugin, which also registers some other template functions.</p>


path: aws_boto-command-line.md
topic: aws
title: Using boto3 from the command line
url: https://github.com/simonw/til/blob/main/aws/boto-command-line.md
created: 2022-08-02T20:34:27-07:00
created_utc: 2022-08-03T03:34:27+00:00
updated: 2022-08-02T20:34:27-07:00
updated_utc: 2022-08-03T03:34:27+00:00
shot_hash: be4e6236df967f2d6d68f8caaf400be9
slug: boto-command-line
body:
I found a useful pattern today for automating more complex AWS processes as pastable command line snippets, using [Boto3](https://aws.amazon.com/sdk-for-python/).

The trick is to take advantage of the fact that `python3 -c '...'` lets you pass in a multi-line Python string which will be executed directly.

I used that to create a new IAM role by running the following:
```bash
python3 -c '
import boto3, json

iam = boto3.client("iam")
create_role_response = iam.create_role(
    Description=("Description of my role"),
    RoleName="my-new-role",
    AssumeRolePolicyDocument=json.dumps(
        {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "AWS": "arn:aws:iam::462092780466:user/s3.read-write.my-previously-created-user"
                    },
                    "Action": "sts:AssumeRole",
                }
            ],
        }
    ),
    MaxSessionDuration=12 * 60 * 60,
)
# Attach AmazonS3FullAccess to it - note that even though we use full access
# on the role itself any time we call sts.assume_role() we attach an additional
# policy to ensure reduced access for the temporary credentials
iam.attach_role_policy(
    RoleName="my-new-role",
    PolicyArn="arn:aws:iam::aws:policy/AmazonS3FullAccess",
)
print(create_role_response["Role"]["Arn"])
'
```

html:
<p>I found a useful pattern today for automating more complex AWS processes as pastable command line snippets, using <a href="https://aws.amazon.com/sdk-for-python/" rel="nofollow">Boto3</a>.</p>
<p>The trick is to take advantage of the fact that <code>python3 -c '...'</code> lets you pass in a multi-line Python string which will be executed directly.</p>
<p>I used that to create a new IAM role by running the following:</p>
<div class="highlight highlight-source-shell"><pre>python3 -c <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s">import boto3, json</span>
<span class="pl-s"></span>
<span class="pl-s">iam = boto3.client("iam")</span>
<span class="pl-s">create_role_response = iam.create_role(</span>
<span class="pl-s">    Description=("Description of my role"),</span>
<span class="pl-s">    RoleName="my-new-role",</span>
<span class="pl-s">    AssumeRolePolicyDocument=json.dumps(</span>
<span class="pl-s">        {</span>
<span class="pl-s">            "Version": "2012-10-17",</span>
<span class="pl-s">            "Statement": [</span>
<span class="pl-s">                {</span>
<span class="pl-s">                    "Effect": "Allow",</span>
<span class="pl-s">                    "Principal": {</span>
<span class="pl-s">                        "AWS": "arn:aws:iam::462092780466:user/s3.read-write.my-previously-created-user"</span>
<span class="pl-s">                    },</span>
<span class="pl-s">                    "Action": "sts:AssumeRole",</span>
<span class="pl-s">                }</span>
<span class="pl-s">            ],</span>
<span class="pl-s">        }</span>
<span class="pl-s">    ),</span>
<span class="pl-s">    MaxSessionDuration=12 * 60 * 60,</span>
<span class="pl-s">)</span>
<span class="pl-s"># Attach AmazonS3FullAccess to it - note that even though we use full access</span>
<span class="pl-s"># on the role itself any time we call sts.assume_role() we attach an additional</span>
<span class="pl-s"># policy to ensure reduced access for the temporary credentials</span>
<span class="pl-s">iam.attach_role_policy(</span>
<span class="pl-s">    RoleName="my-new-role",</span>
<span class="pl-s">    PolicyArn="arn:aws:iam::aws:policy/AmazonS3FullAccess",</span>
<span class="pl-s">)</span>
<span class="pl-s">print(create_role_response["Role"]["Arn"])</span>
<span class="pl-s"><span class="pl-pds">'</span></span></pre></div>


path: sqlite_trying-macos-extensions.md
topic: sqlite
title: Trying out SQLite extensions on macOS
url: https://github.com/simonw/til/blob/main/sqlite/trying-macos-extensions.md
created: 2022-08-03T10:31:02-07:00
created_utc: 2022-08-03T17:31:02+00:00
updated: 2022-08-03T11:34:23-07:00
updated_utc: 2022-08-03T18:34:23+00:00
shot_hash: d5d5ba4774d34e029b11f14cf1741863
slug: trying-macos-extensions
body:
Alex Garcia has been building some really cool new custom extensions for SQLite, working in C and Go. So far he's released two:

- [sqlite-lines](https://github.com/asg017/sqlite-lines), written in C, which adds `lines(text, optional-delimiter)` and `lines_read(filepath, optional-delimiter)` table-valued functions, for processing files line-by-line.
- [sqlite-html](https://github.com/asg017/sqlite-html), 
written in Go, which provides a [whole family of functions](https://github.com/asg017/sqlite-html/blob/main/docs.md) for parsing and constructing HTML strings.

Both of these have interactive demos, which you can try out in your browser in Alex's Observable notebooks:

- [Introducing sqlite-lines - a SQLite extension for reading files line-by-line](https://observablehq.com/@asg017/introducing-sqlite-lines)
- [Introducing sqlite-html: query, parse, and generate HTML in SQLite](https://observablehq.com/@asg017/introducing-sqlite-html)

Getting them to run on a macOS laptop is harder. Here's how I got them to work.

## Don't use the sqlite3 command that came with macOS

The `sqlite3` command that comes built in to macOS has a frustrating limitation: it has been compiled without the ability to load new extensions.

You can confirm this by running the following:

    sqlite3 :memory: 'select * from pragma_compile_options()'

On my machine part of the ouput from this says `OMIT_LOAD_EXTENSION`.

You can run `which sqlite3` to see where it is located - on my system that outputs `/usr/bin/sqlite3`.

If you use Homebrew you can run `brew install sqlite` - this will install a modern version of SQLite, but it won't link it into your path (it's a "key-only package" in Homebrew jargon) to avoid conflicting with the macOS default installation. Running `brew info sqlite` confirms this.

```
% brew info sqlite 
sqlite: stable 3.39.2 (bottled) [keg-only]
Command-line interface for SQLite
https://sqlite.org/index.html
/usr/local/Cellar/sqlite/3.39.2 (11 files, 4.4MB)
  Poured from bottle on 2022-07-24 at 14:46:49
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/sqlite.rb
License: blessing
==> Dependencies
Required: readline ✔
==> Caveats
sqlite is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have sqlite first in your PATH, run:
  echo 'export PATH="/usr/local/opt/sqlite/bin:$PATH"' >> ~/.zshrc
...
```
So the command is installed but is not on your path. It lives at `/usr/local/opt/sqlite/bin/sqlite3` - so you can run this version using that full path.

Running this confirms that it doesn't have that `OMIT_LOAD_EXTENSION` option:

    /usr/local/opt/sqlite/bin/sqlite3 :memory: \
      'select * from pragma_compile_options()'

(It also reveals some exciting extra extensions: `ENABLE_GEOPOLY` and `ENABLE_RTREE` are both listed there.)

You can also download a precompiled SQLite binary for macOS from the [SQLite downloads page](https://www.sqlite.org/download.html) - though this isn't signed, so you'll need to follow the steps described next to get it to run.

## Download the .dylib files using wget

I figured this out after first writing this TIL. If you download a `.dylib` extension using `wget` it will work straight away:
```
% cd /tmp
/tmp % wget https://github.com/asg017/sqlite-lines/releases/download/v0.1.1/lines0.dylib
...
Saving to: ‘lines0.dylib’
/tmp % /usr/local/opt/sqlite/bin/sqlite3
SQLite version 3.39.2 2022-07-21 15:24:47
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite> .load lines0
sqlite> .mode column
sqlite> select
  line ->> 'color' as color, 
  sum(line ->> 'value') as sum
from lines('{"color":"red","value":56}
{"color":"red","value":79}
{"color":"blue","value":52}
{"color":"blue","value":15}') group by color;
color  sum
-----  ---
blue   67 
red    135
```
## And for sqlite-html

Grab the `html0.dylib` extension from Alex's [releases page](https://github.com/asg017/sqlite-html/releases/tag/v0.1.0), again using `wget`:
```
% cd /tmp
% wget https://github.com/asg017/sqlite-html/releases/download/v0.1.0/html0.dylib
 % /usr/local/opt/sqlite/bin/sqlite3
sqlite> .load html0.dylib
sqlite> select name from pragma_function_list where name like 'html_%';
html_valid
html_count
html_text
html_text
html_group_element_div
html_attr_get
html_attribute_has
html_group_element_span
html_attr_has
html_element
html_trim
html_table
html_attribute_get
html_extract
html_version
html_debug
html_escape
html_unescape
sqlite> .mode column
sqlite> select * from html_each('<ul>
<li>Alpha</li>
<li>Bravo</li>
<li>Charlie</li>
<li>Delta</li>', 'li');
html              text   
----------------  -------
<li>Alpha</li>    Alpha  
<li>Bravo</li>    Bravo  
<li>Charlie</li>  Charlie
<li>Delta</li>    Delta  
```

## If you download with your browser you'll have to jump through more hoops

I first tried downloading the `lines0.dylib` file from the [0.1.1 release page](https://github.com/asg017/sqlite-lines/releases/tag/v0.1.1).

The first time I tried loading the extension I got this error:

```
% /usr/local/opt/sqlite/bin/sqlite3
sqlite> .load /Users/simon/Downloads/lines0.dylib
Error: dlopen(/Users/simon/Downloads/lines0.dylib.dylib, 0x000A): tried: '/Users/simon/Downloads/lines0.dylib.dylib' (no such file), '/usr/local/lib/lines0.dylib.dylib' (no such file), '/usr/lib/lines0.dylib.dylib' (no such file)
```
macOS popped up up a helpful dialog box saing what went wrong:

<img src="https://static.simonwillison.net/static/2022/sqlite-lines-computer-says-no.png" width="300" alt="lines0.dylib cannot be opened because the developer cannot be verified. macOS cannot verify that this app is free from malware. Buttons: Move to Trash or Cancel">

This is because the code hasn't been signed. You can still open it though - the trick is to head over to the macOS Security tab in System Preferences:

![The security panel now has an extra line saying "lines0.dylib was blocked from use because it is not from an identified developer" - with a Allow Anyway button.](https://static.simonwillison.net/static/2022/security-lines0.png)

Click "Allow Anyway", then try running the `.load` command again. Click "Open" one more time in this dialog:

<img src="https://static.simonwillison.net/static/2022/lines0-allow.png" width="300" alt="Same dialog a before but now there is an Open button">

And the extension will load from now on!

html:
<p>Alex Garcia has been building some really cool new custom extensions for SQLite, working in C and Go. So far he's released two:</p>
<ul>
<li>
<a href="https://github.com/asg017/sqlite-lines">sqlite-lines</a>, written in C, which adds <code>lines(text, optional-delimiter)</code> and <code>lines_read(filepath, optional-delimiter)</code> table-valued functions, for processing files line-by-line.</li>
<li>
<a href="https://github.com/asg017/sqlite-html">sqlite-html</a>,
written in Go, which provides a <a href="https://github.com/asg017/sqlite-html/blob/main/docs.md">whole family of functions</a> for parsing and constructing HTML strings.</li>
</ul>
<p>Both of these have interactive demos, which you can try out in your browser in Alex's Observable notebooks:</p>
<ul>
<li><a href="https://observablehq.com/@asg017/introducing-sqlite-lines" rel="nofollow">Introducing sqlite-lines - a SQLite extension for reading files line-by-line</a></li>
<li><a href="https://observablehq.com/@asg017/introducing-sqlite-html" rel="nofollow">Introducing sqlite-html: query, parse, and generate HTML in SQLite</a></li>
</ul>
<p>Getting them to run on a macOS laptop is harder. Here's how I got them to work.</p>
<h2>
<a id="user-content-dont-use-the-sqlite3-command-that-came-with-macos" class="anchor" href="#dont-use-the-sqlite3-command-that-came-with-macos" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Don't use the sqlite3 command that came with macOS</h2>
<p>The <code>sqlite3</code> command that comes built in to macOS has a frustrating limitation: it has been compiled without the ability to load new extensions.</p>
<p>You can confirm this by running the following:</p>
<pre><code>sqlite3 :memory: 'select * from pragma_compile_options()'
</code></pre>
<p>On my machine part of the ouput from this says <code>OMIT_LOAD_EXTENSION</code>.</p>
<p>You can run <code>which sqlite3</code> to see where it is located - on my system that outputs <code>/usr/bin/sqlite3</code>.</p>
<p>If you use Homebrew you can run <code>brew install sqlite</code> - this will install a modern version of SQLite, but it won't link it into your path (it's a "key-only package" in Homebrew jargon) to avoid conflicting with the macOS default installation. Running <code>brew info sqlite</code> confirms this.</p>
<pre><code>% brew info sqlite 
sqlite: stable 3.39.2 (bottled) [keg-only]
Command-line interface for SQLite
https://sqlite.org/index.html
/usr/local/Cellar/sqlite/3.39.2 (11 files, 4.4MB)
  Poured from bottle on 2022-07-24 at 14:46:49
From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/sqlite.rb
License: blessing
==&gt; Dependencies
Required: readline ✔
==&gt; Caveats
sqlite is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have sqlite first in your PATH, run:
  echo 'export PATH="/usr/local/opt/sqlite/bin:$PATH"' &gt;&gt; ~/.zshrc
...
</code></pre>
<p>So the command is installed but is not on your path. It lives at <code>/usr/local/opt/sqlite/bin/sqlite3</code> - so you can run this version using that full path.</p>
<p>Running this confirms that it doesn't have that <code>OMIT_LOAD_EXTENSION</code> option:</p>
<pre><code>/usr/local/opt/sqlite/bin/sqlite3 :memory: \
  'select * from pragma_compile_options()'
</code></pre>
<p>(It also reveals some exciting extra extensions: <code>ENABLE_GEOPOLY</code> and <code>ENABLE_RTREE</code> are both listed there.)</p>
<p>You can also download a precompiled SQLite binary for macOS from the <a href="https://www.sqlite.org/download.html" rel="nofollow">SQLite downloads page</a> - though this isn't signed, so you'll need to follow the steps described next to get it to run.</p>
<h2>
<a id="user-content-download-the-dylib-files-using-wget" class="anchor" href="#download-the-dylib-files-using-wget" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Download the .dylib files using wget</h2>
<p>I figured this out after first writing this TIL. If you download a <code>.dylib</code> extension using <code>wget</code> it will work straight away:</p>
<pre><code>% cd /tmp
/tmp % wget https://github.com/asg017/sqlite-lines/releases/download/v0.1.1/lines0.dylib
...
Saving to: ‘lines0.dylib’
/tmp % /usr/local/opt/sqlite/bin/sqlite3
SQLite version 3.39.2 2022-07-21 15:24:47
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
sqlite&gt; .load lines0
sqlite&gt; .mode column
sqlite&gt; select
  line -&gt;&gt; 'color' as color, 
  sum(line -&gt;&gt; 'value') as sum
from lines('{"color":"red","value":56}
{"color":"red","value":79}
{"color":"blue","value":52}
{"color":"blue","value":15}') group by color;
color  sum
-----  ---
blue   67 
red    135
</code></pre>
<h2>
<a id="user-content-and-for-sqlite-html" class="anchor" href="#and-for-sqlite-html" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>And for sqlite-html</h2>
<p>Grab the <code>html0.dylib</code> extension from Alex's <a href="https://github.com/asg017/sqlite-html/releases/tag/v0.1.0">releases page</a>, again using <code>wget</code>:</p>
<pre><code>% cd /tmp
% wget https://github.com/asg017/sqlite-html/releases/download/v0.1.0/html0.dylib
 % /usr/local/opt/sqlite/bin/sqlite3
sqlite&gt; .load html0.dylib
sqlite&gt; select name from pragma_function_list where name like 'html_%';
html_valid
html_count
html_text
html_text
html_group_element_div
html_attr_get
html_attribute_has
html_group_element_span
html_attr_has
html_element
html_trim
html_table
html_attribute_get
html_extract
html_version
html_debug
html_escape
html_unescape
sqlite&gt; .mode column
sqlite&gt; select * from html_each('&lt;ul&gt;
&lt;li&gt;Alpha&lt;/li&gt;
&lt;li&gt;Bravo&lt;/li&gt;
&lt;li&gt;Charlie&lt;/li&gt;
&lt;li&gt;Delta&lt;/li&gt;', 'li');
html              text   
----------------  -------
&lt;li&gt;Alpha&lt;/li&gt;    Alpha  
&lt;li&gt;Bravo&lt;/li&gt;    Bravo  
&lt;li&gt;Charlie&lt;/li&gt;  Charlie
&lt;li&gt;Delta&lt;/li&gt;    Delta  
</code></pre>
<h2>
<a id="user-content-if-you-download-with-your-browser-youll-have-to-jump-through-more-hoops" class="anchor" href="#if-you-download-with-your-browser-youll-have-to-jump-through-more-hoops" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>If you download with your browser you'll have to jump through more hoops</h2>
<p>I first tried downloading the <code>lines0.dylib</code> file from the <a href="https://github.com/asg017/sqlite-lines/releases/tag/v0.1.1">0.1.1 release page</a>.</p>
<p>The first time I tried loading the extension I got this error:</p>
<pre><code>% /usr/local/opt/sqlite/bin/sqlite3
sqlite&gt; .load /Users/simon/Downloads/lines0.dylib
Error: dlopen(/Users/simon/Downloads/lines0.dylib.dylib, 0x000A): tried: '/Users/simon/Downloads/lines0.dylib.dylib' (no such file), '/usr/local/lib/lines0.dylib.dylib' (no such file), '/usr/lib/lines0.dylib.dylib' (no such file)
</code></pre>
<p>macOS popped up up a helpful dialog box saing what went wrong:</p>
<p>This is because the code hasn't been signed. You can still open it though - the trick is to head over to the macOS Security tab in System Preferences:</p>
<p><a href="https://camo.githubusercontent.com/f6fc72b0e70dcbc6784a0059caf978be3cc2449e39416d4383a8b26e3b7b894f/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032322f73656375726974792d6c696e6573302e706e67" target="_blank" rel="nofollow"><img src="https://camo.githubusercontent.com/f6fc72b0e70dcbc6784a0059caf978be3cc2449e39416d4383a8b26e3b7b894f/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032322f73656375726974792d6c696e6573302e706e67" alt='The security panel now has an extra line saying "lines0.dylib was blocked from use because it is not from an identified developer" - with a Allow Anyway button.' data-canonical-src="https://static.simonwillison.net/static/2022/security-lines0.png" style="max-width:100%;"></a></p>
<p>Click "Allow Anyway", then try running the <code>.load</code> command again. Click "Open" one more time in this dialog:</p>
<p><a href="https://camo.githubusercontent.com/de9419db8778ca0fcc48166a86927434622be20c5c4eb3bc23cff3bea6b6bb6c/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032322f6c696e6573302d616c6c6f772e706e67" target="_blank" rel="nofollow"><img src="https://camo.githubusercontent.com/de9419db8778ca0fcc48166a86927434622be20c5c4eb3bc23cff3bea6b6bb6c/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032322f6c696e6573302d616c6c6f772e706e67" width="300" alt="Same dialog a before but now there is an Open button" data-canonical-src="https://static.simonwillison.net/static/2022/lines0-allow.png" style="max-width:100%;"></a></p>
<p>And the extension will load from now on!</p>


path: pytest_mocking-boto.md
topic: pytest
title: Mocking a Textract LimitExceededException with boto
url: https://github.com/simonw/til/blob/main/pytest/mocking-boto.md
created: 2022-08-07T10:37:11-07:00
created_utc: 2022-08-07T17:37:11+00:00
updated: 2022-08-07T12:37:51-07:00
updated_utc: 2022-08-07T19:37:51+00:00
shot_hash: 5d6afcd8d0ce92ceac0bc1dfed171de5
slug: mocking-boto
body:
For [s3-ocr issue #21](https://github.com/simonw/s3-ocr/issues/21) I needed to write a test that simulates what happens when Amazon Textract returns a "LimitExceededException". When using boto this error presents itself as an exception:

> `botocore.errorfactory.LimitExceededException: An error occurred (LimitExceededException) when calling the StartDocumentTextDetection operation: Open jobs exceed maximum concurrent job limit`

I uses [moto](https://github.com/spulec/moto) to simulate AWS in that test suite, but moto does not yet have a mechanism for simulating Textract errors like this one.

I ended up turning to Python mocks, here provided by the the [pytest-mock](https://pypi.org/project/pytest-mock/) fixture. Here's the test I came up with:

```python
def test_limit_exceeded_automatic_retry(s3, mocker):
    mocked = mocker.patch("s3_ocr.cli.start_document_text_extraction")
    # It's going to fail the first time, then succeed
    mocked.side_effect = [
        boto3.client("textract").exceptions.LimitExceededException(
            error_response={},
            operation_name="StartDocumentTextExtraction",
        ),
        {"JobId": "123"},
    ]
    runner = CliRunner()
    result = runner.invoke(cli, ["start", "my-bucket", "--all"])
    assert result.exit_code == 0
    assert result.output == (
        "Found 0 files with .s3-ocr.json out of 1 PDFs\n"
        "An error occurred (Unknown) when calling the StartDocumentTextExtraction operation: Unknown - retrying...\n"
        "Starting OCR for blah.pdf, Job ID: 123\n"
    )
```
Here I'm patching the function identified by the string `"s3_ocr.cli.start_document_text_extraction"`. This is a new function that I wrote specifically to make this mock easier to apply - it lives in `s3_ocr/cli.py` and looks [like this](https://github.com/simonw/s3-ocr/blob/23497aa5741c28c7eee00614a19c398066d61bf7/s3_ocr/cli.py#L552-L554):

```python
def start_document_text_extraction(textract, **kwargs):
    # Wrapper function to make this easier to mock in tests
    return textract.start_document_text_detection(**kwargs)
```
The most confusing thing about working with Python mocks is figuring out the string to use to mock the right piece of code. I like this pattern of refactoring the code under test to make it as simple to mock as possible.

The code I am testing here implements automatic retries. As such, I needed the API method I am simulating to fail the first time and then succeed the second time.

Originally I had done this with a `side_effect()` function - see below - but then [@szotten on Twitter](https://twitter.com/szotten/status/1556337221258575873) pointed out that you can instead set `mock.side_effect` to a list and it will [cycle through those items in turn](https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock.side_effect):

```python
mocked.side_effect = [
    boto3.client("textract").exceptions.LimitExceededException(
        error_response={},
        operation_name="StartDocumentTextExtraction",
    ),
    {"JobId": "123"},
]
```
Any exception objects in that list will be raised by the mocked function; any other kind of object will be returned.

The hardest thing to figure out was how to simulate the exception. The original error message indicated `botocore.errorfactory.LimitExceededException` but that's not actually a class you can import and raise.

Instead, I used `boto3.client("textract").exceptions.LimitExceededException`.

Figuring out that it needed an `error_response` and `operation_name` was tricky too. I eventually [tracked down](https://github.com/boto/botocore/blob/f4ed130b78076fb5683e4384c7df007e82dda71d/botocore/exceptions.py#L500-L517) the `botocore` `ClientError` constructor, which showed me what I needed to provide:

```python
class ClientError(Exception):
    MSG_TEMPLATE = (
        'An error occurred ({error_code}) when calling the {operation_name} '
        'operation{retry_info}: {error_message}'
    )

    def __init__(self, error_response, operation_name):
        retry_info = self._get_retry_info(error_response)
        error = error_response.get('Error', {})
        msg = self.MSG_TEMPLATE.format(
            error_code=error.get('Code', 'Unknown'),
            error_message=error.get('Message', 'Unknown'),
            operation_name=operation_name,
            retry_info=retry_info,
        )
        super().__init__(msg)
        self.response = error_response
        self.operation_name = operation_name
```

## Using a side effect function

Prior to the tip about setting `.side_effect` to a list I used a side effect function instead, with a `nonlocal` variable to change its behaviour the second time it was called.

```python
should_fail = True

def side_effect(*args, **kwargs):
    nonlocal should_fail
    if should_fail:
        should_fail = False
        raise boto3.client("textract").exceptions.LimitExceededException(
            error_response={},
            operation_name="StartDocumentTextExtraction",
        )
    else:
        return {"JobId": "123"}

mocked.side_effect = side_effect
```

html:
<p>For <a href="https://github.com/simonw/s3-ocr/issues/21">s3-ocr issue #21</a> I needed to write a test that simulates what happens when Amazon Textract returns a "LimitExceededException". When using boto this error presents itself as an exception:</p>
<blockquote>
<p><code>botocore.errorfactory.LimitExceededException: An error occurred (LimitExceededException) when calling the StartDocumentTextDetection operation: Open jobs exceed maximum concurrent job limit</code></p>
</blockquote>
<p>I uses <a href="https://github.com/spulec/moto">moto</a> to simulate AWS in that test suite, but moto does not yet have a mechanism for simulating Textract errors like this one.</p>
<p>I ended up turning to Python mocks, here provided by the the <a href="https://pypi.org/project/pytest-mock/" rel="nofollow">pytest-mock</a> fixture. Here's the test I came up with:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">test_limit_exceeded_automatic_retry</span>(<span class="pl-s1">s3</span>, <span class="pl-s1">mocker</span>):
    <span class="pl-s1">mocked</span> <span class="pl-c1">=</span> <span class="pl-s1">mocker</span>.<span class="pl-en">patch</span>(<span class="pl-s">"s3_ocr.cli.start_document_text_extraction"</span>)
    <span class="pl-c"># It's going to fail the first time, then succeed</span>
    <span class="pl-s1">mocked</span>.<span class="pl-s1">side_effect</span> <span class="pl-c1">=</span> [
        <span class="pl-s1">boto3</span>.<span class="pl-en">client</span>(<span class="pl-s">"textract"</span>).<span class="pl-s1">exceptions</span>.<span class="pl-v">LimitExceededException</span>(
            <span class="pl-s1">error_response</span><span class="pl-c1">=</span>{},
            <span class="pl-s1">operation_name</span><span class="pl-c1">=</span><span class="pl-s">"StartDocumentTextExtraction"</span>,
        ),
        {<span class="pl-s">"JobId"</span>: <span class="pl-s">"123"</span>},
    ]
    <span class="pl-s1">runner</span> <span class="pl-c1">=</span> <span class="pl-v">CliRunner</span>()
    <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-s1">runner</span>.<span class="pl-en">invoke</span>(<span class="pl-s1">cli</span>, [<span class="pl-s">"start"</span>, <span class="pl-s">"my-bucket"</span>, <span class="pl-s">"--all"</span>])
    <span class="pl-k">assert</span> <span class="pl-s1">result</span>.<span class="pl-s1">exit_code</span> <span class="pl-c1">==</span> <span class="pl-c1">0</span>
    <span class="pl-k">assert</span> <span class="pl-s1">result</span>.<span class="pl-s1">output</span> <span class="pl-c1">==</span> (
        <span class="pl-s">"Found 0 files with .s3-ocr.json out of 1 PDFs<span class="pl-cce">\n</span>"</span>
        <span class="pl-s">"An error occurred (Unknown) when calling the StartDocumentTextExtraction operation: Unknown - retrying...<span class="pl-cce">\n</span>"</span>
        <span class="pl-s">"Starting OCR for blah.pdf, Job ID: 123<span class="pl-cce">\n</span>"</span>
    )</pre></div>
<p>Here I'm patching the function identified by the string <code>"s3_ocr.cli.start_document_text_extraction"</code>. This is a new function that I wrote specifically to make this mock easier to apply - it lives in <code>s3_ocr/cli.py</code> and looks <a href="https://github.com/simonw/s3-ocr/blob/23497aa5741c28c7eee00614a19c398066d61bf7/s3_ocr/cli.py#L552-L554">like this</a>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">start_document_text_extraction</span>(<span class="pl-s1">textract</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
    <span class="pl-c"># Wrapper function to make this easier to mock in tests</span>
    <span class="pl-k">return</span> <span class="pl-s1">textract</span>.<span class="pl-en">start_document_text_detection</span>(<span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)</pre></div>
<p>The most confusing thing about working with Python mocks is figuring out the string to use to mock the right piece of code. I like this pattern of refactoring the code under test to make it as simple to mock as possible.</p>
<p>The code I am testing here implements automatic retries. As such, I needed the API method I am simulating to fail the first time and then succeed the second time.</p>
<p>Originally I had done this with a <code>side_effect()</code> function - see below - but then <a href="https://twitter.com/szotten/status/1556337221258575873" rel="nofollow">@szotten on Twitter</a> pointed out that you can instead set <code>mock.side_effect</code> to a list and it will <a href="https://docs.python.org/3/library/unittest.mock.html#unittest.mock.Mock.side_effect" rel="nofollow">cycle through those items in turn</a>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">mocked</span>.<span class="pl-s1">side_effect</span> <span class="pl-c1">=</span> [
    <span class="pl-s1">boto3</span>.<span class="pl-en">client</span>(<span class="pl-s">"textract"</span>).<span class="pl-s1">exceptions</span>.<span class="pl-v">LimitExceededException</span>(
        <span class="pl-s1">error_response</span><span class="pl-c1">=</span>{},
        <span class="pl-s1">operation_name</span><span class="pl-c1">=</span><span class="pl-s">"StartDocumentTextExtraction"</span>,
    ),
    {<span class="pl-s">"JobId"</span>: <span class="pl-s">"123"</span>},
]</pre></div>
<p>Any exception objects in that list will be raised by the mocked function; any other kind of object will be returned.</p>
<p>The hardest thing to figure out was how to simulate the exception. The original error message indicated <code>botocore.errorfactory.LimitExceededException</code> but that's not actually a class you can import and raise.</p>
<p>Instead, I used <code>boto3.client("textract").exceptions.LimitExceededException</code>.</p>
<p>Figuring out that it needed an <code>error_response</code> and <code>operation_name</code> was tricky too. I eventually <a href="https://github.com/boto/botocore/blob/f4ed130b78076fb5683e4384c7df007e82dda71d/botocore/exceptions.py#L500-L517">tracked down</a> the <code>botocore</code> <code>ClientError</code> constructor, which showed me what I needed to provide:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">class</span> <span class="pl-v">ClientError</span>(<span class="pl-v">Exception</span>):
    <span class="pl-v">MSG_TEMPLATE</span> <span class="pl-c1">=</span> (
        <span class="pl-s">'An error occurred ({error_code}) when calling the {operation_name} '</span>
        <span class="pl-s">'operation{retry_info}: {error_message}'</span>
    )

    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">error_response</span>, <span class="pl-s1">operation_name</span>):
        <span class="pl-s1">retry_info</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">_get_retry_info</span>(<span class="pl-s1">error_response</span>)
        <span class="pl-s1">error</span> <span class="pl-c1">=</span> <span class="pl-s1">error_response</span>.<span class="pl-en">get</span>(<span class="pl-s">'Error'</span>, {})
        <span class="pl-s1">msg</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-v">MSG_TEMPLATE</span>.<span class="pl-en">format</span>(
            <span class="pl-s1">error_code</span><span class="pl-c1">=</span><span class="pl-s1">error</span>.<span class="pl-en">get</span>(<span class="pl-s">'Code'</span>, <span class="pl-s">'Unknown'</span>),
            <span class="pl-s1">error_message</span><span class="pl-c1">=</span><span class="pl-s1">error</span>.<span class="pl-en">get</span>(<span class="pl-s">'Message'</span>, <span class="pl-s">'Unknown'</span>),
            <span class="pl-s1">operation_name</span><span class="pl-c1">=</span><span class="pl-s1">operation_name</span>,
            <span class="pl-s1">retry_info</span><span class="pl-c1">=</span><span class="pl-s1">retry_info</span>,
        )
        <span class="pl-en">super</span>().<span class="pl-en">__init__</span>(<span class="pl-s1">msg</span>)
        <span class="pl-s1">self</span>.<span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">error_response</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">operation_name</span> <span class="pl-c1">=</span> <span class="pl-s1">operation_name</span></pre></div>
<h2>
<a id="user-content-using-a-side-effect-function" class="anchor" href="#using-a-side-effect-function" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using a side effect function</h2>
<p>Prior to the tip about setting <code>.side_effect</code> to a list I used a side effect function instead, with a <code>nonlocal</code> variable to change its behaviour the second time it was called.</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">should_fail</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>

<span class="pl-k">def</span> <span class="pl-en">side_effect</span>(<span class="pl-c1">*</span><span class="pl-s1">args</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
    <span class="pl-k">nonlocal</span> <span class="pl-s1">should_fail</span>
    <span class="pl-k">if</span> <span class="pl-s1">should_fail</span>:
        <span class="pl-s1">should_fail</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>
        <span class="pl-k">raise</span> <span class="pl-s1">boto3</span>.<span class="pl-en">client</span>(<span class="pl-s">"textract"</span>).<span class="pl-s1">exceptions</span>.<span class="pl-v">LimitExceededException</span>(
            <span class="pl-s1">error_response</span><span class="pl-c1">=</span>{},
            <span class="pl-s1">operation_name</span><span class="pl-c1">=</span><span class="pl-s">"StartDocumentTextExtraction"</span>,
        )
    <span class="pl-k">else</span>:
        <span class="pl-k">return</span> {<span class="pl-s">"JobId"</span>: <span class="pl-s">"123"</span>}

<span class="pl-s1">mocked</span>.<span class="pl-s1">side_effect</span> <span class="pl-c1">=</span> <span class="pl-s1">side_effect</span></pre></div>


path: gpt3_jq.md
topic: gpt3
title: Using GPT-3 to figure out jq recipes
url: https://github.com/simonw/til/blob/main/gpt3/jq.md
created: 2022-08-10T08:23:36-07:00
created_utc: 2022-08-10T15:23:36+00:00
updated: 2022-08-10T08:23:36-07:00
updated_utc: 2022-08-10T15:23:36+00:00
shot_hash: 54d33f9ef915ebcbe1d5da2d6c836f9d
slug: jq
body:
I like [jq](https://stedolan.github.io/jq/), but I always have to think pretty hard about how to construct the right syntax for it.

Here's how I used the [GPT-3 playground](https://simonwillison.net/2022/Jun/5/play-with-gpt3/) to solve a simple `jq` problem.

I wanted to sum up the total number of comments in JSON that looks like this (truncated example):

```json
[
    {
        "id": "31195431",
        "title": "Automatically filing issues when tracked file content changes",
        "url": "https://simonwillison.net/2022/Apr/28/issue-on-changes/",
        "dt": "2022-04-28T17:29:37",
        "points": 14,
        "submitter": "simonw",
        "commentsUrl": "https://news.ycombinator.com/item?id=31195431",
        "numComments": 4
    },
    {
        "id": "31185496",
        "title": "Parallel SQL Queries in Datasette",
        "url": "https://simonwillison.net/2022/Apr/27/parallel-queries/",
        "dt": "2022-04-27T20:39:50",
        "points": 1,
        "submitter": "frogger8",
        "commentsUrl": "https://news.ycombinator.com/item?id=31185496",
        "numComments": 3
    }
]
```
The first prompt I tried was:

```
# Use jq to calculate the sum of the num field in each object in this array:

echo '[
  {"num": 1},
  {"num": 2}
]' | jq
```
I hinted at how I wanted to do this by piping to `jq` but without any argument.

GPT-3 completed that to:

```
echo '[
  {"num": 1},
  {"num": 2}
]' | jq 'reduce .[] as $item (0; . + $item.num)'
```
... then a bunch more output that gave further examples, which I ignored

This is good - that solution works! Here's what happens when I run it:

```
~ % echo '[
  {"num": 1},
  {"num": 2}
]' | jq 'reduce .[] as $item (0; . + $item.num)'
3
```
But I'm not a big fan of `reduce` in general - I always have to think a bit about what it's actually doing.

So I added a follow-up prompt:

```
Now do that without using reduce:
```
And GPT-3 replied:
```
echo '[
  {"num": 1},
  {"num": 2}
]' | jq  '. | map(.num) | add'
```
This is the exact solution I was looking for.

html:
<p>I like <a href="https://stedolan.github.io/jq/" rel="nofollow">jq</a>, but I always have to think pretty hard about how to construct the right syntax for it.</p>
<p>Here's how I used the <a href="https://simonwillison.net/2022/Jun/5/play-with-gpt3/" rel="nofollow">GPT-3 playground</a> to solve a simple <code>jq</code> problem.</p>
<p>I wanted to sum up the total number of comments in JSON that looks like this (truncated example):</p>
<div class="highlight highlight-source-json"><pre>[
    {
        <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>31195431<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"title"</span>: <span class="pl-s"><span class="pl-pds">"</span>Automatically filing issues when tracked file content changes<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"url"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://simonwillison.net/2022/Apr/28/issue-on-changes/<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"dt"</span>: <span class="pl-s"><span class="pl-pds">"</span>2022-04-28T17:29:37<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"points"</span>: <span class="pl-c1">14</span>,
        <span class="pl-ent">"submitter"</span>: <span class="pl-s"><span class="pl-pds">"</span>simonw<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"commentsUrl"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://news.ycombinator.com/item?id=31195431<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"numComments"</span>: <span class="pl-c1">4</span>
    },
    {
        <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>31185496<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"title"</span>: <span class="pl-s"><span class="pl-pds">"</span>Parallel SQL Queries in Datasette<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"url"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://simonwillison.net/2022/Apr/27/parallel-queries/<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"dt"</span>: <span class="pl-s"><span class="pl-pds">"</span>2022-04-27T20:39:50<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"points"</span>: <span class="pl-c1">1</span>,
        <span class="pl-ent">"submitter"</span>: <span class="pl-s"><span class="pl-pds">"</span>frogger8<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"commentsUrl"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://news.ycombinator.com/item?id=31185496<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"numComments"</span>: <span class="pl-c1">3</span>
    }
]</pre></div>
<p>The first prompt I tried was:</p>
<pre><code># Use jq to calculate the sum of the num field in each object in this array:

echo '[
  {"num": 1},
  {"num": 2}
]' | jq
</code></pre>
<p>I hinted at how I wanted to do this by piping to <code>jq</code> but without any argument.</p>
<p>GPT-3 completed that to:</p>
<pre><code>echo '[
  {"num": 1},
  {"num": 2}
]' | jq 'reduce .[] as $item (0; . + $item.num)'
</code></pre>
<p>... then a bunch more output that gave further examples, which I ignored</p>
<p>This is good - that solution works! Here's what happens when I run it:</p>
<pre><code>~ % echo '[
  {"num": 1},
  {"num": 2}
]' | jq 'reduce .[] as $item (0; . + $item.num)'
3
</code></pre>
<p>But I'm not a big fan of <code>reduce</code> in general - I always have to think a bit about what it's actually doing.</p>
<p>So I added a follow-up prompt:</p>
<pre><code>Now do that without using reduce:
</code></pre>
<p>And GPT-3 replied:</p>
<pre><code>echo '[
  {"num": 1},
  {"num": 2}
]' | jq  '. | map(.num) | add'
</code></pre>
<p>This is the exact solution I was looking for.</p>


path: javascript_openseadragon.md
topic: javascript
title: Creating a tiled zoomable image with OpenSeadragon and vips
url: https://github.com/simonw/til/blob/main/javascript/openseadragon.md
created: 2022-08-12T17:34:34-07:00
created_utc: 2022-08-13T00:34:34+00:00
updated: 2022-08-12T20:39:06-07:00
updated_utc: 2022-08-13T03:39:06+00:00
shot_hash: 7c76d37368b6d691305150cf17431365
slug: openseadragon
body:
The San Francisco Microscopical Society has some extremely high resolution scanned images - one of them is a 1.67GB PNG file with a 25,088 × 17,283 pixel resolution.

I wanted to display these in a browser using a zoomed interface, without needing to load in that whole file.

[OpenSeadragon](https://openseadragon.github.io/) looked like a good fit for this:

> An open-source, web-based viewer for **high-resolution zoomable images**, implemented in pure JavaScript, for desktop and mobile.

## Creating the tiles with vips

The first step in using it is to split the source image up into a set of tiles.

OpenSeadragon offers a [range of options](http://openseadragon.github.io/examples/creating-zooming-images/) for doing this. After investigating a few the one that worked best for me was [vips](https://www.libvips.org/), with its [vips dzsave](https://www.libvips.org/API/current/Making-image-pyramids.html) command.

Installing this on macOS with Homebrew:

    brew install vips

Once installed you can split a source image into tiles like this:

    vips dzsave Miniature_Victorian_Slide_Collection.png \
      Miniature_Victorian_Slide_Collection

This command runs _really_ quickly: it took just a few seconds, and Activity Monitor showed it using more than 600% CPU, so it is clearly optimized to make full use of multiple cores.

This command creates an XML file called `Miniature_Victorian_Slide_Collection.dzi` defining the zoomed tile set, and a folder called `Miniature_Victorian_Slide_Collection_files` containing thousands of smaller tiles, arranged in a hierarchy.
```
% find . | grep Miniat | wc -l
    5233
```
That's a LOT of files. They were each around 256 × 256 in resolution - I decided to see if I could make them bigger and reduce their number.

I used the `--tile-size` option to do this:

    vips dzsave Miniature_Victorian_Slide_Collection.png \
      Miniature_Victorian_Slide_Collection-tile-size-512 \
      --tile-size 512

This created a more manageable 1,343 files, totalling 25.2MB.

## Implementing the viewer

Here's the minimal HTML I figured out to create a zoomable viewer that takes up the full browser viewport:

```html
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0">
<style>
#openseadragon1 {
  width: 100vw;
  height: 100vh;
}
body {
  margin: 0;
  padding: 0;
}
</style>

<div id="openseadragon1"></div>
<script src="https://cdn.jsdelivr.net/npm/openseadragon@3.1/build/openseadragon/openseadragon.min.js"></script>
<script type="text/javascript">
var viewer = OpenSeadragon({
  id: "openseadragon1",
  prefixUrl: "https://cdn.jsdelivr.net/npm/openseadragon@3.1.0/build/openseadragon/images/",
  tileSources: "Miniature_Victorian_Slide_Collection-tile-size-512.dzi"
});
</script>
```
Saving this as `index.html` in the same folder as the `.dzi` file and then serving it like this let me start zooming and interacting with the tiled image:

    python3 -m http.server

html:
<p>The San Francisco Microscopical Society has some extremely high resolution scanned images - one of them is a 1.67GB PNG file with a 25,088 × 17,283 pixel resolution.</p>
<p>I wanted to display these in a browser using a zoomed interface, without needing to load in that whole file.</p>
<p><a href="https://openseadragon.github.io/" rel="nofollow">OpenSeadragon</a> looked like a good fit for this:</p>
<blockquote>
<p>An open-source, web-based viewer for <strong>high-resolution zoomable images</strong>, implemented in pure JavaScript, for desktop and mobile.</p>
</blockquote>
<h2><a id="user-content-creating-the-tiles-with-vips" class="anchor" aria-hidden="true" href="#creating-the-tiles-with-vips"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating the tiles with vips</h2>
<p>The first step in using it is to split the source image up into a set of tiles.</p>
<p>OpenSeadragon offers a <a href="http://openseadragon.github.io/examples/creating-zooming-images/" rel="nofollow">range of options</a> for doing this. After investigating a few the one that worked best for me was <a href="https://www.libvips.org/" rel="nofollow">vips</a>, with its <a href="https://www.libvips.org/API/current/Making-image-pyramids.html" rel="nofollow">vips dzsave</a> command.</p>
<p>Installing this on macOS with Homebrew:</p>
<pre><code>brew install vips
</code></pre>
<p>Once installed you can split a source image into tiles like this:</p>
<pre><code>vips dzsave Miniature_Victorian_Slide_Collection.png \
  Miniature_Victorian_Slide_Collection
</code></pre>
<p>This command runs <em>really</em> quickly: it took just a few seconds, and Activity Monitor showed it using more than 600% CPU, so it is clearly optimized to make full use of multiple cores.</p>
<p>This command creates an XML file called <code>Miniature_Victorian_Slide_Collection.dzi</code> defining the zoomed tile set, and a folder called <code>Miniature_Victorian_Slide_Collection_files</code> containing thousands of smaller tiles, arranged in a hierarchy.</p>
<pre><code>% find . | grep Miniat | wc -l
    5233
</code></pre>
<p>That's a LOT of files. They were each around 256 × 256 in resolution - I decided to see if I could make them bigger and reduce their number.</p>
<p>I used the <code>--tile-size</code> option to do this:</p>
<pre><code>vips dzsave Miniature_Victorian_Slide_Collection.png \
  Miniature_Victorian_Slide_Collection-tile-size-512 \
  --tile-size 512
</code></pre>
<p>This created a more manageable 1,343 files, totalling 25.2MB.</p>
<h2><a id="user-content-implementing-the-viewer" class="anchor" aria-hidden="true" href="#implementing-the-viewer"><span aria-hidden="true" class="octicon octicon-link"></span></a>Implementing the viewer</h2>
<p>Here's the minimal HTML I figured out to create a zoomable viewer that takes up the full browser viewport:</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">meta</span> <span class="pl-c1">name</span>="<span class="pl-s">viewport</span>" <span class="pl-c1">content</span>="<span class="pl-s">width=device-width, initial-scale=1.0, maximum-scale=1.0</span>"<span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;</span><span class="pl-ent">style</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">#</span><span class="pl-c1">openseadragon1</span> {
  <span class="pl-c1">width</span><span class="pl-kos">:</span> <span class="pl-c1">100<span class="pl-smi">vw</span></span>;
  <span class="pl-c1">height</span><span class="pl-kos">:</span> <span class="pl-c1">100<span class="pl-smi">vh</span></span>;
}
<span class="pl-ent">body</span> {
  <span class="pl-c1">margin</span><span class="pl-kos">:</span> <span class="pl-c1">0</span>;
  <span class="pl-c1">padding</span><span class="pl-kos">:</span> <span class="pl-c1">0</span>;
}
<span class="pl-kos">&lt;/</span><span class="pl-ent">style</span><span class="pl-kos">&gt;</span>

<span class="pl-kos">&lt;</span><span class="pl-ent">div</span> <span class="pl-c1">id</span>="<span class="pl-s">openseadragon1</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">div</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;</span><span class="pl-ent">script</span> <span class="pl-c1">src</span>="<span class="pl-s">https://cdn.jsdelivr.net/npm/openseadragon@3.1/build/openseadragon/openseadragon.min.js</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;</span><span class="pl-ent">script</span> <span class="pl-c1">type</span>="<span class="pl-s">text/javascript</span>"<span class="pl-kos">&gt;</span>
<span class="pl-k">var</span> <span class="pl-s1">viewer</span> <span class="pl-c1">=</span> <span class="pl-v">OpenSeadragon</span><span class="pl-kos">(</span><span class="pl-kos">{</span>
  <span class="pl-c1">id</span>: <span class="pl-s">"openseadragon1"</span><span class="pl-kos">,</span>
  <span class="pl-c1">prefixUrl</span>: <span class="pl-s">"https://cdn.jsdelivr.net/npm/openseadragon@3.1.0/build/openseadragon/images/"</span><span class="pl-kos">,</span>
  <span class="pl-c1">tileSources</span>: <span class="pl-s">"Miniature_Victorian_Slide_Collection-tile-size-512.dzi"</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">&lt;/</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span></pre></div>
<p>Saving this as <code>index.html</code> in the same folder as the <code>.dzi</code> file and then serving it like this let me start zooming and interacting with the tiled image:</p>
<pre><code>python3 -m http.server
</code></pre>


path: python_setup-py-from-url.md
topic: python
title: Defining setup.py dependencies using a URL
url: https://github.com/simonw/til/blob/main/python/setup-py-from-url.md
created: 2022-08-13T09:31:43-07:00
created_utc: 2022-08-13T16:31:43+00:00
updated: 2022-08-13T09:31:43-07:00
updated_utc: 2022-08-13T16:31:43+00:00
shot_hash: 65226e3ac3722a0636fbb2ed34776ff9
slug: setup-py-from-url
body:
For [sqlite-utils issue 464](https://github.com/simonw/sqlite-utils/issues/464) I implemented a fix to a tiny bug in a dependency in [my own fork](https://github.com/simonw/beanbag-docutils/tree/bytes-in-url) on GitHub.

I wanted to use my fork as a dependency when I ran my build on ReadTheDocs - but for that particular project the ReadTheDocs dependencies are installed using the equivalent of `pip install '.[docs]'` (by [this configuration](https://github.com/simonw/sqlite-utils/blob/45e24deffea042b5db7ab84cd1eb63b3ed9bb9da/.readthedocs.yaml#L9-L12)) - and that `docs` set of extra requirements was defined in my `setup.py` file [like this](https://github.com/simonw/sqlite-utils/blob/271433fdd18e436b0a527ab899cb6f6fa67f23d0/setup.py#L33-L44):

```python
    extras_require={
        "test": ["pytest", "black", "hypothesis", "cogapp"],
        "docs": ["furo", "sphinx-autobuild", "codespell", "sphinx-copybutton"],
        "mypy": [
            "mypy",
            "types-click",
            "types-tabulate",
            "types-python-dateutil",
            "data-science-types",
        ],
        "flake8": ["flake8"],
    },
```

Any branch on GitHub can be installed by `pip` by finding the URL to the zip export of that branch. In this case that URL is:

`https://github.com/simonw/beanbag-docutils/archive/refs/heads/bytes-in-url.zip`

I tried adding that into the list of strings defined for the `"docs"` bundle and got this error when I ran `pip install -e '.[docs]'`:

```
Complete output (1 lines):
    error in sqlite-utils setup command: 'extras_require' must be a dictionary
    whose values are strings or lists of strings containing valid project/version requirement specifiers.
```

It turned out the solution was to use `packagename @ URL` instead, like this:

```python
    extras_require={
        "test": ["pytest", "black", "hypothesis", "cogapp"],
        "docs": [
            "furo",
            "sphinx-autobuild",
            "codespell",
            "sphinx-copybutton",
            "beanbag-docutils @ https://github.com/simonw/beanbag-docutils/archive/refs/heads/bytes-in-url.zip",
        ]
```

html:
<p>For <a href="https://github.com/simonw/sqlite-utils/issues/464">sqlite-utils issue 464</a> I implemented a fix to a tiny bug in a dependency in <a href="https://github.com/simonw/beanbag-docutils/tree/bytes-in-url">my own fork</a> on GitHub.</p>
<p>I wanted to use my fork as a dependency when I ran my build on ReadTheDocs - but for that particular project the ReadTheDocs dependencies are installed using the equivalent of <code>pip install '.[docs]'</code> (by <a href="https://github.com/simonw/sqlite-utils/blob/45e24deffea042b5db7ab84cd1eb63b3ed9bb9da/.readthedocs.yaml#L9-L12">this configuration</a>) - and that <code>docs</code> set of extra requirements was defined in my <code>setup.py</code> file <a href="https://github.com/simonw/sqlite-utils/blob/271433fdd18e436b0a527ab899cb6f6fa67f23d0/setup.py#L33-L44">like this</a>:</p>
<div class="highlight highlight-source-python"><pre>    <span class="pl-s1">extras_require</span><span class="pl-c1">=</span>{
        <span class="pl-s">"test"</span>: [<span class="pl-s">"pytest"</span>, <span class="pl-s">"black"</span>, <span class="pl-s">"hypothesis"</span>, <span class="pl-s">"cogapp"</span>],
        <span class="pl-s">"docs"</span>: [<span class="pl-s">"furo"</span>, <span class="pl-s">"sphinx-autobuild"</span>, <span class="pl-s">"codespell"</span>, <span class="pl-s">"sphinx-copybutton"</span>],
        <span class="pl-s">"mypy"</span>: [
            <span class="pl-s">"mypy"</span>,
            <span class="pl-s">"types-click"</span>,
            <span class="pl-s">"types-tabulate"</span>,
            <span class="pl-s">"types-python-dateutil"</span>,
            <span class="pl-s">"data-science-types"</span>,
        ],
        <span class="pl-s">"flake8"</span>: [<span class="pl-s">"flake8"</span>],
    },</pre></div>
<p>Any branch on GitHub can be installed by <code>pip</code> by finding the URL to the zip export of that branch. In this case that URL is:</p>
<p><code>https://github.com/simonw/beanbag-docutils/archive/refs/heads/bytes-in-url.zip</code></p>
<p>I tried adding that into the list of strings defined for the <code>"docs"</code> bundle and got this error when I ran <code>pip install -e '.[docs]'</code>:</p>
<pre><code>Complete output (1 lines):
    error in sqlite-utils setup command: 'extras_require' must be a dictionary
    whose values are strings or lists of strings containing valid project/version requirement specifiers.
</code></pre>
<p>It turned out the solution was to use <code>packagename @ URL</code> instead, like this:</p>
<div class="highlight highlight-source-python"><pre>    <span class="pl-s1">extras_require</span><span class="pl-c1">=</span>{
        <span class="pl-s">"test"</span>: [<span class="pl-s">"pytest"</span>, <span class="pl-s">"black"</span>, <span class="pl-s">"hypothesis"</span>, <span class="pl-s">"cogapp"</span>],
        <span class="pl-s">"docs"</span>: [
            <span class="pl-s">"furo"</span>,
            <span class="pl-s">"sphinx-autobuild"</span>,
            <span class="pl-s">"codespell"</span>,
            <span class="pl-s">"sphinx-copybutton"</span>,
            <span class="pl-s">"beanbag-docutils @ https://github.com/simonw/beanbag-docutils/archive/refs/heads/bytes-in-url.zip"</span>,
        ]</pre></div>


path: googlecloud_google-cloud-spend-datasette.md
topic: googlecloud
title: Analyzing Google Cloud spend with Datasette
url: https://github.com/simonw/til/blob/main/googlecloud/google-cloud-spend-datasette.md
created: 2022-08-16T15:35:05-07:00
created_utc: 2022-08-16T22:35:05+00:00
updated: 2022-08-16T15:35:05-07:00
updated_utc: 2022-08-16T22:35:05+00:00
shot_hash: 21c3deb7f6885278922d74af20d04fc5
slug: google-cloud-spend-datasette
body:
Google Cloud provide extremely finely grained billing, but you need to access it through BigQuery which I find quite inconvenient.

You can export a dump [from BigQuery](https://console.cloud.google.com/bigquery) to your Google Drive and then download and import it into Datasette.

I started with a `SELECT *` query against the export table it had created for me:

    SELECT * FROM `datasette-222320.datasette_project_billing.gcp_billing_export_resource_v1_00C25B_639CE5_5833F9` 

I tried the CSV export first but it wasn't very easy to work with. Then I spotted this option:

<img alt="The JSONL export option, which saves up to 1GB to Google Drive" src="https://user-images.githubusercontent.com/9599/184989437-a6bbe5bb-fa7e-4090-b2f8-7ba3cef4324b.png" width="400">

This actually saved a 1.3GB newline-delimited JSON file to my Google Drive! I downloaded that to my computer

## Importing it into SQLite with sqlite-utils

`sqlite-utils` can [insert newline-delimited JSON](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-newline-delimited-json). I ran it like this:

    sqlite-utils insert /tmp/billing.db lines bq-results-20220816-213359-1660685720334.json --nl

This took a couple of minutes but gave me a 1GB SQLite file. I opened that in [Datasette Desktop](https://datasette.io/desktop).

I decided to slim it down to make it easier to work with, and to turn some of the nested JSON into regular columns so I could facet by them more easily.

Here's the eventual recipe I figured out for creating that useful subset:

```
sqlite-utils /tmp/subset.db --attach billing /tmp/billing.db '
create table items as select
  json_extract(invoice, "$.month") as month,
  cost,
  json_extract(service, "$.description") as service,
  json_extract(sku, "$.description") as sku_description,
  usage_start_time,
  usage_end_time,
  json_extract(project, "$.id") as project_id,
  json_extract(labels, "$[0].value") as service_name,
  json_extract(location, "$.location") as location,
  json_extract(resource, "$.name") as resource_name,
  currency,
  currency_conversion_rate,
  credits,
  invoice,
  cost_type,
  adjustment_info
from
  billing.lines
'
```
`/tmp/subset.db` is now a 295MB database.

## Some interesting queries

This query showed me a cost breakdown by month:
```sql
select month, count(*), sum(cost) from items group by month
```
This query showed my my most expensive Cloud Run services:

```sql
select service_name, cast('' || sum(cost) as float) as total_cost
from items group by service_name order by total_cost desc;
```

html:
<p>Google Cloud provide extremely finely grained billing, but you need to access it through BigQuery which I find quite inconvenient.</p>
<p>You can export a dump <a href="https://console.cloud.google.com/bigquery" rel="nofollow">from BigQuery</a> to your Google Drive and then download and import it into Datasette.</p>
<p>I started with a <code>SELECT *</code> query against the export table it had created for me:</p>
<pre><code>SELECT * FROM `datasette-222320.datasette_project_billing.gcp_billing_export_resource_v1_00C25B_639CE5_5833F9` 
</code></pre>
<p>I tried the CSV export first but it wasn't very easy to work with. Then I spotted this option:</p>
<p><a target="_blank" rel="noopener noreferrer" href="https://user-images.githubusercontent.com/9599/184989437-a6bbe5bb-fa7e-4090-b2f8-7ba3cef4324b.png"><img alt="The JSONL export option, which saves up to 1GB to Google Drive" src="https://user-images.githubusercontent.com/9599/184989437-a6bbe5bb-fa7e-4090-b2f8-7ba3cef4324b.png" width="400" style="max-width: 100%;"></a></p>
<p>This actually saved a 1.3GB newline-delimited JSON file to my Google Drive! I downloaded that to my computer</p>
<h2><a id="user-content-importing-it-into-sqlite-with-sqlite-utils" class="anchor" aria-hidden="true" href="#importing-it-into-sqlite-with-sqlite-utils"><span aria-hidden="true" class="octicon octicon-link"></span></a>Importing it into SQLite with sqlite-utils</h2>
<p><code>sqlite-utils</code> can <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-newline-delimited-json" rel="nofollow">insert newline-delimited JSON</a>. I ran it like this:</p>
<pre><code>sqlite-utils insert /tmp/billing.db lines bq-results-20220816-213359-1660685720334.json --nl
</code></pre>
<p>This took a couple of minutes but gave me a 1GB SQLite file. I opened that in <a href="https://datasette.io/desktop" rel="nofollow">Datasette Desktop</a>.</p>
<p>I decided to slim it down to make it easier to work with, and to turn some of the nested JSON into regular columns so I could facet by them more easily.</p>
<p>Here's the eventual recipe I figured out for creating that useful subset:</p>
<pre><code>sqlite-utils /tmp/subset.db --attach billing /tmp/billing.db '
create table items as select
  json_extract(invoice, "$.month") as month,
  cost,
  json_extract(service, "$.description") as service,
  json_extract(sku, "$.description") as sku_description,
  usage_start_time,
  usage_end_time,
  json_extract(project, "$.id") as project_id,
  json_extract(labels, "$[0].value") as service_name,
  json_extract(location, "$.location") as location,
  json_extract(resource, "$.name") as resource_name,
  currency,
  currency_conversion_rate,
  credits,
  invoice,
  cost_type,
  adjustment_info
from
  billing.lines
'
</code></pre>
<p><code>/tmp/subset.db</code> is now a 295MB database.</p>
<h2><a id="user-content-some-interesting-queries" class="anchor" aria-hidden="true" href="#some-interesting-queries"><span aria-hidden="true" class="octicon octicon-link"></span></a>Some interesting queries</h2>
<p>This query showed me a cost breakdown by month:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> month, <span class="pl-c1">count</span>(<span class="pl-k">*</span>), <span class="pl-c1">sum</span>(cost) <span class="pl-k">from</span> items <span class="pl-k">group by</span> month</pre></div>
<p>This query showed my my most expensive Cloud Run services:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> service_name, cast(<span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span> <span class="pl-k">||</span> <span class="pl-c1">sum</span>(cost) <span class="pl-k">as</span> float) <span class="pl-k">as</span> total_cost
<span class="pl-k">from</span> items <span class="pl-k">group by</span> service_name <span class="pl-k">order by</span> total_cost <span class="pl-k">desc</span>;</pre></div>


path: sqlite_sort-by-number-of-json-intersections.md
topic: sqlite
title: Sort by number of JSON intersections
url: https://github.com/simonw/til/blob/main/sqlite/sort-by-number-of-json-intersections.md
created: 2022-08-17T16:21:00-07:00
created_utc: 2022-08-17T23:21:00+00:00
updated: 2022-08-17T16:32:39-07:00
updated_utc: 2022-08-17T23:32:39+00:00
shot_hash: 5e852d9989b37614e58ea7d5534e225c
slug: sort-by-number-of-json-intersections
body:
This [post on Reddit](https://www.reddit.com/r/sqlite/comments/wr0wp0/i_have_a_sqlite_database_of_recipes_i_would_like/) asked how to run a query that takes a list of items (in this case ingredients) as the input and returns all rows with at least one of those items in a JSON list, ordered by the most matches.

I jumped on this as a great opportunity to demonstrate [Datasette Lite](https://lite.datasette.io/) as a tool for answering SQL questions.

## Creating an example database in Datasette Lite

I started with the following SQL to create the demo table:
```sql
create table recipes (id integer primary key, name text, ingredients text);
insert into recipes values (1, 'Cake', '["flour", "sugar", "eggs"]');
insert into recipes values (2, 'Pancakes', '["flour", "eggs", "butter"]');
insert into recipes values (3, 'Waffles', '["flour", "milk", "eggs"]');
insert into recipes values (4, 'Pizza', '["flour", "sugar", "eggs", "cheese"]');
```
I actually got GitHub Copilot to write most of that for me:

<img alt="In GitHub Copilot, I start with:  create table recipes (id integer primary key, name text, ingredients text);  I type &quot;insert into&quot; and it autocompletes to:  insert into recipes values (1, 'Pizza', 'Tomato Sauce, Cheese, and Pizza');  I  edit that to 'Cake' instead and it fills in &quot;flour, sugar, eggs, butter'  Then I edit that to be a JSON array instead  It completes (with a few tweaks from me) with:  insert into recipes values (2, 'Pancakes', '[&quot;flour&quot;, &quot;sugar&quot;, &quot;eggs&quot;, &quot;butter&quot;]'); insert into recipes values (3, 'Pizza', '[&quot;vegetables&quot;, &quot;cheese&quot;, &quot;tomatoes&quot;, &quot;sauce&quot;]'); insert into recipes values (4, 'Spaghetti', '[&quot;pasta&quot;, &quot;tomatoes&quot;, &quot;sauce&quot;]');" src="https://user-images.githubusercontent.com/9599/185261271-cf8aff79-67dc-4359-89a2-5f8a857e944a.gif">

I saved that [to a gist](https://gist.github.com/simonw/1f8a91123ccefd8844187225b1832d7a) and opened it in Datasette Lite with this URL:

https://lite.datasette.io/?sql=https://gist.githubusercontent.com/simonw/1f8a91123ccefd8844187225b1832d7a/raw/5069075b86aa79358fbab3d4482d1d269077d632/recipes.sql

## Solving the problem

Here's the SQL recipe I came up with to solve the question:

```sql
select id, name, ingredients, (
  select json_group_array(value) from json_each(ingredients)
  where value in (select value from json_each(:p0))
) as matching_ingredients
from recipes
where json_array_length(matching_ingredients) > 0
order by json_array_length(matching_ingredients) desc
```
[Example of that query](https://lite.datasette.io/?sql=https://gist.githubusercontent.com/simonw/1f8a91123ccefd8844187225b1832d7a/raw/5069075b86aa79358fbab3d4482d1d269077d632/recipes.sql#/data?sql=select+id%2C+name%2C+ingredients%2C+%28%0A++select+json_group_array%28value%29+from+json_each%28ingredients%29%0A++where+value+in+%28select+value+from+json_each%28%3Ap0%29%29%0A%29+as+matching_ingredients%0Afrom+recipes%0Awhere+json_array_length%28matching_ingredients%29+%3E+0%0Aorder+by+json_array_length%28matching_ingredients%29+desc&p0=%5B%22sugar%22%2C+%22cheese%22%5D) with an input for `p0` of `["sugar", "cheese"]`, which returns:

|   id | name   | ingredients                          | matching_ingredients   |
|------|--------|--------------------------------------|------------------------|
|    4 | Pizza  | ["flour", "sugar", "eggs", "cheese"] | ["sugar","cheese"]     |
|    1 | Cake   | ["flour", "sugar", "eggs"]           | ["sugar"]              |

The key trick here is the bit that creates that `matching_ingredients` column, which uses a sub-select like this:
```sql
select json_group_array(value) from json_each(ingredients)
where value in (select value from json_each(:p0))
```
`json_group_array(value)` is an aggregation function that turns the results into a JSON array.

`where value in (select value from json_each('["sugar", "cheese"]')` is the bit that calculates the intersection of the two JSON arrays.

Then at the end I use `json_array_length()` to remove rows with no overlap and then sort with the most matching ingredients first.

html:
<p>This <a href="https://www.reddit.com/r/sqlite/comments/wr0wp0/i_have_a_sqlite_database_of_recipes_i_would_like/" rel="nofollow">post on Reddit</a> asked how to run a query that takes a list of items (in this case ingredients) as the input and returns all rows with at least one of those items in a JSON list, ordered by the most matches.</p>
<p>I jumped on this as a great opportunity to demonstrate <a href="https://lite.datasette.io/" rel="nofollow">Datasette Lite</a> as a tool for answering SQL questions.</p>
<h2><a id="user-content-creating-an-example-database-in-datasette-lite" class="anchor" aria-hidden="true" href="#creating-an-example-database-in-datasette-lite"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating an example database in Datasette Lite</h2>
<p>I started with the following SQL to create the demo table:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">create</span> <span class="pl-k">table</span> <span class="pl-en">recipes</span> (id <span class="pl-k">integer</span> <span class="pl-k">primary key</span>, name <span class="pl-k">text</span>, ingredients <span class="pl-k">text</span>);
<span class="pl-k">insert into</span> recipes <span class="pl-k">values</span> (<span class="pl-c1">1</span>, <span class="pl-s"><span class="pl-pds">'</span>Cake<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>["flour", "sugar", "eggs"]<span class="pl-pds">'</span></span>);
<span class="pl-k">insert into</span> recipes <span class="pl-k">values</span> (<span class="pl-c1">2</span>, <span class="pl-s"><span class="pl-pds">'</span>Pancakes<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>["flour", "eggs", "butter"]<span class="pl-pds">'</span></span>);
<span class="pl-k">insert into</span> recipes <span class="pl-k">values</span> (<span class="pl-c1">3</span>, <span class="pl-s"><span class="pl-pds">'</span>Waffles<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>["flour", "milk", "eggs"]<span class="pl-pds">'</span></span>);
<span class="pl-k">insert into</span> recipes <span class="pl-k">values</span> (<span class="pl-c1">4</span>, <span class="pl-s"><span class="pl-pds">'</span>Pizza<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>["flour", "sugar", "eggs", "cheese"]<span class="pl-pds">'</span></span>);</pre></div>
<p>I actually got GitHub Copilot to write most of that for me:</p>
<p><a target="_blank" rel="noopener noreferrer" href="https://user-images.githubusercontent.com/9599/185261271-cf8aff79-67dc-4359-89a2-5f8a857e944a.gif"><img alt="In GitHub Copilot, I start with:  create table recipes (id integer primary key, name text, ingredients text);  I type &quot;insert into&quot; and it autocompletes to:  insert into recipes values (1, 'Pizza', 'Tomato Sauce, Cheese, and Pizza');  I  edit that to 'Cake' instead and it fills in &quot;flour, sugar, eggs, butter'  Then I edit that to be a JSON array instead  It completes (with a few tweaks from me) with:  insert into recipes values (2, 'Pancakes', '[&quot;flour&quot;, &quot;sugar&quot;, &quot;eggs&quot;, &quot;butter&quot;]'); insert into recipes values (3, 'Pizza', '[&quot;vegetables&quot;, &quot;cheese&quot;, &quot;tomatoes&quot;, &quot;sauce&quot;]'); insert into recipes values (4, 'Spaghetti', '[&quot;pasta&quot;, &quot;tomatoes&quot;, &quot;sauce&quot;]');" src="https://user-images.githubusercontent.com/9599/185261271-cf8aff79-67dc-4359-89a2-5f8a857e944a.gif" style="max-width: 100%;"></a></p>
<p>I saved that <a href="https://gist.github.com/simonw/1f8a91123ccefd8844187225b1832d7a">to a gist</a> and opened it in Datasette Lite with this URL:</p>
<p><a href="https://lite.datasette.io/?sql=https://gist.githubusercontent.com/simonw/1f8a91123ccefd8844187225b1832d7a/raw/5069075b86aa79358fbab3d4482d1d269077d632/recipes.sql" rel="nofollow">https://lite.datasette.io/?sql=https://gist.githubusercontent.com/simonw/1f8a91123ccefd8844187225b1832d7a/raw/5069075b86aa79358fbab3d4482d1d269077d632/recipes.sql</a></p>
<h2><a id="user-content-solving-the-problem" class="anchor" aria-hidden="true" href="#solving-the-problem"><span aria-hidden="true" class="octicon octicon-link"></span></a>Solving the problem</h2>
<p>Here's the SQL recipe I came up with to solve the question:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> id, name, ingredients, (
  <span class="pl-k">select</span> json_group_array(value) <span class="pl-k">from</span> json_each(ingredients)
  <span class="pl-k">where</span> value <span class="pl-k">in</span> (<span class="pl-k">select</span> value <span class="pl-k">from</span> json_each(:p0))
) <span class="pl-k">as</span> matching_ingredients
<span class="pl-k">from</span> recipes
<span class="pl-k">where</span> json_array_length(matching_ingredients) <span class="pl-k">&gt;</span> <span class="pl-c1">0</span>
<span class="pl-k">order by</span> json_array_length(matching_ingredients) <span class="pl-k">desc</span></pre></div>
<p><a href="https://lite.datasette.io/?sql=https://gist.githubusercontent.com/simonw/1f8a91123ccefd8844187225b1832d7a/raw/5069075b86aa79358fbab3d4482d1d269077d632/recipes.sql#/data?sql=select+id%2C+name%2C+ingredients%2C+%28%0A++select+json_group_array%28value%29+from+json_each%28ingredients%29%0A++where+value+in+%28select+value+from+json_each%28%3Ap0%29%29%0A%29+as+matching_ingredients%0Afrom+recipes%0Awhere+json_array_length%28matching_ingredients%29+%3E+0%0Aorder+by+json_array_length%28matching_ingredients%29+desc&amp;p0=%5B%22sugar%22%2C+%22cheese%22%5D" rel="nofollow">Example of that query</a> with an input for <code>p0</code> of <code>["sugar", "cheese"]</code>, which returns:</p>
<table>
<thead>
<tr>
<th>id</th>
<th>name</th>
<th>ingredients</th>
<th>matching_ingredients</th>
</tr>
</thead>
<tbody>
<tr>
<td>4</td>
<td>Pizza</td>
<td>["flour", "sugar", "eggs", "cheese"]</td>
<td>["sugar","cheese"]</td>
</tr>
<tr>
<td>1</td>
<td>Cake</td>
<td>["flour", "sugar", "eggs"]</td>
<td>["sugar"]</td>
</tr>
</tbody>
</table>
<p>The key trick here is the bit that creates that <code>matching_ingredients</code> column, which uses a sub-select like this:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> json_group_array(value) <span class="pl-k">from</span> json_each(ingredients)
<span class="pl-k">where</span> value <span class="pl-k">in</span> (<span class="pl-k">select</span> value <span class="pl-k">from</span> json_each(:p0))</pre></div>
<p><code>json_group_array(value)</code> is an aggregation function that turns the results into a JSON array.</p>
<p><code>where value in (select value from json_each('["sugar", "cheese"]')</code> is the bit that calculates the intersection of the two JSON arrays.</p>
<p>Then at the end I use <code>json_array_length()</code> to remove rows with no overlap and then sort with the most matching ingredients first.</p>


path: quarto_trying-out-quarto.md
topic: quarto
title: Trying out Quarto on macOS
url: https://github.com/simonw/til/blob/main/quarto/trying-out-quarto.md
created: 2022-08-18T13:03:58-07:00
created_utc: 2022-08-18T20:03:58+00:00
updated: 2022-08-18T13:10:56-07:00
updated_utc: 2022-08-18T20:10:56+00:00
shot_hash: 61343dd59ec3ac8a16992a7d54a5916d
slug: trying-out-quarto
body:
I decided to try out [Quarto](https://quarto.org/), the new notebook/markdown/publishing system built on Pandoc.

I followed their [Get started](https://quarto.org/docs/get-started/) guide and downloaded and installed the macOS installer.

Having run the installer, `quarto --help` showed that I had the CLI command installed.

## Setting up VS Code and solving ImportErrors

Next I installed the Quarto extension for VS Code by clicking the "Install" button on this page: https://marketplace.visualstudio.com/items?itemName=quarto.quarto

I started following the tutorial there, which worked... up until the point where I clicked "Render", when I got this error in the VS Code terminal:

```
ModuleNotFoundError: No module named 'nbformat'
```

This looked suspiciously to me like an [xkcd 1987](https://xkcd.com/1987/) problem: which of the many versions of Python on my computer was Quarto using? And why didn't that version have the right packages installed?

I figured out which Python was being used by adding this block to the tutorial Markdown file and executing it in VS Code:

````
```{python}
import sys
print(sys.executable)
```
````
This output:
```
/Applications/Xcode.app/Contents/Developer/usr/bin/python3
```
Now that I knew which Python was being used, I could install the `nbformat` and later the `nbclient` packages that the error messages were complaining about:

```
/Applications/Xcode.app/Contents/Developer/usr/bin/python3 -m pip install nbformat nbclient
```

Having installed those two packages into that particular Python instance, the "Render" button in VS Code worked and gave me a rendered PDF preview.

I filed a bug report about this here: https://github.com/quarto-dev/quarto-cli/issues/1972

html:
<p>I decided to try out <a href="https://quarto.org/" rel="nofollow">Quarto</a>, the new notebook/markdown/publishing system built on Pandoc.</p>
<p>I followed their <a href="https://quarto.org/docs/get-started/" rel="nofollow">Get started</a> guide and downloaded and installed the macOS installer.</p>
<p>Having run the installer, <code>quarto --help</code> showed that I had the CLI command installed.</p>
<h2><a id="user-content-setting-up-vs-code-and-solving-importerrors" class="anchor" aria-hidden="true" href="#setting-up-vs-code-and-solving-importerrors"><span aria-hidden="true" class="octicon octicon-link"></span></a>Setting up VS Code and solving ImportErrors</h2>
<p>Next I installed the Quarto extension for VS Code by clicking the "Install" button on this page: <a href="https://marketplace.visualstudio.com/items?itemName=quarto.quarto" rel="nofollow">https://marketplace.visualstudio.com/items?itemName=quarto.quarto</a></p>
<p>I started following the tutorial there, which worked... up until the point where I clicked "Render", when I got this error in the VS Code terminal:</p>
<pre><code>ModuleNotFoundError: No module named 'nbformat'
</code></pre>
<p>This looked suspiciously to me like an <a href="https://xkcd.com/1987/" rel="nofollow">xkcd 1987</a> problem: which of the many versions of Python on my computer was Quarto using? And why didn't that version have the right packages installed?</p>
<p>I figured out which Python was being used by adding this block to the tutorial Markdown file and executing it in VS Code:</p>
<pre><code>```{python}
import sys
print(sys.executable)
```
</code></pre>
<p>This output:</p>
<pre><code>/Applications/Xcode.app/Contents/Developer/usr/bin/python3
</code></pre>
<p>Now that I knew which Python was being used, I could install the <code>nbformat</code> and later the <code>nbclient</code> packages that the error messages were complaining about:</p>
<pre><code>/Applications/Xcode.app/Contents/Developer/usr/bin/python3 -m pip install nbformat nbclient
</code></pre>
<p>Having installed those two packages into that particular Python instance, the "Render" button in VS Code worked and gave me a rendered PDF preview.</p>
<p>I filed a bug report about this here: <a href="https://github.com/quarto-dev/quarto-cli/issues/1972">https://github.com/quarto-dev/quarto-cli/issues/1972</a></p>


path: sqlite_function-list.md
topic: sqlite
title: Seeing which functions are unique to a specific SQLite / Datasette instance
url: https://github.com/simonw/til/blob/main/sqlite/function-list.md
created: 2022-08-23T14:56:38-07:00
created_utc: 2022-08-23T21:56:38+00:00
updated: 2022-08-24T09:01:14-07:00
updated_utc: 2022-08-24T16:01:14+00:00
shot_hash: fd218eea099e40a31675192dc625ff44
slug: function-list
body:
In reading [Scraping JSON, HTML, and ZIP Files with Pure SQLite](https://observablehq.com/@asg017/scrape-json-html-zip-with-sqlite) by Alex Garcia I got curious to see a full list of functions he had registered in his [sqlite-extension-examples.fly.dev](https://sqlite-extension-examples.fly.dev) Datasette instance that weren't available in a regular Datasette.

Here's how I figured that out.

## pragma_function_list()

You can list all of the functions available to SQLite using `PRAGMA function_list`.

Datasette doesn't allow non-SELECT queries, but it does allow-list a set of `pragma_x()` functions which you can instead call like this:

```sql
select * from pragma_function_list()
```
[Here's that for latest.datasette.io](https://latest.datasette.io/fixtures?sql=select+*+from+pragma_function_list%28%29) (truncated):

| name                          |   builtin | type   | enc   |   narg |   flags |
|-------------------------------|-----------|--------|-------|--------|---------|
| pow                           |         1 | s      | utf8  |      2 | 2099200 |
| group_concat                  |         1 | w      | utf8  |      1 | 2097152 |
| group_concat                  |         1 | w      | utf8  |      2 | 2097152 |
| json_type                     |         1 | s      | utf8  |      1 |    2048 |
| json_type                     |         1 | s      | utf8  |      2 |    2048 |
| julianday                     |         1 | s      | utf8  |     -1 | 2099200 |

I decided to create a comma-separated list of quoted names. I used this query:

```sql
select "'" || group_concat(name, "', '") || "'" from pragma_function_list()
```
Run [against latest.datasette.io](https://latest.datasette.io/fixtures?sql=select+%22%27%22+||+group_concat(name%2C+%22%27%2C+%27%22)+||+%22%27%22+from+pragma_function_list()) this returned the following:

`'pow', 'group_concat', 'group_concat', 'json_type', 'json_type', 'julianday', 'ntile', 'nullif', 'sqlite_compileoption_get', 'json_valid', 'json_quote', 'json_patch', '->', 'json_array', 'current_timestamp', 'power', 'sqlite_compileoption_used', 'json_remove', 'json_object', 'json_insert', '->>', 'sin', 'sum', 'quote', 'printf', 'likelihood', 'json_replace', 'json_extract', 'last_value', 'rank', 'sign', 'sqrt', 'sinh', 'tan', 'round', 'round', 'rtrim', 'rtrim', 'nth_value', 'tanh', 'random', 'trim', 'trim', 'time', 'radians', 'trunc', 'total', 'substr', 'substr', 'replace', 'upper', 'subtype', 'typeof', 'load_extension', 'load_extension', 'soundex', 'json_group_array', 'avg', 'abs', 'json_group_object', 'json_array_length', 'json_array_length', 'strftime', 'atan', 'asin', 'acos', 'substring', 'substring', 'randomblob', 'unicode', 'percent_rank', 'row_number', 'atanh', 'asinh', 'acosh', 'cos', 'atan2', 'last_insert_rowid', 'sqlite_log', 'unlikely', 'cosh', 'ceil', 'char', 'unixepoch', 'exp', 'count', 'count', 'date', 'ceiling', 'total_changes', 'changes', 'sqlite_version', 'degrees', 'floor', 'coalesce', 'glob', 'zeroblob', 'hex', 'iif', 'sqlite_source_id', 'format', 'datetime', 'cume_dist', 'ln', 'instr', 'json', 'dense_rank', 'log', 'log', 'ifnull', 'current_date', 'current_time', 'lag', 'lag', 'lag', 'mod', 'log2', 'like', 'like', 'max', 'max', 'min', 'min', 'lead', 'lead', 'lead', 'log10', 'lower', 'ltrim', 'ltrim', 'first_value', 'pi', 'length', 'likely', 'json_set', 'escape_fts', 'prepare_connection_args', 'convert_units', 'sleep', 'rtreedepth', 'match', 'snippet', 'fts5_source_id', 'offsets', 'matchinfo', 'matchinfo', 'optimize', 'rtreecheck', 'rtreenode', 'highlight', 'bm25', 'fts3_tokenizer', 'fts3_tokenizer', 'fts5'`

## Comparing via copy-and-paste

To see the functions that were registered for https://sqlite-extension-examples.fly.dev/ but not for https://latest.datasette.io/ I used the above output to construct the following query:

`select name from pragma_function_list() where name not in ('pow', 'group_concat', 'group_concat', 'json_type', 'json_type', 'julianday', 'ntile', 'nullif', 'sqlite_compileoption_get', 'json_valid', 'json_quote', 'json_patch', '->', 'json_array', 'current_timestamp', 'power', 'sqlite_compileoption_used', 'json_remove', 'json_object', 'json_insert', '->>', 'sin', 'sum', 'quote', 'printf', 'likelihood', 'json_replace', 'json_extract', 'last_value', 'rank', 'sign', 'sqrt', 'sinh', 'tan', 'round', 'round', 'rtrim', 'rtrim', 'nth_value', 'tanh', 'random', 'trim', 'trim', 'time', 'radians', 'trunc', 'total', 'substr', 'substr', 'replace', 'upper', 'subtype', 'typeof', 'load_extension', 'load_extension', 'soundex', 'json_group_array', 'avg', 'abs', 'json_group_object', 'json_array_length', 'json_array_length', 'strftime', 'atan', 'asin', 'acos', 'substring', 'substring', 'randomblob', 'unicode', 'percent_rank', 'row_number', 'atanh', 'asinh', 'acosh', 'cos', 'atan2', 'last_insert_rowid', 'sqlite_log', 'unlikely', 'cosh', 'ceil', 'char', 'unixepoch', 'exp', 'count', 'count', 'date', 'ceiling', 'total_changes', 'changes', 'sqlite_version', 'degrees', 'floor', 'coalesce', 'glob', 'zeroblob', 'hex', 'iif', 'sqlite_source_id', 'format', 'datetime', 'cume_dist', 'ln', 'instr', 'json', 'dense_rank', 'log', 'log', 'ifnull', 'current_date', 'current_time', 'lag', 'lag', 'lag', 'mod', 'log2', 'like', 'like', 'max', 'max', 'min', 'min', 'lead', 'lead', 'lead', 'log10', 'lower', 'ltrim', 'ltrim', 'first_value', 'pi', 'length', 'likely', 'json_set', 'escape_fts', 'prepare_connection_args', 'convert_units', 'sleep', 'rtreedepth', 'match', 'snippet', 'fts5_source_id', 'offsets', 'matchinfo', 'matchinfo', 'optimize', 'rtreecheck', 'rtreenode', 'highlight', 'bm25', 'fts3_tokenizer', 'fts3_tokenizer', 'fts5')`

This [returned]([https://latest.datasette.io/fixtures?sql=select+%22%27%22+||+group_concat(name%2C+%22%27%2C+%27%22)+||+%22%27%22+from+pragma_function_list()](https://sqlite-extension-examples.fly.dev/data?sql=select+name+from+pragma_function_list%28%29+where+name+not+in+%28%27pow%27%2C+%27group_concat%27%2C+%27group_concat%27%2C+%27json_type%27%2C+%27json_type%27%2C+%27julianday%27%2C+%27ntile%27%2C+%27nullif%27%2C+%27sqlite_compileoption_get%27%2C+%27json_valid%27%2C+%27json_quote%27%2C+%27json_patch%27%2C+%27-%3E%27%2C+%27json_array%27%2C+%27current_timestamp%27%2C+%27power%27%2C+%27sqlite_compileoption_used%27%2C+%27json_remove%27%2C+%27json_object%27%2C+%27json_insert%27%2C+%27-%3E%3E%27%2C+%27sin%27%2C+%27sum%27%2C+%27quote%27%2C+%27printf%27%2C+%27likelihood%27%2C+%27json_replace%27%2C+%27json_extract%27%2C+%27last_value%27%2C+%27rank%27%2C+%27sign%27%2C+%27sqrt%27%2C+%27sinh%27%2C+%27tan%27%2C+%27round%27%2C+%27round%27%2C+%27rtrim%27%2C+%27rtrim%27%2C+%27nth_value%27%2C+%27tanh%27%2C+%27random%27%2C+%27trim%27%2C+%27trim%27%2C+%27time%27%2C+%27radians%27%2C+%27trunc%27%2C+%27total%27%2C+%27substr%27%2C+%27substr%27%2C+%27replace%27%2C+%27upper%27%2C+%27subtype%27%2C+%27typeof%27%2C+%27load_extension%27%2C+%27load_extension%27%2C+%27soundex%27%2C+%27json_group_array%27%2C+%27avg%27%2C+%27abs%27%2C+%27json_group_object%27%2C+%27json_array_length%27%2C+%27json_array_length%27%2C+%27strftime%27%2C+%27atan%27%2C+%27asin%27%2C+%27acos%27%2C+%27substring%27%2C+%27substring%27%2C+%27randomblob%27%2C+%27unicode%27%2C+%27percent_rank%27%2C+%27row_number%27%2C+%27atanh%27%2C+%27asinh%27%2C+%27acosh%27%2C+%27cos%27%2C+%27atan2%27%2C+%27last_insert_rowid%27%2C+%27sqlite_log%27%2C+%27unlikely%27%2C+%27cosh%27%2C+%27ceil%27%2C+%27char%27%2C+%27unixepoch%27%2C+%27exp%27%2C+%27count%27%2C+%27count%27%2C+%27date%27%2C+%27ceiling%27%2C+%27total_changes%27%2C+%27changes%27%2C+%27sqlite_version%27%2C+%27degrees%27%2C+%27floor%27%2C+%27coalesce%27%2C+%27glob%27%2C+%27zeroblob%27%2C+%27hex%27%2C+%27iif%27%2C+%27sqlite_source_id%27%2C+%27format%27%2C+%27datetime%27%2C+%27cume_dist%27%2C+%27ln%27%2C+%27instr%27%2C+%27json%27%2C+%27dense_rank%27%2C+%27log%27%2C+%27log%27%2C+%27ifnull%27%2C+%27current_date%27%2C+%27current_time%27%2C+%27lag%27%2C+%27lag%27%2C+%27lag%27%2C+%27mod%27%2C+%27log2%27%2C+%27like%27%2C+%27like%27%2C+%27max%27%2C+%27max%27%2C+%27min%27%2C+%27min%27%2C+%27lead%27%2C+%27lead%27%2C+%27lead%27%2C+%27log10%27%2C+%27lower%27%2C+%27ltrim%27%2C+%27ltrim%27%2C+%27first_value%27%2C+%27pi%27%2C+%27length%27%2C+%27likely%27%2C+%27json_set%27%2C+%27escape_fts%27%2C+%27prepare_connection_args%27%2C+%27convert_units%27%2C+%27sleep%27%2C+%27rtreedepth%27%2C+%27match%27%2C+%27snippet%27%2C+%27fts5_source_id%27%2C+%27offsets%27%2C+%27matchinfo%27%2C+%27matchinfo%27%2C+%27optimize%27%2C+%27rtreecheck%27%2C+%27rtreenode%27%2C+%27highlight%27%2C+%27bm25%27%2C+%27fts3_tokenizer%27%2C+%27fts3_tokenizer%27%2C+%27fts5%27%29)) the following list:

```
http_cookies
http_headers_date
http_headers
http_debug
html_count
html_text
html_text
html_extract
html_group_element_span
html_group_element_div
html_element
http_version
html_escape
html
fts5_rowid
fts5_decode_none
html_unescape
html_trim
fts5_decode
http_headers_get
html_attr_has
fts5_expr_tcl
html_attr_get
fts5_expr
fts5_isalnum
html_version
http_headers_has
fts5_fold
html_valid
html_table
html_debug
html_attribute_get
html_attribute_has
```

## A better solution using json_group_array() and json_each()

[Alex pointed out](https://twitter.com/agarcia_me/status/1562462457976590336) an alternative solution using SQLite's JSON functions, which is actually better because it avoids any risk of commas or quotation marks in the values breaking the resulting string.

```sql
select json_group_array(distinct name)
from pragma_function_list()
```
[Try that against latest.datasette.io](https://latest.datasette.io/_memory?sql=select+json_group_array%28distinct+name%29%0D%0Afrom+pragma_function_list%28%29)

Output:

`["pow","group_concat","json_type","julianday","ntile","nullif","sqlite_compileoption_get","json_valid","json_quote","json_patch","->","json_array","current_timestamp","power","sqlite_compileoption_used","json_remove","json_object","json_insert","->>","sin","sum","quote","printf","likelihood","json_replace","json_extract","last_value","rank","sign","sqrt","sinh","tan","round","rtrim","nth_value","tanh","random","trim","time","radians","trunc","total","substr","replace","upper","subtype","typeof","load_extension","soundex","json_group_array","avg","abs","json_group_object","json_array_length","strftime","atan","asin","acos","substring","randomblob","unicode","percent_rank","row_number","atanh","asinh","acosh","cos","atan2","last_insert_rowid","sqlite_log","unlikely","cosh","ceil","char","unixepoch","exp","count","date","ceiling","total_changes","changes","sqlite_version","degrees","floor","coalesce","glob","zeroblob","hex","iif","sqlite_source_id","format","datetime","cume_dist","ln","instr","json","dense_rank","log","ifnull","current_date","current_time","lag","mod","log2","like","max","min","lead","log10","lower","ltrim","first_value","pi","length","likely","json_set","escape_fts","prepare_connection_args","convert_units","sleep","rtreedepth","match","snippet","fts5_source_id","offsets","matchinfo","optimize","rtreecheck","rtreenode","highlight","bm25","fts3_tokenizer","fts5"]`

Then use this query and run it on the other instance:
```sql
select
  name
from
  pragma_function_list()
where
  name not in (
    select
      value
    from
      json_each(:json)
  )
```
A neat thing about this alternative is you can pass the single JSON string as a named parameter, rather than needing to paste the list of IN terms into the SQL query itself.

[Try that against sqlite-extension-examples.fly.dev](https://sqlite-extension-examples.fly.dev/data?sql=select%0D%0A++name%0D%0Afrom%0D%0A++pragma_function_list()%0D%0Awhere%0D%0A++name+not+in+(%0D%0A++++select%0D%0A++++++value%0D%0A++++from%0D%0A++++++json_each(%3Ajson)%0D%0A++)&json=[%22pow%22%2C%22group_concat%22%2C%22json_type%22%2C%22julianday%22%2C%22ntile%22%2C%22nullif%22%2C%22sqlite_compileoption_get%22%2C%22json_valid%22%2C%22json_quote%22%2C%22json_patch%22%2C%22-%3E%22%2C%22json_array%22%2C%22current_timestamp%22%2C%22power%22%2C%22sqlite_compileoption_used%22%2C%22json_remove%22%2C%22json_object%22%2C%22json_insert%22%2C%22-%3E%3E%22%2C%22sin%22%2C%22sum%22%2C%22quote%22%2C%22printf%22%2C%22likelihood%22%2C%22json_replace%22%2C%22json_extract%22%2C%22last_value%22%2C%22rank%22%2C%22sign%22%2C%22sqrt%22%2C%22sinh%22%2C%22tan%22%2C%22round%22%2C%22rtrim%22%2C%22nth_value%22%2C%22tanh%22%2C%22random%22%2C%22trim%22%2C%22time%22%2C%22radians%22%2C%22trunc%22%2C%22total%22%2C%22substr%22%2C%22replace%22%2C%22upper%22%2C%22subtype%22%2C%22typeof%22%2C%22load_extension%22%2C%22soundex%22%2C%22json_group_array%22%2C%22avg%22%2C%22abs%22%2C%22json_group_object%22%2C%22json_array_length%22%2C%22strftime%22%2C%22atan%22%2C%22asin%22%2C%22acos%22%2C%22substring%22%2C%22randomblob%22%2C%22unicode%22%2C%22percent_rank%22%2C%22row_number%22%2C%22atanh%22%2C%22asinh%22%2C%22acosh%22%2C%22cos%22%2C%22atan2%22%2C%22last_insert_rowid%22%2C%22sqlite_log%22%2C%22unlikely%22%2C%22cosh%22%2C%22ceil%22%2C%22char%22%2C%22unixepoch%22%2C%22exp%22%2C%22count%22%2C%22date%22%2C%22ceiling%22%2C%22total_changes%22%2C%22changes%22%2C%22sqlite_version%22%2C%22degrees%22%2C%22floor%22%2C%22coalesce%22%2C%22glob%22%2C%22zeroblob%22%2C%22hex%22%2C%22iif%22%2C%22sqlite_source_id%22%2C%22format%22%2C%22datetime%22%2C%22cume_dist%22%2C%22ln%22%2C%22instr%22%2C%22json%22%2C%22dense_rank%22%2C%22log%22%2C%22ifnull%22%2C%22current_date%22%2C%22current_time%22%2C%22lag%22%2C%22mod%22%2C%22log2%22%2C%22like%22%2C%22max%22%2C%22min%22%2C%22lead%22%2C%22log10%22%2C%22lower%22%2C%22ltrim%22%2C%22first_value%22%2C%22pi%22%2C%22length%22%2C%22likely%22%2C%22json_set%22%2C%22escape_fts%22%2C%22prepare_connection_args%22%2C%22convert_units%22%2C%22sleep%22%2C%22rtreedepth%22%2C%22match%22%2C%22snippet%22%2C%22fts5_source_id%22%2C%22offsets%22%2C%22matchinfo%22%2C%22optimize%22%2C%22rtreecheck%22%2C%22rtreenode%22%2C%22highlight%22%2C%22bm25%22%2C%22fts3_tokenizer%22%2C%22fts5%22]).

html:
<p>In reading <a href="https://observablehq.com/@asg017/scrape-json-html-zip-with-sqlite" rel="nofollow">Scraping JSON, HTML, and ZIP Files with Pure SQLite</a> by Alex Garcia I got curious to see a full list of functions he had registered in his <a href="https://sqlite-extension-examples.fly.dev" rel="nofollow">sqlite-extension-examples.fly.dev</a> Datasette instance that weren't available in a regular Datasette.</p>
<p>Here's how I figured that out.</p>
<h2><a id="user-content-pragma_function_list" class="anchor" aria-hidden="true" href="#pragma_function_list"><span aria-hidden="true" class="octicon octicon-link"></span></a>pragma_function_list()</h2>
<p>You can list all of the functions available to SQLite using <code>PRAGMA function_list</code>.</p>
<p>Datasette doesn't allow non-SELECT queries, but it does allow-list a set of <code>pragma_x()</code> functions which you can instead call like this:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_function_list()</pre></div>
<p><a href="https://latest.datasette.io/fixtures?sql=select+*+from+pragma_function_list%28%29" rel="nofollow">Here's that for latest.datasette.io</a> (truncated):</p>
<table>
<thead>
<tr>
<th>name</th>
<th>builtin</th>
<th>type</th>
<th>enc</th>
<th>narg</th>
<th>flags</th>
</tr>
</thead>
<tbody>
<tr>
<td>pow</td>
<td>1</td>
<td>s</td>
<td>utf8</td>
<td>2</td>
<td>2099200</td>
</tr>
<tr>
<td>group_concat</td>
<td>1</td>
<td>w</td>
<td>utf8</td>
<td>1</td>
<td>2097152</td>
</tr>
<tr>
<td>group_concat</td>
<td>1</td>
<td>w</td>
<td>utf8</td>
<td>2</td>
<td>2097152</td>
</tr>
<tr>
<td>json_type</td>
<td>1</td>
<td>s</td>
<td>utf8</td>
<td>1</td>
<td>2048</td>
</tr>
<tr>
<td>json_type</td>
<td>1</td>
<td>s</td>
<td>utf8</td>
<td>2</td>
<td>2048</td>
</tr>
<tr>
<td>julianday</td>
<td>1</td>
<td>s</td>
<td>utf8</td>
<td>-1</td>
<td>2099200</td>
</tr>
</tbody>
</table>
<p>I decided to create a comma-separated list of quoted names. I used this query:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">"</span>'<span class="pl-pds">"</span></span> <span class="pl-k">||</span> group_concat(name, <span class="pl-s"><span class="pl-pds">"</span>', '<span class="pl-pds">"</span></span>) <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">"</span>'<span class="pl-pds">"</span></span> <span class="pl-k">from</span> pragma_function_list()</pre></div>
<p>Run <a href="https://latest.datasette.io/fixtures?sql=select+%22%27%22+%7C%7C+group_concat(name%2C+%22%27%2C+%27%22)+%7C%7C+%22%27%22+from+pragma_function_list()" rel="nofollow">against latest.datasette.io</a> this returned the following:</p>
<p><code>'pow', 'group_concat', 'group_concat', 'json_type', 'json_type', 'julianday', 'ntile', 'nullif', 'sqlite_compileoption_get', 'json_valid', 'json_quote', 'json_patch', '-&gt;', 'json_array', 'current_timestamp', 'power', 'sqlite_compileoption_used', 'json_remove', 'json_object', 'json_insert', '-&gt;&gt;', 'sin', 'sum', 'quote', 'printf', 'likelihood', 'json_replace', 'json_extract', 'last_value', 'rank', 'sign', 'sqrt', 'sinh', 'tan', 'round', 'round', 'rtrim', 'rtrim', 'nth_value', 'tanh', 'random', 'trim', 'trim', 'time', 'radians', 'trunc', 'total', 'substr', 'substr', 'replace', 'upper', 'subtype', 'typeof', 'load_extension', 'load_extension', 'soundex', 'json_group_array', 'avg', 'abs', 'json_group_object', 'json_array_length', 'json_array_length', 'strftime', 'atan', 'asin', 'acos', 'substring', 'substring', 'randomblob', 'unicode', 'percent_rank', 'row_number', 'atanh', 'asinh', 'acosh', 'cos', 'atan2', 'last_insert_rowid', 'sqlite_log', 'unlikely', 'cosh', 'ceil', 'char', 'unixepoch', 'exp', 'count', 'count', 'date', 'ceiling', 'total_changes', 'changes', 'sqlite_version', 'degrees', 'floor', 'coalesce', 'glob', 'zeroblob', 'hex', 'iif', 'sqlite_source_id', 'format', 'datetime', 'cume_dist', 'ln', 'instr', 'json', 'dense_rank', 'log', 'log', 'ifnull', 'current_date', 'current_time', 'lag', 'lag', 'lag', 'mod', 'log2', 'like', 'like', 'max', 'max', 'min', 'min', 'lead', 'lead', 'lead', 'log10', 'lower', 'ltrim', 'ltrim', 'first_value', 'pi', 'length', 'likely', 'json_set', 'escape_fts', 'prepare_connection_args', 'convert_units', 'sleep', 'rtreedepth', 'match', 'snippet', 'fts5_source_id', 'offsets', 'matchinfo', 'matchinfo', 'optimize', 'rtreecheck', 'rtreenode', 'highlight', 'bm25', 'fts3_tokenizer', 'fts3_tokenizer', 'fts5'</code></p>
<h2><a id="user-content-comparing-via-copy-and-paste" class="anchor" aria-hidden="true" href="#comparing-via-copy-and-paste"><span aria-hidden="true" class="octicon octicon-link"></span></a>Comparing via copy-and-paste</h2>
<p>To see the functions that were registered for <a href="https://sqlite-extension-examples.fly.dev/" rel="nofollow">https://sqlite-extension-examples.fly.dev/</a> but not for <a href="https://latest.datasette.io/" rel="nofollow">https://latest.datasette.io/</a> I used the above output to construct the following query:</p>
<p><code>select name from pragma_function_list() where name not in ('pow', 'group_concat', 'group_concat', 'json_type', 'json_type', 'julianday', 'ntile', 'nullif', 'sqlite_compileoption_get', 'json_valid', 'json_quote', 'json_patch', '-&gt;', 'json_array', 'current_timestamp', 'power', 'sqlite_compileoption_used', 'json_remove', 'json_object', 'json_insert', '-&gt;&gt;', 'sin', 'sum', 'quote', 'printf', 'likelihood', 'json_replace', 'json_extract', 'last_value', 'rank', 'sign', 'sqrt', 'sinh', 'tan', 'round', 'round', 'rtrim', 'rtrim', 'nth_value', 'tanh', 'random', 'trim', 'trim', 'time', 'radians', 'trunc', 'total', 'substr', 'substr', 'replace', 'upper', 'subtype', 'typeof', 'load_extension', 'load_extension', 'soundex', 'json_group_array', 'avg', 'abs', 'json_group_object', 'json_array_length', 'json_array_length', 'strftime', 'atan', 'asin', 'acos', 'substring', 'substring', 'randomblob', 'unicode', 'percent_rank', 'row_number', 'atanh', 'asinh', 'acosh', 'cos', 'atan2', 'last_insert_rowid', 'sqlite_log', 'unlikely', 'cosh', 'ceil', 'char', 'unixepoch', 'exp', 'count', 'count', 'date', 'ceiling', 'total_changes', 'changes', 'sqlite_version', 'degrees', 'floor', 'coalesce', 'glob', 'zeroblob', 'hex', 'iif', 'sqlite_source_id', 'format', 'datetime', 'cume_dist', 'ln', 'instr', 'json', 'dense_rank', 'log', 'log', 'ifnull', 'current_date', 'current_time', 'lag', 'lag', 'lag', 'mod', 'log2', 'like', 'like', 'max', 'max', 'min', 'min', 'lead', 'lead', 'lead', 'log10', 'lower', 'ltrim', 'ltrim', 'first_value', 'pi', 'length', 'likely', 'json_set', 'escape_fts', 'prepare_connection_args', 'convert_units', 'sleep', 'rtreedepth', 'match', 'snippet', 'fts5_source_id', 'offsets', 'matchinfo', 'matchinfo', 'optimize', 'rtreecheck', 'rtreenode', 'highlight', 'bm25', 'fts3_tokenizer', 'fts3_tokenizer', 'fts5')</code></p>
<p>This returned the following list:</p>
<pre><code>http_cookies
http_headers_date
http_headers
http_debug
html_count
html_text
html_text
html_extract
html_group_element_span
html_group_element_div
html_element
http_version
html_escape
html
fts5_rowid
fts5_decode_none
html_unescape
html_trim
fts5_decode
http_headers_get
html_attr_has
fts5_expr_tcl
html_attr_get
fts5_expr
fts5_isalnum
html_version
http_headers_has
fts5_fold
html_valid
html_table
html_debug
html_attribute_get
html_attribute_has
</code></pre>
<h2><a id="user-content-a-better-solution-using-json_group_array-and-json_each" class="anchor" aria-hidden="true" href="#a-better-solution-using-json_group_array-and-json_each"><span aria-hidden="true" class="octicon octicon-link"></span></a>A better solution using json_group_array() and json_each()</h2>
<p><a href="https://twitter.com/agarcia_me/status/1562462457976590336" rel="nofollow">Alex pointed out</a> an alternative solution using SQLite's JSON functions, which is actually better because it avoids any risk of commas or quotation marks in the values breaking the resulting string.</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> json_group_array(distinct name)
<span class="pl-k">from</span> pragma_function_list()</pre></div>
<p><a href="https://latest.datasette.io/_memory?sql=select+json_group_array%28distinct+name%29%0D%0Afrom+pragma_function_list%28%29" rel="nofollow">Try that against latest.datasette.io</a></p>
<p>Output:</p>
<p><code>["pow","group_concat","json_type","julianday","ntile","nullif","sqlite_compileoption_get","json_valid","json_quote","json_patch","-&gt;","json_array","current_timestamp","power","sqlite_compileoption_used","json_remove","json_object","json_insert","-&gt;&gt;","sin","sum","quote","printf","likelihood","json_replace","json_extract","last_value","rank","sign","sqrt","sinh","tan","round","rtrim","nth_value","tanh","random","trim","time","radians","trunc","total","substr","replace","upper","subtype","typeof","load_extension","soundex","json_group_array","avg","abs","json_group_object","json_array_length","strftime","atan","asin","acos","substring","randomblob","unicode","percent_rank","row_number","atanh","asinh","acosh","cos","atan2","last_insert_rowid","sqlite_log","unlikely","cosh","ceil","char","unixepoch","exp","count","date","ceiling","total_changes","changes","sqlite_version","degrees","floor","coalesce","glob","zeroblob","hex","iif","sqlite_source_id","format","datetime","cume_dist","ln","instr","json","dense_rank","log","ifnull","current_date","current_time","lag","mod","log2","like","max","min","lead","log10","lower","ltrim","first_value","pi","length","likely","json_set","escape_fts","prepare_connection_args","convert_units","sleep","rtreedepth","match","snippet","fts5_source_id","offsets","matchinfo","optimize","rtreecheck","rtreenode","highlight","bm25","fts3_tokenizer","fts5"]</code></p>
<p>Then use this query and run it on the other instance:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  name
<span class="pl-k">from</span>
  pragma_function_list()
<span class="pl-k">where</span>
  name not <span class="pl-k">in</span> (
    <span class="pl-k">select</span>
      value
    <span class="pl-k">from</span>
      json_each(:json)
  )</pre></div>
<p>A neat thing about this alternative is you can pass the single JSON string as a named parameter, rather than needing to paste the list of IN terms into the SQL query itself.</p>
<p><a href="https://sqlite-extension-examples.fly.dev/data?sql=select%0D%0A++name%0D%0Afrom%0D%0A++pragma_function_list()%0D%0Awhere%0D%0A++name+not+in+(%0D%0A++++select%0D%0A++++++value%0D%0A++++from%0D%0A++++++json_each(%3Ajson)%0D%0A++)&amp;json=%5B%22pow%22%2C%22group_concat%22%2C%22json_type%22%2C%22julianday%22%2C%22ntile%22%2C%22nullif%22%2C%22sqlite_compileoption_get%22%2C%22json_valid%22%2C%22json_quote%22%2C%22json_patch%22%2C%22-%3E%22%2C%22json_array%22%2C%22current_timestamp%22%2C%22power%22%2C%22sqlite_compileoption_used%22%2C%22json_remove%22%2C%22json_object%22%2C%22json_insert%22%2C%22-%3E%3E%22%2C%22sin%22%2C%22sum%22%2C%22quote%22%2C%22printf%22%2C%22likelihood%22%2C%22json_replace%22%2C%22json_extract%22%2C%22last_value%22%2C%22rank%22%2C%22sign%22%2C%22sqrt%22%2C%22sinh%22%2C%22tan%22%2C%22round%22%2C%22rtrim%22%2C%22nth_value%22%2C%22tanh%22%2C%22random%22%2C%22trim%22%2C%22time%22%2C%22radians%22%2C%22trunc%22%2C%22total%22%2C%22substr%22%2C%22replace%22%2C%22upper%22%2C%22subtype%22%2C%22typeof%22%2C%22load_extension%22%2C%22soundex%22%2C%22json_group_array%22%2C%22avg%22%2C%22abs%22%2C%22json_group_object%22%2C%22json_array_length%22%2C%22strftime%22%2C%22atan%22%2C%22asin%22%2C%22acos%22%2C%22substring%22%2C%22randomblob%22%2C%22unicode%22%2C%22percent_rank%22%2C%22row_number%22%2C%22atanh%22%2C%22asinh%22%2C%22acosh%22%2C%22cos%22%2C%22atan2%22%2C%22last_insert_rowid%22%2C%22sqlite_log%22%2C%22unlikely%22%2C%22cosh%22%2C%22ceil%22%2C%22char%22%2C%22unixepoch%22%2C%22exp%22%2C%22count%22%2C%22date%22%2C%22ceiling%22%2C%22total_changes%22%2C%22changes%22%2C%22sqlite_version%22%2C%22degrees%22%2C%22floor%22%2C%22coalesce%22%2C%22glob%22%2C%22zeroblob%22%2C%22hex%22%2C%22iif%22%2C%22sqlite_source_id%22%2C%22format%22%2C%22datetime%22%2C%22cume_dist%22%2C%22ln%22%2C%22instr%22%2C%22json%22%2C%22dense_rank%22%2C%22log%22%2C%22ifnull%22%2C%22current_date%22%2C%22current_time%22%2C%22lag%22%2C%22mod%22%2C%22log2%22%2C%22like%22%2C%22max%22%2C%22min%22%2C%22lead%22%2C%22log10%22%2C%22lower%22%2C%22ltrim%22%2C%22first_value%22%2C%22pi%22%2C%22length%22%2C%22likely%22%2C%22json_set%22%2C%22escape_fts%22%2C%22prepare_connection_args%22%2C%22convert_units%22%2C%22sleep%22%2C%22rtreedepth%22%2C%22match%22%2C%22snippet%22%2C%22fts5_source_id%22%2C%22offsets%22%2C%22matchinfo%22%2C%22optimize%22%2C%22rtreecheck%22%2C%22rtreenode%22%2C%22highlight%22%2C%22bm25%22%2C%22fts3_tokenizer%22%2C%22fts5%22%5D" rel="nofollow">Try that against sqlite-extension-examples.fly.dev</a>.</p>


path: sqlite_vacum-disk-full.md
topic: sqlite
title: SQLite VACUUM: database or disk is full
url: https://github.com/simonw/til/blob/main/sqlite/vacum-disk-full.md
created: 2022-08-29T11:15:03-07:00
created_utc: 2022-08-29T18:15:03+00:00
updated: 2022-08-29T11:15:03-07:00
updated_utc: 2022-08-29T18:15:03+00:00
shot_hash: b7aaae8a8ef03b1f4865a57438ec7c64
slug: vacum-disk-full
body:
I was trying to run `VACUUM` against a large SQLite database file (~7GB) using `sqlite-utils vacuum data.db` and I got this error:

    sqlite3.OperationalError: database or disk is full

The `/data` volume that the database lived in was 20GB in size, so there should have been enough room to run the operation.

I realized that this was because VACUUM uses the `/tmp` directory, and on this machine that was on a separate volume that did not have enough space.

The fix was to set the `SQLITE_TMPDIR` environment variable to the current directory (or any directory on a volume with enough space):

    SQLITE_TMPDIR=/data sqlite-utils vacuum data.db

html:
<p>I was trying to run <code>VACUUM</code> against a large SQLite database file (~7GB) using <code>sqlite-utils vacuum data.db</code> and I got this error:</p>
<pre><code>sqlite3.OperationalError: database or disk is full
</code></pre>
<p>The <code>/data</code> volume that the database lived in was 20GB in size, so there should have been enough room to run the operation.</p>
<p>I realized that this was because VACUUM uses the <code>/tmp</code> directory, and on this machine that was on a separate volume that did not have enough space.</p>
<p>The fix was to set the <code>SQLITE_TMPDIR</code> environment variable to the current directory (or any directory on a volume with enough space):</p>
<pre><code>SQLITE_TMPDIR=/data sqlite-utils vacuum data.db
</code></pre>


path: fly_scp.md
topic: fly
title: How to scp files to and from Fly
url: https://github.com/simonw/til/blob/main/fly/scp.md
created: 2022-09-02T15:39:04-07:00
created_utc: 2022-09-02T22:39:04+00:00
updated: 2022-09-02T15:39:04-07:00
updated_utc: 2022-09-02T22:39:04+00:00
shot_hash: 6f6d33491b25a9e67f82d62fc61ed9c9
slug: scp
body:
I have a Fly instance with a 20GB volume, and I wanted to copy files to and from the instance from my computer using `scp`.

Here's the process that worked for me.

1. Connect to Fly's WireGuard network. Fly have [step by step instructions](https://fly.io/docs/reference/private-networking/#step-by-step) for this - you need to install a WireGuard app (I used the [official WireGuard macOS app](https://www.wireguard.com/install/)) and use the `fly wireguard create` command to configure it.
2. Generate 24 hour limited SSH credentials for your Fly organization: Run `fly ssh issue`, follow the prompt to select your organization and then tell it where to put the credentials. I saved them to `/tmp/fly` since they will only work for 24 hours.
3. Find the IPv6 private address for the instance you want to connect to. My instance is in the `laion-aesthetic` application so I did this by running: `fly ips private -a laion-aesthetic`
4. If the image you used to build the instance doesn't have `scp` installed you'll need to install it. On Ubuntu or Debian machines you can do that by attaching using `fly ssh console -a name-of-app` and then running `apt-get update && install openssh-client -y`. Any time you restart the container you'll have to run this step again, so if you're going to do it often you should instead update the image you are using to include this package.
6. Run the `scp` like this: `scp -i /tmp/fly root@\[fdaa:0:4ef:a7b:ad0:1:9c23:2\]:/data/data.db /tmp` - note how the IPv6 address is enclosed in `\[...\]`.

html:
<p>I have a Fly instance with a 20GB volume, and I wanted to copy files to and from the instance from my computer using <code>scp</code>.</p>
<p>Here's the process that worked for me.</p>
<ol>
<li>Connect to Fly's WireGuard network. Fly have <a href="https://fly.io/docs/reference/private-networking/#step-by-step" rel="nofollow">step by step instructions</a> for this - you need to install a WireGuard app (I used the <a href="https://www.wireguard.com/install/" rel="nofollow">official WireGuard macOS app</a>) and use the <code>fly wireguard create</code> command to configure it.</li>
<li>Generate 24 hour limited SSH credentials for your Fly organization: Run <code>fly ssh issue</code>, follow the prompt to select your organization and then tell it where to put the credentials. I saved them to <code>/tmp/fly</code> since they will only work for 24 hours.</li>
<li>Find the IPv6 private address for the instance you want to connect to. My instance is in the <code>laion-aesthetic</code> application so I did this by running: <code>fly ips private -a laion-aesthetic</code>
</li>
<li>If the image you used to build the instance doesn't have <code>scp</code> installed you'll need to install it. On Ubuntu or Debian machines you can do that by attaching using <code>fly ssh console -a name-of-app</code> and then running <code>apt-get update &amp;&amp; install openssh-client -y</code>. Any time you restart the container you'll have to run this step again, so if you're going to do it often you should instead update the image you are using to include this package.</li>
<li>Run the <code>scp</code> like this: <code>scp -i /tmp/fly root@\[fdaa:0:4ef:a7b:ad0:1:9c23:2\]:/data/data.db /tmp</code> - note how the IPv6 address is enclosed in <code>\[...\]</code>.</li>
</ol>


path: twitter_birdwatch-sqlite.md
topic: twitter
title: Loading Twitter Birdwatch into SQLite for analysis with Datasette
url: https://github.com/simonw/til/blob/main/twitter/birdwatch-sqlite.md
created: 2022-09-03T08:37:05-07:00
created_utc: 2022-09-03T15:37:05+00:00
updated: 2022-09-03T09:18:41-07:00
updated_utc: 2022-09-03T16:18:41+00:00
shot_hash: 273d0ddb3a5a328ec207382d607ac8ba
slug: birdwatch-sqlite
body:
[Twitter Birdwatch](https://twitter.github.io/birdwatch/) is "a collaborative way to add helpful context to Tweets and keep people better informed".

Data collected by the program is [made available for download](https://twitter.github.io/birdwatch/download-data/) as a trio of TSV files.

You can obtain those files from [this page](https://twitter.com/i/birdwatch/download-data) (Twitter login required). The files are:

- `notes-0000.tsv`
- `ratings-0000.tsv`
- `noteStatusHistory-0000.tsv`

As far as I can tell they only include notes and ratings from the past 48 hours. This means the files are quite small - when I downloaded them on 3rd September 2022 they were:

```
2.7M  noteStatusHistory-00000.tsv
16M   notes-00000.tsv
45M   ratings-00000.tsv
```

## Import into SQLite

I used [sqlite-utils insert](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data) to insert the data into a `birdwatch.db` SQLite database:

```
sqlite-utils insert birdwatch.db notes notes-00000.tsv --tsv --detect-types
sqlite-utils insert birdwatch.db ratings ratings-00000.tsv --tsv --detect-types
sqlite-utils insert birdwatch.db noteStatusHistory noteStatusHistory-00000.tsv --tsv --detect-types
```
The `--detect-types` option ensures we get a mix of `integer` and `text` columns - without this we would just get `text`.

Having run this, the schema of the resulting database file looks like this:

```
sqlite-utils schema birdwatch.db
```
```sql
CREATE TABLE "notes" (
   [noteId] INTEGER,
   [participantId] TEXT,
   [createdAtMillis] INTEGER,
   [tweetId] INTEGER,
   [classification] TEXT,
   [believable] TEXT,
   [harmful] TEXT,
   [validationDifficulty] TEXT,
   [misleadingOther] INTEGER,
   [misleadingFactualError] INTEGER,
   [misleadingManipulatedMedia] INTEGER,
   [misleadingOutdatedInformation] INTEGER,
   [misleadingMissingImportantContext] INTEGER,
   [misleadingUnverifiedClaimAsFact] INTEGER,
   [misleadingSatire] INTEGER,
   [notMisleadingOther] INTEGER,
   [notMisleadingFactuallyCorrect] INTEGER,
   [notMisleadingOutdatedButNotWhenWritten] INTEGER,
   [notMisleadingClearlySatire] INTEGER,
   [notMisleadingPersonalOpinion] INTEGER,
   [trustworthySources] INTEGER,
   [summary] TEXT
);
CREATE TABLE "ratings" (
   [noteId] INTEGER,
   [participantId] TEXT,
   [createdAtMillis] INTEGER,
   [version] INTEGER,
   [agree] INTEGER,
   [disagree] INTEGER,
   [helpful] INTEGER,
   [notHelpful] INTEGER,
   [helpfulnessLevel] TEXT,
   [helpfulOther] INTEGER,
   [helpfulInformative] INTEGER,
   [helpfulClear] INTEGER,
   [helpfulEmpathetic] INTEGER,
   [helpfulGoodSources] INTEGER,
   [helpfulUniqueContext] INTEGER,
   [helpfulAddressesClaim] INTEGER,
   [helpfulImportantContext] INTEGER,
   [helpfulUnbiasedLanguage] INTEGER,
   [notHelpfulOther] INTEGER,
   [notHelpfulIncorrect] INTEGER,
   [notHelpfulSourcesMissingOrUnreliable] INTEGER,
   [notHelpfulOpinionSpeculationOrBias] INTEGER,
   [notHelpfulMissingKeyPoints] INTEGER,
   [notHelpfulOutdated] INTEGER,
   [notHelpfulHardToUnderstand] INTEGER,
   [notHelpfulArgumentativeOrBiased] INTEGER,
   [notHelpfulOffTopic] INTEGER,
   [notHelpfulSpamHarassmentOrAbuse] INTEGER,
   [notHelpfulIrrelevantSources] INTEGER,
   [notHelpfulOpinionSpeculation] INTEGER,
   [notHelpfulNoteNotNeeded] INTEGER
);
CREATE TABLE "noteStatusHistory" (
   [noteId] INTEGER,
   [participantId] TEXT,
   [createdAtMillis] INTEGER,
   [timestampMillisOfFirstNonNMRStatus] INTEGER,
   [firstNonNMRStatus] TEXT,
   [timestampMillisOfCurrentStatus] INTEGER,
   [currentStatus] TEXT,
   [timestampMillisOfMostRecentNonNMRStatus] INTEGER,
   [mostRecentNonNMRStatus] TEXT
);
```

## Enabling search

The `summary` column in the `notes` table is the most interesting with respect to search. We can enable SQLite full-text search on it like this:

```
sqlite-utils enable-its birdwatch.db notes summary
```

## Downloading the full tweets

The Birdwatch data includes plenty of tweet IDs - in the `tweetId` column in the `notes` table - but it doesn't include the full details of those tweets.

If you have old credentials for v1 of the Twitter API you can use [twitter-to-sqlite](https://datasette.io/tools/twitter-to-sqlite) to download the full details of those tweets like this:

    twitter-to-sqlite statuses-lookup birdwatch.db --sql 'select distinct tweetId from notes'

This fetches full tweets (and authors and attachments and suchlike) for every tweet with an ID matching one from the SQL query `select distinct tweetId from notes`.

This command shows a progress bar while it works:

    Importing 27,909 tweets  [#####-------------------]   17%  00:10:26

If you don't have Twitter APIv1 credentials consider using [twarc](https://twarc-project.readthedocs.io/) instead (maybe with [this plugin](https://github.com/DocNow/twarc-csv)).

The final database schema looks like this:

```sql
CREATE TABLE "notes" (
   [noteId] INTEGER,
   [participantId] TEXT,
   [createdAtMillis] INTEGER,
   [tweetId] INTEGER,
   [classification] TEXT,
   [believable] TEXT,
   [harmful] TEXT,
   [validationDifficulty] TEXT,
   [misleadingOther] INTEGER,
   [misleadingFactualError] INTEGER,
   [misleadingManipulatedMedia] INTEGER,
   [misleadingOutdatedInformation] INTEGER,
   [misleadingMissingImportantContext] INTEGER,
   [misleadingUnverifiedClaimAsFact] INTEGER,
   [misleadingSatire] INTEGER,
   [notMisleadingOther] INTEGER,
   [notMisleadingFactuallyCorrect] INTEGER,
   [notMisleadingOutdatedButNotWhenWritten] INTEGER,
   [notMisleadingClearlySatire] INTEGER,
   [notMisleadingPersonalOpinion] INTEGER,
   [trustworthySources] INTEGER,
   [summary] TEXT
);
CREATE TABLE "ratings" (
   [noteId] INTEGER,
   [participantId] TEXT,
   [createdAtMillis] INTEGER,
   [version] INTEGER,
   [agree] INTEGER,
   [disagree] INTEGER,
   [helpful] INTEGER,
   [notHelpful] INTEGER,
   [helpfulnessLevel] TEXT,
   [helpfulOther] INTEGER,
   [helpfulInformative] INTEGER,
   [helpfulClear] INTEGER,
   [helpfulEmpathetic] INTEGER,
   [helpfulGoodSources] INTEGER,
   [helpfulUniqueContext] INTEGER,
   [helpfulAddressesClaim] INTEGER,
   [helpfulImportantContext] INTEGER,
   [helpfulUnbiasedLanguage] INTEGER,
   [notHelpfulOther] INTEGER,
   [notHelpfulIncorrect] INTEGER,
   [notHelpfulSourcesMissingOrUnreliable] INTEGER,
   [notHelpfulOpinionSpeculationOrBias] INTEGER,
   [notHelpfulMissingKeyPoints] INTEGER,
   [notHelpfulOutdated] INTEGER,
   [notHelpfulHardToUnderstand] INTEGER,
   [notHelpfulArgumentativeOrBiased] INTEGER,
   [notHelpfulOffTopic] INTEGER,
   [notHelpfulSpamHarassmentOrAbuse] INTEGER,
   [notHelpfulIrrelevantSources] INTEGER,
   [notHelpfulOpinionSpeculation] INTEGER,
   [notHelpfulNoteNotNeeded] INTEGER
);
CREATE TABLE "noteStatusHistory" (
   [noteId] INTEGER,
   [participantId] TEXT,
   [createdAtMillis] INTEGER,
   [timestampMillisOfFirstNonNMRStatus] INTEGER,
   [firstNonNMRStatus] TEXT,
   [timestampMillisOfCurrentStatus] INTEGER,
   [currentStatus] TEXT,
   [timestampMillisOfMostRecentNonNMRStatus] INTEGER,
   [mostRecentNonNMRStatus] TEXT
);
CREATE TABLE 'notes_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'notes_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE 'notes_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE 'notes_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
CREATE TABLE [migrations] (
   [name] TEXT PRIMARY KEY,
   [applied] TEXT
);
CREATE TABLE [places] (
   [id] TEXT PRIMARY KEY
, [url] TEXT, [place_type] TEXT, [name] TEXT, [full_name] TEXT, [country_code] TEXT, [country] TEXT, [contained_within] TEXT, [bounding_box] TEXT, [attributes] TEXT);
CREATE TABLE [sources] (
   [id] TEXT PRIMARY KEY,
   [name] TEXT,
   [url] TEXT
);
CREATE TABLE [users] (
   [id] INTEGER PRIMARY KEY,
   [screen_name] TEXT,
   [name] TEXT,
   [description] TEXT,
   [location] TEXT
, [url] TEXT, [protected] INTEGER, [followers_count] INTEGER, [friends_count] INTEGER, [listed_count] INTEGER, [created_at] TEXT, [favourites_count] INTEGER, [utc_offset] TEXT, [time_zone] TEXT, [geo_enabled] INTEGER, [verified] INTEGER, [statuses_count] INTEGER, [lang] TEXT, [contributors_enabled] INTEGER, [is_translator] INTEGER, [is_translation_enabled] INTEGER, [profile_background_color] TEXT, [profile_background_image_url] TEXT, [profile_background_image_url_https] TEXT, [profile_background_tile] INTEGER, [profile_image_url] TEXT, [profile_image_url_https] TEXT, [profile_banner_url] TEXT, [profile_link_color] TEXT, [profile_sidebar_border_color] TEXT, [profile_sidebar_fill_color] TEXT, [profile_text_color] TEXT, [profile_use_background_image] INTEGER, [has_extended_profile] INTEGER, [default_profile] INTEGER, [default_profile_image] INTEGER, [following] INTEGER, [follow_request_sent] INTEGER, [notifications] INTEGER, [translator_type] TEXT, [withheld_in_countries] TEXT);
CREATE TABLE 'users_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'users_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE 'users_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE 'users_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
CREATE TABLE [tweets] (
   [id] INTEGER PRIMARY KEY,
   [user] INTEGER REFERENCES [users]([id]),
   [created_at] TEXT,
   [full_text] TEXT,
   [retweeted_status] INTEGER,
   [quoted_status] INTEGER,
   [place] TEXT REFERENCES [places]([id]),
   [source] TEXT REFERENCES [sources]([id]), [truncated] INTEGER, [display_text_range] TEXT, [in_reply_to_status_id] TEXT, [in_reply_to_user_id] TEXT, [in_reply_to_screen_name] TEXT, [geo] TEXT, [coordinates] TEXT, [contributors] TEXT, [is_quote_status] INTEGER, [retweet_count] INTEGER, [favorite_count] INTEGER, [favorited] INTEGER, [retweeted] INTEGER, [lang] TEXT, [possibly_sensitive] INTEGER, [scopes] TEXT, [withheld_in_countries] TEXT, [withheld_scope] TEXT, [withheld_copyright] INTEGER,
   FOREIGN KEY([retweeted_status]) REFERENCES [tweets]([id]),
   FOREIGN KEY([quoted_status]) REFERENCES [tweets]([id])
);
CREATE TABLE 'tweets_fts_data'(id INTEGER PRIMARY KEY, block BLOB);
CREATE TABLE 'tweets_fts_idx'(segid, term, pgno, PRIMARY KEY(segid, term)) WITHOUT ROWID;
CREATE TABLE 'tweets_fts_docsize'(id INTEGER PRIMARY KEY, sz BLOB);
CREATE TABLE 'tweets_fts_config'(k PRIMARY KEY, v) WITHOUT ROWID;
CREATE VIRTUAL TABLE [notes_fts] USING FTS5 (
    [summary],
    content=[notes]
);
CREATE TRIGGER [notes_ai] AFTER INSERT ON [notes] BEGIN
  INSERT INTO [notes_fts] (rowid, [summary]) VALUES (new.rowid, new.[summary]);
END;
CREATE TRIGGER [notes_ad] AFTER DELETE ON [notes] BEGIN
  INSERT INTO [notes_fts] ([notes_fts], rowid, [summary]) VALUES('delete', old.rowid, old.[summary]);
END;
CREATE TRIGGER [notes_au] AFTER UPDATE ON [notes] BEGIN
  INSERT INTO [notes_fts] ([notes_fts], rowid, [summary]) VALUES('delete', old.rowid, old.[summary]);
  INSERT INTO [notes_fts] (rowid, [summary]) VALUES (new.rowid, new.[summary]);
END;
CREATE VIRTUAL TABLE [users_fts] USING FTS5 (
    [name], [screen_name], [description], [location],
    content=[users]
);
CREATE TRIGGER [users_ai] AFTER INSERT ON [users] BEGIN
  INSERT INTO [users_fts] (rowid, [name], [screen_name], [description], [location]) VALUES (new.rowid, new.[name], new.[screen_name], new.[description], new.[location]);
END;
CREATE TRIGGER [users_ad] AFTER DELETE ON [users] BEGIN
  INSERT INTO [users_fts] ([users_fts], rowid, [name], [screen_name], [description], [location]) VALUES('delete', old.rowid, old.[name], old.[screen_name], old.[description], old.[location]);
END;
CREATE TRIGGER [users_au] AFTER UPDATE ON [users] BEGIN
  INSERT INTO [users_fts] ([users_fts], rowid, [name], [screen_name], [description], [location]) VALUES('delete', old.rowid, old.[name], old.[screen_name], old.[description], old.[location]);
  INSERT INTO [users_fts] (rowid, [name], [screen_name], [description], [location]) VALUES (new.rowid, new.[name], new.[screen_name], new.[description], new.[location]);
END;
CREATE VIRTUAL TABLE [tweets_fts] USING FTS5 (
    [full_text],
    content=[tweets]
);
CREATE TRIGGER [tweets_ai] AFTER INSERT ON [tweets] BEGIN
  INSERT INTO [tweets_fts] (rowid, [full_text]) VALUES (new.rowid, new.[full_text]);
END;
CREATE TRIGGER [tweets_ad] AFTER DELETE ON [tweets] BEGIN
  INSERT INTO [tweets_fts] ([tweets_fts], rowid, [full_text]) VALUES('delete', old.rowid, old.[full_text]);
END;
CREATE TRIGGER [tweets_au] AFTER UPDATE ON [tweets] BEGIN
  INSERT INTO [tweets_fts] ([tweets_fts], rowid, [full_text]) VALUES('delete', old.rowid, old.[full_text]);
  INSERT INTO [tweets_fts] (rowid, [full_text]) VALUES (new.rowid, new.[full_text]);
END;
CREATE TABLE [following] (
   [followed_id] INTEGER REFERENCES [users]([id]),
   [follower_id] INTEGER REFERENCES [users]([id]),
   [first_seen] TEXT,
   PRIMARY KEY ([followed_id], [follower_id])
);
CREATE INDEX [idx_following_followed_id]
    ON [following] ([followed_id]);
CREATE INDEX [idx_following_follower_id]
    ON [following] ([follower_id]);
CREATE TABLE [since_id_types] (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT
);
CREATE TABLE [since_ids] (
   [type] INTEGER REFERENCES [since_id_types]([id]),
   [key] TEXT,
   [since_id] INTEGER,
   PRIMARY KEY ([type], [key])
);
CREATE TABLE [count_history_types] (
   [id] INTEGER PRIMARY KEY,
   [name] TEXT
);
CREATE TABLE [count_history] (
   [type] INTEGER REFERENCES [count_history_types]([id]),
   [user] INTEGER REFERENCES [users]([id]),
   [datetime] TEXT,
   [count] INTEGER,
   PRIMARY KEY ([type], [user], [datetime])
);
CREATE TABLE [media] (
   [id] INTEGER PRIMARY KEY,
   [id_str] TEXT,
   [indices] TEXT,
   [media_url] TEXT,
   [media_url_https] TEXT,
   [url] TEXT,
   [display_url] TEXT,
   [expanded_url] TEXT,
   [type] TEXT,
   [sizes] TEXT
, [video_info] TEXT, [additional_media_info] TEXT, [source_status_id] INTEGER, [source_status_id_str] TEXT, [source_user_id] INTEGER, [source_user_id_str] TEXT);
CREATE TABLE [media_tweets] (
   [media_id] INTEGER REFERENCES [media]([id]),
   [tweets_id] INTEGER REFERENCES [tweets]([id]),
   PRIMARY KEY ([media_id], [tweets_id])
);
```

## A warning about IDs

The resulting database from all of this stores twitter IDs as integers.

If you are processing these using JavaScript, you may run into problems with JavaScript's maximum integer size being 9007199254740991 - some twitter IDs may exceed this.

If this is a problem for you, dropping the `--detect-types` options from the initial TSV import should result in IDs being stored as text instead which will work around the issue.

## Now open it in Datasette

You can use the command-line version of [Datasette](https://datasette.io/):

    pip install datasette
    # OR
    brew install datasette
    # OR
    pipx install datasette
    datasette birdwatch.db
    INFO:     Started server process [58650]
    INFO:     Waiting for application startup.
    INFO:     Application startup complete.
    INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)

Or you can install [Datasette Desktop](https://datasette.io/desktop) for macOS and double-click the SQLite file to open it.

html:
<p><a href="https://twitter.github.io/birdwatch/" rel="nofollow">Twitter Birdwatch</a> is "a collaborative way to add helpful context to Tweets and keep people better informed".</p>
<p>Data collected by the program is <a href="https://twitter.github.io/birdwatch/download-data/" rel="nofollow">made available for download</a> as a trio of TSV files.</p>
<p>You can obtain those files from <a href="https://twitter.com/i/birdwatch/download-data" rel="nofollow">this page</a> (Twitter login required). The files are:</p>
<ul>
<li><code>notes-0000.tsv</code></li>
<li><code>ratings-0000.tsv</code></li>
<li><code>noteStatusHistory-0000.tsv</code></li>
</ul>
<p>As far as I can tell they only include notes and ratings from the past 48 hours. This means the files are quite small - when I downloaded them on 3rd September 2022 they were:</p>
<pre><code>2.7M  noteStatusHistory-00000.tsv
16M   notes-00000.tsv
45M   ratings-00000.tsv
</code></pre>
<h2><a id="user-content-import-into-sqlite" class="anchor" aria-hidden="true" href="#import-into-sqlite"><span aria-hidden="true" class="octicon octicon-link"></span></a>Import into SQLite</h2>
<p>I used <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data" rel="nofollow">sqlite-utils insert</a> to insert the data into a <code>birdwatch.db</code> SQLite database:</p>
<pre><code>sqlite-utils insert birdwatch.db notes notes-00000.tsv --tsv --detect-types
sqlite-utils insert birdwatch.db ratings ratings-00000.tsv --tsv --detect-types
sqlite-utils insert birdwatch.db noteStatusHistory noteStatusHistory-00000.tsv --tsv --detect-types
</code></pre>
<p>The <code>--detect-types</code> option ensures we get a mix of <code>integer</code> and <code>text</code> columns - without this we would just get <code>text</code>.</p>
<p>Having run this, the schema of the resulting database file looks like this:</p>
<pre><code>sqlite-utils schema birdwatch.db
</code></pre>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> "<span class="pl-en">notes</span>" (
   [noteId] <span class="pl-k">INTEGER</span>,
   [participantId] <span class="pl-k">TEXT</span>,
   [createdAtMillis] <span class="pl-k">INTEGER</span>,
   [tweetId] <span class="pl-k">INTEGER</span>,
   [classification] <span class="pl-k">TEXT</span>,
   [believable] <span class="pl-k">TEXT</span>,
   [harmful] <span class="pl-k">TEXT</span>,
   [validationDifficulty] <span class="pl-k">TEXT</span>,
   [misleadingOther] <span class="pl-k">INTEGER</span>,
   [misleadingFactualError] <span class="pl-k">INTEGER</span>,
   [misleadingManipulatedMedia] <span class="pl-k">INTEGER</span>,
   [misleadingOutdatedInformation] <span class="pl-k">INTEGER</span>,
   [misleadingMissingImportantContext] <span class="pl-k">INTEGER</span>,
   [misleadingUnverifiedClaimAsFact] <span class="pl-k">INTEGER</span>,
   [misleadingSatire] <span class="pl-k">INTEGER</span>,
   [notMisleadingOther] <span class="pl-k">INTEGER</span>,
   [notMisleadingFactuallyCorrect] <span class="pl-k">INTEGER</span>,
   [notMisleadingOutdatedButNotWhenWritten] <span class="pl-k">INTEGER</span>,
   [notMisleadingClearlySatire] <span class="pl-k">INTEGER</span>,
   [notMisleadingPersonalOpinion] <span class="pl-k">INTEGER</span>,
   [trustworthySources] <span class="pl-k">INTEGER</span>,
   [summary] <span class="pl-k">TEXT</span>
);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> "<span class="pl-en">ratings</span>" (
   [noteId] <span class="pl-k">INTEGER</span>,
   [participantId] <span class="pl-k">TEXT</span>,
   [createdAtMillis] <span class="pl-k">INTEGER</span>,
   [version] <span class="pl-k">INTEGER</span>,
   [agree] <span class="pl-k">INTEGER</span>,
   [disagree] <span class="pl-k">INTEGER</span>,
   [helpful] <span class="pl-k">INTEGER</span>,
   [notHelpful] <span class="pl-k">INTEGER</span>,
   [helpfulnessLevel] <span class="pl-k">TEXT</span>,
   [helpfulOther] <span class="pl-k">INTEGER</span>,
   [helpfulInformative] <span class="pl-k">INTEGER</span>,
   [helpfulClear] <span class="pl-k">INTEGER</span>,
   [helpfulEmpathetic] <span class="pl-k">INTEGER</span>,
   [helpfulGoodSources] <span class="pl-k">INTEGER</span>,
   [helpfulUniqueContext] <span class="pl-k">INTEGER</span>,
   [helpfulAddressesClaim] <span class="pl-k">INTEGER</span>,
   [helpfulImportantContext] <span class="pl-k">INTEGER</span>,
   [helpfulUnbiasedLanguage] <span class="pl-k">INTEGER</span>,
   [notHelpfulOther] <span class="pl-k">INTEGER</span>,
   [notHelpfulIncorrect] <span class="pl-k">INTEGER</span>,
   [notHelpfulSourcesMissingOrUnreliable] <span class="pl-k">INTEGER</span>,
   [notHelpfulOpinionSpeculationOrBias] <span class="pl-k">INTEGER</span>,
   [notHelpfulMissingKeyPoints] <span class="pl-k">INTEGER</span>,
   [notHelpfulOutdated] <span class="pl-k">INTEGER</span>,
   [notHelpfulHardToUnderstand] <span class="pl-k">INTEGER</span>,
   [notHelpfulArgumentativeOrBiased] <span class="pl-k">INTEGER</span>,
   [notHelpfulOffTopic] <span class="pl-k">INTEGER</span>,
   [notHelpfulSpamHarassmentOrAbuse] <span class="pl-k">INTEGER</span>,
   [notHelpfulIrrelevantSources] <span class="pl-k">INTEGER</span>,
   [notHelpfulOpinionSpeculation] <span class="pl-k">INTEGER</span>,
   [notHelpfulNoteNotNeeded] <span class="pl-k">INTEGER</span>
);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> "<span class="pl-en">noteStatusHistory</span>" (
   [noteId] <span class="pl-k">INTEGER</span>,
   [participantId] <span class="pl-k">TEXT</span>,
   [createdAtMillis] <span class="pl-k">INTEGER</span>,
   [timestampMillisOfFirstNonNMRStatus] <span class="pl-k">INTEGER</span>,
   [firstNonNMRStatus] <span class="pl-k">TEXT</span>,
   [timestampMillisOfCurrentStatus] <span class="pl-k">INTEGER</span>,
   [currentStatus] <span class="pl-k">TEXT</span>,
   [timestampMillisOfMostRecentNonNMRStatus] <span class="pl-k">INTEGER</span>,
   [mostRecentNonNMRStatus] <span class="pl-k">TEXT</span>
);</pre></div>
<h2><a id="user-content-enabling-search" class="anchor" aria-hidden="true" href="#enabling-search"><span aria-hidden="true" class="octicon octicon-link"></span></a>Enabling search</h2>
<p>The <code>summary</code> column in the <code>notes</code> table is the most interesting with respect to search. We can enable SQLite full-text search on it like this:</p>
<pre><code>sqlite-utils enable-its birdwatch.db notes summary
</code></pre>
<h2><a id="user-content-downloading-the-full-tweets" class="anchor" aria-hidden="true" href="#downloading-the-full-tweets"><span aria-hidden="true" class="octicon octicon-link"></span></a>Downloading the full tweets</h2>
<p>The Birdwatch data includes plenty of tweet IDs - in the <code>tweetId</code> column in the <code>notes</code> table - but it doesn't include the full details of those tweets.</p>
<p>If you have old credentials for v1 of the Twitter API you can use <a href="https://datasette.io/tools/twitter-to-sqlite" rel="nofollow">twitter-to-sqlite</a> to download the full details of those tweets like this:</p>
<pre><code>twitter-to-sqlite statuses-lookup birdwatch.db --sql 'select distinct tweetId from notes'
</code></pre>
<p>This fetches full tweets (and authors and attachments and suchlike) for every tweet with an ID matching one from the SQL query <code>select distinct tweetId from notes</code>.</p>
<p>This command shows a progress bar while it works:</p>
<pre><code>Importing 27,909 tweets  [#####-------------------]   17%  00:10:26
</code></pre>
<p>If you don't have Twitter APIv1 credentials consider using <a href="https://twarc-project.readthedocs.io/" rel="nofollow">twarc</a> instead (maybe with <a href="https://github.com/DocNow/twarc-csv">this plugin</a>).</p>
<p>The final database schema looks like this:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> "<span class="pl-en">notes</span>" (
   [noteId] <span class="pl-k">INTEGER</span>,
   [participantId] <span class="pl-k">TEXT</span>,
   [createdAtMillis] <span class="pl-k">INTEGER</span>,
   [tweetId] <span class="pl-k">INTEGER</span>,
   [classification] <span class="pl-k">TEXT</span>,
   [believable] <span class="pl-k">TEXT</span>,
   [harmful] <span class="pl-k">TEXT</span>,
   [validationDifficulty] <span class="pl-k">TEXT</span>,
   [misleadingOther] <span class="pl-k">INTEGER</span>,
   [misleadingFactualError] <span class="pl-k">INTEGER</span>,
   [misleadingManipulatedMedia] <span class="pl-k">INTEGER</span>,
   [misleadingOutdatedInformation] <span class="pl-k">INTEGER</span>,
   [misleadingMissingImportantContext] <span class="pl-k">INTEGER</span>,
   [misleadingUnverifiedClaimAsFact] <span class="pl-k">INTEGER</span>,
   [misleadingSatire] <span class="pl-k">INTEGER</span>,
   [notMisleadingOther] <span class="pl-k">INTEGER</span>,
   [notMisleadingFactuallyCorrect] <span class="pl-k">INTEGER</span>,
   [notMisleadingOutdatedButNotWhenWritten] <span class="pl-k">INTEGER</span>,
   [notMisleadingClearlySatire] <span class="pl-k">INTEGER</span>,
   [notMisleadingPersonalOpinion] <span class="pl-k">INTEGER</span>,
   [trustworthySources] <span class="pl-k">INTEGER</span>,
   [summary] <span class="pl-k">TEXT</span>
);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> "<span class="pl-en">ratings</span>" (
   [noteId] <span class="pl-k">INTEGER</span>,
   [participantId] <span class="pl-k">TEXT</span>,
   [createdAtMillis] <span class="pl-k">INTEGER</span>,
   [version] <span class="pl-k">INTEGER</span>,
   [agree] <span class="pl-k">INTEGER</span>,
   [disagree] <span class="pl-k">INTEGER</span>,
   [helpful] <span class="pl-k">INTEGER</span>,
   [notHelpful] <span class="pl-k">INTEGER</span>,
   [helpfulnessLevel] <span class="pl-k">TEXT</span>,
   [helpfulOther] <span class="pl-k">INTEGER</span>,
   [helpfulInformative] <span class="pl-k">INTEGER</span>,
   [helpfulClear] <span class="pl-k">INTEGER</span>,
   [helpfulEmpathetic] <span class="pl-k">INTEGER</span>,
   [helpfulGoodSources] <span class="pl-k">INTEGER</span>,
   [helpfulUniqueContext] <span class="pl-k">INTEGER</span>,
   [helpfulAddressesClaim] <span class="pl-k">INTEGER</span>,
   [helpfulImportantContext] <span class="pl-k">INTEGER</span>,
   [helpfulUnbiasedLanguage] <span class="pl-k">INTEGER</span>,
   [notHelpfulOther] <span class="pl-k">INTEGER</span>,
   [notHelpfulIncorrect] <span class="pl-k">INTEGER</span>,
   [notHelpfulSourcesMissingOrUnreliable] <span class="pl-k">INTEGER</span>,
   [notHelpfulOpinionSpeculationOrBias] <span class="pl-k">INTEGER</span>,
   [notHelpfulMissingKeyPoints] <span class="pl-k">INTEGER</span>,
   [notHelpfulOutdated] <span class="pl-k">INTEGER</span>,
   [notHelpfulHardToUnderstand] <span class="pl-k">INTEGER</span>,
   [notHelpfulArgumentativeOrBiased] <span class="pl-k">INTEGER</span>,
   [notHelpfulOffTopic] <span class="pl-k">INTEGER</span>,
   [notHelpfulSpamHarassmentOrAbuse] <span class="pl-k">INTEGER</span>,
   [notHelpfulIrrelevantSources] <span class="pl-k">INTEGER</span>,
   [notHelpfulOpinionSpeculation] <span class="pl-k">INTEGER</span>,
   [notHelpfulNoteNotNeeded] <span class="pl-k">INTEGER</span>
);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> "<span class="pl-en">noteStatusHistory</span>" (
   [noteId] <span class="pl-k">INTEGER</span>,
   [participantId] <span class="pl-k">TEXT</span>,
   [createdAtMillis] <span class="pl-k">INTEGER</span>,
   [timestampMillisOfFirstNonNMRStatus] <span class="pl-k">INTEGER</span>,
   [firstNonNMRStatus] <span class="pl-k">TEXT</span>,
   [timestampMillisOfCurrentStatus] <span class="pl-k">INTEGER</span>,
   [currentStatus] <span class="pl-k">TEXT</span>,
   [timestampMillisOfMostRecentNonNMRStatus] <span class="pl-k">INTEGER</span>,
   [mostRecentNonNMRStatus] <span class="pl-k">TEXT</span>
);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> '<span class="pl-en">notes_fts_data</span>'(id <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>, block BLOB);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> '<span class="pl-en">notes_fts_idx</span>'(segid, term, pgno, <span class="pl-k">PRIMARY KEY</span>(segid, term)) WITHOUT ROWID;
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> '<span class="pl-en">notes_fts_docsize</span>'(id <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>, sz BLOB);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> '<span class="pl-en">notes_fts_config</span>'(k <span class="pl-k">PRIMARY KEY</span>, v) WITHOUT ROWID;
CREATE TABLE [migrations] (
   [name] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
   [applied] <span class="pl-k">TEXT</span>
);
CREATE TABLE [places] (
   [id] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>
, [url] <span class="pl-k">TEXT</span>, [place_type] <span class="pl-k">TEXT</span>, [name] <span class="pl-k">TEXT</span>, [full_name] <span class="pl-k">TEXT</span>, [country_code] <span class="pl-k">TEXT</span>, [country] <span class="pl-k">TEXT</span>, [contained_within] <span class="pl-k">TEXT</span>, [bounding_box] <span class="pl-k">TEXT</span>, [attributes] <span class="pl-k">TEXT</span>);
CREATE TABLE [sources] (
   [id] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
   [name] <span class="pl-k">TEXT</span>,
   [url] <span class="pl-k">TEXT</span>
);
CREATE TABLE [users] (
   [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
   [screen_name] <span class="pl-k">TEXT</span>,
   [name] <span class="pl-k">TEXT</span>,
   [description] <span class="pl-k">TEXT</span>,
   [location] <span class="pl-k">TEXT</span>
, [url] <span class="pl-k">TEXT</span>, [protected] <span class="pl-k">INTEGER</span>, [followers_count] <span class="pl-k">INTEGER</span>, [friends_count] <span class="pl-k">INTEGER</span>, [listed_count] <span class="pl-k">INTEGER</span>, [created_at] <span class="pl-k">TEXT</span>, [favourites_count] <span class="pl-k">INTEGER</span>, [utc_offset] <span class="pl-k">TEXT</span>, [time_zone] <span class="pl-k">TEXT</span>, [geo_enabled] <span class="pl-k">INTEGER</span>, [verified] <span class="pl-k">INTEGER</span>, [statuses_count] <span class="pl-k">INTEGER</span>, [lang] <span class="pl-k">TEXT</span>, [contributors_enabled] <span class="pl-k">INTEGER</span>, [is_translator] <span class="pl-k">INTEGER</span>, [is_translation_enabled] <span class="pl-k">INTEGER</span>, [profile_background_color] <span class="pl-k">TEXT</span>, [profile_background_image_url] <span class="pl-k">TEXT</span>, [profile_background_image_url_https] <span class="pl-k">TEXT</span>, [profile_background_tile] <span class="pl-k">INTEGER</span>, [profile_image_url] <span class="pl-k">TEXT</span>, [profile_image_url_https] <span class="pl-k">TEXT</span>, [profile_banner_url] <span class="pl-k">TEXT</span>, [profile_link_color] <span class="pl-k">TEXT</span>, [profile_sidebar_border_color] <span class="pl-k">TEXT</span>, [profile_sidebar_fill_color] <span class="pl-k">TEXT</span>, [profile_text_color] <span class="pl-k">TEXT</span>, [profile_use_background_image] <span class="pl-k">INTEGER</span>, [has_extended_profile] <span class="pl-k">INTEGER</span>, [default_profile] <span class="pl-k">INTEGER</span>, [default_profile_image] <span class="pl-k">INTEGER</span>, [following] <span class="pl-k">INTEGER</span>, [follow_request_sent] <span class="pl-k">INTEGER</span>, [notifications] <span class="pl-k">INTEGER</span>, [translator_type] <span class="pl-k">TEXT</span>, [withheld_in_countries] <span class="pl-k">TEXT</span>);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> '<span class="pl-en">users_fts_data</span>'(id <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>, block BLOB);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> '<span class="pl-en">users_fts_idx</span>'(segid, term, pgno, <span class="pl-k">PRIMARY KEY</span>(segid, term)) WITHOUT ROWID;
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> '<span class="pl-en">users_fts_docsize</span>'(id <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>, sz BLOB);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> '<span class="pl-en">users_fts_config</span>'(k <span class="pl-k">PRIMARY KEY</span>, v) WITHOUT ROWID;
CREATE TABLE [tweets] (
   [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
   [user] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [users]([id]),
   [created_at] <span class="pl-k">TEXT</span>,
   [full_text] <span class="pl-k">TEXT</span>,
   [retweeted_status] <span class="pl-k">INTEGER</span>,
   [quoted_status] <span class="pl-k">INTEGER</span>,
   [place] <span class="pl-k">TEXT</span> <span class="pl-k">REFERENCES</span> [places]([id]),
   [source] <span class="pl-k">TEXT</span> <span class="pl-k">REFERENCES</span> [sources]([id]), [truncated] <span class="pl-k">INTEGER</span>, [display_text_range] <span class="pl-k">TEXT</span>, [in_reply_to_status_id] <span class="pl-k">TEXT</span>, [in_reply_to_user_id] <span class="pl-k">TEXT</span>, [in_reply_to_screen_name] <span class="pl-k">TEXT</span>, [geo] <span class="pl-k">TEXT</span>, [coordinates] <span class="pl-k">TEXT</span>, [contributors] <span class="pl-k">TEXT</span>, [is_quote_status] <span class="pl-k">INTEGER</span>, [retweet_count] <span class="pl-k">INTEGER</span>, [favorite_count] <span class="pl-k">INTEGER</span>, [favorited] <span class="pl-k">INTEGER</span>, [retweeted] <span class="pl-k">INTEGER</span>, [lang] <span class="pl-k">TEXT</span>, [possibly_sensitive] <span class="pl-k">INTEGER</span>, [scopes] <span class="pl-k">TEXT</span>, [withheld_in_countries] <span class="pl-k">TEXT</span>, [withheld_scope] <span class="pl-k">TEXT</span>, [withheld_copyright] <span class="pl-k">INTEGER</span>,
   <span class="pl-k">FOREIGN KEY</span>([retweeted_status]) <span class="pl-k">REFERENCES</span> [tweets]([id]),
   <span class="pl-k">FOREIGN KEY</span>([quoted_status]) <span class="pl-k">REFERENCES</span> [tweets]([id])
);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> '<span class="pl-en">tweets_fts_data</span>'(id <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>, block BLOB);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> '<span class="pl-en">tweets_fts_idx</span>'(segid, term, pgno, <span class="pl-k">PRIMARY KEY</span>(segid, term)) WITHOUT ROWID;
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> '<span class="pl-en">tweets_fts_docsize</span>'(id <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>, sz BLOB);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> '<span class="pl-en">tweets_fts_config</span>'(k <span class="pl-k">PRIMARY KEY</span>, v) WITHOUT ROWID;
CREATE VIRTUAL TABLE [notes_fts] USING FTS5 (
    [summary],
    content<span class="pl-k">=</span>[notes]
);
CREATE TRIGGER [notes_ai] AFTER INSERT <span class="pl-k">ON</span> [notes] <span class="pl-k">BEGIN</span>
  <span class="pl-k">INSERT INTO</span> [notes_fts] (rowid, [summary]) <span class="pl-k">VALUES</span> (<span class="pl-c1">new</span>.<span class="pl-c1">rowid</span>, new.[summary]);
END;
CREATE TRIGGER [notes_ad] AFTER <span class="pl-k">DELETE</span> <span class="pl-k">ON</span> [notes] <span class="pl-k">BEGIN</span>
  <span class="pl-k">INSERT INTO</span> [notes_fts] ([notes_fts], rowid, [summary]) <span class="pl-k">VALUES</span>(<span class="pl-s"><span class="pl-pds">'</span>delete<span class="pl-pds">'</span></span>, <span class="pl-c1">old</span>.<span class="pl-c1">rowid</span>, old.[summary]);
END;
CREATE TRIGGER [notes_au] AFTER <span class="pl-k">UPDATE</span> <span class="pl-k">ON</span> [notes] <span class="pl-k">BEGIN</span>
  <span class="pl-k">INSERT INTO</span> [notes_fts] ([notes_fts], rowid, [summary]) <span class="pl-k">VALUES</span>(<span class="pl-s"><span class="pl-pds">'</span>delete<span class="pl-pds">'</span></span>, <span class="pl-c1">old</span>.<span class="pl-c1">rowid</span>, old.[summary]);
  <span class="pl-k">INSERT INTO</span> [notes_fts] (rowid, [summary]) <span class="pl-k">VALUES</span> (<span class="pl-c1">new</span>.<span class="pl-c1">rowid</span>, new.[summary]);
END;
CREATE VIRTUAL TABLE [users_fts] USING FTS5 (
    [name], [screen_name], [description], [location],
    content<span class="pl-k">=</span>[users]
);
CREATE TRIGGER [users_ai] AFTER INSERT <span class="pl-k">ON</span> [users] <span class="pl-k">BEGIN</span>
  <span class="pl-k">INSERT INTO</span> [users_fts] (rowid, [name], [screen_name], [description], [location]) <span class="pl-k">VALUES</span> (<span class="pl-c1">new</span>.<span class="pl-c1">rowid</span>, new.[name], new.[screen_name], new.[description], new.[location]);
END;
CREATE TRIGGER [users_ad] AFTER <span class="pl-k">DELETE</span> <span class="pl-k">ON</span> [users] <span class="pl-k">BEGIN</span>
  <span class="pl-k">INSERT INTO</span> [users_fts] ([users_fts], rowid, [name], [screen_name], [description], [location]) <span class="pl-k">VALUES</span>(<span class="pl-s"><span class="pl-pds">'</span>delete<span class="pl-pds">'</span></span>, <span class="pl-c1">old</span>.<span class="pl-c1">rowid</span>, old.[name], old.[screen_name], old.[description], old.[location]);
END;
CREATE TRIGGER [users_au] AFTER <span class="pl-k">UPDATE</span> <span class="pl-k">ON</span> [users] <span class="pl-k">BEGIN</span>
  <span class="pl-k">INSERT INTO</span> [users_fts] ([users_fts], rowid, [name], [screen_name], [description], [location]) <span class="pl-k">VALUES</span>(<span class="pl-s"><span class="pl-pds">'</span>delete<span class="pl-pds">'</span></span>, <span class="pl-c1">old</span>.<span class="pl-c1">rowid</span>, old.[name], old.[screen_name], old.[description], old.[location]);
  <span class="pl-k">INSERT INTO</span> [users_fts] (rowid, [name], [screen_name], [description], [location]) <span class="pl-k">VALUES</span> (<span class="pl-c1">new</span>.<span class="pl-c1">rowid</span>, new.[name], new.[screen_name], new.[description], new.[location]);
END;
CREATE VIRTUAL TABLE [tweets_fts] USING FTS5 (
    [full_text],
    content<span class="pl-k">=</span>[tweets]
);
CREATE TRIGGER [tweets_ai] AFTER INSERT <span class="pl-k">ON</span> [tweets] <span class="pl-k">BEGIN</span>
  <span class="pl-k">INSERT INTO</span> [tweets_fts] (rowid, [full_text]) <span class="pl-k">VALUES</span> (<span class="pl-c1">new</span>.<span class="pl-c1">rowid</span>, new.[full_text]);
END;
CREATE TRIGGER [tweets_ad] AFTER <span class="pl-k">DELETE</span> <span class="pl-k">ON</span> [tweets] <span class="pl-k">BEGIN</span>
  <span class="pl-k">INSERT INTO</span> [tweets_fts] ([tweets_fts], rowid, [full_text]) <span class="pl-k">VALUES</span>(<span class="pl-s"><span class="pl-pds">'</span>delete<span class="pl-pds">'</span></span>, <span class="pl-c1">old</span>.<span class="pl-c1">rowid</span>, old.[full_text]);
END;
CREATE TRIGGER [tweets_au] AFTER <span class="pl-k">UPDATE</span> <span class="pl-k">ON</span> [tweets] <span class="pl-k">BEGIN</span>
  <span class="pl-k">INSERT INTO</span> [tweets_fts] ([tweets_fts], rowid, [full_text]) <span class="pl-k">VALUES</span>(<span class="pl-s"><span class="pl-pds">'</span>delete<span class="pl-pds">'</span></span>, <span class="pl-c1">old</span>.<span class="pl-c1">rowid</span>, old.[full_text]);
  <span class="pl-k">INSERT INTO</span> [tweets_fts] (rowid, [full_text]) <span class="pl-k">VALUES</span> (<span class="pl-c1">new</span>.<span class="pl-c1">rowid</span>, new.[full_text]);
END;
CREATE TABLE [following] (
   [followed_id] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [users]([id]),
   [follower_id] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [users]([id]),
   [first_seen] <span class="pl-k">TEXT</span>,
   <span class="pl-k">PRIMARY KEY</span> ([followed_id], [follower_id])
);
CREATE INDEX [idx_following_followed_id]
    <span class="pl-k">ON</span> [following] ([followed_id]);
CREATE INDEX [idx_following_follower_id]
    <span class="pl-k">ON</span> [following] ([follower_id]);
CREATE TABLE [since_id_types] (
   [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
   [name] <span class="pl-k">TEXT</span>
);
CREATE TABLE [since_ids] (
   [type] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [since_id_types]([id]),
   [key] <span class="pl-k">TEXT</span>,
   [since_id] <span class="pl-k">INTEGER</span>,
   <span class="pl-k">PRIMARY KEY</span> ([type], [key])
);
CREATE TABLE [count_history_types] (
   [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
   [name] <span class="pl-k">TEXT</span>
);
CREATE TABLE [count_history] (
   [type] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [count_history_types]([id]),
   [user] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [users]([id]),
   [datetime] <span class="pl-k">TEXT</span>,
   [count] <span class="pl-k">INTEGER</span>,
   <span class="pl-k">PRIMARY KEY</span> ([type], [user], [datetime])
);
CREATE TABLE [media] (
   [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
   [id_str] <span class="pl-k">TEXT</span>,
   [indices] <span class="pl-k">TEXT</span>,
   [media_url] <span class="pl-k">TEXT</span>,
   [media_url_https] <span class="pl-k">TEXT</span>,
   [url] <span class="pl-k">TEXT</span>,
   [display_url] <span class="pl-k">TEXT</span>,
   [expanded_url] <span class="pl-k">TEXT</span>,
   [type] <span class="pl-k">TEXT</span>,
   [sizes] <span class="pl-k">TEXT</span>
, [video_info] <span class="pl-k">TEXT</span>, [additional_media_info] <span class="pl-k">TEXT</span>, [source_status_id] <span class="pl-k">INTEGER</span>, [source_status_id_str] <span class="pl-k">TEXT</span>, [source_user_id] <span class="pl-k">INTEGER</span>, [source_user_id_str] <span class="pl-k">TEXT</span>);
CREATE TABLE [media_tweets] (
   [media_id] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [media]([id]),
   [tweets_id] <span class="pl-k">INTEGER</span> <span class="pl-k">REFERENCES</span> [tweets]([id]),
   <span class="pl-k">PRIMARY KEY</span> ([media_id], [tweets_id])
);</pre></div>
<h2><a id="user-content-a-warning-about-ids" class="anchor" aria-hidden="true" href="#a-warning-about-ids"><span aria-hidden="true" class="octicon octicon-link"></span></a>A warning about IDs</h2>
<p>The resulting database from all of this stores twitter IDs as integers.</p>
<p>If you are processing these using JavaScript, you may run into problems with JavaScript's maximum integer size being 9007199254740991 - some twitter IDs may exceed this.</p>
<p>If this is a problem for you, dropping the <code>--detect-types</code> options from the initial TSV import should result in IDs being stored as text instead which will work around the issue.</p>
<h2><a id="user-content-now-open-it-in-datasette" class="anchor" aria-hidden="true" href="#now-open-it-in-datasette"><span aria-hidden="true" class="octicon octicon-link"></span></a>Now open it in Datasette</h2>
<p>You can use the command-line version of <a href="https://datasette.io/" rel="nofollow">Datasette</a>:</p>
<pre><code>pip install datasette
# OR
brew install datasette
# OR
pipx install datasette
datasette birdwatch.db
INFO:     Started server process [58650]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8001 (Press CTRL+C to quit)
</code></pre>
<p>Or you can install <a href="https://datasette.io/desktop" rel="nofollow">Datasette Desktop</a> for macOS and double-click the SQLite file to open it.</p>


path: docker_pytest-docker.md
topic: docker
title: Run pytest against a specific Python version using Docker
url: https://github.com/simonw/til/blob/main/docker/pytest-docker.md
created: 2022-09-05T16:23:06-07:00
created_utc: 2022-09-05T23:23:06+00:00
updated: 2022-09-06T10:51:25-07:00
updated_utc: 2022-09-06T17:51:25+00:00
shot_hash: 7b5c417fbdee6e3968459b3d0db5c6a9
slug: pytest-docker
body:
For [datasette issue #1802](https://github.com/simonw/datasette/issues/1802) I needed to run my `pytest` test suite using a specific version of Python 3.7.

I decided to do this using Docker, using the official [python:3.7-buster](https://hub.docker.com/_/python/tags?page=1&name=3.7-buster) image.

Here's the recipe that worked for me:
```bash
docker run --rm -it -v `pwd`:/code \
  python:3.7-buster \
  bash -c "cd /code && pip install -e '.[test]' && pytest"
```

This command runs interactively so I can see the output (the `-it` option).

It mounts the current directory (with my testable application in it - I ran this in the root of a `datasette` checkout) as the `/code` volume inside the container.

The `--rm` option ensures that the container used for the test will be deleted once the test has completed (not just stopped).

It then runs the following using `bash -c`:

    cd /code && pip install -e '.[test]' && pytest

This installs my project's dependencies and test dependencies and then runs `pytest`.

The truncated output looks like this:
```
% docker run -it -v `pwd`:/code \
  python:3.7-buster \
  bash -c "cd /code && pip install -e '.[test]' && pytest"
Obtaining file:///code
  Preparing metadata (setup.py) ... done
Collecting asgiref>=3.2.10
  Downloading asgiref-3.5.2-py3-none-any.whl (22 kB)
...
Installing collected packages: rfc3986, mypy-extensions, iniconfig, zipp, typing-extensions, typed-ast, tomli, soupsieve, sniffio, six, PyYAML, pyparsing, pycparser, py, platformdirs, pathspec, mergedeep, MarkupSafe, itsdangerous, idna, hupper, h11, execnet, cogapp, certifi, attrs, aiofiles, python-multipart, packaging, Jinja2, janus, importlib-metadata, cffi, beautifulsoup4, asgiref, anyio, pluggy, pint, httpcore, cryptography, click, asgi-csrf, uvicorn, trustme, pytest, httpx, click-default-group-wheel, black, pytest-timeout, pytest-forked, pytest-asyncio, datasette, blacken-docs, pytest-xdist
  Running setup.py develop for datasette
...
========================================================= test session starts ==========================================================
platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0
SQLite: 3.27.2
rootdir: /code, configfile: pytest.ini
plugins: asyncio-0.19.0, anyio-3.6.1, timeout-2.1.0, xdist-2.5.0, forked-1.4.0
asyncio: mode=strict
collected 1054 items                                                                                                                   

tests/test_package.py ..                                                                                                         [  0%]
tests/test_cli.py .                                                                                                              [  0%]
tests/test_cli_serve_get.py ..                                                                                                   [  0%]
tests/test_cli.py .                                                                                                              [  0%]
tests/test_black.py .                                                                                                            [  0%]
tests/test_api.py ..................................................                                                             [  5%]
```

html:
<p>For <a href="https://github.com/simonw/datasette/issues/1802">datasette issue #1802</a> I needed to run my <code>pytest</code> test suite using a specific version of Python 3.7.</p>
<p>I decided to do this using Docker, using the official <a href="https://hub.docker.com/_/python/tags?page=1&amp;name=3.7-buster" rel="nofollow">python:3.7-buster</a> image.</p>
<p>Here's the recipe that worked for me:</p>
<div class="highlight highlight-source-shell"><pre>docker run --rm -it -v <span class="pl-s"><span class="pl-pds">`</span>pwd<span class="pl-pds">`</span></span>:/code \
  python:3.7-buster \
  bash -c <span class="pl-s"><span class="pl-pds">"</span>cd /code &amp;&amp; pip install -e '.[test]' &amp;&amp; pytest<span class="pl-pds">"</span></span></pre></div>
<p>This command runs interactively so I can see the output (the <code>-it</code> option).</p>
<p>It mounts the current directory (with my testable application in it - I ran this in the root of a <code>datasette</code> checkout) as the <code>/code</code> volume inside the container.</p>
<p>The <code>--rm</code> option ensures that the container used for the test will be deleted once the test has completed (not just stopped).</p>
<p>It then runs the following using <code>bash -c</code>:</p>
<pre><code>cd /code &amp;&amp; pip install -e '.[test]' &amp;&amp; pytest
</code></pre>
<p>This installs my project's dependencies and test dependencies and then runs <code>pytest</code>.</p>
<p>The truncated output looks like this:</p>
<pre><code>% docker run -it -v `pwd`:/code \
  python:3.7-buster \
  bash -c "cd /code &amp;&amp; pip install -e '.[test]' &amp;&amp; pytest"
Obtaining file:///code
  Preparing metadata (setup.py) ... done
Collecting asgiref&gt;=3.2.10
  Downloading asgiref-3.5.2-py3-none-any.whl (22 kB)
...
Installing collected packages: rfc3986, mypy-extensions, iniconfig, zipp, typing-extensions, typed-ast, tomli, soupsieve, sniffio, six, PyYAML, pyparsing, pycparser, py, platformdirs, pathspec, mergedeep, MarkupSafe, itsdangerous, idna, hupper, h11, execnet, cogapp, certifi, attrs, aiofiles, python-multipart, packaging, Jinja2, janus, importlib-metadata, cffi, beautifulsoup4, asgiref, anyio, pluggy, pint, httpcore, cryptography, click, asgi-csrf, uvicorn, trustme, pytest, httpx, click-default-group-wheel, black, pytest-timeout, pytest-forked, pytest-asyncio, datasette, blacken-docs, pytest-xdist
  Running setup.py develop for datasette
...
========================================================= test session starts ==========================================================
platform linux -- Python 3.7.13, pytest-7.1.3, pluggy-1.0.0
SQLite: 3.27.2
rootdir: /code, configfile: pytest.ini
plugins: asyncio-0.19.0, anyio-3.6.1, timeout-2.1.0, xdist-2.5.0, forked-1.4.0
asyncio: mode=strict
collected 1054 items                                                                                                                   

tests/test_package.py ..                                                                                                         [  0%]
tests/test_cli.py .                                                                                                              [  0%]
tests/test_cli_serve_get.py ..                                                                                                   [  0%]
tests/test_cli.py .                                                                                                              [  0%]
tests/test_black.py .                                                                                                            [  0%]
tests/test_api.py ..................................................                                                             [  5%]
</code></pre>


path: github_clone-and-push-gist.md
topic: github
title: Clone, edit and push files that live in a Gist
url: https://github.com/simonw/til/blob/main/github/clone-and-push-gist.md
created: 2022-09-08T13:47:18-07:00
created_utc: 2022-09-08T20:47:18+00:00
updated: 2022-09-08T15:03:45-07:00
updated_utc: 2022-09-08T22:03:45+00:00
shot_hash: f8ee8edd513a204b905b5de5404f2831
slug: clone-and-push-gist
body:
GitHub [Gists](https://gist.github.com/) are full Git repositories, and can be cloned and pushed to.

You can clone them anonymously (read-only) just using their URL:

    git clone https://gist.github.com/simonw/0a30d52feeb3ff60f7d8636b0bde296b

But if you want to be able to make local edits and then push them back, you need to use this recipe instead:

    git clone git@gist.github.com:0a30d52feeb3ff60f7d8636b0bde296b.git

You can find this in the "Embed" menu, as the "Clone via SSH" option.

This only uses the Gist's ID, the `simonw/` part from the URL is omitted.

This uses your existing GitHub SSH credentials.

You can then edit files in that repository and commit and push them like this:

    cd 0a30d52feeb3ff60f7d8636b0bde296b
    # Edit files here
    git commit -m "Edited some files" -a
    git push

html:
<p>GitHub <a href="https://gist.github.com/">Gists</a> are full Git repositories, and can be cloned and pushed to.</p>
<p>You can clone them anonymously (read-only) just using their URL:</p>
<pre><code>git clone https://gist.github.com/simonw/0a30d52feeb3ff60f7d8636b0bde296b
</code></pre>
<p>But if you want to be able to make local edits and then push them back, you need to use this recipe instead:</p>
<pre><code>git clone git@gist.github.com:0a30d52feeb3ff60f7d8636b0bde296b.git
</code></pre>
<p>You can find this in the "Embed" menu, as the "Clone via SSH" option.</p>
<p>This only uses the Gist's ID, the <code>simonw/</code> part from the URL is omitted.</p>
<p>This uses your existing GitHub SSH credentials.</p>
<p>You can then edit files in that repository and commit and push them like this:</p>
<pre><code>cd 0a30d52feeb3ff60f7d8636b0bde296b
# Edit files here
git commit -m "Edited some files" -a
git push
</code></pre>


path: macos_external-display-laptop.md
topic: macos
title: Driving an external display from a Mac laptop
url: https://github.com/simonw/til/blob/main/macos/external-display-laptop.md
created: 2022-09-10T08:38:28-07:00
created_utc: 2022-09-10T15:38:28+00:00
updated: 2022-09-10T08:38:28-07:00
updated_utc: 2022-09-10T15:38:28+00:00
shot_hash: d27be3ad101030177a05791dffb45a9f
slug: external-display-laptop
body:
For a friend's wedding I needed to run a Google Photos slideshow on some large televisions.

I had two MacBook Pro laptops. Here's how I got it to work.

Hardware:

- Two MacBook laptops
- Two laptop power supplies (battery only doesn't work)
- Two HDMI cables
- Two HDMI-to-UBC-C dongles

I wanted to leave the laptops closed, tucked away within HDMI cable distance of the TVs.

The power supplies were necessary because macOS won't let you close the lid of a laptop that is connected to an external display and have it stay powered up.

## Settings

It took a few iterations to find the right settings. The key thing was that I needed the laptop and the display to stay on, for several hours, despite the laptop itself being closed.

There are three separate preference panels involved in this.

In "Battery", switch to the "Power Adapter" tab and move "Turn display off after:" to "Never" and check the "Prevent your Mac from automatically sleeping when the display is off" checkbox:

![Screenshot of power adapter settings](https://user-images.githubusercontent.com/9599/189490465-a8b6da6c-1215-452b-9eda-874b4542fcef.png)

In "Desktop & Screen Saver" turn off the option to "Show screensaver after X minutes":

![Screenshot of screensaver settings](https://user-images.githubusercontent.com/9599/189490561-10b3b056-28f5-4b96-b3cb-56c978231bef.png)

In "Security and Privacy", turn off the option to "Require password ... after sleep or screen saver begins":

![Screenshot of security and privacy settings](https://user-images.githubusercontent.com/9599/189490619-cc9fdc4c-b55b-41dd-b7a8-61d14b5e5093.png)

Don't forget to undo these changes when your laptop returns to its regular use!

html:
<p>For a friend's wedding I needed to run a Google Photos slideshow on some large televisions.</p>
<p>I had two MacBook Pro laptops. Here's how I got it to work.</p>
<p>Hardware:</p>
<ul>
<li>Two MacBook laptops</li>
<li>Two laptop power supplies (battery only doesn't work)</li>
<li>Two HDMI cables</li>
<li>Two HDMI-to-UBC-C dongles</li>
</ul>
<p>I wanted to leave the laptops closed, tucked away within HDMI cable distance of the TVs.</p>
<p>The power supplies were necessary because macOS won't let you close the lid of a laptop that is connected to an external display and have it stay powered up.</p>
<h2><a id="user-content-settings" class="anchor" aria-hidden="true" href="#settings"><span aria-hidden="true" class="octicon octicon-link"></span></a>Settings</h2>
<p>It took a few iterations to find the right settings. The key thing was that I needed the laptop and the display to stay on, for several hours, despite the laptop itself being closed.</p>
<p>There are three separate preference panels involved in this.</p>
<p>In "Battery", switch to the "Power Adapter" tab and move "Turn display off after:" to "Never" and check the "Prevent your Mac from automatically sleeping when the display is off" checkbox:</p>
<p><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/9599/189490465-a8b6da6c-1215-452b-9eda-874b4542fcef.png"><img src="https://user-images.githubusercontent.com/9599/189490465-a8b6da6c-1215-452b-9eda-874b4542fcef.png" alt="Screenshot of power adapter settings" style="max-width: 100%;"></a></p>
<p>In "Desktop &amp; Screen Saver" turn off the option to "Show screensaver after X minutes":</p>
<p><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/9599/189490561-10b3b056-28f5-4b96-b3cb-56c978231bef.png"><img src="https://user-images.githubusercontent.com/9599/189490561-10b3b056-28f5-4b96-b3cb-56c978231bef.png" alt="Screenshot of screensaver settings" style="max-width: 100%;"></a></p>
<p>In "Security and Privacy", turn off the option to "Require password ... after sleep or screen saver begins":</p>
<p><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/9599/189490619-cc9fdc4c-b55b-41dd-b7a8-61d14b5e5093.png"><img src="https://user-images.githubusercontent.com/9599/189490619-cc9fdc4c-b55b-41dd-b7a8-61d14b5e5093.png" alt="Screenshot of security and privacy settings" style="max-width: 100%;"></a></p>
<p>Don't forget to undo these changes when your laptop returns to its regular use!</p>


path: macos_ifuse-iphone.md
topic: macos
title: Browse files (including SQLite databases) on your iPhone with ifuse
url: https://github.com/simonw/til/blob/main/macos/ifuse-iphone.md
created: 2022-09-13T21:03:15-07:00
created_utc: 2022-09-14T04:03:15+00:00
updated: 2022-09-13T21:33:31-07:00
updated_utc: 2022-09-14T04:33:31+00:00
shot_hash: 2fae92e6413bfcc39ae0bb6090bf59ef
slug: ifuse-iphone
body:
I spotted an intriguing note in the release notes for [osxphotos 0.51.7](https://github.com/RhetTbull/osxphotos/releases/tag/v0.51.7):

> Added ability to read Photos.sqlite from iPhone

This lead me to [osxphotos issue #745](https://github.com/RhetTbull/osxphotos/issues/745) which lead me to [How to access iPhone files with a disk mount](https://reincubate.com/support/how-to/mount-iphone-files/) which lead me to [ifuse](https://github.com/libimobiledevice/ifuse) and [ homebrew-fuse
](https://github.com/gromgit/homebrew-fuse).

Here's what worked for me.

1. Connect the iPhone to a Mac using a USB-C to Lightning cable
2. Find the serial number of your iPhone, by running `system_profiler SPUSBDataType -detailLevel mini | grep -e iPhone -e Serial` and looking for the first number. It will look something like `00008110001631C105BA801E`.
3. Add a hyphen to that serial number after the 8th character: `00008110-001631C105BA801E` - you will need this later
4. `brew install macfuse`
5. `brew install gromgit/fuse/ifuse-mac`
6. A tricky bit: this has installed an unsigned binary, which macOS really doesn't want to execute. So...
7. `mkdir /tmp/iPhone`
8. `ifuse /tmp/iPhone --udid 00008110-001631C105BA801E` - using the serial number you figured out earlier
9. This should fail with an error complaining about the unsigned binary
10. Open System Preferences -> Security & Privacy and allow that binary to run. You may then need to restart your computer.
11. After the reboot, `mkdir /tmp/iPhone` again.
12. Now try `ifuse /tmp/iPhone --udid 00008110-001631C105BA801E` again - this time it should work

If everything goes right, you can run `open /tmp/iPhone` to open a Finder window showing at least some of the contents of your phone. You can also `cd /tmp/iPhone` to start poking around from the terminal.

## Finding SQLite databases

The easiest way to find SQLite databases to explore is to run:

    find /tmp/iPhone | grep 'wal'

This searches for files with names like `downloads.28.sqlitedb-wal` - which indicate a SQLite database that has been opened in WAL mode.

On my iPhone I get these:
```
/tmp/iPhone/Downloads/downloads.28.sqlitedb-wal
/tmp/iPhone/Books/MetadataStore/BookMetadataStore.sqlite-wal
/tmp/iPhone/Books/Sync/Database/OutstandingAssets_4.sqlite-wal
/tmp/iPhone/Radio/Radio.db-wal
/tmp/iPhone/iTunes_Control/iTunes/MediaLibrary.sqlitedb-wal
/tmp/iPhone/MediaAnalysis/mediaanalysis.db-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSBusinessCategoryCache.Nature.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSContactCache.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/musiccache.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSLocationCache.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSPublicEventCache.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/PhotosGraph/construction-photosgraph.kgdb-wal
/tmp/iPhone/PhotoData/Caches/GraphService/PhotosGraph/photosgraph-tmp.kgdb-wal
/tmp/iPhone/PhotoData/Caches/GraphService/PhotosGraph/photosgraph.kgdb-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSBusinessCategoryCache.POI.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSBusinessCategoryCache.ROI.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/PGCurationCache.sqlite.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSBusinessCategoryCache.AOI.sqlite-wal
/tmp/iPhone/PhotoData/Caches/search/psi.sqlite-wal
/tmp/iPhone/PhotoData/CPL/storage/store.cloudphotodb-wal
/tmp/iPhone/PhotoData/Photos.sqlite-wal
```
You can then open them in Datasette by specifying the filename without the `-wal` extension. For example:

    datasette /tmp/iPhone/Radio/Radio.db

html:
<p>I spotted an intriguing note in the release notes for <a href="https://github.com/RhetTbull/osxphotos/releases/tag/v0.51.7">osxphotos 0.51.7</a>:</p>
<blockquote>
<p>Added ability to read Photos.sqlite from iPhone</p>
</blockquote>
<p>This lead me to <a href="https://github.com/RhetTbull/osxphotos/issues/745">osxphotos issue #745</a> which lead me to <a href="https://reincubate.com/support/how-to/mount-iphone-files/" rel="nofollow">How to access iPhone files with a disk mount</a> which lead me to <a href="https://github.com/libimobiledevice/ifuse">ifuse</a> and <a href="https://github.com/gromgit/homebrew-fuse"> homebrew-fuse
</a>.</p>
<p>Here's what worked for me.</p>
<ol>
<li>Connect the iPhone to a Mac using a USB-C to Lightning cable</li>
<li>Find the serial number of your iPhone, by running <code>system_profiler SPUSBDataType -detailLevel mini | grep -e iPhone -e Serial</code> and looking for the first number. It will look something like <code>00008110001631C105BA801E</code>.</li>
<li>Add a hyphen to that serial number after the 8th character: <code>00008110-001631C105BA801E</code> - you will need this later</li>
<li><code>brew install macfuse</code></li>
<li><code>brew install gromgit/fuse/ifuse-mac</code></li>
<li>A tricky bit: this has installed an unsigned binary, which macOS really doesn't want to execute. So...</li>
<li><code>mkdir /tmp/iPhone</code></li>
<li>
<code>ifuse /tmp/iPhone --udid 00008110-001631C105BA801E</code> - using the serial number you figured out earlier</li>
<li>This should fail with an error complaining about the unsigned binary</li>
<li>Open System Preferences -&gt; Security &amp; Privacy and allow that binary to run. You may then need to restart your computer.</li>
<li>After the reboot, <code>mkdir /tmp/iPhone</code> again.</li>
<li>Now try <code>ifuse /tmp/iPhone --udid 00008110-001631C105BA801E</code> again - this time it should work</li>
</ol>
<p>If everything goes right, you can run <code>open /tmp/iPhone</code> to open a Finder window showing at least some of the contents of your phone. You can also <code>cd /tmp/iPhone</code> to start poking around from the terminal.</p>
<h2><a id="user-content-finding-sqlite-databases" class="anchor" aria-hidden="true" href="#finding-sqlite-databases"><span aria-hidden="true" class="octicon octicon-link"></span></a>Finding SQLite databases</h2>
<p>The easiest way to find SQLite databases to explore is to run:</p>
<pre><code>find /tmp/iPhone | grep 'wal'
</code></pre>
<p>This searches for files with names like <code>downloads.28.sqlitedb-wal</code> - which indicate a SQLite database that has been opened in WAL mode.</p>
<p>On my iPhone I get these:</p>
<pre><code>/tmp/iPhone/Downloads/downloads.28.sqlitedb-wal
/tmp/iPhone/Books/MetadataStore/BookMetadataStore.sqlite-wal
/tmp/iPhone/Books/Sync/Database/OutstandingAssets_4.sqlite-wal
/tmp/iPhone/Radio/Radio.db-wal
/tmp/iPhone/iTunes_Control/iTunes/MediaLibrary.sqlitedb-wal
/tmp/iPhone/MediaAnalysis/mediaanalysis.db-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSBusinessCategoryCache.Nature.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSContactCache.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/musiccache.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSLocationCache.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSPublicEventCache.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/PhotosGraph/construction-photosgraph.kgdb-wal
/tmp/iPhone/PhotoData/Caches/GraphService/PhotosGraph/photosgraph-tmp.kgdb-wal
/tmp/iPhone/PhotoData/Caches/GraphService/PhotosGraph/photosgraph.kgdb-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSBusinessCategoryCache.POI.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSBusinessCategoryCache.ROI.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/PGCurationCache.sqlite.sqlite-wal
/tmp/iPhone/PhotoData/Caches/GraphService/CLSBusinessCategoryCache.AOI.sqlite-wal
/tmp/iPhone/PhotoData/Caches/search/psi.sqlite-wal
/tmp/iPhone/PhotoData/CPL/storage/store.cloudphotodb-wal
/tmp/iPhone/PhotoData/Photos.sqlite-wal
</code></pre>
<p>You can then open them in Datasette by specifying the filename without the <code>-wal</code> extension. For example:</p>
<pre><code>datasette /tmp/iPhone/Radio/Radio.db
</code></pre>


path: python_pypy-macos.md
topic: python
title: Running PyPy on macOS using Homebrew
url: https://github.com/simonw/til/blob/main/python/pypy-macos.md
created: 2022-09-14T11:39:18-07:00
created_utc: 2022-09-14T18:39:18+00:00
updated: 2022-09-14T11:39:18-07:00
updated_utc: 2022-09-14T18:39:18+00:00
shot_hash: ab83a689be106c74d6bddb2081b4fd25
slug: pypy-macos
body:
[Towards Inserting One Billion Rows in SQLite Under A Minute](https://avi.im/blag/2021/fast-sqlite-inserts/) includes this snippet:

> All I had to do was run my existing code, without any change, using PyPy. It worked and the speed bump was phenomenal. The batched version took only 2.5 minutes to insert 100M rows. I got close to 3.5x speed :)

I decided to try this out against my own Python tool for [inserting CSV files](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data), `sqlite-utils`.

I installed PyPy using Homebrew:

    brew install pypy3

Having run this, `pypy3` was available on my command-line.

I used that to create a PyPy virtual environment in my `/tmp` directory:

```
cd /tmp
pypy3 -m venv venv
source venv/bin/activate
```
Running `python --version` confirmed that this had worked:
```
% python --version
Python 3.7.13 (7e0ae751533460d5f89f3ac48ce366d8642d1db5, Apr 26 2022, 09:29:08)
[PyPy 7.3.9 with GCC Apple LLVM 13.1.6 (clang-1316.0.21.2)]
```
Then I installed `sqlite-utils` into that virtual environment like so:
```
pip install sqlite-utils
```
And confirmed the installation like this:
```
(venv) /tmp % which sqlite-utils
/private/tmp/venv/bin/sqlite-utils
(venv) /tmp % head $(which sqlite-utils)
#!/private/tmp/venv/bin/pypy3
# -*- coding: utf-8 -*-
import re
import sys
from sqlite_utils.cli import cli
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(cli())
```
Then I tested an import against a large CSV file like so:
```
(venv) /tmp % time sqlite-utils insert pypy.db t /tmp/en.openfoodfacts.org.products.csv --csv
  [------------------------------------]    0%
  [###################################-]   99%
  12.67s user 2.53s system 92% cpu 16.514 total
```
I tried the same thing using regular Python `sqlite-utils` too:
```
~ % time sqlite-utils insert pydb t /tmp/en.openfoodfacts.org.products.csv --csv 
  [------------------------------------]    0%
  [###################################-]   99%
  12.74s user 2.40s system 93% cpu 16.172 total
```
Surprisingly I didn't get any meaningful difference in performance between the two. But at least I know how to run things using PyPy now.

html:
<p><a href="https://avi.im/blag/2021/fast-sqlite-inserts/" rel="nofollow">Towards Inserting One Billion Rows in SQLite Under A Minute</a> includes this snippet:</p>
<blockquote>
<p>All I had to do was run my existing code, without any change, using PyPy. It worked and the speed bump was phenomenal. The batched version took only 2.5 minutes to insert 100M rows. I got close to 3.5x speed :)</p>
</blockquote>
<p>I decided to try this out against my own Python tool for <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-csv-or-tsv-data" rel="nofollow">inserting CSV files</a>, <code>sqlite-utils</code>.</p>
<p>I installed PyPy using Homebrew:</p>
<pre><code>brew install pypy3
</code></pre>
<p>Having run this, <code>pypy3</code> was available on my command-line.</p>
<p>I used that to create a PyPy virtual environment in my <code>/tmp</code> directory:</p>
<pre><code>cd /tmp
pypy3 -m venv venv
source venv/bin/activate
</code></pre>
<p>Running <code>python --version</code> confirmed that this had worked:</p>
<pre><code>% python --version
Python 3.7.13 (7e0ae751533460d5f89f3ac48ce366d8642d1db5, Apr 26 2022, 09:29:08)
[PyPy 7.3.9 with GCC Apple LLVM 13.1.6 (clang-1316.0.21.2)]
</code></pre>
<p>Then I installed <code>sqlite-utils</code> into that virtual environment like so:</p>
<pre><code>pip install sqlite-utils
</code></pre>
<p>And confirmed the installation like this:</p>
<pre><code>(venv) /tmp % which sqlite-utils
/private/tmp/venv/bin/sqlite-utils
(venv) /tmp % head $(which sqlite-utils)
#!/private/tmp/venv/bin/pypy3
# -*- coding: utf-8 -*-
import re
import sys
from sqlite_utils.cli import cli
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(cli())
</code></pre>
<p>Then I tested an import against a large CSV file like so:</p>
<pre><code>(venv) /tmp % time sqlite-utils insert pypy.db t /tmp/en.openfoodfacts.org.products.csv --csv
  [------------------------------------]    0%
  [###################################-]   99%
  12.67s user 2.53s system 92% cpu 16.514 total
</code></pre>
<p>I tried the same thing using regular Python <code>sqlite-utils</code> too:</p>
<pre><code>~ % time sqlite-utils insert pydb t /tmp/en.openfoodfacts.org.products.csv --csv 
  [------------------------------------]    0%
  [###################################-]   99%
  12.74s user 2.40s system 93% cpu 16.172 total
</code></pre>
<p>Surprisingly I didn't get any meaningful difference in performance between the two. But at least I know how to run things using PyPy now.</p>


path: sqlite_related-rows-single-query.md
topic: sqlite
title: Returning related rows in a single SQL query using JSON
url: https://github.com/simonw/til/blob/main/sqlite/related-rows-single-query.md
created: 2022-09-16T12:31:47-07:00
created_utc: 2022-09-16T19:31:47+00:00
updated: 2022-09-16T15:20:56-07:00
updated_utc: 2022-09-16T22:20:56+00:00
shot_hash: 46f2fbf738069a3f8e2c9f5b1d8c3582
slug: related-rows-single-query
body:
When building database-backed applications you'll often find yourself wanting to return a row from the database along with its related rows.

A few examples:

- Retrieving a list of congressional legislators and their terms, following a foreign key relationship
- Return blog entries and their tags in one go, via a many-to-many table

You can do this in SQLite using the [json_group_array() aggregation function](https://sqlite.org/json1.html#jgrouparray). A couple of examples.

## Legislators and their terms, via a foreign key

Simplified schema for [this database](https://congress-legislators.datasettes.com/legislators):

```sql
CREATE TABLE [legislators] (
   [id] TEXT PRIMARY KEY,
   [name] TEXT,
   [bio_birthday] TEXT
);
CREATE TABLE [legislator_terms] (
   [legislator_id] TEXT REFERENCES [legislators]([id]),
   [type] TEXT,
   [state] TEXT,
   [start] TEXT,
   [end] TEXT,
   [party] TEXT
);
```
Here's a query that returns each legislator along with a JSON array of their terms:
```sql
select
  legislators.id,
  legislators.name,
  json_group_array(json_object(
    'type', legislator_terms.type,
    'state', legislator_terms.state,
    'start', legislator_terms.start,
    'end', legislator_terms.end,
    'party', legislator_terms.party
   )) as terms,
   count(*) as num_terms
from
  legislators join legislator_terms on legislator_terms.legislator_id = legislators.id
  group by legislators.id
order by
  id
limit
  10
```
And [the result](https://congress-legislators.datasettes.com/legislators?sql=select%0D%0A++legislators.id%2C%0D%0A++legislators.name%2C%0D%0A++json_group_array(json_object(%0D%0A++++%27type%27%2C+legislator_terms.type%2C%0D%0A++++%27state%27%2C+legislator_terms.state%2C%0D%0A++++%27start%27%2C+legislator_terms.start%2C%0D%0A++++%27end%27%2C+legislator_terms.end%2C%0D%0A++++%27party%27%2C+legislator_terms.party%0D%0A+++))+as+terms%2C%0D%0A+++count(*)+as+num_terms%0D%0Afrom%0D%0A++legislators+join+legislator_terms+on+legislator_terms.legislator_id+%3D+legislators.id%0D%0A++group+by+legislators.id%0D%0Aorder+by%0D%0A++id%0D%0Alimit%0D%0A++10):

<img width="593" alt="Screenshot of a query result. There is a terms column containing a JSON list of terms." src="https://user-images.githubusercontent.com/9599/190714817-4959daca-2450-4d28-9601-c7d59c02aa6e.png">

Note that this query does `group by legislators.id` which is allowed in SQLite but may not work in other databases, which might require `group by legislators.id, legislators.name` instead.

## Tags on blog entries, via a many-to-many table

Simplified schema:
```sql
CREATE TABLE [blog_entry] (
   [id] INTEGER PRIMARY KEY,
   [title] TEXT
);

CREATE TABLE [blog_tag] (
   [id] INTEGER PRIMARY KEY,
   [tag] TEXT
);

CREATE TABLE [blog_entry_tags] (
   [id] INTEGER PRIMARY KEY,
   [entry_id] INTEGER,
   [tag_id] INTEGER,
   FOREIGN KEY([entry_id]) REFERENCES [blog_entry]([id]),
   FOREIGN KEY([tag_id]) REFERENCES [blog_tag]([id])
);
```
Query to retrieve entries with their tags:
```sql
select
  blog_entry.id,
  blog_entry.title,
  json_group_array(json_object('tag', blog_tag.tag)) as tags
from
  blog_entry
  join blog_entry_tags on blog_entry.id = blog_entry_tags.entry_id
  join blog_tag on blog_tag.id = blog_entry_tags.tag_id
group by
  blog_entry.id
order by
  blog_entry.id desc
```
[Result](https://datasette.simonwillison.net/simonwillisonblog?sql=select+blog_entry.id%2C+blog_entry.title%2C+json_group_array(json_object(%27tag%27%2C+blog_tag.tag))+as+tags%0D%0Afrom+blog_entry+join+blog_entry_tags+on+blog_entry.id+%3D+blog_entry_tags.entry_id%0D%0Ajoin+blog_tag+on+blog_tag.id+%3D+blog_entry_tags.tag_id%0D%0Agroup+by+blog_entry.id%0D%0Aorder+by+blog_entry.id+desc):

| id | title | tags |
| --- | --- | --- |
| 8191 | I don't know how to solve prompt injection | [{"tag":"ai"},{"tag":"security"},{"tag":"openai"}] |
| 8190 | Weeknotes: Datasette Lite, s3-credentials, shot-scraper, datasette-edit-templates and more | [{"tag":"shotscraper"},{"tag":"datasette"},{"tag":"plugins"},{"tag":"datasettelite"},{"tag":"projects"},{"tag":"s3credentials"},{"tag":"weeknotes"}] |
| 8189 | Prompt injection attacks against GPT-3 | [{"tag":"ai"},{"tag":"gpt3"},{"tag":"security"},{"tag":"openai"}] |

There's a subtle bug in the above: if an entry has no tags at all it will be excluded from the query results entirely.

You can fix that using left joins like this:

```sql
select
  blog_entry.id,
  blog_entry.title,
  json_group_array(json_object('tag', blog_tag.tag)) as tags
from
  blog_entry
  left join blog_entry_tags on blog_entry.id = blog_entry_tags.entry_id
  left join blog_tag on blog_tag.id = blog_entry_tags.tag_id
where blog_entry.id < 4
group by
  blog_entry.id
order by
  blog_entry.id desc
```
This almost works, but [it outputs the following](https://datasette.simonwillison.net/simonwillisonblog?sql=select%0D%0A++blog_entry.id%2C%0D%0A++blog_entry.title%2C%0D%0A++json_group_array(json_object(%27tag%27%2C+blog_tag.tag))+as+tags%0D%0Afrom%0D%0A++blog_entry%0D%0A++left+join+blog_entry_tags+on+blog_entry.id+%3D+blog_entry_tags.entry_id%0D%0A++left+join+blog_tag+on+blog_tag.id+%3D+blog_entry_tags.tag_id%0D%0Awhere+blog_entry.id+%3C+4%0D%0Agroup+by%0D%0A++blog_entry.id%0D%0Aorder+by%0D%0A++blog_entry.id+desc) returning `{"tag": null}` for entries with no tags:

| id | title | tags |
| --- | --- | --- |
| 3 | Todo list | [{"tag":null}] |
| 2 | Blogging aint easy | [{"tag":null}] |
| 1 | WaSP Phase II | [{"tag":null}] |

David Fetter [showed me](https://twitter.com/davidfetter/status/1570882233631903745?s=46&t=S0XOcgrqscFA3YNL8BYWBw) the solution:

```sql
select
  blog_entry.id,
  blog_entry.title,
  json_group_array(
    json_object('tag', blog_tag.tag) 
  ) filter (
    where
      blog_tag.tag is not null
  ) as tags
from
  blog_entry
  left join blog_entry_tags on blog_entry.id = blog_entry_tags.entry_id
  left join blog_tag on blog_tag.id = blog_entry_tags.tag_id
group by
  blog_entry.id
order by
  blog_entry.id
```

That extra filter on the aggregation [does the trick](https://datasette.simonwillison.net/simonwillisonblog?sql=select%0D%0A++blog_entry.id%2C%0D%0A++blog_entry.title%2C%0D%0A++json_group_array%28%0D%0A++++json_object%28%27tag%27%2C+blog_tag.tag%29+%0D%0A++%29+filter+%28%0D%0A++++where%0D%0A++++++blog_tag.tag+is+not+null%0D%0A++%29+as+tags%0D%0Afrom%0D%0A++blog_entry%0D%0A++left+join+blog_entry_tags+on+blog_entry.id+%3D+blog_entry_tags.entry_id%0D%0A++left+join+blog_tag+on+blog_tag.id+%3D+blog_entry_tags.tag_id%0D%0Agroup+by%0D%0A++blog_entry.id%0D%0Aorder+by%0D%0A++blog_entry.id)!

## Other databases

Other databases are capable of the same thing, but using different functions. PostgreSQL has [json_agg()](https://www.postgresql.org/docs/9.5/functions-aggregate.html) for example, which is also available in Django as [JSONBAgg](https://docs.djangoproject.com/en/4.1/ref/contrib/postgres/aggregates/#jsonbagg).

Here's an equivalent query in PostgreSQL syntax:

```sql
select
  blog_entry.id,
  title,
  slug,
  created,
  coalesce(json_agg(json_build_object(blog_tag.id, blog_tag.tag)) filter (
    where
      blog_tag.tag is not null
  ), json_build_array()) as tags
from
  blog_entry
  left join blog_entry_tags on blog_entry.id = blog_entry_tags.entry_id
  left join blog_tag on blog_entry_tags.tag_id = blog_tag.id
group by
  blog_entry.id
order by
  blog_entry.id
```
[See that running here](https://simonwillison.net/dashboard/json-agg-example/) in `django-sql-dashboard`.

html:
<p>When building database-backed applications you'll often find yourself wanting to return a row from the database along with its related rows.</p>
<p>A few examples:</p>
<ul>
<li>Retrieving a list of congressional legislators and their terms, following a foreign key relationship</li>
<li>Return blog entries and their tags in one go, via a many-to-many table</li>
</ul>
<p>You can do this in SQLite using the <a href="https://sqlite.org/json1.html#jgrouparray" rel="nofollow">json_group_array() aggregation function</a>. A couple of examples.</p>
<h2><a id="user-content-legislators-and-their-terms-via-a-foreign-key" class="anchor" aria-hidden="true" href="#legislators-and-their-terms-via-a-foreign-key"><span aria-hidden="true" class="octicon octicon-link"></span></a>Legislators and their terms, via a foreign key</h2>
<p>Simplified schema for <a href="https://congress-legislators.datasettes.com/legislators" rel="nofollow">this database</a>:</p>
<div class="highlight highlight-source-sql"><pre>CREATE TABLE [legislators] (
   [id] <span class="pl-k">TEXT</span> <span class="pl-k">PRIMARY KEY</span>,
   [name] <span class="pl-k">TEXT</span>,
   [bio_birthday] <span class="pl-k">TEXT</span>
);
CREATE TABLE [legislator_terms] (
   [legislator_id] <span class="pl-k">TEXT</span> <span class="pl-k">REFERENCES</span> [legislators]([id]),
   [type] <span class="pl-k">TEXT</span>,
   [state] <span class="pl-k">TEXT</span>,
   [start] <span class="pl-k">TEXT</span>,
   [end] <span class="pl-k">TEXT</span>,
   [party] <span class="pl-k">TEXT</span>
);</pre></div>
<p>Here's a query that returns each legislator along with a JSON array of their terms:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  <span class="pl-c1">legislators</span>.<span class="pl-c1">id</span>,
  <span class="pl-c1">legislators</span>.<span class="pl-c1">name</span>,
  json_group_array(json_object(
    <span class="pl-s"><span class="pl-pds">'</span>type<span class="pl-pds">'</span></span>, <span class="pl-c1">legislator_terms</span>.<span class="pl-c1">type</span>,
    <span class="pl-s"><span class="pl-pds">'</span>state<span class="pl-pds">'</span></span>, <span class="pl-c1">legislator_terms</span>.<span class="pl-c1">state</span>,
    <span class="pl-s"><span class="pl-pds">'</span>start<span class="pl-pds">'</span></span>, <span class="pl-c1">legislator_terms</span>.<span class="pl-c1">start</span>,
    <span class="pl-s"><span class="pl-pds">'</span>end<span class="pl-pds">'</span></span>, <span class="pl-c1">legislator_terms</span>.<span class="pl-c1">end</span>,
    <span class="pl-s"><span class="pl-pds">'</span>party<span class="pl-pds">'</span></span>, <span class="pl-c1">legislator_terms</span>.<span class="pl-c1">party</span>
   )) <span class="pl-k">as</span> terms,
   <span class="pl-c1">count</span>(<span class="pl-k">*</span>) <span class="pl-k">as</span> num_terms
<span class="pl-k">from</span>
  legislators <span class="pl-k">join</span> legislator_terms <span class="pl-k">on</span> <span class="pl-c1">legislator_terms</span>.<span class="pl-c1">legislator_id</span> <span class="pl-k">=</span> <span class="pl-c1">legislators</span>.<span class="pl-c1">id</span>
  <span class="pl-k">group by</span> <span class="pl-c1">legislators</span>.<span class="pl-c1">id</span>
<span class="pl-k">order by</span>
  id
<span class="pl-k">limit</span>
  <span class="pl-c1">10</span></pre></div>
<p>And <a href="https://congress-legislators.datasettes.com/legislators?sql=select%0D%0A++legislators.id%2C%0D%0A++legislators.name%2C%0D%0A++json_group_array(json_object(%0D%0A++++%27type%27%2C+legislator_terms.type%2C%0D%0A++++%27state%27%2C+legislator_terms.state%2C%0D%0A++++%27start%27%2C+legislator_terms.start%2C%0D%0A++++%27end%27%2C+legislator_terms.end%2C%0D%0A++++%27party%27%2C+legislator_terms.party%0D%0A+++))+as+terms%2C%0D%0A+++count(*)+as+num_terms%0D%0Afrom%0D%0A++legislators+join+legislator_terms+on+legislator_terms.legislator_id+%3D+legislators.id%0D%0A++group+by+legislators.id%0D%0Aorder+by%0D%0A++id%0D%0Alimit%0D%0A++10" rel="nofollow">the result</a>:</p>
<p><a target="_blank" rel="noopener noreferrer" href="https://user-images.githubusercontent.com/9599/190714817-4959daca-2450-4d28-9601-c7d59c02aa6e.png"><img width="593" alt="Screenshot of a query result. There is a terms column containing a JSON list of terms." src="https://user-images.githubusercontent.com/9599/190714817-4959daca-2450-4d28-9601-c7d59c02aa6e.png" style="max-width: 100%;"></a></p>
<p>Note that this query does <code>group by legislators.id</code> which is allowed in SQLite but may not work in other databases, which might require <code>group by legislators.id, legislators.name</code> instead.</p>
<h2><a id="user-content-tags-on-blog-entries-via-a-many-to-many-table" class="anchor" aria-hidden="true" href="#tags-on-blog-entries-via-a-many-to-many-table"><span aria-hidden="true" class="octicon octicon-link"></span></a>Tags on blog entries, via a many-to-many table</h2>
<p>Simplified schema:</p>
<div class="highlight highlight-source-sql"><pre>CREATE TABLE [blog_entry] (
   [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
   [title] <span class="pl-k">TEXT</span>
);

CREATE TABLE [blog_tag] (
   [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
   [tag] <span class="pl-k">TEXT</span>
);

CREATE TABLE [blog_entry_tags] (
   [id] <span class="pl-k">INTEGER</span> <span class="pl-k">PRIMARY KEY</span>,
   [entry_id] <span class="pl-k">INTEGER</span>,
   [tag_id] <span class="pl-k">INTEGER</span>,
   <span class="pl-k">FOREIGN KEY</span>([entry_id]) <span class="pl-k">REFERENCES</span> [blog_entry]([id]),
   <span class="pl-k">FOREIGN KEY</span>([tag_id]) <span class="pl-k">REFERENCES</span> [blog_tag]([id])
);</pre></div>
<p>Query to retrieve entries with their tags:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span>,
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">title</span>,
  json_group_array(json_object(<span class="pl-s"><span class="pl-pds">'</span>tag<span class="pl-pds">'</span></span>, <span class="pl-c1">blog_tag</span>.<span class="pl-c1">tag</span>)) <span class="pl-k">as</span> tags
<span class="pl-k">from</span>
  blog_entry
  <span class="pl-k">join</span> blog_entry_tags <span class="pl-k">on</span> <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">blog_entry_tags</span>.<span class="pl-c1">entry_id</span>
  <span class="pl-k">join</span> blog_tag <span class="pl-k">on</span> <span class="pl-c1">blog_tag</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">blog_entry_tags</span>.<span class="pl-c1">tag_id</span>
<span class="pl-k">group by</span>
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span>
<span class="pl-k">order by</span>
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span> <span class="pl-k">desc</span></pre></div>
<p><a href="https://datasette.simonwillison.net/simonwillisonblog?sql=select+blog_entry.id%2C+blog_entry.title%2C+json_group_array(json_object(%27tag%27%2C+blog_tag.tag))+as+tags%0D%0Afrom+blog_entry+join+blog_entry_tags+on+blog_entry.id+%3D+blog_entry_tags.entry_id%0D%0Ajoin+blog_tag+on+blog_tag.id+%3D+blog_entry_tags.tag_id%0D%0Agroup+by+blog_entry.id%0D%0Aorder+by+blog_entry.id+desc" rel="nofollow">Result</a>:</p>
<table>
<thead>
<tr>
<th>id</th>
<th>title</th>
<th>tags</th>
</tr>
</thead>
<tbody>
<tr>
<td>8191</td>
<td>I don't know how to solve prompt injection</td>
<td>[{"tag":"ai"},{"tag":"security"},{"tag":"openai"}]</td>
</tr>
<tr>
<td>8190</td>
<td>Weeknotes: Datasette Lite, s3-credentials, shot-scraper, datasette-edit-templates and more</td>
<td>[{"tag":"shotscraper"},{"tag":"datasette"},{"tag":"plugins"},{"tag":"datasettelite"},{"tag":"projects"},{"tag":"s3credentials"},{"tag":"weeknotes"}]</td>
</tr>
<tr>
<td>8189</td>
<td>Prompt injection attacks against GPT-3</td>
<td>[{"tag":"ai"},{"tag":"gpt3"},{"tag":"security"},{"tag":"openai"}]</td>
</tr>
</tbody>
</table>
<p>There's a subtle bug in the above: if an entry has no tags at all it will be excluded from the query results entirely.</p>
<p>You can fix that using left joins like this:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span>,
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">title</span>,
  json_group_array(json_object(<span class="pl-s"><span class="pl-pds">'</span>tag<span class="pl-pds">'</span></span>, <span class="pl-c1">blog_tag</span>.<span class="pl-c1">tag</span>)) <span class="pl-k">as</span> tags
<span class="pl-k">from</span>
  blog_entry
  <span class="pl-k">left join</span> blog_entry_tags <span class="pl-k">on</span> <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">blog_entry_tags</span>.<span class="pl-c1">entry_id</span>
  <span class="pl-k">left join</span> blog_tag <span class="pl-k">on</span> <span class="pl-c1">blog_tag</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">blog_entry_tags</span>.<span class="pl-c1">tag_id</span>
<span class="pl-k">where</span> <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span> <span class="pl-k">&lt;</span> <span class="pl-c1">4</span>
<span class="pl-k">group by</span>
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span>
<span class="pl-k">order by</span>
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span> <span class="pl-k">desc</span></pre></div>
<p>This almost works, but <a href="https://datasette.simonwillison.net/simonwillisonblog?sql=select%0D%0A++blog_entry.id%2C%0D%0A++blog_entry.title%2C%0D%0A++json_group_array(json_object(%27tag%27%2C+blog_tag.tag))+as+tags%0D%0Afrom%0D%0A++blog_entry%0D%0A++left+join+blog_entry_tags+on+blog_entry.id+%3D+blog_entry_tags.entry_id%0D%0A++left+join+blog_tag+on+blog_tag.id+%3D+blog_entry_tags.tag_id%0D%0Awhere+blog_entry.id+%3C+4%0D%0Agroup+by%0D%0A++blog_entry.id%0D%0Aorder+by%0D%0A++blog_entry.id+desc" rel="nofollow">it outputs the following</a> returning <code>{"tag": null}</code> for entries with no tags:</p>
<table>
<thead>
<tr>
<th>id</th>
<th>title</th>
<th>tags</th>
</tr>
</thead>
<tbody>
<tr>
<td>3</td>
<td>Todo list</td>
<td>[{"tag":null}]</td>
</tr>
<tr>
<td>2</td>
<td>Blogging aint easy</td>
<td>[{"tag":null}]</td>
</tr>
<tr>
<td>1</td>
<td>WaSP Phase II</td>
<td>[{"tag":null}]</td>
</tr>
</tbody>
</table>
<p>David Fetter <a href="https://twitter.com/davidfetter/status/1570882233631903745?s=46&amp;t=S0XOcgrqscFA3YNL8BYWBw" rel="nofollow">showed me</a> the solution:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span>,
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">title</span>,
  json_group_array(
    json_object(<span class="pl-s"><span class="pl-pds">'</span>tag<span class="pl-pds">'</span></span>, <span class="pl-c1">blog_tag</span>.<span class="pl-c1">tag</span>) 
  ) filter (
    <span class="pl-k">where</span>
      <span class="pl-c1">blog_tag</span>.<span class="pl-c1">tag</span> <span class="pl-k">is not null</span>
  ) <span class="pl-k">as</span> tags
<span class="pl-k">from</span>
  blog_entry
  <span class="pl-k">left join</span> blog_entry_tags <span class="pl-k">on</span> <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">blog_entry_tags</span>.<span class="pl-c1">entry_id</span>
  <span class="pl-k">left join</span> blog_tag <span class="pl-k">on</span> <span class="pl-c1">blog_tag</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">blog_entry_tags</span>.<span class="pl-c1">tag_id</span>
<span class="pl-k">group by</span>
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span>
<span class="pl-k">order by</span>
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span></pre></div>
<p>That extra filter on the aggregation <a href="https://datasette.simonwillison.net/simonwillisonblog?sql=select%0D%0A++blog_entry.id%2C%0D%0A++blog_entry.title%2C%0D%0A++json_group_array%28%0D%0A++++json_object%28%27tag%27%2C+blog_tag.tag%29+%0D%0A++%29+filter+%28%0D%0A++++where%0D%0A++++++blog_tag.tag+is+not+null%0D%0A++%29+as+tags%0D%0Afrom%0D%0A++blog_entry%0D%0A++left+join+blog_entry_tags+on+blog_entry.id+%3D+blog_entry_tags.entry_id%0D%0A++left+join+blog_tag+on+blog_tag.id+%3D+blog_entry_tags.tag_id%0D%0Agroup+by%0D%0A++blog_entry.id%0D%0Aorder+by%0D%0A++blog_entry.id" rel="nofollow">does the trick</a>!</p>
<h2><a id="user-content-other-databases" class="anchor" aria-hidden="true" href="#other-databases"><span aria-hidden="true" class="octicon octicon-link"></span></a>Other databases</h2>
<p>Other databases are capable of the same thing, but using different functions. PostgreSQL has <a href="https://www.postgresql.org/docs/9.5/functions-aggregate.html" rel="nofollow">json_agg()</a> for example, which is also available in Django as <a href="https://docs.djangoproject.com/en/4.1/ref/contrib/postgres/aggregates/#jsonbagg" rel="nofollow">JSONBAgg</a>.</p>
<p>Here's an equivalent query in PostgreSQL syntax:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span>,
  title,
  slug,
  created,
  coalesce(json_agg(json_build_object(<span class="pl-c1">blog_tag</span>.<span class="pl-c1">id</span>, <span class="pl-c1">blog_tag</span>.<span class="pl-c1">tag</span>)) filter (
    <span class="pl-k">where</span>
      <span class="pl-c1">blog_tag</span>.<span class="pl-c1">tag</span> <span class="pl-k">is not null</span>
  ), json_build_array()) <span class="pl-k">as</span> tags
<span class="pl-k">from</span>
  blog_entry
  <span class="pl-k">left join</span> blog_entry_tags <span class="pl-k">on</span> <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">blog_entry_tags</span>.<span class="pl-c1">entry_id</span>
  <span class="pl-k">left join</span> blog_tag <span class="pl-k">on</span> <span class="pl-c1">blog_entry_tags</span>.<span class="pl-c1">tag_id</span> <span class="pl-k">=</span> <span class="pl-c1">blog_tag</span>.<span class="pl-c1">id</span>
<span class="pl-k">group by</span>
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span>
<span class="pl-k">order by</span>
  <span class="pl-c1">blog_entry</span>.<span class="pl-c1">id</span></pre></div>
<p><a href="https://simonwillison.net/dashboard/json-agg-example/" rel="nofollow">See that running here</a> in <code>django-sql-dashboard</code>.</p>


path: duckdb_parquet.md
topic: duckdb
title: Using DuckDB in Python to access Parquet data
url: https://github.com/simonw/til/blob/main/duckdb/parquet.md
created: 2022-09-16T19:47:28-07:00
created_utc: 2022-09-17T02:47:28+00:00
updated: 2022-09-16T19:47:28-07:00
updated_utc: 2022-09-17T02:47:28+00:00
shot_hash: d353e1d0386aa26d77be763d6400c173
slug: parquet
body:
Did a quick experiment with [DuckDB](https://duckdb.org/) today, inspired by the [bmschmidt/hathi-binary](https://github.com/bmschmidt/hathi-binary) repo.

That repo includes 3GB of data in 68 parquet files. Those files are 45MB each.

DuckDB can run queries against Parquet data _really fast_.

I checked out the repo like this:

    cd /tmp
    git clone https://github.com/bmschmidt/hathi-binary
    cd hathi-binary

To install it:

    pip install duckdb

Then in a Python console:

```pycon
>>> import duckdb
>>> db = duckdb.connect() # No need to pass a file name, we will use a VIEW
>>> db.execute("CREATE VIEW hamming AS SELECT * FROM parquet_scan('parquet/*.parquet')")
<duckdb.DuckDBPyConnection object at 0x110eab530>
>>> db.execute("select count(*) from hamming").fetchall()
[(17123746,)]
>>> db.execute("select sum(A), sum(B), sum(C) from hamming").fetchall()
[(19478990546114240096822710, 16303362475198894881395004, 43191807707832192976154883)]
```
There are 17,123,746 records in the 3GB of Parquet data.

I switched to iPython so I could time a query. First I ran a query to see what a record looks like, using `.df().to_dict()` to convert the result into a DataFrame and then a Python dictionary:

```
In [17]: db.execute("select * from hamming limit 1").df().to_dict()
Out[17]: 
{'htid': {0: 'uc1.b3209520'},
 'A': {0: -3968610387004385723},
 'B': {0: 7528965001168362882},
 'C': {0: 5017761927246436345},
 'D': {0: 2866021664979717155},
 'E': {0: -8718458467632335109},
 'F': {0: 3783827906913154091},
 'G': {0: -883843087282811188},
 'H': {0: 4045142741717613284},
 'I': {0: -9144138405661797607},
 'J': {0: 3285280333149952194},
 'K': {0: -3352555231043531556},
 'L': {0: 2071206943103704211},
 'M': {0: -5859914591541496612},
 'N': {0: -4209182319449999971},
 'O': {0: 2040176595216801886},
 'P': {0: 860910514658882647},
 'Q': {0: 3505065119653024843},
 'R': {0: -3110594979418944378},
 'S': {0: -8591743965043807123},
 'T': {0: -3262129165685658773}}
```
Then I timed an aggregate query using `%time`:
```
In [18]: %time db.execute("select sum(A), sum(B), sum(C) from hamming").fetchall()
CPU times: user 1.13 s, sys: 488 ms, total: 1.62 s
Wall time: 206 ms
Out[18]: 
[(19478990546114240096822710,
  16303362475198894881395004,
  43191807707832192976154883)]
```
206ms to sum three columns across 17 million records is pretty fast!

html:
<p>Did a quick experiment with <a href="https://duckdb.org/" rel="nofollow">DuckDB</a> today, inspired by the <a href="https://github.com/bmschmidt/hathi-binary">bmschmidt/hathi-binary</a> repo.</p>
<p>That repo includes 3GB of data in 68 parquet files. Those files are 45MB each.</p>
<p>DuckDB can run queries against Parquet data <em>really fast</em>.</p>
<p>I checked out the repo like this:</p>
<pre><code>cd /tmp
git clone https://github.com/bmschmidt/hathi-binary
cd hathi-binary
</code></pre>
<p>To install it:</p>
<pre><code>pip install duckdb
</code></pre>
<p>Then in a Python console:</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; <span class="pl-k">import</span> duckdb
&gt;&gt;&gt; db <span class="pl-k">=</span> duckdb.connect() <span class="pl-c"><span class="pl-c">#</span> No need to pass a file name, we will use a VIEW</span>
&gt;&gt;&gt; db.execute(<span class="pl-s"><span class="pl-pds">"</span>CREATE VIEW hamming AS SELECT * FROM parquet_scan('parquet/*.parquet')<span class="pl-pds">"</span></span>)
&lt;duckdb.DuckDBPyConnection object at 0x110eab530&gt;
&gt;&gt;&gt; db.execute(<span class="pl-s"><span class="pl-pds">"</span>select count(*) from hamming<span class="pl-pds">"</span></span>).fetchall()
[(17123746,)]
&gt;&gt;&gt; db.execute(<span class="pl-s"><span class="pl-pds">"</span>select sum(A), sum(B), sum(C) from hamming<span class="pl-pds">"</span></span>).fetchall()
[(19478990546114240096822710, 16303362475198894881395004, 43191807707832192976154883)]</pre></div>
<p>There are 17,123,746 records in the 3GB of Parquet data.</p>
<p>I switched to iPython so I could time a query. First I ran a query to see what a record looks like, using <code>.df().to_dict()</code> to convert the result into a DataFrame and then a Python dictionary:</p>
<pre><code>In [17]: db.execute("select * from hamming limit 1").df().to_dict()
Out[17]: 
{'htid': {0: 'uc1.b3209520'},
 'A': {0: -3968610387004385723},
 'B': {0: 7528965001168362882},
 'C': {0: 5017761927246436345},
 'D': {0: 2866021664979717155},
 'E': {0: -8718458467632335109},
 'F': {0: 3783827906913154091},
 'G': {0: -883843087282811188},
 'H': {0: 4045142741717613284},
 'I': {0: -9144138405661797607},
 'J': {0: 3285280333149952194},
 'K': {0: -3352555231043531556},
 'L': {0: 2071206943103704211},
 'M': {0: -5859914591541496612},
 'N': {0: -4209182319449999971},
 'O': {0: 2040176595216801886},
 'P': {0: 860910514658882647},
 'Q': {0: 3505065119653024843},
 'R': {0: -3110594979418944378},
 'S': {0: -8591743965043807123},
 'T': {0: -3262129165685658773}}
</code></pre>
<p>Then I timed an aggregate query using <code>%time</code>:</p>
<pre><code>In [18]: %time db.execute("select sum(A), sum(B), sum(C) from hamming").fetchall()
CPU times: user 1.13 s, sys: 488 ms, total: 1.62 s
Wall time: 206 ms
Out[18]: 
[(19478990546114240096822710,
  16303362475198894881395004,
  43191807707832192976154883)]
</code></pre>
<p>206ms to sum three columns across 17 million records is pretty fast!</p>


path: awslambda_asgi-mangum.md
topic: awslambda
title: Deploying Python web apps as AWS Lambda functions
url: https://github.com/simonw/til/blob/main/awslambda/asgi-mangum.md
created: 2022-09-18T20:08:14-07:00
created_utc: 2022-09-19T03:08:14+00:00
updated: 2022-09-19T11:39:51-07:00
updated_utc: 2022-09-19T18:39:51+00:00
shot_hash: d6285f78938f820ca587824fc6eba035
slug: asgi-mangum
body:
I've been wanting to figure out how to do this for years. Today I finally put all of the pieces together for it.

[AWS Lambda](https://aws.amazon.com/lambda/) can host functions written in Python. These are "scale to zero" - my favourite definition of serverless! - which means you only pay for the traffic that they serve. A project with no traffic costs nothing to run.

You used to have to jump through a whole bunch of extra hoops to get a working URL that triggered those functions, but in April 2022 they [released Lambda Function URLs](https://aws.amazon.com/blogs/aws/announcing-aws-lambda-function-urls-built-in-https-endpoints-for-single-function-microservices/) and dramatically simplified that process.

There are still a lot of steps involved though. Here's how to deploy a Python web application as a Lambda function.

## Set up the AWS CLI tool

I did this so long ago I can't remember how. You need an AWS account and you need to have the [AWS CLI tool](https://aws.amazon.com/cli/) installed and configured.

The `aws --version` should return a version number of `1.22.90` or higher, as [that's when they added function URL support](https://github.com/simonw/help-scraper/commit/d217b9d7f44a1200d0582d02aeccf27e006b8b78).

I found I had too old a version of the tool. I ended up figuring out this as the way to upgrade it:

```bash
head -n 1 $(which aws)
```
Output:
```
#!/usr/local/opt/python@3.9/bin/python3.9
```
This showed me the location of the Python environment that contained the tool. I could then edit that path to upgrade it like so:
```bash
/usr/local/opt/python@3.9/bin/pip3 install -U awscli
```
## Create a Python handler function

This is "hello world" as a Python handler function. Put it in `lambda_function.py`:

```python
def lambda_handler(event, context): 
    return {
        "statusCode": 200,
        "headers": {
            "Content-Type": "text/html"
        },
        "body": "<h1>Hello World from Python</h1>"
    }
```

## Add that to a zip file

This is the part of the process that I found most unintuitive. Lambda functions are deployed as zip files. The zip file needs to contain both the Python code AND all of its dependencies - more on that to come.

Our first function doesn't have any dependencies, which makes things a lot easier. Here's how to turn it into a zip file ready to be deployed:
```bash
zip function.zip lambda_function.py
```
## Create a role with a policy

You only have to do this the first time you deploy a Lambda function. You need an IAM role that you can use for the other steps.

This command creates a role called `lambda-ex`:
```
aws iam create-role \
  --role-name lambda-ex \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"}
    ]}'
```
Then you have to run this. I don't know why this can't be handled as part of the `create-role` command, but it's necessary:
```
aws iam attach-role-policy \
  --role-name lambda-ex \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
```

## Find your AWS account ID

You need to know your AWS account ID for the next step.

You can find it by running this command:
```bash
aws sts get-caller-identity \
  --query "Account" --output text
```
I assigned it to an environment variable so I could use it later like this:
```bash
export AWS_ACCOUNT_ID=$(
  aws sts get-caller-identity \
  --query "Account" --output text
)
```
Run this to confirm that worked:
```bash
echo $AWS_ACCOUNT_ID
```

## Deploy that function

Now we can deploy the zip file as a new Lambda function!

Pick a unique function name - I picked `lambda-python-hello-world`.

Then run the following:
```bash
aws lambda create-function \
  --function-name lambda-python-hello-world \
  --zip-file fileb://function.zip \
  --runtime python3.9 \
  --handler "lambda_function.lambda_handler" \
  --role "arn:aws:iam::${AWS_ACCOUNT_ID}:role/lambda-ex"
```
We're telling it to deploy our `function.zip` file using the `python3.9` runtime.

We list `lambda_function.lambda_handler` as the handler because our Python file was called `lambda_function.py` and the function was called `lambda_handler`.

If all goes well you should get back a response from that command that looks something like this:

```json
{
    "FunctionName": "lambda-python-hello-world",
    "FunctionArn": "arn:aws:lambda:us-east-1:462092780466:function:lambda-python-hello-world",
    "Runtime": "python3.9",
    "Role": "arn:aws:iam::462092780466:role/lambda-ex",
    "Handler": "lambda_function.lambda_handler",
    "CodeSize": 332,
    "Description": "",
    "Timeout": 3,
    "MemorySize": 128,
    "LastModified": "2022-09-19T02:27:18.213+0000",
    "CodeSha256": "Y1nCZLN6KvU9vUmhHAgcAkYfvgu6uBhmdGVprq8c97Y=",
    "Version": "$LATEST",
    "TracingConfig": {
        "Mode": "PassThrough"
    },
    "RevisionId": "316481f5-7934-4e54-914f-6b075bb7d9dd",
    "State": "Pending",
    "StateReason": "The function is being created.",
    "StateReasonCode": "Creating",
    "PackageType": "Zip",
    "Architectures": [
        "x86_64"
    ],
    "EphemeralStorage": {
        "Size": 512
    }
}
```

## Grant permission for it to be executed

This magic command is also necessary for everything to work:

```bash
aws lambda add-permission \
  --function-name lambda-python-hello-world \
  --action lambda:InvokeFunctionUrl \
  --principal "*" \
  --function-url-auth-type "NONE" \
  --statement-id url
```

## Give it a Function URL

We need a URL that we can access in a browser to trigger our function.

Here's how to add a new Function URL to our deployed function:

```
aws lambda create-function-url-config \
  --function-name lambda-python-hello-world \
  --auth-type NONE
```

That `--auth-type NONE` means anyone on the internet will be able to trigger the function by visiting the URL.

This should return something like the following:

```json
{
    "FunctionUrl": "https://m2jatdfy4bulhvsfcrfc6sfw2i0bjfpx.lambda-url.us-east-1.on.aws/",
    "FunctionArn": "arn:aws:lambda:us-east-1:462092780466:function:lambda-python-hello-world",
    "AuthType": "NONE",
    "CreationTime": "2022-09-19T02:27:48.356967Z"
}
```
And sure enough, https://m2jatdfy4bulhvsfcrfc6sfw2i0bjfpx.lambda-url.us-east-1.on.aws/ now returns "Hello World from Python".

## Updating the function

Having deployed the function, updating it is pleasantly easy.

You create a new `function.zip` file - which I do like this:

```bash
rm -f function.zip # Delete if it exists
zip function.zip lambda_function.py 
```
And then deploy the update like so:
```bash
aws lambda update-function-code \
  --function-name lambda-python-hello-world \
  --zip-file fileb://function.zip
```

## Adding pure Python dependencies

Adding dependencies to the project was by far the most confusing aspect of this whole process.

Eventually I found a [good way to do it](https://github.com/pixegami/fastapi-tutorial/blob/4ec9247faf53e4c399ea18a4ac27c0e85a137955/README.md#deploying-fastapi-to-aws-lambda) thanks to the example code published to accompany [this YouTube video](https://www.youtube.com/watch?v=RGIM4JfsSk0) by Pixegami.

The trick is to include ALL of your dependencies _in the root of your zip file_.

Forget about `requirements.txt` and suchlike - you need to install copies of the actual dependencies themselves.

Here's the recipe that works for me. First, create a `requirements.txt` file listing your dependencies:

```
cowsay
```

Now use the `pip install -t` command to install those requirements into a specific directory - I use `lib`:
```bash
python3 -m pip install -t lib -r requirements.txt
```
Run `ls -lah lib` to confirm that the files are in there.
```
ls lib | cat
```
```
bin
cowsay
cowsay-5.0-py3.10.egg-info
```
Now use this recipe to add everything in `lib` to the root of your zip file:
```bash
(cd lib; zip ../function.zip -r .)
```
You can run this command to see the list of files in the zip:
```bash
unzip -l function.zip
```

Let's update `lambda_function.py` to demonstrate the `cowsay` library:
```python
import cowsay


def lambda_handler(event, context): 
    return {
        "statusCode": 200,
        "headers": {
            "Content-Type": "text/plain"
        },
        "body": cowsay.get_output_string("pig", "Hello world, I am a pig")
    }
```
Add that updated `lambda_function.py` to the zip file again:
```bash
zip function.zip lambda_function.py
```
Deploy the update:
```
aws lambda update-function-code \
  --function-name lambda-python-hello-world \
  --zip-file fileb://function.zip
```
Hit refresh on the URL from earlier and you should see:
```
  _______________________
| Hello world, I am a pig |
  =======================
                       \
                        \
                         \
                          \
                                    ,.
                                   (_|,.
                                   ,' /, )_______   _
                               __j o``-'        `.'-)'
                               (")                 \'
                               `-j                |
                                 `-._(           /
                                    |_\  |--^.  /
                                   /_]'|_| /_)_/
                                       /_]'  /_]'
```
## Advanced Python dependencies

The above recipe works fine for dependencies that are written only in Python.

Where things get more complicated is when you want to use a dependency that includes native code.

I use a Mac. If I run `pip install -t lib -r requirements.txt` I'll get the Mac versions of those dependencies.

But AWS Lambda functions run on Amazon Linux. So we need to include version of our packages that are built for that platform in our zip file.

I first had to do this because I realized the `python3.9` Lambda runtime includes a truly ancient version of SQLite - version 3.7.17 [from 2013-05-20](http://www.sqlite.org/releaselog/3_7_17.html).

The [pysqlite3-binary](https://pypi.org/project/pysqlite3-binary/) package provides a much more recent SQLite, and [Datasette](https://datasette.io/) uses that automatically if it's installed.

I figured the best way to do this would be to run the `pip install` command inside an Amazon Linux Docker container. After much head scratching, I came up with this recipe for doing that:
```bash
docker run -t -v $(pwd):/mnt \
  public.ecr.aws/sam/build-python3.9:latest \
  /bin/sh -c "pip install -r /mnt/requirements.txt -t /mnt/lib"
```

- The `-v $(pwd):/mnt` flag mounts the current directory as `/mnt` inside the container.
- The `public.ecr.aws/sam/build-python3.9:latest` image is the official AWS Lambda Python 3.9 image.
- `/bin/sh -c "pip install -r /mnt/requirements.txt -t /mnt/lib"` runs `pip install` inside the container, but ensures the files are written to `/mnt/lib` which is the `lib` folder in our current directory.

This recipe works! The result is a `lib/` folder full of Amazon Linux Python packages, ready to be zipped up and deployed.

## Running an ASGI application

I want to deploy [Datasette](https://datasette.io/).

Datasette is an [ASGI application](https://simonwillison.net/2019/Jun/23/datasette-asgi/).

But... AWS Lambda functions have their own weird interface to HTTP - the `event` and `context` parameters shown above.

[Mangum](https://github.com/jordaneremieff/mangum) is a well regarded library that bridges the gap between the two.

Here's how I got Datasette and Mangum working. It was surprisingly straight-forward!

I added the following to my `requirements.txt` file:

```
datasette
pysqlite3-binary
mangum
```
I deleted my `lib` folder:
```
rm -rf lib
```
Then I ran the magic incantation from above:

```bash
docker run -t -v $(pwd):/mnt \
  public.ecr.aws/sam/build-python3.9:latest \
  /bin/sh -c "pip install -r /mnt/requirements.txt -t /mnt/lib"
```
I added the dependencies to a new `function.zip` file:
```bash
rm -rf function.zip
(cd lib; zip ../function.zip -r .)
```
Then I added the following to `lambda_function.py`:
```python
import asyncio
from datasette.app import Datasette
import mangum


ds = Datasette(["fixtures.db"])
# Handler wraps the Datasette ASGI app with Mangum:
lambda_handler = mangum.Mangum(ds.app())
```
I added that to the zip:
```
zip function.zip lambda_function.py
```
Finally, I grabbed a copy of the standard Datasette `fixtures.db` database file and added that to the zip as well:

```bash
wget https://latest.datasette.io/fixtures.db
zip function.zip fixtures.db
```
The finished `function.zip` is 7.1MB. Time to deploy it:
```
aws lambda update-function-code \
  --function-name lambda-python-hello-world \
  --zip-file fileb://function.zip
```
This did the trick! I now had a Datasette instance running on Lambda: https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/

The default Lambda configuration only provides 128MB of RAM, and I was getting occasional timeout errors. Bumping that up to 256MB fixed the problem:

```bash
aws lambda update-function-configuration \
  --function-name lambda-python-hello-world \
  --memory-size 256
```
## This should work for Starlette and FastAPI too

Mangum works with ASGI apps, so any app built using [Starlette](https://www.starlette.io/) or [FastAPI](https://fastapi.tiangolo.com/) should work exactly the same way.

## Pretty URLs

One thing I haven't figured out yet is how to assign a custom domain name to a Lambda function.

I understand doing that involves several other AWS services, potentially API Gateway, CloudFront and Route53. I'll update this once I figure those out.

html:
<p>I've been wanting to figure out how to do this for years. Today I finally put all of the pieces together for it.</p>
<p><a href="https://aws.amazon.com/lambda/" rel="nofollow">AWS Lambda</a> can host functions written in Python. These are "scale to zero" - my favourite definition of serverless! - which means you only pay for the traffic that they serve. A project with no traffic costs nothing to run.</p>
<p>You used to have to jump through a whole bunch of extra hoops to get a working URL that triggered those functions, but in April 2022 they <a href="https://aws.amazon.com/blogs/aws/announcing-aws-lambda-function-urls-built-in-https-endpoints-for-single-function-microservices/" rel="nofollow">released Lambda Function URLs</a> and dramatically simplified that process.</p>
<p>There are still a lot of steps involved though. Here's how to deploy a Python web application as a Lambda function.</p>
<h2><a id="user-content-set-up-the-aws-cli-tool" class="anchor" aria-hidden="true" href="#set-up-the-aws-cli-tool"><span aria-hidden="true" class="octicon octicon-link"></span></a>Set up the AWS CLI tool</h2>
<p>I did this so long ago I can't remember how. You need an AWS account and you need to have the <a href="https://aws.amazon.com/cli/" rel="nofollow">AWS CLI tool</a> installed and configured.</p>
<p>The <code>aws --version</code> should return a version number of <code>1.22.90</code> or higher, as <a href="https://github.com/simonw/help-scraper/commit/d217b9d7f44a1200d0582d02aeccf27e006b8b78">that's when they added function URL support</a>.</p>
<p>I found I had too old a version of the tool. I ended up figuring out this as the way to upgrade it:</p>
<div class="highlight highlight-source-shell"><pre>head -n 1 <span class="pl-s"><span class="pl-pds">$(</span>which aws<span class="pl-pds">)</span></span></pre></div>
<p>Output:</p>
<pre><code>#!/usr/local/opt/python@3.9/bin/python3.9
</code></pre>
<p>This showed me the location of the Python environment that contained the tool. I could then edit that path to upgrade it like so:</p>
<div class="highlight highlight-source-shell"><pre>/usr/local/opt/python@3.9/bin/pip3 install -U awscli</pre></div>
<h2><a id="user-content-create-a-python-handler-function" class="anchor" aria-hidden="true" href="#create-a-python-handler-function"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create a Python handler function</h2>
<p>This is "hello world" as a Python handler function. Put it in <code>lambda_function.py</code>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">lambda_handler</span>(<span class="pl-s1">event</span>, <span class="pl-s1">context</span>): 
    <span class="pl-k">return</span> {
        <span class="pl-s">"statusCode"</span>: <span class="pl-c1">200</span>,
        <span class="pl-s">"headers"</span>: {
            <span class="pl-s">"Content-Type"</span>: <span class="pl-s">"text/html"</span>
        },
        <span class="pl-s">"body"</span>: <span class="pl-s">"&lt;h1&gt;Hello World from Python&lt;/h1&gt;"</span>
    }</pre></div>
<h2><a id="user-content-add-that-to-a-zip-file" class="anchor" aria-hidden="true" href="#add-that-to-a-zip-file"><span aria-hidden="true" class="octicon octicon-link"></span></a>Add that to a zip file</h2>
<p>This is the part of the process that I found most unintuitive. Lambda functions are deployed as zip files. The zip file needs to contain both the Python code AND all of its dependencies - more on that to come.</p>
<p>Our first function doesn't have any dependencies, which makes things a lot easier. Here's how to turn it into a zip file ready to be deployed:</p>
<div class="highlight highlight-source-shell"><pre>zip function.zip lambda_function.py</pre></div>
<h2><a id="user-content-create-a-role-with-a-policy" class="anchor" aria-hidden="true" href="#create-a-role-with-a-policy"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create a role with a policy</h2>
<p>You only have to do this the first time you deploy a Lambda function. You need an IAM role that you can use for the other steps.</p>
<p>This command creates a role called <code>lambda-ex</code>:</p>
<pre><code>aws iam create-role \
  --role-name lambda-ex \
  --assume-role-policy-document '{
    "Version": "2012-10-17",
    "Statement": [{
      "Effect": "Allow",
      "Principal": {
        "Service": "lambda.amazonaws.com"
      },
      "Action": "sts:AssumeRole"}
    ]}'
</code></pre>
<p>Then you have to run this. I don't know why this can't be handled as part of the <code>create-role</code> command, but it's necessary:</p>
<pre><code>aws iam attach-role-policy \
  --role-name lambda-ex \
  --policy-arn arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
</code></pre>
<h2><a id="user-content-find-your-aws-account-id" class="anchor" aria-hidden="true" href="#find-your-aws-account-id"><span aria-hidden="true" class="octicon octicon-link"></span></a>Find your AWS account ID</h2>
<p>You need to know your AWS account ID for the next step.</p>
<p>You can find it by running this command:</p>
<div class="highlight highlight-source-shell"><pre>aws sts get-caller-identity \
  --query <span class="pl-s"><span class="pl-pds">"</span>Account<span class="pl-pds">"</span></span> --output text</pre></div>
<p>I assigned it to an environment variable so I could use it later like this:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-k">export</span> AWS_ACCOUNT_ID=<span class="pl-s"><span class="pl-pds">$(</span></span>
<span class="pl-s">  aws sts get-caller-identity \</span>
<span class="pl-s">  --query <span class="pl-s"><span class="pl-pds">"</span>Account<span class="pl-pds">"</span></span> --output text</span>
<span class="pl-s"><span class="pl-pds">)</span></span></pre></div>
<p>Run this to confirm that worked:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c1">echo</span> <span class="pl-smi">$AWS_ACCOUNT_ID</span></pre></div>
<h2><a id="user-content-deploy-that-function" class="anchor" aria-hidden="true" href="#deploy-that-function"><span aria-hidden="true" class="octicon octicon-link"></span></a>Deploy that function</h2>
<p>Now we can deploy the zip file as a new Lambda function!</p>
<p>Pick a unique function name - I picked <code>lambda-python-hello-world</code>.</p>
<p>Then run the following:</p>
<div class="highlight highlight-source-shell"><pre>aws lambda create-function \
  --function-name lambda-python-hello-world \
  --zip-file fileb://function.zip \
  --runtime python3.9 \
  --handler <span class="pl-s"><span class="pl-pds">"</span>lambda_function.lambda_handler<span class="pl-pds">"</span></span> \
  --role <span class="pl-s"><span class="pl-pds">"</span>arn:aws:iam::<span class="pl-smi">${AWS_ACCOUNT_ID}</span>:role/lambda-ex<span class="pl-pds">"</span></span></pre></div>
<p>We're telling it to deploy our <code>function.zip</code> file using the <code>python3.9</code> runtime.</p>
<p>We list <code>lambda_function.lambda_handler</code> as the handler because our Python file was called <code>lambda_function.py</code> and the function was called <code>lambda_handler</code>.</p>
<p>If all goes well you should get back a response from that command that looks something like this:</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-ent">"FunctionName"</span>: <span class="pl-s"><span class="pl-pds">"</span>lambda-python-hello-world<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"FunctionArn"</span>: <span class="pl-s"><span class="pl-pds">"</span>arn:aws:lambda:us-east-1:462092780466:function:lambda-python-hello-world<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"Runtime"</span>: <span class="pl-s"><span class="pl-pds">"</span>python3.9<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"Role"</span>: <span class="pl-s"><span class="pl-pds">"</span>arn:aws:iam::462092780466:role/lambda-ex<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"Handler"</span>: <span class="pl-s"><span class="pl-pds">"</span>lambda_function.lambda_handler<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"CodeSize"</span>: <span class="pl-c1">332</span>,
    <span class="pl-ent">"Description"</span>: <span class="pl-s"><span class="pl-pds">"</span><span class="pl-pds">"</span></span>,
    <span class="pl-ent">"Timeout"</span>: <span class="pl-c1">3</span>,
    <span class="pl-ent">"MemorySize"</span>: <span class="pl-c1">128</span>,
    <span class="pl-ent">"LastModified"</span>: <span class="pl-s"><span class="pl-pds">"</span>2022-09-19T02:27:18.213+0000<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"CodeSha256"</span>: <span class="pl-s"><span class="pl-pds">"</span>Y1nCZLN6KvU9vUmhHAgcAkYfvgu6uBhmdGVprq8c97Y=<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"Version"</span>: <span class="pl-s"><span class="pl-pds">"</span>$LATEST<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"TracingConfig"</span>: {
        <span class="pl-ent">"Mode"</span>: <span class="pl-s"><span class="pl-pds">"</span>PassThrough<span class="pl-pds">"</span></span>
    },
    <span class="pl-ent">"RevisionId"</span>: <span class="pl-s"><span class="pl-pds">"</span>316481f5-7934-4e54-914f-6b075bb7d9dd<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"State"</span>: <span class="pl-s"><span class="pl-pds">"</span>Pending<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"StateReason"</span>: <span class="pl-s"><span class="pl-pds">"</span>The function is being created.<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"StateReasonCode"</span>: <span class="pl-s"><span class="pl-pds">"</span>Creating<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"PackageType"</span>: <span class="pl-s"><span class="pl-pds">"</span>Zip<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"Architectures"</span>: [
        <span class="pl-s"><span class="pl-pds">"</span>x86_64<span class="pl-pds">"</span></span>
    ],
    <span class="pl-ent">"EphemeralStorage"</span>: {
        <span class="pl-ent">"Size"</span>: <span class="pl-c1">512</span>
    }
}</pre></div>
<h2><a id="user-content-grant-permission-for-it-to-be-executed" class="anchor" aria-hidden="true" href="#grant-permission-for-it-to-be-executed"><span aria-hidden="true" class="octicon octicon-link"></span></a>Grant permission for it to be executed</h2>
<p>This magic command is also necessary for everything to work:</p>
<div class="highlight highlight-source-shell"><pre>aws lambda add-permission \
  --function-name lambda-python-hello-world \
  --action lambda:InvokeFunctionUrl \
  --principal <span class="pl-s"><span class="pl-pds">"</span>*<span class="pl-pds">"</span></span> \
  --function-url-auth-type <span class="pl-s"><span class="pl-pds">"</span>NONE<span class="pl-pds">"</span></span> \
  --statement-id url</pre></div>
<h2><a id="user-content-give-it-a-function-url" class="anchor" aria-hidden="true" href="#give-it-a-function-url"><span aria-hidden="true" class="octicon octicon-link"></span></a>Give it a Function URL</h2>
<p>We need a URL that we can access in a browser to trigger our function.</p>
<p>Here's how to add a new Function URL to our deployed function:</p>
<pre><code>aws lambda create-function-url-config \
  --function-name lambda-python-hello-world \
  --auth-type NONE
</code></pre>
<p>That <code>--auth-type NONE</code> means anyone on the internet will be able to trigger the function by visiting the URL.</p>
<p>This should return something like the following:</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-ent">"FunctionUrl"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://m2jatdfy4bulhvsfcrfc6sfw2i0bjfpx.lambda-url.us-east-1.on.aws/<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"FunctionArn"</span>: <span class="pl-s"><span class="pl-pds">"</span>arn:aws:lambda:us-east-1:462092780466:function:lambda-python-hello-world<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"AuthType"</span>: <span class="pl-s"><span class="pl-pds">"</span>NONE<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"CreationTime"</span>: <span class="pl-s"><span class="pl-pds">"</span>2022-09-19T02:27:48.356967Z<span class="pl-pds">"</span></span>
}</pre></div>
<p>And sure enough, <a href="https://m2jatdfy4bulhvsfcrfc6sfw2i0bjfpx.lambda-url.us-east-1.on.aws/" rel="nofollow">https://m2jatdfy4bulhvsfcrfc6sfw2i0bjfpx.lambda-url.us-east-1.on.aws/</a> now returns "Hello World from Python".</p>
<h2><a id="user-content-updating-the-function" class="anchor" aria-hidden="true" href="#updating-the-function"><span aria-hidden="true" class="octicon octicon-link"></span></a>Updating the function</h2>
<p>Having deployed the function, updating it is pleasantly easy.</p>
<p>You create a new <code>function.zip</code> file - which I do like this:</p>
<div class="highlight highlight-source-shell"><pre>rm -f function.zip <span class="pl-c"><span class="pl-c">#</span> Delete if it exists</span>
zip function.zip lambda_function.py </pre></div>
<p>And then deploy the update like so:</p>
<div class="highlight highlight-source-shell"><pre>aws lambda update-function-code \
  --function-name lambda-python-hello-world \
  --zip-file fileb://function.zip</pre></div>
<h2><a id="user-content-adding-pure-python-dependencies" class="anchor" aria-hidden="true" href="#adding-pure-python-dependencies"><span aria-hidden="true" class="octicon octicon-link"></span></a>Adding pure Python dependencies</h2>
<p>Adding dependencies to the project was by far the most confusing aspect of this whole process.</p>
<p>Eventually I found a <a href="https://github.com/pixegami/fastapi-tutorial/blob/4ec9247faf53e4c399ea18a4ac27c0e85a137955/README.md#deploying-fastapi-to-aws-lambda">good way to do it</a> thanks to the example code published to accompany <a href="https://www.youtube.com/watch?v=RGIM4JfsSk0" rel="nofollow">this YouTube video</a> by Pixegami.</p>
<p>The trick is to include ALL of your dependencies <em>in the root of your zip file</em>.</p>
<p>Forget about <code>requirements.txt</code> and suchlike - you need to install copies of the actual dependencies themselves.</p>
<p>Here's the recipe that works for me. First, create a <code>requirements.txt</code> file listing your dependencies:</p>
<pre><code>cowsay
</code></pre>
<p>Now use the <code>pip install -t</code> command to install those requirements into a specific directory - I use <code>lib</code>:</p>
<div class="highlight highlight-source-shell"><pre>python3 -m pip install -t lib -r requirements.txt</pre></div>
<p>Run <code>ls -lah lib</code> to confirm that the files are in there.</p>
<pre><code>ls lib | cat
</code></pre>
<pre><code>bin
cowsay
cowsay-5.0-py3.10.egg-info
</code></pre>
<p>Now use this recipe to add everything in <code>lib</code> to the root of your zip file:</p>
<div class="highlight highlight-source-shell"><pre>(cd lib<span class="pl-k">;</span> zip ../function.zip -r .)</pre></div>
<p>You can run this command to see the list of files in the zip:</p>
<div class="highlight highlight-source-shell"><pre>unzip -l function.zip</pre></div>
<p>Let's update <code>lambda_function.py</code> to demonstrate the <code>cowsay</code> library:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">cowsay</span>


<span class="pl-k">def</span> <span class="pl-en">lambda_handler</span>(<span class="pl-s1">event</span>, <span class="pl-s1">context</span>): 
    <span class="pl-k">return</span> {
        <span class="pl-s">"statusCode"</span>: <span class="pl-c1">200</span>,
        <span class="pl-s">"headers"</span>: {
            <span class="pl-s">"Content-Type"</span>: <span class="pl-s">"text/plain"</span>
        },
        <span class="pl-s">"body"</span>: <span class="pl-s1">cowsay</span>.<span class="pl-en">get_output_string</span>(<span class="pl-s">"pig"</span>, <span class="pl-s">"Hello world, I am a pig"</span>)
    }</pre></div>
<p>Add that updated <code>lambda_function.py</code> to the zip file again:</p>
<div class="highlight highlight-source-shell"><pre>zip function.zip lambda_function.py</pre></div>
<p>Deploy the update:</p>
<pre><code>aws lambda update-function-code \
  --function-name lambda-python-hello-world \
  --zip-file fileb://function.zip
</code></pre>
<p>Hit refresh on the URL from earlier and you should see:</p>
<pre><code>  _______________________
| Hello world, I am a pig |
  =======================
                       \
                        \
                         \
                          \
                                    ,.
                                   (_|,.
                                   ,' /, )_______   _
                               __j o``-'        `.'-)'
                               (")                 \'
                               `-j                |
                                 `-._(           /
                                    |_\  |--^.  /
                                   /_]'|_| /_)_/
                                       /_]'  /_]'
</code></pre>
<h2><a id="user-content-advanced-python-dependencies" class="anchor" aria-hidden="true" href="#advanced-python-dependencies"><span aria-hidden="true" class="octicon octicon-link"></span></a>Advanced Python dependencies</h2>
<p>The above recipe works fine for dependencies that are written only in Python.</p>
<p>Where things get more complicated is when you want to use a dependency that includes native code.</p>
<p>I use a Mac. If I run <code>pip install -t lib -r requirements.txt</code> I'll get the Mac versions of those dependencies.</p>
<p>But AWS Lambda functions run on Amazon Linux. So we need to include version of our packages that are built for that platform in our zip file.</p>
<p>I first had to do this because I realized the <code>python3.9</code> Lambda runtime includes a truly ancient version of SQLite - version 3.7.17 <a href="http://www.sqlite.org/releaselog/3_7_17.html" rel="nofollow">from 2013-05-20</a>.</p>
<p>The <a href="https://pypi.org/project/pysqlite3-binary/" rel="nofollow">pysqlite3-binary</a> package provides a much more recent SQLite, and <a href="https://datasette.io/" rel="nofollow">Datasette</a> uses that automatically if it's installed.</p>
<p>I figured the best way to do this would be to run the <code>pip install</code> command inside an Amazon Linux Docker container. After much head scratching, I came up with this recipe for doing that:</p>
<div class="highlight highlight-source-shell"><pre>docker run -t -v <span class="pl-s"><span class="pl-pds">$(</span>pwd<span class="pl-pds">)</span></span>:/mnt \
  public.ecr.aws/sam/build-python3.9:latest \
  /bin/sh -c <span class="pl-s"><span class="pl-pds">"</span>pip install -r /mnt/requirements.txt -t /mnt/lib<span class="pl-pds">"</span></span></pre></div>
<ul>
<li>The <code>-v $(pwd):/mnt</code> flag mounts the current directory as <code>/mnt</code> inside the container.</li>
<li>The <code>public.ecr.aws/sam/build-python3.9:latest</code> image is the official AWS Lambda Python 3.9 image.</li>
<li>
<code>/bin/sh -c "pip install -r /mnt/requirements.txt -t /mnt/lib"</code> runs <code>pip install</code> inside the container, but ensures the files are written to <code>/mnt/lib</code> which is the <code>lib</code> folder in our current directory.</li>
</ul>
<p>This recipe works! The result is a <code>lib/</code> folder full of Amazon Linux Python packages, ready to be zipped up and deployed.</p>
<h2><a id="user-content-running-an-asgi-application" class="anchor" aria-hidden="true" href="#running-an-asgi-application"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running an ASGI application</h2>
<p>I want to deploy <a href="https://datasette.io/" rel="nofollow">Datasette</a>.</p>
<p>Datasette is an <a href="https://simonwillison.net/2019/Jun/23/datasette-asgi/" rel="nofollow">ASGI application</a>.</p>
<p>But... AWS Lambda functions have their own weird interface to HTTP - the <code>event</code> and <code>context</code> parameters shown above.</p>
<p><a href="https://github.com/jordaneremieff/mangum">Mangum</a> is a well regarded library that bridges the gap between the two.</p>
<p>Here's how I got Datasette and Mangum working. It was surprisingly straight-forward!</p>
<p>I added the following to my <code>requirements.txt</code> file:</p>
<pre><code>datasette
pysqlite3-binary
mangum
</code></pre>
<p>I deleted my <code>lib</code> folder:</p>
<pre><code>rm -rf lib
</code></pre>
<p>Then I ran the magic incantation from above:</p>
<div class="highlight highlight-source-shell"><pre>docker run -t -v <span class="pl-s"><span class="pl-pds">$(</span>pwd<span class="pl-pds">)</span></span>:/mnt \
  public.ecr.aws/sam/build-python3.9:latest \
  /bin/sh -c <span class="pl-s"><span class="pl-pds">"</span>pip install -r /mnt/requirements.txt -t /mnt/lib<span class="pl-pds">"</span></span></pre></div>
<p>I added the dependencies to a new <code>function.zip</code> file:</p>
<div class="highlight highlight-source-shell"><pre>rm -rf function.zip
(cd lib<span class="pl-k">;</span> zip ../function.zip -r .)</pre></div>
<p>Then I added the following to <code>lambda_function.py</code>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">asyncio</span>
<span class="pl-k">from</span> <span class="pl-s1">datasette</span>.<span class="pl-s1">app</span> <span class="pl-k">import</span> <span class="pl-v">Datasette</span>
<span class="pl-k">import</span> <span class="pl-s1">mangum</span>


<span class="pl-s1">ds</span> <span class="pl-c1">=</span> <span class="pl-v">Datasette</span>([<span class="pl-s">"fixtures.db"</span>])
<span class="pl-c"># Handler wraps the Datasette ASGI app with Mangum:</span>
<span class="pl-s1">lambda_handler</span> <span class="pl-c1">=</span> <span class="pl-s1">mangum</span>.<span class="pl-v">Mangum</span>(<span class="pl-s1">ds</span>.<span class="pl-en">app</span>())</pre></div>
<p>I added that to the zip:</p>
<pre><code>zip function.zip lambda_function.py
</code></pre>
<p>Finally, I grabbed a copy of the standard Datasette <code>fixtures.db</code> database file and added that to the zip as well:</p>
<div class="highlight highlight-source-shell"><pre>wget https://latest.datasette.io/fixtures.db
zip function.zip fixtures.db</pre></div>
<p>The finished <code>function.zip</code> is 7.1MB. Time to deploy it:</p>
<pre><code>aws lambda update-function-code \
  --function-name lambda-python-hello-world \
  --zip-file fileb://function.zip
</code></pre>
<p>This did the trick! I now had a Datasette instance running on Lambda: <a href="https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/" rel="nofollow">https://fnkvspusjrl5dxytaxnuwidxem0hverw.lambda-url.us-east-1.on.aws/</a></p>
<p>The default Lambda configuration only provides 128MB of RAM, and I was getting occasional timeout errors. Bumping that up to 256MB fixed the problem:</p>
<div class="highlight highlight-source-shell"><pre>aws lambda update-function-configuration \
  --function-name lambda-python-hello-world \
  --memory-size 256</pre></div>
<h2><a id="user-content-this-should-work-for-starlette-and-fastapi-too" class="anchor" aria-hidden="true" href="#this-should-work-for-starlette-and-fastapi-too"><span aria-hidden="true" class="octicon octicon-link"></span></a>This should work for Starlette and FastAPI too</h2>
<p>Mangum works with ASGI apps, so any app built using <a href="https://www.starlette.io/" rel="nofollow">Starlette</a> or <a href="https://fastapi.tiangolo.com/" rel="nofollow">FastAPI</a> should work exactly the same way.</p>
<h2><a id="user-content-pretty-urls" class="anchor" aria-hidden="true" href="#pretty-urls"><span aria-hidden="true" class="octicon octicon-link"></span></a>Pretty URLs</h2>
<p>One thing I haven't figured out yet is how to assign a custom domain name to a Lambda function.</p>
<p>I understand doing that involves several other AWS services, potentially API Gateway, CloudFront and Route53. I'll update this once I figure those out.</p>


path: cocktails_whisky-sour.md
topic: cocktails
title: Whisky sour
url: https://github.com/simonw/til/blob/main/cocktails/whisky-sour.md
created: 2022-09-25T09:24:00-07:00
created_utc: 2022-09-25T16:24:00+00:00
updated: 2022-09-26T20:08:28-07:00
updated_utc: 2022-09-27T03:08:28+00:00
shot_hash: 02cffdf51d48cd639d9f59c3241d45c8
slug: whisky-sour
body:
I picked up the recipe for this one from [this video](https://www.tiktok.com/t/ZTRaxyxQP/) by [@notjustabartender](https://www.tiktok.com/@notjustabartender) on TikTok.

![A tasty looking whisky sour](https://static.simonwillison.net/static/2022/whisky-sour.jpg)

## Ingredients per drink

- 1 egg white
- 1/2 oz rich Demerara syrup (see below - needs Demerara sugar, water and a bit of vodka)
- .75oz lemon juice
- 1.5oz rye (I used Rittenhouse)
- Angostura bitters

I made two drinks in one go, so I doubled these.

I tried this recipe once with a fancy scotch but it wasn't nearly as nice as the one made with rye.
## Equipment

- Jiggers
- Cocktail shaker (I use two metal shaker cups that fit together)
- Hand juicer
- Strainer
- 2 double glasses
- Small saucepan and scales bottle if making Demerara syrup

## Rich Demerara syrup

The syrup is a 2/1 ratio of sugar to water - so start with the saucepan on the scales and measure in around 20 units of Demerara sugar, then add 10 units of water.

Heat and stir to dissolve together, without boiling too much. Stir in a tiny splash of vodka. Empty into the glass bottle with the funnel and leave to cool.

## Making two whisky sours

1. Put the two glasses in the ice box in the freezer to chill
2. Separate two egg whites. Put the egg whites in the shaker
3. Add 1.5 floz freshly squeezed lemon juice
4. Add 1 floz rich Demerara syrup
5. Add 3 floz rye
6. Dry shake vigorously in the shaker. Dry shaking means shaking without ice. I went about 15 seconds. The result should be a foamy egg white mix.
7. Now add ice - I added 6 cubes. Shake vigorously again - I did another 15s.
8. Strain into the chilled glasses, each containing another four ice cubes (or a big square ice cube if you have them)
9. Add a couple of drops of angostura bitters on the top

html:
<p>I picked up the recipe for this one from <a href="https://www.tiktok.com/t/ZTRaxyxQP/" rel="nofollow">this video</a> by <a href="https://www.tiktok.com/@notjustabartender" rel="nofollow">@notjustabartender</a> on TikTok.</p>
<p><a target="_blank" rel="noopener noreferrer nofollow" href="https://camo.githubusercontent.com/23b3b498a074e949733cc11ff8d2a91ae8cd96ae60fa2aca603d02caf3ec77ed/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032322f776869736b792d736f75722e6a7067"><img src="https://camo.githubusercontent.com/23b3b498a074e949733cc11ff8d2a91ae8cd96ae60fa2aca603d02caf3ec77ed/68747470733a2f2f7374617469632e73696d6f6e77696c6c69736f6e2e6e65742f7374617469632f323032322f776869736b792d736f75722e6a7067" alt="A tasty looking whisky sour" data-canonical-src="https://static.simonwillison.net/static/2022/whisky-sour.jpg" style="max-width: 100%;"></a></p>
<h2><a id="user-content-ingredients-per-drink" class="anchor" aria-hidden="true" href="#ingredients-per-drink"><span aria-hidden="true" class="octicon octicon-link"></span></a>Ingredients per drink</h2>
<ul>
<li>1 egg white</li>
<li>1/2 oz rich Demerara syrup (see below - needs Demerara sugar, water and a bit of vodka)</li>
<li>.75oz lemon juice</li>
<li>1.5oz rye (I used Rittenhouse)</li>
<li>Angostura bitters</li>
</ul>
<p>I made two drinks in one go, so I doubled these.</p>
<p>I tried this recipe once with a fancy scotch but it wasn't nearly as nice as the one made with rye.</p>
<h2><a id="user-content-equipment" class="anchor" aria-hidden="true" href="#equipment"><span aria-hidden="true" class="octicon octicon-link"></span></a>Equipment</h2>
<ul>
<li>Jiggers</li>
<li>Cocktail shaker (I use two metal shaker cups that fit together)</li>
<li>Hand juicer</li>
<li>Strainer</li>
<li>2 double glasses</li>
<li>Small saucepan and scales bottle if making Demerara syrup</li>
</ul>
<h2><a id="user-content-rich-demerara-syrup" class="anchor" aria-hidden="true" href="#rich-demerara-syrup"><span aria-hidden="true" class="octicon octicon-link"></span></a>Rich Demerara syrup</h2>
<p>The syrup is a 2/1 ratio of sugar to water - so start with the saucepan on the scales and measure in around 20 units of Demerara sugar, then add 10 units of water.</p>
<p>Heat and stir to dissolve together, without boiling too much. Stir in a tiny splash of vodka. Empty into the glass bottle with the funnel and leave to cool.</p>
<h2><a id="user-content-making-two-whisky-sours" class="anchor" aria-hidden="true" href="#making-two-whisky-sours"><span aria-hidden="true" class="octicon octicon-link"></span></a>Making two whisky sours</h2>
<ol>
<li>Put the two glasses in the ice box in the freezer to chill</li>
<li>Separate two egg whites. Put the egg whites in the shaker</li>
<li>Add 1.5 floz freshly squeezed lemon juice</li>
<li>Add 1 floz rich Demerara syrup</li>
<li>Add 3 floz rye</li>
<li>Dry shake vigorously in the shaker. Dry shaking means shaking without ice. I went about 15 seconds. The result should be a foamy egg white mix.</li>
<li>Now add ice - I added 6 cubes. Shake vigorously again - I did another 15s.</li>
<li>Strain into the chilled glasses, each containing another four ice cubes (or a big square ice cube if you have them)</li>
<li>Add a couple of drops of angostura bitters on the top</li>
</ol>


