# til

path: docker_docker-compose-for-django-development.md
topic: docker
title: Docker Compose for Django development
url: https://github.com/simonw/til/blob/main/docker/docker-compose-for-django-development.md
created: 2021-05-24T22:08:23-07:00
created_utc: 2021-05-25T05:08:23+00:00
updated: 2021-05-26T11:32:27-07:00
updated_utc: 2021-05-26T18:32:27+00:00
shot_hash: 52bf5cb1337977b903d48ff3d30949df
slug: docker-compose-for-django-development
body:
I had to get Docker Compose working for a Django project, primarily to make it easier for other developers to get a working development environment.

Some features of this project:

- Uses GeoDjango, so needs GDAL etc for the Django app plus a PostgreSQL server running PostGIS
- Already has a `Dockerfile` used for the production deployment, but needed a separate one for the development environment
- Makes extensive use of Django migrations (over 100 and counting)

I ended up with this `docker-compose.yml` file in the root of the project:

```yaml
version: "3.1"

volumes:
  postgresql-data:
services:
  database:
    image: postgis/postgis:13-3.1
    restart: always
    expose:
      - "5432"
    ports:
      - "5432:5432"
    volumes:
      - postgresql-data:/var/lib/postgresql/data
    environment:
      POSTGRES_USER: postgres
      POSTGRES_DB: mydb
      POSTGRES_PASSWORD: postgres
  web:
    container_name: myapp
    platform: linux/amd64
    build:
      context: .
      dockerfile: Dockerfile.dev
    command: python manage.py runserver 0.0.0.0:3000
    environment:
      DATABASE_URL: postgres://postgres:postgres@database:5432/mydb
      DEBUG: 1
    volumes:
      - .:/app
    ports:
      - "3000:3000"
    depends_on:
      - migrations
      - database
  migrations:
    platform: linux/amd64
    build:
      context: .
      dockerfile: Dockerfile.dev
    command: python manage.py migrate --noinput
    environment:
      DATABASE_URL: postgres://postgres:postgres@database:5432/mydb
    volumes:
      - .:/app
    depends_on:
      - database
```
The `db` container runs PostGIS. It uses a named volume to persist PostgreSQL data in between container restarts.

The `web` container runs the Django development server, built using the custom `Dockerfile.dev` Dockerfile.

The `migrations` container simply runs the apps migrations and then terminates - with `depends_on` used to ensure that migrations run after the hdatabase server starts and before the web server.

Both `web` and `migrations` include a `platform: linux/amd64` property - this ensures they will work on M1 Macs even if the Python dependencies are not yet easily compiled for that architecture, see [Running Docker on an M1 Mac](https://til.simonwillison.net/macos/running-docker-on-remote-m1).

The `container_name: myapp` field on the `web` container is a convenience which means you can later run commands like this:

    docker exec -it myapp ./manage.py collectstatic

Here's `Dockerfile.dev`:

```dockerfile
FROM python:3.9-slim

ENV APP_HOME /app
WORKDIR $APP_HOME

ENV PYTHONUNBUFFERED 1

# gdal for GeoDjango
RUN apt-get update && apt-get install -y \
    binutils \
    gdal-bin \
    libproj-dev \
    git \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

WORKDIR $APP_HOME/myapp

RUN ./manage.py collectstatic --no-input

CMD python manage.py runserver 0.0.0.0:3000
```

html:
<p>I had to get Docker Compose working for a Django project, primarily to make it easier for other developers to get a working development environment.</p>
<p>Some features of this project:</p>
<ul>
<li>Uses GeoDjango, so needs GDAL etc for the Django app plus a PostgreSQL server running PostGIS</li>
<li>Already has a <code>Dockerfile</code> used for the production deployment, but needed a separate one for the development environment</li>
<li>Makes extensive use of Django migrations (over 100 and counting)</li>
</ul>
<p>I ended up with this <code>docker-compose.yml</code> file in the root of the project:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">version</span>: <span class="pl-s"><span class="pl-pds">"</span>3.1<span class="pl-pds">"</span></span>

<span class="pl-ent">volumes</span>:
  <span class="pl-ent">postgresql-data</span>:
<span class="pl-ent">services</span>:
  <span class="pl-ent">database</span>:
    <span class="pl-ent">image</span>: <span class="pl-s">postgis/postgis:13-3.1</span>
    <span class="pl-ent">restart</span>: <span class="pl-s">always</span>
    <span class="pl-ent">expose</span>:
      - <span class="pl-s"><span class="pl-pds">"</span>5432<span class="pl-pds">"</span></span>
    <span class="pl-ent">ports</span>:
      - <span class="pl-s"><span class="pl-pds">"</span>5432:5432<span class="pl-pds">"</span></span>
    <span class="pl-ent">volumes</span>:
      - <span class="pl-s">postgresql-data:/var/lib/postgresql/data</span>
    <span class="pl-ent">environment</span>:
      <span class="pl-ent">POSTGRES_USER</span>: <span class="pl-s">postgres</span>
      <span class="pl-ent">POSTGRES_DB</span>: <span class="pl-s">mydb</span>
      <span class="pl-ent">POSTGRES_PASSWORD</span>: <span class="pl-s">postgres</span>
  <span class="pl-ent">web</span>:
    <span class="pl-ent">container_name</span>: <span class="pl-s">myapp</span>
    <span class="pl-ent">platform</span>: <span class="pl-s">linux/amd64</span>
    <span class="pl-ent">build</span>:
      <span class="pl-ent">context</span>: <span class="pl-s">.</span>
      <span class="pl-ent">dockerfile</span>: <span class="pl-s">Dockerfile.dev</span>
    <span class="pl-ent">command</span>: <span class="pl-s">python manage.py runserver 0.0.0.0:3000</span>
    <span class="pl-ent">environment</span>:
      <span class="pl-ent">DATABASE_URL</span>: <span class="pl-s">postgres://postgres:postgres@database:5432/mydb</span>
      <span class="pl-ent">DEBUG</span>: <span class="pl-c1">1</span>
    <span class="pl-ent">volumes</span>:
      - <span class="pl-s">.:/app</span>
    <span class="pl-ent">ports</span>:
      - <span class="pl-s"><span class="pl-pds">"</span>3000:3000<span class="pl-pds">"</span></span>
    <span class="pl-ent">depends_on</span>:
      - <span class="pl-s">migrations</span>
      - <span class="pl-s">database</span>
  <span class="pl-ent">migrations</span>:
    <span class="pl-ent">platform</span>: <span class="pl-s">linux/amd64</span>
    <span class="pl-ent">build</span>:
      <span class="pl-ent">context</span>: <span class="pl-s">.</span>
      <span class="pl-ent">dockerfile</span>: <span class="pl-s">Dockerfile.dev</span>
    <span class="pl-ent">command</span>: <span class="pl-s">python manage.py migrate --noinput</span>
    <span class="pl-ent">environment</span>:
      <span class="pl-ent">DATABASE_URL</span>: <span class="pl-s">postgres://postgres:postgres@database:5432/mydb</span>
    <span class="pl-ent">volumes</span>:
      - <span class="pl-s">.:/app</span>
    <span class="pl-ent">depends_on</span>:
      - <span class="pl-s">database</span></pre></div>
<p>The <code>db</code> container runs PostGIS. It uses a named volume to persist PostgreSQL data in between container restarts.</p>
<p>The <code>web</code> container runs the Django development server, built using the custom <code>Dockerfile.dev</code> Dockerfile.</p>
<p>The <code>migrations</code> container simply runs the apps migrations and then terminates - with <code>depends_on</code> used to ensure that migrations run after the hdatabase server starts and before the web server.</p>
<p>Both <code>web</code> and <code>migrations</code> include a <code>platform: linux/amd64</code> property - this ensures they will work on M1 Macs even if the Python dependencies are not yet easily compiled for that architecture, see <a href="https://til.simonwillison.net/macos/running-docker-on-remote-m1" rel="nofollow">Running Docker on an M1 Mac</a>.</p>
<p>The <code>container_name: myapp</code> field on the <code>web</code> container is a convenience which means you can later run commands like this:</p>
<pre><code>docker exec -it myapp ./manage.py collectstatic
</code></pre>
<p>Here's <code>Dockerfile.dev</code>:</p>
<div class="highlight highlight-source-dockerfile"><pre><span class="pl-k">FROM</span> python:3.9-slim

<span class="pl-k">ENV</span> APP_HOME /app
<span class="pl-k">WORKDIR</span> $APP_HOME

<span class="pl-k">ENV</span> PYTHONUNBUFFERED 1

<span class="pl-c"><span class="pl-c">#</span> gdal for GeoDjango</span>
<span class="pl-k">RUN</span> apt-get update &amp;&amp; apt-get install -y \
    binutils \
    gdal-bin \
    libproj-dev \
    git \
    &amp;&amp; rm -rf /var/lib/apt/lists/*

<span class="pl-k">COPY</span> requirements.txt .
<span class="pl-k">RUN</span> pip install --no-cache-dir -r requirements.txt

<span class="pl-k">WORKDIR</span> $APP_HOME/myapp

<span class="pl-k">RUN</span> ./manage.py collectstatic --no-input

<span class="pl-k">CMD</span> python manage.py runserver 0.0.0.0:3000</pre></div>


path: jq_array-of-array-to-objects.md
topic: jq
title: Turning an array of arrays into objects with jq
url: https://github.com/simonw/til/blob/main/jq/array-of-array-to-objects.md
created: 2021-05-25T08:33:46-07:00
created_utc: 2021-05-25T15:33:46+00:00
updated: 2021-05-25T08:33:46-07:00
updated_utc: 2021-05-25T15:33:46+00:00
shot_hash: 699c39e721ab73c2f2300c45acdbd8f3
slug: array-of-array-to-objects
body:
Input:
```json
[
    ["mm.domus.SW230"," A LA RONDE","Buildings:Houses:Medium houses",50.642781,-3.405508],
    ["mm.domus.SW193"," ALEXANDER KEILLER MUSEUM","Archaeology:Prehistory",51.427927,-1.857344],
    ["mm.domus.SE416"," ANNE OF CLEVES HOUSE MUSEUM","Buildings:Houses:Medium houses",50.869227,0.005329],
]
```
I want an array of objects. Here's what I came up with, using [jqplay.org](https://jqplay.org/):

```jq
[.[] | {id: .[0], name: .[1], category: .[2], latitude: .[3], longitude: .[4]}]
```
This outputs:
```json
[
  {
    "id": "mm.domus.SW230",
    "name": " A LA RONDE",
    "category": "Buildings:Houses:Medium houses",
    "latitude": 50.642781,
    "longitude": -3.405508
  },
  {
    "id": "mm.domus.SW193",
    "name": " ALEXANDER KEILLER MUSEUM",
    "category": "Archaeology:Prehistory",
    "latitude": 51.427927,
    "longitude": -1.857344
  },
  {
    "id": "mm.domus.SE416",
    "name": " ANNE OF CLEVES HOUSE MUSEUM",
    "category": "Buildings:Houses:Medium houses",
    "latitude": 50.869227,
    "longitude": 0.005329
  }
]
```
If you remove the outer `[` and `]` and use the "Compact output" option you get back this instead:
```
{"id":"mm.domus.SW230","name":" A LA RONDE","category":"Buildings:Houses:Medium houses","latitude":50.642781,"longitude":-3.405508}
{"id":"mm.domus.SW193","name":" ALEXANDER KEILLER MUSEUM","category":"Archaeology:Prehistory","latitude":51.427927,"longitude":-1.857344}
{"id":"mm.domus.SE416","name":" ANNE OF CLEVES HOUSE MUSEUM","category":"Buildings:Houses:Medium houses","latitude":50.869227,"longitude":0.005329}

```

html:
<p>Input:</p>
<div class="highlight highlight-source-json"><pre>[
    [<span class="pl-s"><span class="pl-pds">"</span>mm.domus.SW230<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span> A LA RONDE<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span>Buildings:Houses:Medium houses<span class="pl-pds">"</span></span>,<span class="pl-c1">50.642781</span>,<span class="pl-c1">-3.405508</span>],
    [<span class="pl-s"><span class="pl-pds">"</span>mm.domus.SW193<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span> ALEXANDER KEILLER MUSEUM<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span>Archaeology:Prehistory<span class="pl-pds">"</span></span>,<span class="pl-c1">51.427927</span>,<span class="pl-c1">-1.857344</span>],
    [<span class="pl-s"><span class="pl-pds">"</span>mm.domus.SE416<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span> ANNE OF CLEVES HOUSE MUSEUM<span class="pl-pds">"</span></span>,<span class="pl-s"><span class="pl-pds">"</span>Buildings:Houses:Medium houses<span class="pl-pds">"</span></span>,<span class="pl-c1">50.869227</span>,<span class="pl-c1">0.005329</span>],
]</pre></div>
<p>I want an array of objects. Here's what I came up with, using <a href="https://jqplay.org/" rel="nofollow">jqplay.org</a>:</p>
<div class="highlight highlight-source-jq"><pre>[<span class="pl-k">.</span>[] <span class="pl-k">|</span> {<span class="pl-e">id</span>: <span class="pl-k">.</span>[<span class="pl-c1">0</span>], <span class="pl-e">name</span>: <span class="pl-k">.</span>[<span class="pl-c1">1</span>], <span class="pl-e">category</span>: <span class="pl-k">.</span>[<span class="pl-c1">2</span>], <span class="pl-e">latitude</span>: <span class="pl-k">.</span>[<span class="pl-c1">3</span>], <span class="pl-e">longitude</span>: <span class="pl-k">.</span>[<span class="pl-c1">4</span>]}]</pre></div>
<p>This outputs:</p>
<div class="highlight highlight-source-json"><pre>[
  {
    <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>mm.domus.SW230<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span> A LA RONDE<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>category<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Buildings:Houses:Medium houses<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>latitude<span class="pl-pds">"</span></span>: <span class="pl-c1">50.642781</span>,
    <span class="pl-s"><span class="pl-pds">"</span>longitude<span class="pl-pds">"</span></span>: <span class="pl-c1">-3.405508</span>
  },
  {
    <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>mm.domus.SW193<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span> ALEXANDER KEILLER MUSEUM<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>category<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Archaeology:Prehistory<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>latitude<span class="pl-pds">"</span></span>: <span class="pl-c1">51.427927</span>,
    <span class="pl-s"><span class="pl-pds">"</span>longitude<span class="pl-pds">"</span></span>: <span class="pl-c1">-1.857344</span>
  },
  {
    <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>mm.domus.SE416<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span> ANNE OF CLEVES HOUSE MUSEUM<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>category<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Buildings:Houses:Medium houses<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>latitude<span class="pl-pds">"</span></span>: <span class="pl-c1">50.869227</span>,
    <span class="pl-s"><span class="pl-pds">"</span>longitude<span class="pl-pds">"</span></span>: <span class="pl-c1">0.005329</span>
  }
]</pre></div>
<p>If you remove the outer <code>[</code> and <code>]</code> and use the "Compact output" option you get back this instead:</p>
<pre><code>{"id":"mm.domus.SW230","name":" A LA RONDE","category":"Buildings:Houses:Medium houses","latitude":50.642781,"longitude":-3.405508}
{"id":"mm.domus.SW193","name":" ALEXANDER KEILLER MUSEUM","category":"Archaeology:Prehistory","latitude":51.427927,"longitude":-1.857344}
{"id":"mm.domus.SE416","name":" ANNE OF CLEVES HOUSE MUSEUM","category":"Buildings:Houses:Medium houses","latitude":50.869227,"longitude":0.005329}

</code></pre>


path: macos_running-docker-on-remote-m1.md
topic: macos
title: Running Docker on an M1 Mac
url: https://github.com/simonw/til/blob/main/macos/running-docker-on-remote-m1.md
created: 2021-05-25T16:30:18-07:00
created_utc: 2021-05-25T23:30:18+00:00
updated: 2021-05-26T08:17:17-07:00
updated_utc: 2021-05-26T15:17:17+00:00
shot_hash: 978287f940981270b7336e0694b4aee0
slug: running-docker-on-remote-m1
body:
*TLDR: Use `platform: linux/amd64` in `docker-compose.yml` and your containers will probably work!* 

I was trying to figure out how to get a development environment for a Django project running inside Docker in a M1 Mac.

Since I don't have an M1 Mac, I decided to rent one. AWS haven't launched this yet (at least anywhere I could find it), and Scaleway were out of stock on their machines which you can rent by the day, so I tried using a $109/month M1 Mac Mini [from MacStadium](https://www.macstadium.com/m1-mini).

The machine became available within a minute of me entering the card details, and gave me an IP address I could connect to plus an administrator account username and password. These worked for both VNC and SSH.

For VNC, navigating to `vnc://ip.address.here` in Safari opened the macOS Screen Sharing app and prompted me to connect. For SSH, `ssh administrator@ip.addresse.here` and then entering the password did the trick.

## Installing Homebrew and Docker

First, most important lesson: do _not_ attempt to install these large pieces of software over SSH! Both of these had points within the install process that triggered desktop UI interactions of some sort, and trying to start them over SSH lead to a bewildering array of permission errors. I ended up having to turn the machine off through the MacStadium web interface, waiting a while and then turning it back on again in order to clear the errors (a soft reboot didn't help).

So be sure to open a terminal in the Screen Sharing app over VNC and run the install commands there.

Pasting in the installation command from https://brew.sh/ did the trick. I then installed Docker using `brew install --cask docker`. These both needed to be done not over SSH.

Docker for Mac required Rosetta 2 - thankfully installing that is a case of running `softwareupdate --install-rosetta` - which for me only took a couple of seconds and displayed an error message which appeared not to matter at all, since Docker for Mac then started working fine.

When I tried to run Docker containers with mounted volumes I ran into an issue where a UI prompt displayed on the macOS desktop stating that Docker wanted access to my filesystem - another reason to run the first set of commands over VNC rather than SSH.

Eventually I got to a point where `docker run` commands were safe to run via SSH instead of a terminal over VNC.

## Editing files with VS Code

Not unique to remote macOS but this was the first time I used the VS Code [Remote Development using SSH](https://code.visualstudio.com/docs/remote/ssh) extension and it worked flawlessly - I gave it the IP address, username and password that I had used over SSH and I was able to edit files on the remote Mac in the same way I edit files on my local machine. Fantastic.

## And then I almost gave up...

Getting things to work in Docker on an M1 is really, really hard. Python dependencies using wheels such as `psycopg2-binary` failed to install. `apt-get install ...` inside a Docker container seemed to work for most packages - I upgraded to `FROM python:3.9-buster` as my base image and was able to run `apt-get install python-psycopg2` to talk to PostgreSQL.

The I ran into this error on container startup:

```
docker run -it -v "$(pwd):/app" vial_web bash
root@a9ae5a2956a5:/app/vaccinate# python -i ./manage.py runserver 0.0.0.0:3000
Performing system checks...
free(): invalid pointer
Traceback (most recent call last):
  File "./manage.py", line 29, in <module>
    main()
  File "./manage.py", line 25, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 61, in execute
    super().execute(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 96, in handle
    self.run(**options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 103, in run
    autoreload.run_with_reloader(self.inner_run, **options)
  File "/usr/local/lib/python3.8/site-packages/django/utils/autoreload.py", line 640, in run_with_reloader
    sys.exit(exit_code)
SystemExit: -6
```
That `free(): invalid pointer` error completely stumped me, since it didn't come with a detailed enough stacktrace that I could figure out which dependency (I have a hunch it was caused by a C deopendency) was causing it, and I couldn't figure out debugging techniques from searching the internet.

Armin Ronacher [said on Twitter](https://twitter.com/mitsuhiko/status/1397266788262584325):

> M1 macs add a whole new world of crazy to Python. Apple Python and homebrew Python and the different types of pyenv pythons and different cpu archs add completely new sources of frustration and errors. I still get random compiler errors from macos version mismatches on big sur.

Which made me feel a little bit better about being defeated! Until...

## platform: linux/amd64 to the rescue!

Mo McRoberts on Twitter [pointed out that](https://twitter.com/nevali/status/1397337506962677763):

> also Docker for M1 runs linux/amd64 containers *really well*, not ideal, but good enough for a lot of uses

So I dug in a bit.... and it turns out you can tell Docker for Mac to build and run containers using the `linux/amd64` architecture and everything... just works!

I'm using Docker Compose (see [Docker Compose for Django development](https://til.simonwillison.net/docker/docker-compose-for-django-development) so I added the `platform: linux/amd64` YAML key for my container:

```yaml
  web:
    platform: linux/amd64
    build:
      context: .
      dockerfile: Dockerfile.dev
    command: python manage.py runserver 0.0.0.0:3000
```
And that did the trick! I ran `docker-compose build` to rebuild the containers and `docker-compose up` to start them running and everything worked exactly the same as it did on my non-M1 Mac laptop.

html:
<p><em>TLDR: Use <code>platform: linux/amd64</code> in <code>docker-compose.yml</code> and your containers will probably work!</em></p>
<p>I was trying to figure out how to get a development environment for a Django project running inside Docker in a M1 Mac.</p>
<p>Since I don't have an M1 Mac, I decided to rent one. AWS haven't launched this yet (at least anywhere I could find it), and Scaleway were out of stock on their machines which you can rent by the day, so I tried using a $109/month M1 Mac Mini <a href="https://www.macstadium.com/m1-mini" rel="nofollow">from MacStadium</a>.</p>
<p>The machine became available within a minute of me entering the card details, and gave me an IP address I could connect to plus an administrator account username and password. These worked for both VNC and SSH.</p>
<p>For VNC, navigating to <code>vnc://ip.address.here</code> in Safari opened the macOS Screen Sharing app and prompted me to connect. For SSH, <code>ssh administrator@ip.addresse.here</code> and then entering the password did the trick.</p>
<h2>
<a id="user-content-installing-homebrew-and-docker" class="anchor" href="#installing-homebrew-and-docker" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing Homebrew and Docker</h2>
<p>First, most important lesson: do <em>not</em> attempt to install these large pieces of software over SSH! Both of these had points within the install process that triggered desktop UI interactions of some sort, and trying to start them over SSH lead to a bewildering array of permission errors. I ended up having to turn the machine off through the MacStadium web interface, waiting a while and then turning it back on again in order to clear the errors (a soft reboot didn't help).</p>
<p>So be sure to open a terminal in the Screen Sharing app over VNC and run the install commands there.</p>
<p>Pasting in the installation command from <a href="https://brew.sh/" rel="nofollow">https://brew.sh/</a> did the trick. I then installed Docker using <code>brew install --cask docker</code>. These both needed to be done not over SSH.</p>
<p>Docker for Mac required Rosetta 2 - thankfully installing that is a case of running <code>softwareupdate --install-rosetta</code> - which for me only took a couple of seconds and displayed an error message which appeared not to matter at all, since Docker for Mac then started working fine.</p>
<p>When I tried to run Docker containers with mounted volumes I ran into an issue where a UI prompt displayed on the macOS desktop stating that Docker wanted access to my filesystem - another reason to run the first set of commands over VNC rather than SSH.</p>
<p>Eventually I got to a point where <code>docker run</code> commands were safe to run via SSH instead of a terminal over VNC.</p>
<h2>
<a id="user-content-editing-files-with-vs-code" class="anchor" href="#editing-files-with-vs-code" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Editing files with VS Code</h2>
<p>Not unique to remote macOS but this was the first time I used the VS Code <a href="https://code.visualstudio.com/docs/remote/ssh" rel="nofollow">Remote Development using SSH</a> extension and it worked flawlessly - I gave it the IP address, username and password that I had used over SSH and I was able to edit files on the remote Mac in the same way I edit files on my local machine. Fantastic.</p>
<h2>
<a id="user-content-and-then-i-almost-gave-up" class="anchor" href="#and-then-i-almost-gave-up" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>And then I almost gave up...</h2>
<p>Getting things to work in Docker on an M1 is really, really hard. Python dependencies using wheels such as <code>psycopg2-binary</code> failed to install. <code>apt-get install ...</code> inside a Docker container seemed to work for most packages - I upgraded to <code>FROM python:3.9-buster</code> as my base image and was able to run <code>apt-get install python-psycopg2</code> to talk to PostgreSQL.</p>
<p>The I ran into this error on container startup:</p>
<pre><code>docker run -it -v "$(pwd):/app" vial_web bash
root@a9ae5a2956a5:/app/vaccinate# python -i ./manage.py runserver 0.0.0.0:3000
Performing system checks...
free(): invalid pointer
Traceback (most recent call last):
  File "./manage.py", line 29, in &lt;module&gt;
    main()
  File "./manage.py", line 25, in main
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 61, in execute
    super().execute(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 96, in handle
    self.run(**options)
  File "/usr/local/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 103, in run
    autoreload.run_with_reloader(self.inner_run, **options)
  File "/usr/local/lib/python3.8/site-packages/django/utils/autoreload.py", line 640, in run_with_reloader
    sys.exit(exit_code)
SystemExit: -6
</code></pre>
<p>That <code>free(): invalid pointer</code> error completely stumped me, since it didn't come with a detailed enough stacktrace that I could figure out which dependency (I have a hunch it was caused by a C deopendency) was causing it, and I couldn't figure out debugging techniques from searching the internet.</p>
<p>Armin Ronacher <a href="https://twitter.com/mitsuhiko/status/1397266788262584325" rel="nofollow">said on Twitter</a>:</p>
<blockquote>
<p>M1 macs add a whole new world of crazy to Python. Apple Python and homebrew Python and the different types of pyenv pythons and different cpu archs add completely new sources of frustration and errors. I still get random compiler errors from macos version mismatches on big sur.</p>
</blockquote>
<p>Which made me feel a little bit better about being defeated! Until...</p>
<h2>
<a id="user-content-platform-linuxamd64-to-the-rescue" class="anchor" href="#platform-linuxamd64-to-the-rescue" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>platform: linux/amd64 to the rescue!</h2>
<p>Mo McRoberts on Twitter <a href="https://twitter.com/nevali/status/1397337506962677763" rel="nofollow">pointed out that</a>:</p>
<blockquote>
<p>also Docker for M1 runs linux/amd64 containers <em>really well</em>, not ideal, but good enough for a lot of uses</p>
</blockquote>
<p>So I dug in a bit.... and it turns out you can tell Docker for Mac to build and run containers using the <code>linux/amd64</code> architecture and everything... just works!</p>
<p>I'm using Docker Compose (see <a href="https://til.simonwillison.net/docker/docker-compose-for-django-development" rel="nofollow">Docker Compose for Django development</a> so I added the <code>platform: linux/amd64</code> YAML key for my container:</p>
<div class="highlight highlight-source-yaml"><pre>  <span class="pl-ent">web</span>:
    <span class="pl-ent">platform</span>: <span class="pl-s">linux/amd64</span>
    <span class="pl-ent">build</span>:
      <span class="pl-ent">context</span>: <span class="pl-s">.</span>
      <span class="pl-ent">dockerfile</span>: <span class="pl-s">Dockerfile.dev</span>
    <span class="pl-ent">command</span>: <span class="pl-s">python manage.py runserver 0.0.0.0:3000</span></pre></div>
<p>And that did the trick! I ran <code>docker-compose build</code> to rebuild the containers and <code>docker-compose up</code> to start them running and everything worked exactly the same as it did on my non-M1 Mac laptop.</p>


path: bash_finding-bom-csv-files-with-ripgrep.md
topic: bash
title: Finding CSV files that start with a BOM using ripgrep
url: https://github.com/simonw/til/blob/main/bash/finding-bom-csv-files-with-ripgrep.md
created: 2021-05-28T22:23:45-07:00
created_utc: 2021-05-29T05:23:45+00:00
updated: 2021-05-28T22:23:45-07:00
updated_utc: 2021-05-29T05:23:45+00:00
shot_hash: 708508f8876dcdb33cc2e58461643886
slug: finding-bom-csv-files-with-ripgrep
body:
For [sqlite-utils issue 250](https://github.com/simonw/sqlite-utils/issues/250) I needed to locate some test CSV files that start with a UTF-8 BOM.

Here's how I did that using [ripgrep](https://github.com/BurntSushi/ripgrep):
```
$ rg --multiline --encoding none '^(?-u:\xEF\xBB\xBF)' --glob '*.csv' .
```
The `--multiline` option means the search spans multiple lines - I only want to match entire files that begin with my search term, so this means that `^` will match the start of the file, not the start of individual lines.

`--encoding none` runs the search against the raw bytes of the file, disabling ripgrep's default BOM detection.

`--glob '*.csv'` causes ripgrep to search only CSV files.

The regular expression itself looks like this:

    ^(?-u:\xEF\xBB\xBF)

This is [rust regex](https://docs.rs/regex/1.5.4/regex/#syntax) syntax.

`(?-u:` means "turn OFF the `u` flag for the duration of this block" - the `u` flag, which is on by default, causes the Rust regex engine to interpret input as unicode. So within the rest of that `(...)` block we can use escaped byte sequences.

Finally, `\xEF\xBB\xBF` is the byte sequence for the UTF-8 BOM itself.

html:
<p>For <a href="https://github.com/simonw/sqlite-utils/issues/250">sqlite-utils issue 250</a> I needed to locate some test CSV files that start with a UTF-8 BOM.</p>
<p>Here's how I did that using <a href="https://github.com/BurntSushi/ripgrep">ripgrep</a>:</p>
<pre><code>$ rg --multiline --encoding none '^(?-u:\xEF\xBB\xBF)' --glob '*.csv' .
</code></pre>
<p>The <code>--multiline</code> option means the search spans multiple lines - I only want to match entire files that begin with my search term, so this means that <code>^</code> will match the start of the file, not the start of individual lines.</p>
<p><code>--encoding none</code> runs the search against the raw bytes of the file, disabling ripgrep's default BOM detection.</p>
<p><code>--glob '*.csv'</code> causes ripgrep to search only CSV files.</p>
<p>The regular expression itself looks like this:</p>
<pre><code>^(?-u:\xEF\xBB\xBF)
</code></pre>
<p>This is <a href="https://docs.rs/regex/1.5.4/regex/#syntax" rel="nofollow">rust regex</a> syntax.</p>
<p><code>(?-u:</code> means "turn OFF the <code>u</code> flag for the duration of this block" - the <code>u</code> flag, which is on by default, causes the Rust regex engine to interpret input as unicode. So within the rest of that <code>(...)</code> block we can use escaped byte sequences.</p>
<p>Finally, <code>\xEF\xBB\xBF</code> is the byte sequence for the UTF-8 BOM itself.</p>


path: sql_django-group-permissions-markdown.md
topic: sql
title: Building a Markdown summary of Django group permissions
url: https://github.com/simonw/til/blob/main/sql/django-group-permissions-markdown.md
created: 2021-06-03T09:15:07-07:00
created_utc: 2021-06-03T16:15:07+00:00
updated: 2021-06-04T12:07:08-07:00
updated_utc: 2021-06-04T19:07:08+00:00
shot_hash: 73cbbef707e318c9274a2fecaf2507db
slug: django-group-permissions-markdown
body:
[Django SQL Dashboard](https://django-sql-dashboard.datasette.io/) can display rendered markdown generated by a SQL query.

Here's a query I figured out today for showing a summary of all Django groups, their members and the list of permissions those groups hold:

```sql
select
  (
    '# ' || (
      select
        name
      from
        auth_group
      where
        id = auth_group_original.id
    ) || '

' || (
      select
        count(*)
      from
        auth_user_groups
      where
        auth_user_groups.group_id = auth_group_original.id
    ) || ' members:

' || (
      select
        string_agg(auth_user.username, ', ')
      from
        auth_user
        join auth_user_groups on auth_user_groups.user_id = auth_user.id
      where
        auth_user_groups.group_id = auth_group_original.id
    ) || '

Permissions: 

' || (
      select
        coalesce(
          string_agg(
            '* ' || auth_permission.name,
            '
'
          ),
          'No permissions'
        )
      from
        auth_permission
        join auth_group_permissions on auth_group_permissions.permission_id = auth_permission.id
      where
        auth_group_permissions.group_id = auth_group_original.id
    )
  ) as markdown
from
  auth_group auth_group_original
```
The output looks like this:

<img width="473" alt="Groups_and_permissions" src="https://user-images.githubusercontent.com/9599/120677616-1ee7d300-c44c-11eb-95c6-ca5053338621.png">

The first time I ran this I got back `null` for the Markdown for one of the records - it turned out that one of my groups had no permissions at all, so this query:

```sql
select
  string_agg(
    '* ' || auth_permission.name,
    '
'
  )
from
  auth_permission
  join auth_group_permissions on auth_group_permissions.permission_id = auth_permission.id
where
  auth_group_permissions.group_id = auth_group_original.id
```
Returned `null` - and concatenating `null` to everything else produced `null` too.

The fix was to add a `coalesce()` - which returns the first not-null argument - like this:

```sql
select coalesce(string_agg(...), 'No permissions')
```

html:
<p><a href="https://django-sql-dashboard.datasette.io/" rel="nofollow">Django SQL Dashboard</a> can display rendered markdown generated by a SQL query.</p>
<p>Here's a query I figured out today for showing a summary of all Django groups, their members and the list of permissions those groups hold:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  (
    <span class="pl-s"><span class="pl-pds">'</span># <span class="pl-pds">'</span></span> <span class="pl-k">||</span> (
      <span class="pl-k">select</span>
        name
      <span class="pl-k">from</span>
        auth_group
      <span class="pl-k">where</span>
        id <span class="pl-k">=</span> <span class="pl-c1">auth_group_original</span>.<span class="pl-c1">id</span>
    ) <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s"></span>
<span class="pl-s"><span class="pl-pds">'</span></span> <span class="pl-k">||</span> (
      <span class="pl-k">select</span>
        <span class="pl-c1">count</span>(<span class="pl-k">*</span>)
      <span class="pl-k">from</span>
        auth_user_groups
      <span class="pl-k">where</span>
        <span class="pl-c1">auth_user_groups</span>.<span class="pl-c1">group_id</span> <span class="pl-k">=</span> <span class="pl-c1">auth_group_original</span>.<span class="pl-c1">id</span>
    ) <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span> members:</span>
<span class="pl-s"></span>
<span class="pl-s"><span class="pl-pds">'</span></span> <span class="pl-k">||</span> (
      <span class="pl-k">select</span>
        string_agg(<span class="pl-c1">auth_user</span>.<span class="pl-c1">username</span>, <span class="pl-s"><span class="pl-pds">'</span>, <span class="pl-pds">'</span></span>)
      <span class="pl-k">from</span>
        auth_user
        <span class="pl-k">join</span> auth_user_groups <span class="pl-k">on</span> <span class="pl-c1">auth_user_groups</span>.<span class="pl-c1">user_id</span> <span class="pl-k">=</span> <span class="pl-c1">auth_user</span>.<span class="pl-c1">id</span>
      <span class="pl-k">where</span>
        <span class="pl-c1">auth_user_groups</span>.<span class="pl-c1">group_id</span> <span class="pl-k">=</span> <span class="pl-c1">auth_group_original</span>.<span class="pl-c1">id</span>
    ) <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s"></span>
<span class="pl-s">Permissions: </span>
<span class="pl-s"></span>
<span class="pl-s"><span class="pl-pds">'</span></span> <span class="pl-k">||</span> (
      <span class="pl-k">select</span>
        coalesce(
          string_agg(
            <span class="pl-s"><span class="pl-pds">'</span>* <span class="pl-pds">'</span></span> <span class="pl-k">||</span> <span class="pl-c1">auth_permission</span>.<span class="pl-c1">name</span>,
            <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s"><span class="pl-pds">'</span></span>
          ),
          <span class="pl-s"><span class="pl-pds">'</span>No permissions<span class="pl-pds">'</span></span>
        )
      <span class="pl-k">from</span>
        auth_permission
        <span class="pl-k">join</span> auth_group_permissions <span class="pl-k">on</span> <span class="pl-c1">auth_group_permissions</span>.<span class="pl-c1">permission_id</span> <span class="pl-k">=</span> <span class="pl-c1">auth_permission</span>.<span class="pl-c1">id</span>
      <span class="pl-k">where</span>
        <span class="pl-c1">auth_group_permissions</span>.<span class="pl-c1">group_id</span> <span class="pl-k">=</span> <span class="pl-c1">auth_group_original</span>.<span class="pl-c1">id</span>
    )
  ) <span class="pl-k">as</span> markdown
<span class="pl-k">from</span>
  auth_group auth_group_original</pre></div>
<p>The output looks like this:</p>
<p><a href="https://user-images.githubusercontent.com/9599/120677616-1ee7d300-c44c-11eb-95c6-ca5053338621.png" target="_blank" rel="nofollow"><img width="473" alt="Groups_and_permissions" src="https://user-images.githubusercontent.com/9599/120677616-1ee7d300-c44c-11eb-95c6-ca5053338621.png" style="max-width:100%;"></a></p>
<p>The first time I ran this I got back <code>null</code> for the Markdown for one of the records - it turned out that one of my groups had no permissions at all, so this query:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  string_agg(
    <span class="pl-s"><span class="pl-pds">'</span>* <span class="pl-pds">'</span></span> <span class="pl-k">||</span> <span class="pl-c1">auth_permission</span>.<span class="pl-c1">name</span>,
    <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s"><span class="pl-pds">'</span></span>
  )
<span class="pl-k">from</span>
  auth_permission
  <span class="pl-k">join</span> auth_group_permissions <span class="pl-k">on</span> <span class="pl-c1">auth_group_permissions</span>.<span class="pl-c1">permission_id</span> <span class="pl-k">=</span> <span class="pl-c1">auth_permission</span>.<span class="pl-c1">id</span>
<span class="pl-k">where</span>
  <span class="pl-c1">auth_group_permissions</span>.<span class="pl-c1">group_id</span> <span class="pl-k">=</span> <span class="pl-c1">auth_group_original</span>.<span class="pl-c1">id</span></pre></div>
<p>Returned <code>null</code> - and concatenating <code>null</code> to everything else produced <code>null</code> too.</p>
<p>The fix was to add a <code>coalesce()</code> - which returns the first not-null argument - like this:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> coalesce(string_agg(...), <span class="pl-s"><span class="pl-pds">'</span>No permissions<span class="pl-pds">'</span></span>)</pre></div>


path: amplitude_export-events-to-datasette.md
topic: amplitude
title: Exporting Amplitude events to SQLite
url: https://github.com/simonw/til/blob/main/amplitude/export-events-to-datasette.md
created: 2021-06-06T13:56:09-07:00
created_utc: 2021-06-06T20:56:09+00:00
updated: 2021-06-06T13:56:09-07:00
updated_utc: 2021-06-06T20:56:09+00:00
shot_hash: e12c89da0cb3c1fabcf189aaed72925a
slug: export-events-to-datasette
body:
[Amplitude](https://amplitude.com/) offers an "Export Data" button in the project settings page. This can export up to 365 days of events (up to 4GB per export), where the export is a zip file containing `*.json.gz` gzipped newline-delimited JSON.

You can export multiple times, so if you have more than a year of events you can export them by specifying different date ranges. It's OK to overlap these ranges as each event has a uniue `uuid` that can be used to de-duplicate them.

Here's how to import that into a SQLite database using `sqlite-utils`:
```
unzip export # The exported file does not have a .zip extension for some reason
cd DIRECTORY_CREATED_FROM_EXPORT
gzcat *.json.gz | sqlite-utils insert amplitude.db events --nl --alter --pk uuid --ignore -
```
Since we are using `--pk uuid` and `--ignore` it's safe to run this against as many exported `*.json.gz` files as you like, including exports that overlap each other.

Then run `datasette amplitude.db` to browse the results.

html:
<p><a href="https://amplitude.com/" rel="nofollow">Amplitude</a> offers an "Export Data" button in the project settings page. This can export up to 365 days of events (up to 4GB per export), where the export is a zip file containing <code>*.json.gz</code> gzipped newline-delimited JSON.</p>
<p>You can export multiple times, so if you have more than a year of events you can export them by specifying different date ranges. It's OK to overlap these ranges as each event has a uniue <code>uuid</code> that can be used to de-duplicate them.</p>
<p>Here's how to import that into a SQLite database using <code>sqlite-utils</code>:</p>
<pre><code>unzip export # The exported file does not have a .zip extension for some reason
cd DIRECTORY_CREATED_FROM_EXPORT
gzcat *.json.gz | sqlite-utils insert amplitude.db events --nl --alter --pk uuid --ignore -
</code></pre>
<p>Since we are using <code>--pk uuid</code> and <code>--ignore</code> it's safe to run this against as many exported <code>*.json.gz</code> files as you like, including exports that overlap each other.</p>
<p>Then run <code>datasette amplitude.db</code> to browse the results.</p>


path: homebrew_mysql-homebrew.md
topic: homebrew
title: Running a MySQL server using Homebrew
url: https://github.com/simonw/til/blob/main/homebrew/mysql-homebrew.md
created: 2021-06-11T11:48:21-07:00
created_utc: 2021-06-11T18:48:21+00:00
updated: 2021-06-11T13:04:50-07:00
updated_utc: 2021-06-11T20:04:50+00:00
shot_hash: a8e8de6f0bc0b9b4febbfcb67e5f073d
slug: mysql-homebrew
body:
First, install MySQL like so:

    brew install mysql

This installs the server but doesn't run it. You  can run it in the background like this:
```
% mysql.server start
Starting MySQL
.. SUCCESS! 
%
```
Then later on you can stop it like so:
```
% mysql.server stop 
Shutting down MySQL
. SUCCESS! 
%
```
While it's running it defaults to having a root account that only accepts connections from localhost with no password:
```
% mysql -u root       
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
...
mysql> 
```
Running `mysql_secure_installation` runs a wizard that helps set up a password.

When you first install it, Homebrew says:
```
To have launchd start mysql now and restart at login:
  brew services start mysql
```
You can re-display that message by running `brew reinstall mysql`.

## Installing the mysqlclient Python library

This took me a long time to figure out. Eventually this worked:

    MYSQLCLIENT_CFLAGS=`pkg-config mysqlclient --cflags` \
      MYSQLCLIENT_LDFLAGS=`pkg-config mysqlclient --libs` \
      pip install mysqlclient

html:
<p>First, install MySQL like so:</p>
<pre><code>brew install mysql
</code></pre>
<p>This installs the server but doesn't run it. You  can run it in the background like this:</p>
<pre><code>% mysql.server start
Starting MySQL
.. SUCCESS! 
%
</code></pre>
<p>Then later on you can stop it like so:</p>
<pre><code>% mysql.server stop 
Shutting down MySQL
. SUCCESS! 
%
</code></pre>
<p>While it's running it defaults to having a root account that only accepts connections from localhost with no password:</p>
<pre><code>% mysql -u root       
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
...
mysql&gt; 
</code></pre>
<p>Running <code>mysql_secure_installation</code> runs a wizard that helps set up a password.</p>
<p>When you first install it, Homebrew says:</p>
<pre><code>To have launchd start mysql now and restart at login:
  brew services start mysql
</code></pre>
<p>You can re-display that message by running <code>brew reinstall mysql</code>.</p>
<h2>
<a id="user-content-installing-the-mysqlclient-python-library" class="anchor" href="#installing-the-mysqlclient-python-library" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing the mysqlclient Python library</h2>
<p>This took me a long time to figure out. Eventually this worked:</p>
<pre><code>MYSQLCLIENT_CFLAGS=`pkg-config mysqlclient --cflags` \
  MYSQLCLIENT_LDFLAGS=`pkg-config mysqlclient --libs` \
  pip install mysqlclient
</code></pre>


path: vim_mouse-support-in-vim.md
topic: vim
title: Mouse support in vim
url: https://github.com/simonw/til/blob/main/vim/mouse-support-in-vim.md
created: 2021-06-19T14:57:53-07:00
created_utc: 2021-06-19T21:57:53+00:00
updated: 2021-06-19T14:57:53-07:00
updated_utc: 2021-06-19T21:57:53+00:00
shot_hash: 7dd1d9ceef1a67f444c619c720f0880e
slug: mouse-support-in-vim
body:
Today I learned that if you hit `Esc` in vim and then type `:set mouse=a` and hit enter... vim grows mouse support! In your terminal!

You can use the mouse to select blocks of text and move the insertion cursor around, then hit `del` to delete it or type to replace it.

I learned this after tweeting [a demo video](https://twitter.com/simonw/status/1406336417500860423) of Will McGugan's brilliant new [textual](https://github.com/willmcgugan/textual) Python library for building TUIs - terminal user interfaces - and marveling at how his demo application can already respond to mouseover events and scroll wheel activation while running in the terminal.

html:
<p>Today I learned that if you hit <code>Esc</code> in vim and then type <code>:set mouse=a</code> and hit enter... vim grows mouse support! In your terminal!</p>
<p>You can use the mouse to select blocks of text and move the insertion cursor around, then hit <code>del</code> to delete it or type to replace it.</p>
<p>I learned this after tweeting <a href="https://twitter.com/simonw/status/1406336417500860423" rel="nofollow">a demo video</a> of Will McGugan's brilliant new <a href="https://github.com/willmcgugan/textual">textual</a> Python library for building TUIs - terminal user interfaces - and marveling at how his demo application can already respond to mouseover events and scroll wheel activation while running in the terminal.</p>


path: reddit_scraping-reddit-json.md
topic: reddit
title: Scraping Reddit via their JSON API
url: https://github.com/simonw/til/blob/main/reddit/scraping-reddit-json.md
created: 2021-06-21T14:01:13-07:00
created_utc: 2021-06-21T21:01:13+00:00
updated: 2021-06-21T14:01:13-07:00
updated_utc: 2021-06-21T21:01:13+00:00
shot_hash: 63252b7a410f049730d658143c9ccf35
slug: scraping-reddit-json
body:
Reddit have long had an unofficial (I think) API where you can add `.json` to the end of any URL to get back the data for that page as JSON.

I wanted to track new posts on Reddit that mention my domain `simonwillison.net`.

https://www.reddit.com/domain/simonwillison.net/new/ shows recent posts from a specific domain.

https://www.reddit.com/domain/simonwillison.net/new.json is that data as JSON, which looks like this:

```json
{
  "kind": "Listing",
  "data": {
    "modhash": "la6xmexs8u301d6d105d24f94cdaa4457a00a1ea042c95f6e2",
    "dist": 25,
    "children": [
      {
        "kind": "t3",
        "data": {
          "approved_at_utc": null,
          "subreddit": "programming",
          "selftext": "",
          "author_fullname": "t2_2ks9",
          "saved": false,
          "mod_reason_title": null,
          "gilded": 0,
          "clicked": false,
          "title": "Joining CSV and JSON data with an in-memory SQLite database",
          "link_flair_richtext": [],
          "subreddit_name_prefixed": "r/programming"
```
Attempting to fetch this data with `curl` shows an error:
```
$ curl 'https://www.reddit.com/domain/simonwillison.net/new.json'
{"message": "Too Many Requests", "error": 429}
```
Turns out this rate limiting is [based on user-agent](https://www.reddit.com/r/redditdev/comments/3qbll8/429_too_many_requests/) - so to avoid it, set a custom user-agent:

```
$ curl --user-agent 'simonw/fetch-reddit' 'https://www.reddit.com/domain/simonwillison.net/new.json'
{"kind": "Listing", "data": ...
```
I used `jq` to tidy this up like so:

```jq
[.data.children[] | .data |  {
  id: .id,
  subreddit: .subreddit,
  url: .url,
  created_utc: .created_utc | todate,
  permalink: .permalink,
  num_comments: .num_comments
}]
```
Combined:
```
$ curl \
  --user-agent 'simonw/fetch-reddit' \
  'https://www.reddit.com/domain/simonwillison.net/new.json' \
  | jq '[.data.children[] | .data |  {
    id: .id,
    subreddit: .subreddit,
    url: .url,
    created_utc: .created_utc | todate,
    permalink: .permalink,
    num_comments: .num_comments
  }]' > simonwillison-net.json
```
Output looks like this:
```json
[
  {
    "id": "o3tjsx",
    "subreddit": "programming",
    "url": "https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/",
    "created_utc": "2021-06-20T00:25:51Z",
    "permalink": "/r/programming/comments/o3tjsx/joining_csv_and_json_data_with_an_inmemory_sqlite/",
    "num_comments": 10
  },
  {
    "id": "nnsww6",
    "subreddit": "patient_hackernews",
    "url": "https://til.simonwillison.net/bash/finding-bom-csv-files-with-ripgrep",
    "created_utc": "2021-05-29T18:04:38Z",
    "permalink": "/r/patient_hackernews/comments/nnsww6/finding_csv_files_that_start_with_a_bom_using/",
    "num_comments": 1
  }
]
```

html:
<p>Reddit have long had an unofficial (I think) API where you can add <code>.json</code> to the end of any URL to get back the data for that page as JSON.</p>
<p>I wanted to track new posts on Reddit that mention my domain <code>simonwillison.net</code>.</p>
<p><a href="https://www.reddit.com/domain/simonwillison.net/new/" rel="nofollow">https://www.reddit.com/domain/simonwillison.net/new/</a> shows recent posts from a specific domain.</p>
<p><a href="https://www.reddit.com/domain/simonwillison.net/new.json" rel="nofollow">https://www.reddit.com/domain/simonwillison.net/new.json</a> is that data as JSON, which looks like this:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-s"><span class="pl-pds">"</span>kind<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Listing<span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>data<span class="pl-pds">"</span></span>: {
    <span class="pl-s"><span class="pl-pds">"</span>modhash<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>la6xmexs8u301d6d105d24f94cdaa4457a00a1ea042c95f6e2<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>dist<span class="pl-pds">"</span></span>: <span class="pl-c1">25</span>,
    <span class="pl-s"><span class="pl-pds">"</span>children<span class="pl-pds">"</span></span>: [
      {
        <span class="pl-s"><span class="pl-pds">"</span>kind<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>t3<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>data<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>approved_at_utc<span class="pl-pds">"</span></span>: <span class="pl-c1">null</span>,
          <span class="pl-s"><span class="pl-pds">"</span>subreddit<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>programming<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>selftext<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span><span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>author_fullname<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>t2_2ks9<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>saved<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
          <span class="pl-s"><span class="pl-pds">"</span>mod_reason_title<span class="pl-pds">"</span></span>: <span class="pl-c1">null</span>,
          <span class="pl-s"><span class="pl-pds">"</span>gilded<span class="pl-pds">"</span></span>: <span class="pl-c1">0</span>,
          <span class="pl-s"><span class="pl-pds">"</span>clicked<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
          <span class="pl-s"><span class="pl-pds">"</span>title<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Joining CSV and JSON data with an in-memory SQLite database<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>link_flair_richtext<span class="pl-pds">"</span></span>: [],
          <span class="pl-s"><span class="pl-pds">"</span>subreddit_name_prefixed<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>r/programming<span class="pl-pds">"</span></span></pre></div>
<p>Attempting to fetch this data with <code>curl</code> shows an error:</p>
<pre><code>$ curl 'https://www.reddit.com/domain/simonwillison.net/new.json'
{"message": "Too Many Requests", "error": 429}
</code></pre>
<p>Turns out this rate limiting is <a href="https://www.reddit.com/r/redditdev/comments/3qbll8/429_too_many_requests/" rel="nofollow">based on user-agent</a> - so to avoid it, set a custom user-agent:</p>
<pre><code>$ curl --user-agent 'simonw/fetch-reddit' 'https://www.reddit.com/domain/simonwillison.net/new.json'
{"kind": "Listing", "data": ...
</code></pre>
<p>I used <code>jq</code> to tidy this up like so:</p>
<div class="highlight highlight-source-jq"><pre>[<span class="pl-e">.data.children</span>[] <span class="pl-k">|</span> <span class="pl-e">.data</span> <span class="pl-k">|</span>  {
  <span class="pl-e">id</span>: <span class="pl-e">.id</span>,
  <span class="pl-e">subreddit</span>: <span class="pl-e">.subreddit</span>,
  <span class="pl-e">url</span>: <span class="pl-e">.url</span>,
  <span class="pl-e">created_utc</span>: <span class="pl-e">.created_utc</span> <span class="pl-k">|</span> <span class="pl-c1">todate</span>,
  <span class="pl-e">permalink</span>: <span class="pl-e">.permalink</span>,
  <span class="pl-e">num_comments</span>: <span class="pl-e">.num_comments</span>
}]</pre></div>
<p>Combined:</p>
<pre><code>$ curl \
  --user-agent 'simonw/fetch-reddit' \
  'https://www.reddit.com/domain/simonwillison.net/new.json' \
  | jq '[.data.children[] | .data |  {
    id: .id,
    subreddit: .subreddit,
    url: .url,
    created_utc: .created_utc | todate,
    permalink: .permalink,
    num_comments: .num_comments
  }]' &gt; simonwillison-net.json
</code></pre>
<p>Output looks like this:</p>
<div class="highlight highlight-source-json"><pre>[
  {
    <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>o3tjsx<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>subreddit<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>programming<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>created_utc<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-06-20T00:25:51Z<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>permalink<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>/r/programming/comments/o3tjsx/joining_csv_and_json_data_with_an_inmemory_sqlite/<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>num_comments<span class="pl-pds">"</span></span>: <span class="pl-c1">10</span>
  },
  {
    <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>nnsww6<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>subreddit<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>patient_hackernews<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://til.simonwillison.net/bash/finding-bom-csv-files-with-ripgrep<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>created_utc<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-05-29T18:04:38Z<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>permalink<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>/r/patient_hackernews/comments/nnsww6/finding_csv_files_that_start_with_a_bom_using/<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>num_comments<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
  }
]</pre></div>


path: github-actions_different-postgresql-versions.md
topic: github-actions
title: Installing different PostgreSQL server versions in GitHub Actions
url: https://github.com/simonw/til/blob/main/github-actions/different-postgresql-versions.md
created: 2021-07-05T17:43:13-07:00
created_utc: 2021-07-06T00:43:13+00:00
updated: 2021-07-05T17:43:13-07:00
updated_utc: 2021-07-06T00:43:13+00:00
shot_hash: 2001828a598aa0775483b5934c907bb8
slug: different-postgresql-versions
body:
The GitHub Actions `ubuntu-latest` default runner currently includes an installation of PostgreSQL 13. The server is not running by default but you can interact with it like this:
```
$ /usr/lib/postgresql/13/bin/postgres --version
postgres (PostgreSQL) 13.3 (Ubuntu 13.3-1.pgdg20.04+1)
```
You can install alternative PostgreSQL versions by following the [PostgreSQL Ubuntu instructions](https://www.postgresql.org/download/linux/ubuntu/) - like this:
```
sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get -y install postgresql-12
```
This works with `postgresql-10` and `postgresql-11` as well as `postgresql-12`.

I wanted to use a GitHub Actions matrix to run my tests against all four versions of PostgreSQL. Here's [my complete workflow](https://github.com/simonw/django-sql-dashboard/blob/1.0.1/.github/workflows/test.yml) - the relevant sections are below:
```yaml
name: Test

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        postgresql-version: [10, 11, 12, 13]
    steps:
    - name: Install PostgreSQL
      env:
        POSTGRESQL_VERSION: ${{ matrix.postgresql-version }}
      run: |
        sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
        wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
        sudo apt-get update
        sudo apt-get -y install "postgresql-$POSTGRESQL_VERSION"
    - name: Run tests
      env:
        POSTGRESQL_VERSION: ${{ matrix.postgresql-version }}
      run: |
        export POSTGRESQL_PATH="/usr/lib/postgresql/$POSTGRESQL_VERSION/bin/postgres"
        export INITDB_PATH="/usr/lib/postgresql/$POSTGRESQL_VERSION/bin/initdb"
        pytest
```
I modified my tests to call the `postgres` and `initdb` binaries specified by the `POSTGRESQL_PATH` and `INITDB_PATH` environment variables.

html:
<p>The GitHub Actions <code>ubuntu-latest</code> default runner currently includes an installation of PostgreSQL 13. The server is not running by default but you can interact with it like this:</p>
<pre><code>$ /usr/lib/postgresql/13/bin/postgres --version
postgres (PostgreSQL) 13.3 (Ubuntu 13.3-1.pgdg20.04+1)
</code></pre>
<p>You can install alternative PostgreSQL versions by following the <a href="https://www.postgresql.org/download/linux/ubuntu/" rel="nofollow">PostgreSQL Ubuntu instructions</a> - like this:</p>
<pre><code>sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" &gt; /etc/apt/sources.list.d/pgdg.list'
wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
sudo apt-get update
sudo apt-get -y install postgresql-12
</code></pre>
<p>This works with <code>postgresql-10</code> and <code>postgresql-11</code> as well as <code>postgresql-12</code>.</p>
<p>I wanted to use a GitHub Actions matrix to run my tests against all four versions of PostgreSQL. Here's <a href="https://github.com/simonw/django-sql-dashboard/blob/1.0.1/.github/workflows/test.yml">my complete workflow</a> - the relevant sections are below:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Test</span>

<span class="pl-ent">on</span>: <span class="pl-s">[push]</span>

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">test</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">strategy</span>:
      <span class="pl-ent">matrix</span>:
        <span class="pl-ent">postgresql-version</span>: <span class="pl-s">[10, 11, 12, 13]</span>
    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">name</span>: <span class="pl-s">Install PostgreSQL</span>
      <span class="pl-ent">env</span>:
        <span class="pl-ent">POSTGRESQL_VERSION</span>: <span class="pl-s">${{ matrix.postgresql-version }}</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" &gt; /etc/apt/sources.list.d/pgdg.list'</span>
<span class="pl-s">        wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -</span>
<span class="pl-s">        sudo apt-get update</span>
<span class="pl-s">        sudo apt-get -y install "postgresql-$POSTGRESQL_VERSION"</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Run tests</span>
      <span class="pl-ent">env</span>:
        <span class="pl-ent">POSTGRESQL_VERSION</span>: <span class="pl-s">${{ matrix.postgresql-version }}</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        export POSTGRESQL_PATH="/usr/lib/postgresql/$POSTGRESQL_VERSION/bin/postgres"</span>
<span class="pl-s">        export INITDB_PATH="/usr/lib/postgresql/$POSTGRESQL_VERSION/bin/initdb"</span>
<span class="pl-s">        pytest</span></pre></div>
<p>I modified my tests to call the <code>postgres</code> and <code>initdb</code> binaries specified by the <code>POSTGRESQL_PATH</code> and <code>INITDB_PATH</code> environment variables.</p>


path: javascript_preventing-double-form-submission.md
topic: javascript
title: Preventing double form submissions with JavaScript
url: https://github.com/simonw/til/blob/main/javascript/preventing-double-form-submission.md
created: 2021-07-08T14:53:22-07:00
created_utc: 2021-07-08T21:53:22+00:00
updated: 2021-07-08T14:53:22-07:00
updated_utc: 2021-07-08T21:53:22+00:00
shot_hash: b04a7324695e5f0389b5f00c8025f8fa
slug: preventing-double-form-submission
body:
I needed this for [VIAL issue 722](https://github.com/CAVaccineInventory/vial/issues/722). I decided to disable form submissions for two seconds after they are submitted, to protect against accidental double submissions without risk of unexpected issues that could cause the form to be permanently disabled even though it should still be able to submit it.

I ended up adding this code to a custom Django Admin base template:

```javascript
function protectForm(form) {
  var locked = false;
  form.addEventListener('submit', (ev) => {
    if (locked) {
      ev.preventDefault();
      return;
    }
    locked = true;
    setTimeout(() => {
      locked = false;
    }, 2000);
  });
}
window.addEventListener('load', () => {
  Array.from(document.querySelectorAll('form')).forEach(protectForm);
});
```

html:
<p>I needed this for <a href="https://github.com/CAVaccineInventory/vial/issues/722">VIAL issue 722</a>. I decided to disable form submissions for two seconds after they are submitted, to protect against accidental double submissions without risk of unexpected issues that could cause the form to be permanently disabled even though it should still be able to submit it.</p>
<p>I ended up adding this code to a custom Django Admin base template:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">function</span> <span class="pl-en">protectForm</span><span class="pl-kos">(</span><span class="pl-s1">form</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
  <span class="pl-k">var</span> <span class="pl-s1">locked</span> <span class="pl-c1">=</span> <span class="pl-c1">false</span><span class="pl-kos">;</span>
  <span class="pl-s1">form</span><span class="pl-kos">.</span><span class="pl-en">addEventListener</span><span class="pl-kos">(</span><span class="pl-s">'submit'</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-s1">ev</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">locked</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
      <span class="pl-s1">ev</span><span class="pl-kos">.</span><span class="pl-en">preventDefault</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-k">return</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span>
    <span class="pl-s1">locked</span> <span class="pl-c1">=</span> <span class="pl-c1">true</span><span class="pl-kos">;</span>
    <span class="pl-en">setTimeout</span><span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
      <span class="pl-s1">locked</span> <span class="pl-c1">=</span> <span class="pl-c1">false</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span><span class="pl-kos">,</span> <span class="pl-c1">2000</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span>
<span class="pl-smi">window</span><span class="pl-kos">.</span><span class="pl-en">addEventListener</span><span class="pl-kos">(</span><span class="pl-s">'load'</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
  <span class="pl-v">Array</span><span class="pl-kos">.</span><span class="pl-en">from</span><span class="pl-kos">(</span><span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">querySelectorAll</span><span class="pl-kos">(</span><span class="pl-s">'form'</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">forEach</span><span class="pl-kos">(</span><span class="pl-s1">protectForm</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>


path: cloudrun_increase-cloud-scheduler-time-limit.md
topic: cloudrun
title: Increasing the time limit for a Google Cloud Scheduler task
url: https://github.com/simonw/til/blob/main/cloudrun/increase-cloud-scheduler-time-limit.md
created: 2021-07-08T17:38:15-07:00
created_utc: 2021-07-09T00:38:15+00:00
updated: 2021-07-08T17:40:18-07:00
updated_utc: 2021-07-09T00:40:18+00:00
shot_hash: d4b13be5a8c25bc542739471b7b680e0
slug: increase-cloud-scheduler-time-limit
body:
In [VIAL issue 724](https://github.com/CAVaccineInventory/vial/issues/724) a Cloud Scheduler job which triggered a Cloud Run hosted export script - by sending an HTTP POST to an endpoint - was returning an error. The logs showed the error happened exactly three minutes after the task started executing.

Turns out the HTTP endpoint (which does a lot of work) was taking longer than three minutes, which is the undocumented default time limit for Cloud Scheduler jobs.

Unfortunately it's not possible to increase this time limit using the Cloud Scheduler web console, but it IS possible to increase the limit using the CLI `gcloud` tool.

To list the scheduler jobs:
```
~ % gcloud beta scheduler jobs list --project django-vaccinateca
ID                                        LOCATION  SCHEDULE (TZ)                                                    TARGET_TYPE  STATE
api-export-production                     us-west2  every 1 minutes (America/Los_Angeles)                            HTTP         ENABLED
api-export-staging                        us-west2  every 1 minutes (America/Los_Angeles)                            HTTP         ENABLED
mapbox-export                             us-west2  0 2,9,10,11,12,13,14,15,16,17,18,21 * * * (America/Los_Angeles)  HTTP         ENABLED
resolve-missing-counties-production       us-west2  */10 * * * * (America/Los_Angeles)                               HTTP         ENABLED
resolve-missing-counties-staging          us-west2  */10 * * * * (America/Los_Angeles)                               HTTP         ENABLED
vaccinatethestates-api-export-production  us-west2  */10 * * * * (America/Los_Angeles)                               HTTP         ENABLED
vaccinatethestates-api-export-staging     us-west2  */10 * * * * (America/Los_Angeles)                               HTTP         ENABLED
```
To increase the limit for one of them by name:
```
gcloud beta scheduler jobs update http \
  vaccinatethestates-api-export-production \
  --attempt-deadline=540s \
  --project django-vaccinateca
```
You can see the limit using `describe`:
```
~ % gcloud beta scheduler jobs describe vaccinatethestates-api-export-production --project django-vaccinateca
attemptDeadline: 180s
description: Hit /api/exportVaccinateTheStates to export to api.vaccinatethestates.com
  bucket
httpTarget:
  headers:
    Authorization: Bearer 27:...
    User-Agent: Google-Cloud-Scheduler
  httpMethod: POST
  uri: https://vial.calltheshots.us/api/exportVaccinateTheStates
lastAttemptTime: '2021-07-08T23:40:00.992830Z'
name: projects/django-vaccinateca/locations/us-west2/jobs/vaccinatethestates-api-export-production
retryConfig:
  maxBackoffDuration: 3600s
  maxDoublings: 5
  maxRetryDuration: 0s
  minBackoffDuration: 5s
schedule: '*/10 * * * *'
scheduleTime: '2021-07-08T23:50:00.061563Z'
state: ENABLED
status:
  code: 2
timeZone: America/Los_Angeles
userUpdateTime: '2021-06-10T23:34:48Z'
```

html:
<p>In <a href="https://github.com/CAVaccineInventory/vial/issues/724">VIAL issue 724</a> a Cloud Scheduler job which triggered a Cloud Run hosted export script - by sending an HTTP POST to an endpoint - was returning an error. The logs showed the error happened exactly three minutes after the task started executing.</p>
<p>Turns out the HTTP endpoint (which does a lot of work) was taking longer than three minutes, which is the undocumented default time limit for Cloud Scheduler jobs.</p>
<p>Unfortunately it's not possible to increase this time limit using the Cloud Scheduler web console, but it IS possible to increase the limit using the CLI <code>gcloud</code> tool.</p>
<p>To list the scheduler jobs:</p>
<pre><code>~ % gcloud beta scheduler jobs list --project django-vaccinateca
ID                                        LOCATION  SCHEDULE (TZ)                                                    TARGET_TYPE  STATE
api-export-production                     us-west2  every 1 minutes (America/Los_Angeles)                            HTTP         ENABLED
api-export-staging                        us-west2  every 1 minutes (America/Los_Angeles)                            HTTP         ENABLED
mapbox-export                             us-west2  0 2,9,10,11,12,13,14,15,16,17,18,21 * * * (America/Los_Angeles)  HTTP         ENABLED
resolve-missing-counties-production       us-west2  */10 * * * * (America/Los_Angeles)                               HTTP         ENABLED
resolve-missing-counties-staging          us-west2  */10 * * * * (America/Los_Angeles)                               HTTP         ENABLED
vaccinatethestates-api-export-production  us-west2  */10 * * * * (America/Los_Angeles)                               HTTP         ENABLED
vaccinatethestates-api-export-staging     us-west2  */10 * * * * (America/Los_Angeles)                               HTTP         ENABLED
</code></pre>
<p>To increase the limit for one of them by name:</p>
<pre><code>gcloud beta scheduler jobs update http \
  vaccinatethestates-api-export-production \
  --attempt-deadline=540s \
  --project django-vaccinateca
</code></pre>
<p>You can see the limit using <code>describe</code>:</p>
<pre><code>~ % gcloud beta scheduler jobs describe vaccinatethestates-api-export-production --project django-vaccinateca
attemptDeadline: 180s
description: Hit /api/exportVaccinateTheStates to export to api.vaccinatethestates.com
  bucket
httpTarget:
  headers:
    Authorization: Bearer 27:...
    User-Agent: Google-Cloud-Scheduler
  httpMethod: POST
  uri: https://vial.calltheshots.us/api/exportVaccinateTheStates
lastAttemptTime: '2021-07-08T23:40:00.992830Z'
name: projects/django-vaccinateca/locations/us-west2/jobs/vaccinatethestates-api-export-production
retryConfig:
  maxBackoffDuration: 3600s
  maxDoublings: 5
  maxRetryDuration: 0s
  minBackoffDuration: 5s
schedule: '*/10 * * * *'
scheduleTime: '2021-07-08T23:50:00.061563Z'
state: ENABLED
status:
  code: 2
timeZone: America/Los_Angeles
userUpdateTime: '2021-06-10T23:34:48Z'
</code></pre>


path: sqlite_pysqlite3-on-macos.md
topic: sqlite
title: Using pysqlite3 on macOS
url: https://github.com/simonw/til/blob/main/sqlite/pysqlite3-on-macos.md
created: 2021-07-10T12:21:23-07:00
created_utc: 2021-07-10T19:21:23+00:00
updated: 2021-07-10T12:21:23-07:00
updated_utc: 2021-07-10T19:21:23+00:00
shot_hash: 724fe9423d80d3ce7ab39b9d06a71478
slug: pysqlite3-on-macos
body:
While trying to use [pysqlite3](https://github.com/coleifer/pysqlite3) on macOS I got the following error:

```pycon
>>> import pysqlite3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pysqlite3/__init__.py", line 23, in <module>
    from pysqlite3.dbapi2 import *
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pysqlite3/dbapi2.py", line 28, in <module>
    from pysqlite3._sqlite3 import *
ImportError: dlopen(/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pysqlite3/_sqlite3.cpython-38-darwin.so, 2): Symbol not found: _sqlite3_enable_load_extension
  Referenced from: /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pysqlite3/_sqlite3.cpython-38-darwin.so
  Expected in: flat namespace
 in /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pysqlite3/_sqlite3.cpython-38-darwin.so
```
Thanks to [this tip](https://stackoverflow.com/a/60046923/6083) on Stack Overflow I found that installing a fresh SQLite with `brew install sqlite3` and then running the following first fixed the error:

    export DYLD_LIBRARY_PATH=/usr/local/opt/sqlite/lib:/usr/lib

Then:
```pycon
>>> import pysqlite3
>>> pysqlite3.connect(":memory:").execute("select sqlite_version()").fetchall()
[('3.36.0',)]
```
This is because the Homebrew SQLite is "keg-only", which means that it does not become the default on the system to avoid intefering with the existing SQLite that ships with macOS.

As illustrated by the following:

```
% which sqlite3
/usr/bin/sqlite3
% echo "select sqlite_version()" | /usr/bin/sqlite3
3.28.0
% echo "select sqlite_version()" | /usr/local/opt/sqlite3/bin/sqlite3 
3.36.0
```

html:
<p>While trying to use <a href="https://github.com/coleifer/pysqlite3">pysqlite3</a> on macOS I got the following error:</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; <span class="pl-k">import</span> pysqlite3
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pysqlite3/__init__.py", line 23, in &lt;module&gt;
    from pysqlite3.dbapi2 import *
  File "/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pysqlite3/dbapi2.py", line 28, in &lt;module&gt;
    from pysqlite3._sqlite3 import *
ImportError: dlopen(/Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pysqlite3/_sqlite3.cpython-38-darwin.so, 2): Symbol not found: _sqlite3_enable_load_extension
  Referenced from: /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pysqlite3/_sqlite3.cpython-38-darwin.so
  Expected in: flat namespace
 in /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/lib/python3.8/site-packages/pysqlite3/_sqlite3.cpython-38-darwin.so</pre></div>
<p>Thanks to <a href="https://stackoverflow.com/a/60046923/6083" rel="nofollow">this tip</a> on Stack Overflow I found that installing a fresh SQLite with <code>brew install sqlite3</code> and then running the following first fixed the error:</p>
<pre><code>export DYLD_LIBRARY_PATH=/usr/local/opt/sqlite/lib:/usr/lib
</code></pre>
<p>Then:</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; <span class="pl-k">import</span> pysqlite3
&gt;&gt;&gt; pysqlite3.connect(<span class="pl-s"><span class="pl-pds">"</span>:memory:<span class="pl-pds">"</span></span>).execute(<span class="pl-s"><span class="pl-pds">"</span>select sqlite_version()<span class="pl-pds">"</span></span>).fetchall()
[('3.36.0',)]</pre></div>
<p>This is because the Homebrew SQLite is "keg-only", which means that it does not become the default on the system to avoid intefering with the existing SQLite that ships with macOS.</p>
<p>As illustrated by the following:</p>
<pre><code>% which sqlite3
/usr/bin/sqlite3
% echo "select sqlite_version()" | /usr/bin/sqlite3
3.28.0
% echo "select sqlite_version()" | /usr/local/opt/sqlite3/bin/sqlite3 
3.36.0
</code></pre>


path: nginx_proxy-domain-sockets.md
topic: nginx
title: Using nginx to proxy to a Unix domain socket
url: https://github.com/simonw/til/blob/main/nginx/proxy-domain-sockets.md
created: 2021-07-10T18:12:03-07:00
created_utc: 2021-07-11T01:12:03+00:00
updated: 2021-07-10T18:12:03-07:00
updated_utc: 2021-07-11T01:12:03+00:00
shot_hash: 025db6d03f3cf0130549515eb3464186
slug: proxy-domain-sockets
body:
I figured this out while adding `--uds` support to Datasette in [#1388](https://github.com/simonw/datasette/issues/1388). Save the following in `nginx.conf`:

```nginx
daemon off;
events {
  worker_connections  1024;
}
http {
  server {
    listen 8092;
    location / {
      proxy_pass http://datasette;
      proxy_set_header Host $host;
    }
  }
  upstream datasette {
    server unix:/tmp/datasette.sock;
  }
}
```
Start `nginx` against that configuration file - this works without root provided you listen on a high port:

    nginx -c $PWD/nginx.conf

(The `$PWD` seems necessary to avoid `nginx` looking in its default directory.)

Start something listening on the `/tmp/datasette.sock` path - with the latest Datasette you can do this:

    datasette --uds /tmp/datasette.sock

Now hits to `http://localhost:8092/` will proxy through to Datasette.

html:
<p>I figured this out while adding <code>--uds</code> support to Datasette in <a href="https://github.com/simonw/datasette/issues/1388">#1388</a>. Save the following in <code>nginx.conf</code>:</p>
<div class="highlight highlight-source-nginx"><pre><span class="pl-k">daemon</span><span class="pl-c1"> off</span>;
<span class="pl-k">events</span> {
  <span class="pl-k">worker_connections</span>  <span class="pl-s">1024</span>;
}
<span class="pl-k">http</span> {
  <span class="pl-k">server</span> {
    <span class="pl-k">listen</span> <span class="pl-s">8092</span>;
    <span class="pl-k">location</span> <span class="pl-en">/ </span>{
      <span class="pl-k">proxy_pass</span> http://datasette;
      <span class="pl-k">proxy_set_header</span> Host <span class="pl-smi">$host</span>;
    }
  }
  <span class="pl-k">upstream</span> <span class="pl-en">datasette </span>{
    <span class="pl-k">server</span> unix:/tmp/datasette.sock;
  }
}</pre></div>
<p>Start <code>nginx</code> against that configuration file - this works without root provided you listen on a high port:</p>
<pre><code>nginx -c $PWD/nginx.conf
</code></pre>
<p>(The <code>$PWD</code> seems necessary to avoid <code>nginx</code> looking in its default directory.)</p>
<p>Start something listening on the <code>/tmp/datasette.sock</code> path - with the latest Datasette you can do this:</p>
<pre><code>datasette --uds /tmp/datasette.sock
</code></pre>
<p>Now hits to <code>http://localhost:8092/</code> will proxy through to Datasette.</p>


path: sqlite_import-csv.md
topic: sqlite
title: Importing CSV data into SQLite with .import
url: https://github.com/simonw/til/blob/main/sqlite/import-csv.md
created: 2021-07-13T22:01:05-07:00
created_utc: 2021-07-14T05:01:05+00:00
updated: 2021-07-13T22:01:05-07:00
updated_utc: 2021-07-14T05:01:05+00:00
shot_hash: 6b1ad7d89e30047a32c46361de3ce399
slug: import-csv
body:
I usually use my `sqlite-utils insert blah.db tablename file.csv --csv` command to import CSV data into SQLite, but for large CSV files (like a 750MB one) this can take quite a long time - over half an hour in this case.

SQLite can import CSV data directly, and when I tried it on this file it completed in 45 seconds!

Here's how I scripted that from the command-line:
```bash
sqlite3 data.db <<EOS
.mode csv
.import school.csv schools
.import state.csv states
EOS
```
The `.mode csv` ensures the imported files are treated as CSV, then the `.import filename.csv tablename` lines import the data.

Every column was imported as `TEXT` - so I used `sqlite-utils transform` to transform some of the column types afterwards:
```
sqlite-utils transform data.db schools \
    --type school_nces_id integer \
    --type year integer
```

html:
<p>I usually use my <code>sqlite-utils insert blah.db tablename file.csv --csv</code> command to import CSV data into SQLite, but for large CSV files (like a 750MB one) this can take quite a long time - over half an hour in this case.</p>
<p>SQLite can import CSV data directly, and when I tried it on this file it completed in 45 seconds!</p>
<p>Here's how I scripted that from the command-line:</p>
<div class="highlight highlight-source-shell"><pre>sqlite3 data.db <span class="pl-s"><span class="pl-k">&lt;&lt;</span><span class="pl-k">EOS</span></span>
<span class="pl-s">.mode csv</span>
<span class="pl-s">.import school.csv schools</span>
<span class="pl-s">.import state.csv states</span>
<span class="pl-s"><span class="pl-k">EOS</span></span></pre></div>
<p>The <code>.mode csv</code> ensures the imported files are treated as CSV, then the <code>.import filename.csv tablename</code> lines import the data.</p>
<p>Every column was imported as <code>TEXT</code> - so I used <code>sqlite-utils transform</code> to transform some of the column types afterwards:</p>
<pre><code>sqlite-utils transform data.db schools \
    --type school_nces_id integer \
    --type year integer
</code></pre>


path: tesseract_tesseract-cli.md
topic: tesseract
title: Using the tesseract CLI tool
url: https://github.com/simonw/til/blob/main/tesseract/tesseract-cli.md
created: 2021-07-18T09:56:08-07:00
created_utc: 2021-07-18T16:56:08+00:00
updated: 2021-07-18T09:56:08-07:00
updated_utc: 2021-07-18T16:56:08+00:00
shot_hash: 282c8c674d0dc22990e8ce763bc91e9a
slug: tesseract-cli
body:
Tesseract OCR has a command-line utility which is woefully under-documented. Thanks to [Alexandru Nedelcu](https://alexn.org/blog/2020/11/11/organize-index-screenshots-ocr-macos.html) I figured out how to use it today.

To install on macOS:

    brew install tesseract

To convert an image into an annotated PDF (which you can then copy and paste text out of, and which will be correctly indexed by Spotlight):

    tesseract image.png output-file -l eng pdf
    
The second `output-file` argument there is the path and filename of the output - note that I didn't include a `.pdf` extension because Tesseract adds that automatically - so the output will be in a file called `output-file.pdf`.

To get out just the plain text:

    tesseract image.png output-file -l eng txt

html:
<p>Tesseract OCR has a command-line utility which is woefully under-documented. Thanks to <a href="https://alexn.org/blog/2020/11/11/organize-index-screenshots-ocr-macos.html" rel="nofollow">Alexandru Nedelcu</a> I figured out how to use it today.</p>
<p>To install on macOS:</p>
<pre><code>brew install tesseract
</code></pre>
<p>To convert an image into an annotated PDF (which you can then copy and paste text out of, and which will be correctly indexed by Spotlight):</p>
<pre><code>tesseract image.png output-file -l eng pdf
</code></pre>
<p>The second <code>output-file</code> argument there is the path and filename of the output - note that I didn't include a <code>.pdf</code> extension because Tesseract adds that automatically - so the output will be in a file called <code>output-file.pdf</code>.</p>
<p>To get out just the plain text:</p>
<pre><code>tesseract image.png output-file -l eng txt
</code></pre>


path: jq_extracting-objects-recursively.md
topic: jq
title: Extracting objects recursively with jq
url: https://github.com/simonw/til/blob/main/jq/extracting-objects-recursively.md
created: 2021-07-24T20:58:35-07:00
created_utc: 2021-07-25T03:58:35+00:00
updated: 2021-07-24T20:58:35-07:00
updated_utc: 2021-07-25T03:58:35+00:00
shot_hash: b9c64fde7d490c3083c97580b1d7d7d3
slug: extracting-objects-recursively
body:
The Algolia-powered Hacker News API returns nested comment threads that look like this: https://hn.algolia.com/api/v1/items/27941108

(For this story: https://news.ycombinator.com/item?id=27941108)

```json
{
    "id": 27941108,
    "created_at": "2021-07-24T14:15:05.000Z",
    "type": "story",
    "author": "edward",
    "title": "Fun with Unix domain sockets",
    "url": "https://simonwillison.net/2021/Jul/13/unix-domain-sockets/",
    "children": [
        {
            "id": 27942287,
            "created_at": "2021-07-24T16:31:18.000Z",
            "type": "comment",
            "author": "DesiLurker",
            "text": "<p>one lesser known...",
            "children": []
        },
        {
            "id": 27944615,
            "created_at": "2021-07-24T21:26:33.000Z",
            "type": "comment",
            "author": "galaxyLogic",
            "text": "<p>I read this from Wikipedia...",
            "children": [
                {
                    "id": 27944746,
                    "created_at": "2021-07-24T21:49:07.000Z",
                    "type": "comment",
                    "author": "hughrr",
                    "text": "<p>Yes although I ...",
                    "children": []
                }
            ]
        }
    ]
}
```
I wanted to flatten this into an array of items so I could send it to `sqlite-utils insert`. This recipe worked:

```
curl 'https://hn.algolia.com/api/v1/items/27941108' \
  | jq '[recurse(.children[]) | del(.children)]' \
  | sqlite-utils insert hn.db items - --pk id
```
The `jq` recipe here is:

```jq
[recurse(.children[]) | del(.children)]
```

The first `recurse(.children[])` recurses through a list of everything in a `.children` array.

The `| del(.children)` then deletes that array from the returned objects.

Wrapping it all in `[ ]` ensures the overall result will be an array.

Applied against the above example, this returns:

```json
[
    {
        "id": 27941108,
        "created_at": "2021-07-24T14:15:05.000Z",
        "type": "story",
        "author": "edward",
        "title": "Fun with Unix domain sockets",
        "url": "https://simonwillison.net/2021/Jul/13/unix-domain-sockets/"
    },
    {
        "id": 27942287,
        "created_at": "2021-07-24T16:31:18.000Z",
        "type": "comment",
        "author": "DesiLurker",
        "text": "<p>one lesser known..."
    },
    {
        "id": 27944615,
        "created_at": "2021-07-24T21:26:33.000Z",
        "type": "comment",
        "author": "galaxyLogic",
        "text": "<p>I read this from Wikipedia..."
    },
    {
        "id": 27944746,
        "created_at": "2021-07-24T21:49:07.000Z",
        "type": "comment",
        "author": "hughrr",
        "text": "<p>Yes although I ..."
    }
]
```

html:
<p>The Algolia-powered Hacker News API returns nested comment threads that look like this: <a href="https://hn.algolia.com/api/v1/items/27941108" rel="nofollow">https://hn.algolia.com/api/v1/items/27941108</a></p>
<p>(For this story: <a href="https://news.ycombinator.com/item?id=27941108" rel="nofollow">https://news.ycombinator.com/item?id=27941108</a>)</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-c1">27941108</span>,
    <span class="pl-s"><span class="pl-pds">"</span>created_at<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-07-24T14:15:05.000Z<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>story<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>author<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>edward<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>title<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Fun with Unix domain sockets<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://simonwillison.net/2021/Jul/13/unix-domain-sockets/<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>children<span class="pl-pds">"</span></span>: [
        {
            <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-c1">27942287</span>,
            <span class="pl-s"><span class="pl-pds">"</span>created_at<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-07-24T16:31:18.000Z<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>comment<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>author<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>DesiLurker<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>text<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>&lt;p&gt;one lesser known...<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>children<span class="pl-pds">"</span></span>: []
        },
        {
            <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-c1">27944615</span>,
            <span class="pl-s"><span class="pl-pds">"</span>created_at<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-07-24T21:26:33.000Z<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>comment<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>author<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>galaxyLogic<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>text<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>&lt;p&gt;I read this from Wikipedia...<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>children<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-c1">27944746</span>,
                    <span class="pl-s"><span class="pl-pds">"</span>created_at<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-07-24T21:49:07.000Z<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>comment<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>author<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>hughrr<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>text<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>&lt;p&gt;Yes although I ...<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>children<span class="pl-pds">"</span></span>: []
                }
            ]
        }
    ]
}</pre></div>
<p>I wanted to flatten this into an array of items so I could send it to <code>sqlite-utils insert</code>. This recipe worked:</p>
<pre><code>curl 'https://hn.algolia.com/api/v1/items/27941108' \
  | jq '[recurse(.children[]) | del(.children)]' \
  | sqlite-utils insert hn.db items - --pk id
</code></pre>
<p>The <code>jq</code> recipe here is:</p>
<div class="highlight highlight-source-jq"><pre>[<span class="pl-c1">recurse</span>(<span class="pl-e">.children</span>[]) <span class="pl-k">|</span> <span class="pl-c1">del</span>(<span class="pl-e">.children</span>)]</pre></div>
<p>The first <code>recurse(.children[])</code> recurses through a list of everything in a <code>.children</code> array.</p>
<p>The <code>| del(.children)</code> then deletes that array from the returned objects.</p>
<p>Wrapping it all in <code>[ ]</code> ensures the overall result will be an array.</p>
<p>Applied against the above example, this returns:</p>
<div class="highlight highlight-source-json"><pre>[
    {
        <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-c1">27941108</span>,
        <span class="pl-s"><span class="pl-pds">"</span>created_at<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-07-24T14:15:05.000Z<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>story<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>author<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>edward<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>title<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Fun with Unix domain sockets<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://simonwillison.net/2021/Jul/13/unix-domain-sockets/<span class="pl-pds">"</span></span>
    },
    {
        <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-c1">27942287</span>,
        <span class="pl-s"><span class="pl-pds">"</span>created_at<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-07-24T16:31:18.000Z<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>comment<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>author<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>DesiLurker<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>text<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>&lt;p&gt;one lesser known...<span class="pl-pds">"</span></span>
    },
    {
        <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-c1">27944615</span>,
        <span class="pl-s"><span class="pl-pds">"</span>created_at<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-07-24T21:26:33.000Z<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>comment<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>author<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>galaxyLogic<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>text<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>&lt;p&gt;I read this from Wikipedia...<span class="pl-pds">"</span></span>
    },
    {
        <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-c1">27944746</span>,
        <span class="pl-s"><span class="pl-pds">"</span>created_at<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-07-24T21:49:07.000Z<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>comment<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>author<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>hughrr<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>text<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>&lt;p&gt;Yes although I ...<span class="pl-pds">"</span></span>
    }
]</pre></div>


path: vscode_vs-code-regular-expressions.md
topic: vscode
title: Search and replace with regular expressions in VS Code
url: https://github.com/simonw/til/blob/main/vscode/vs-code-regular-expressions.md
created: 2021-08-02T14:30:11-07:00
created_utc: 2021-08-02T21:30:11+00:00
updated: 2021-08-02T14:30:11-07:00
updated_utc: 2021-08-02T21:30:11+00:00
shot_hash: 4f46ae285ccb5c9cc4a5a7c6204c6f5d
slug: vs-code-regular-expressions
body:
I wanted to replace all instances of this:

    `#90 <https://github.com/simonw/sqlite-utils/issues/90>`__

With this:

    :issue:`90`

For [sqlite-utils issue #306](https://github.com/simonw/sqlite-utils/issues/306).

I used the VS Code's Find and Replace tool with regular expression mode turned on (the `.*` button). I used the following for the find:

    `#(\d+) <https://github.com/simonw/sqlite-utils/issues/\1>`__

Note the `\1` reference to say "the same thing I captured earlier with parenthesis". Then I used this as the replace string:

    :issue:`$1`

Here the `$1` means "the first thing captured with parenthesis".

<img width="983" alt="Screenshot of the find and replace dialog" src="https://user-images.githubusercontent.com/9599/127925903-156f2f74-d8d3-4ce3-b65b-aed29a279253.png">

The resulting change can be seen in [commit e83aef95](https://github.com/simonw/sqlite-utils/commit/e83aef951bd3e8c179511faddb607239a5fa8682).

html:
<p>I wanted to replace all instances of this:</p>
<pre><code>`#90 &lt;https://github.com/simonw/sqlite-utils/issues/90&gt;`__
</code></pre>
<p>With this:</p>
<pre><code>:issue:`90`
</code></pre>
<p>For <a href="https://github.com/simonw/sqlite-utils/issues/306">sqlite-utils issue #306</a>.</p>
<p>I used the VS Code's Find and Replace tool with regular expression mode turned on (the <code>.*</code> button). I used the following for the find:</p>
<pre><code>`#(\d+) &lt;https://github.com/simonw/sqlite-utils/issues/\1&gt;`__
</code></pre>
<p>Note the <code>\1</code> reference to say "the same thing I captured earlier with parenthesis". Then I used this as the replace string:</p>
<pre><code>:issue:`$1`
</code></pre>
<p>Here the <code>$1</code> means "the first thing captured with parenthesis".</p>
<p><a href="https://user-images.githubusercontent.com/9599/127925903-156f2f74-d8d3-4ce3-b65b-aed29a279253.png" target="_blank" rel="nofollow"><img width="983" alt="Screenshot of the find and replace dialog" src="https://user-images.githubusercontent.com/9599/127925903-156f2f74-d8d3-4ce3-b65b-aed29a279253.png" style="max-width:100%;"></a></p>
<p>The resulting change can be seen in <a href="https://github.com/simonw/sqlite-utils/commit/e83aef951bd3e8c179511faddb607239a5fa8682">commit e83aef95</a>.</p>


path: python_codespell.md
topic: python
title: Check spelling using codespell
url: https://github.com/simonw/til/blob/main/python/codespell.md
created: 2021-08-03T09:34:02-07:00
created_utc: 2021-08-03T16:34:02+00:00
updated: 2021-08-03T10:09:50-07:00
updated_utc: 2021-08-03T17:09:50+00:00
shot_hash: 162ece1035b392fc54f4fdcd845947a2
slug: codespell
body:
Today I discovered [codespell](https://github.com/codespell-project/codespell/) via [this Rich commit](https://github.com/willmcgugan/rich/commit/9c12a4537499797c43725fff5276ef0da62423ef#diff-ce84a1b2c9eb4ab3ea22f610cad7111cb9a2f66365c3b24679901376a2a73ab2). `codespell` is a really simple spell checker that can be run locally or incorporated into a CI flow.

codespell is designed to run against source code. Instead of using a dictionary of correctly spelled words it instead uses a dictionary of known common spelling mistakes, derived from English Wikipedia ([defined here](https://github.com/codespell-project/codespell/tree/master/codespell_lib/data)). This makes it less likely to be confused by variable or function names, while still being able to spot spelling mistakes in comments.

Basic usage:

    pip install codespell
    codespell
    # Or point it at a folder, or files in that folder:
    codespell docs/*.rst

This outputs any spelling errors it finds in those files. I got this the first time I ran it against the Datasette documentation:

```
docs/authentication.rst:63: perfom ==> perform
docs/authentication.rst:76: perfom ==> perform
docs/changelog.rst:429: repsonse ==> response
docs/changelog.rst:503: permissons ==> permissions
docs/changelog.rst:717: compatibilty ==> compatibility
docs/changelog.rst:1172: browseable ==> browsable
docs/deploying.rst:191: similiar ==> similar
docs/internals.rst:434: Respons ==> Response, respond
docs/internals.rst:440: Respons ==> Response, respond
docs/internals.rst:717: tha ==> than, that, the
docs/performance.rst:42: databse ==> database
docs/plugin_hooks.rst:667: utilites ==> utilities
docs/publish.rst:168: countainer ==> container
docs/settings.rst:352: inalid ==> invalid
docs/sql_queries.rst:406: preceeded ==> preceded, proceeded
```
You can create a file of additional words that it should ignore and pass that using the `--ignore-words` option:

    codespell docs/*.rst --ignore-words docs/codespell-ignore-words.txt

Since I don't have any words in that file yet I added one fake word, so my file looks like this:

```
AddWordsToIgnoreHere
```
Each ignored word should be on a separate line.

I added it to my GitHub Actions CI like this:
```yaml

name: Check spelling in documentation

on: [push, pull_request]

jobs:
  spellcheck:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v2
      with:
        python-version: 3.9
    - uses: actions/cache@v2
      name: Configure pip caching
      with:
        path: ~/.cache/pip
        key: ${{ runner.os }}-pip-spellcheck
        restore-keys: |
          ${{ runner.os }}-pip-spellcheck
    - name: Install dependencies
      run: |
        pip install codespell
    - name: Check spelling
      run: codespell docs/*.rst --ignore-words docs/codespell-ignore-words.txt
```
Now any push or pull request will have the spell checker applied to it, and will fail if any new incorrectly spelled words are detected.

Here's [the full PR](https://github.com/simonw/datasette/pull/1418) where I added this to Datasette, and [the commit](https://github.com/simonw/sqlite-utils/commit/991cf56ae2840aaefda2af828a5c40396d2506ca) where I added this to `sqlite-utils`.

html:
<p>Today I discovered <a href="https://github.com/codespell-project/codespell/">codespell</a> via <a href="https://github.com/willmcgugan/rich/commit/9c12a4537499797c43725fff5276ef0da62423ef#diff-ce84a1b2c9eb4ab3ea22f610cad7111cb9a2f66365c3b24679901376a2a73ab2">this Rich commit</a>. <code>codespell</code> is a really simple spell checker that can be run locally or incorporated into a CI flow.</p>
<p>codespell is designed to run against source code. Instead of using a dictionary of correctly spelled words it instead uses a dictionary of known common spelling mistakes, derived from English Wikipedia (<a href="https://github.com/codespell-project/codespell/tree/master/codespell_lib/data">defined here</a>). This makes it less likely to be confused by variable or function names, while still being able to spot spelling mistakes in comments.</p>
<p>Basic usage:</p>
<pre><code>pip install codespell
codespell
# Or point it at a folder, or files in that folder:
codespell docs/*.rst
</code></pre>
<p>This outputs any spelling errors it finds in those files. I got this the first time I ran it against the Datasette documentation:</p>
<pre><code>docs/authentication.rst:63: perfom ==&gt; perform
docs/authentication.rst:76: perfom ==&gt; perform
docs/changelog.rst:429: repsonse ==&gt; response
docs/changelog.rst:503: permissons ==&gt; permissions
docs/changelog.rst:717: compatibilty ==&gt; compatibility
docs/changelog.rst:1172: browseable ==&gt; browsable
docs/deploying.rst:191: similiar ==&gt; similar
docs/internals.rst:434: Respons ==&gt; Response, respond
docs/internals.rst:440: Respons ==&gt; Response, respond
docs/internals.rst:717: tha ==&gt; than, that, the
docs/performance.rst:42: databse ==&gt; database
docs/plugin_hooks.rst:667: utilites ==&gt; utilities
docs/publish.rst:168: countainer ==&gt; container
docs/settings.rst:352: inalid ==&gt; invalid
docs/sql_queries.rst:406: preceeded ==&gt; preceded, proceeded
</code></pre>
<p>You can create a file of additional words that it should ignore and pass that using the <code>--ignore-words</code> option:</p>
<pre><code>codespell docs/*.rst --ignore-words docs/codespell-ignore-words.txt
</code></pre>
<p>Since I don't have any words in that file yet I added one fake word, so my file looks like this:</p>
<pre><code>AddWordsToIgnoreHere
</code></pre>
<p>Each ignored word should be on a separate line.</p>
<p>I added it to my GitHub Actions CI like this:</p>
<div class="highlight highlight-source-yaml"><pre>
<span class="pl-ent">name</span>: <span class="pl-s">Check spelling in documentation</span>

<span class="pl-ent">on</span>: <span class="pl-s">[push, pull_request]</span>

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">spellcheck</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v2</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Set up Python ${{ matrix.python-version }}</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">actions/setup-python@v2</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">python-version</span>: <span class="pl-c1">3.9</span>
    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/cache@v2</span>
      <span class="pl-ent">name</span>: <span class="pl-s">Configure pip caching</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">path</span>: <span class="pl-s">~/.cache/pip</span>
        <span class="pl-ent">key</span>: <span class="pl-s">${{ runner.os }}-pip-spellcheck</span>
        <span class="pl-ent">restore-keys</span>: <span class="pl-s">|</span>
<span class="pl-s">          ${{ runner.os }}-pip-spellcheck</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Install dependencies</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        pip install codespell</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Check spelling</span>
      <span class="pl-ent">run</span>: <span class="pl-s">codespell docs/*.rst --ignore-words docs/codespell-ignore-words.txt</span></pre></div>
<p>Now any push or pull request will have the spell checker applied to it, and will fail if any new incorrectly spelled words are detected.</p>
<p>Here's <a href="https://github.com/simonw/datasette/pull/1418">the full PR</a> where I added this to Datasette, and <a href="https://github.com/simonw/sqlite-utils/commit/991cf56ae2840aaefda2af828a5c40396d2506ca">the commit</a> where I added this to <code>sqlite-utils</code>.</p>


path: imagemagick_set-a-gif-to-loop.md
topic: imagemagick
title: Set a GIF to loop using ImageMagick
url: https://github.com/simonw/til/blob/main/imagemagick/set-a-gif-to-loop.md
created: 2021-08-03T14:54:35-07:00
created_utc: 2021-08-03T21:54:35+00:00
updated: 2021-08-03T14:54:35-07:00
updated_utc: 2021-08-03T21:54:35+00:00
shot_hash: 06960a86d19ef9d513cbffe020d4aa1e
slug: set-a-gif-to-loop
body:
I managed to accidentally create a GIF that ran once without looping. I think this is because I created it in [LICEcap](https://www.cockos.com/licecap/) but then deleted some frames and re-saved it using macOS Preview.

I used ImageMagick to get it to loop like this:

    convert chrome-samesite-missing.gif -loop 0 chrome-samesite-missing-loop.gif

Note that the output filename comes last, AFTER the `-loop 0` option.

I installed ImageMagick on macOS using `brew install imagemagick`

Here's the before GIF:

![This loops once](https://static.simonwillison.net/static/2021/chrome-samesite-missing.gif)

And the after GIF:

![This loops forever](https://static.simonwillison.net/static/2021/chrome-samesite-missing-loop.gif)

html:
<p>I managed to accidentally create a GIF that ran once without looping. I think this is because I created it in <a href="https://www.cockos.com/licecap/" rel="nofollow">LICEcap</a> but then deleted some frames and re-saved it using macOS Preview.</p>
<p>I used ImageMagick to get it to loop like this:</p>
<pre><code>convert chrome-samesite-missing.gif -loop 0 chrome-samesite-missing-loop.gif
</code></pre>
<p>Note that the output filename comes last, AFTER the <code>-loop 0</code> option.</p>
<p>I installed ImageMagick on macOS using <code>brew install imagemagick</code></p>
<p>Here's the before GIF:</p>
<p>And the after GIF:</p>


path: sqlite_sqlite-aggregate-filter-clauses.md
topic: sqlite
title: SQLite aggregate filter clauses
url: https://github.com/simonw/til/blob/main/sqlite/sqlite-aggregate-filter-clauses.md
created: 2021-08-04T14:08:58-07:00
created_utc: 2021-08-04T21:08:58+00:00
updated: 2021-08-13T12:01:05-07:00
updated_utc: 2021-08-13T19:01:05+00:00
shot_hash: e4557112e414cb540bf3a4f2a20338c4
slug: sqlite-aggregate-filter-clauses
body:
SQLite supports aggregate filter clauses (as of [3.30.0, released 2019-10-04](https://www.sqlite.org/changes.html#version_3_30_0)), as described in this [SQL Pivot in all databases](https://modern-sql.com/use-case/pivot) tutorial.

An example query:
```sql
select
  year,
  sum(revenue) filter (where month = 1) as jan_revenue,
  sum(revenue) filter (where month = 2) as feb_revenue
from invoices
group by year
```
Here's an example using `sqlite-utils` to initially populate a database table:
```
/tmp % echo 'year,month,revenue
2019,1,110
2019,1,30
2019,2,34
2019,2,112
2020,1,40
2020,1,50
2020,2,110' | sqlite-utils insert data.db invoices - --csv
/tmp % sqlite-utils rows data.db invoices
[{"year": "2019", "month": "1", "revenue": "110"},
 {"year": "2019", "month": "1", "revenue": "30"},
 {"year": "2019", "month": "2", "revenue": "34"},
 {"year": "2019", "month": "2", "revenue": "112"},
 {"year": "2020", "month": "1", "revenue": "40"},
 {"year": "2020", "month": "1", "revenue": "50"},
 {"year": "2020", "month": "2", "revenue": "110"}]
```
And the results of that query:
```
/tmp % sqlite-utils data.db 'select
  year,
  sum(revenue) filter (where month = 1) as jan_revenue,
  sum(revenue) filter (where month = 2) as feb_revenue
from invoices
group by year' -t
  year    jan_revenue    feb_revenue
------  -------------  -------------
  2019            140            146
  2020             90            110
```

html:
<p>SQLite supports aggregate filter clauses (as of <a href="https://www.sqlite.org/changes.html#version_3_30_0" rel="nofollow">3.30.0, released 2019-10-04</a>), as described in this <a href="https://modern-sql.com/use-case/pivot" rel="nofollow">SQL Pivot in all databases</a> tutorial.</p>
<p>An example query:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  year,
  <span class="pl-c1">sum</span>(revenue) filter (<span class="pl-k">where</span> month <span class="pl-k">=</span> <span class="pl-c1">1</span>) <span class="pl-k">as</span> jan_revenue,
  <span class="pl-c1">sum</span>(revenue) filter (<span class="pl-k">where</span> month <span class="pl-k">=</span> <span class="pl-c1">2</span>) <span class="pl-k">as</span> feb_revenue
<span class="pl-k">from</span> invoices
<span class="pl-k">group by</span> year</pre></div>
<p>Here's an example using <code>sqlite-utils</code> to initially populate a database table:</p>
<pre><code>/tmp % echo 'year,month,revenue
2019,1,110
2019,1,30
2019,2,34
2019,2,112
2020,1,40
2020,1,50
2020,2,110' | sqlite-utils insert data.db invoices - --csv
/tmp % sqlite-utils rows data.db invoices
[{"year": "2019", "month": "1", "revenue": "110"},
 {"year": "2019", "month": "1", "revenue": "30"},
 {"year": "2019", "month": "2", "revenue": "34"},
 {"year": "2019", "month": "2", "revenue": "112"},
 {"year": "2020", "month": "1", "revenue": "40"},
 {"year": "2020", "month": "1", "revenue": "50"},
 {"year": "2020", "month": "2", "revenue": "110"}]
</code></pre>
<p>And the results of that query:</p>
<pre><code>/tmp % sqlite-utils data.db 'select
  year,
  sum(revenue) filter (where month = 1) as jan_revenue,
  sum(revenue) filter (where month = 2) as feb_revenue
from invoices
group by year' -t
  year    jan_revenue    feb_revenue
------  -------------  -------------
  2019            140            146
  2020             90            110
</code></pre>


path: imagemagick_compress-animated-gif.md
topic: imagemagick
title: Compressing an animated GIF with gifsicle or ImageMagick mogrify
url: https://github.com/simonw/til/blob/main/imagemagick/compress-animated-gif.md
created: 2021-08-05T22:02:29-07:00
created_utc: 2021-08-06T05:02:29+00:00
updated: 2021-09-13T13:01:48-07:00
updated_utc: 2021-09-13T20:01:48+00:00
shot_hash: c8fff8ea33b635315e881353e746df7d
slug: compress-animated-gif
body:
## Using gifsicle

Tip via Mark Norman Francis [on Twitter](https://twitter.com/cackhanded/status/1423526489623044098):

> Saw your GIF size TIL, and `gifsicle -O3 --colors 48 --lossy` gets it down to 320k. You can tweak the number of colours and loss to get it smaller but that’s when it starts to look worse to my eyes

I installed it using `brew install gifsicle` and ran it like this:

```
/tmp % ls -lah datasette-launch.gif 
-rw-r--r--@ 1 simon  wheel   3.7M Sep 13 12:52 datasette-launch.gif
/tmp % gifsicle -O3 --colors 48 --lossy -o datasette-launch-smaller.gif datasette-launch.gif 
/tmp % ls -lah datasette-launch*                                                            
-rw-r--r--  1 simon  wheel   613K Sep 13 12:54 datasette-launch-smaller.gif
-rw-r--r--@ 1 simon  wheel   3.7M Sep 13 12:52 datasette-launch.gif
```

Original: 3.7MB file:

![datasette-launch](https://user-images.githubusercontent.com/9599/133148193-89a01999-7fb4-407c-bb02-2bc79a70bd44.gif)

Compressed 613KB file:

![datasette-launch-smaller](https://user-images.githubusercontent.com/9599/133148197-e52db60d-442d-4db3-bf7c-28d5579e3b8a.gif)

The reduced colours there were a bit too much for me, especially for the purple gradient buttons at the end. So I tried this instead:

```
gifsicle -O3 --colors 128 --lossy -o datasette-launch-smaller-2.gif datasette-launch.gif
```
Which gave me a 723KB file which I think looks good enough for my purposes:

![datasette-launch-smaller-2](https://user-images.githubusercontent.com/9599/133148592-b98d5e78-f7fa-49e5-84d3-7c257c0bff17.gif)

## Using ImageMagick mogrify

Found this tip [on Stack Overflow](https://stackoverflow.com/a/47343340/6083): to reduce the site of an animated GIF, you can use the `mogrify` tool like this:

    mogrify -layers 'optimize' -fuzz 7%  sqlite-convert-demo.gif

This saves over the original, so make a copy of it first.

I ran this against this 1.3MB animated GIF:

![A demo of my sqlite-utils convert command](https://static.simonwillison.net/static/2021/sqlite-convert-demo-raw.gif)

The result was this 401KB GIF:

![Same demo of my sqlite-utils convert command, but a smaller file](https://static.simonwillison.net/static/2021/sqlite-convert-demo.gif)

The `-fuzz 7%` option is [documented here](https://imagemagick.org/script/command-line-options.php#fuzz) - it treats similar colours as the same colour:

> The distance can be in absolute intensity units or, by appending % as a percentage of the maximum possible intensity (255, 65535, or 4294967295).

html:
<h2>
<a id="user-content-using-gifsicle" class="anchor" href="#using-gifsicle" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using gifsicle</h2>
<p>Tip via Mark Norman Francis <a href="https://twitter.com/cackhanded/status/1423526489623044098" rel="nofollow">on Twitter</a>:</p>
<blockquote>
<p>Saw your GIF size TIL, and <code>gifsicle -O3 --colors 48 --lossy</code> gets it down to 320k. You can tweak the number of colours and loss to get it smaller but that’s when it starts to look worse to my eyes</p>
</blockquote>
<p>I installed it using <code>brew install gifsicle</code> and ran it like this:</p>
<pre><code>/tmp % ls -lah datasette-launch.gif 
-rw-r--r--@ 1 simon  wheel   3.7M Sep 13 12:52 datasette-launch.gif
/tmp % gifsicle -O3 --colors 48 --lossy -o datasette-launch-smaller.gif datasette-launch.gif 
/tmp % ls -lah datasette-launch*                                                            
-rw-r--r--  1 simon  wheel   613K Sep 13 12:54 datasette-launch-smaller.gif
-rw-r--r--@ 1 simon  wheel   3.7M Sep 13 12:52 datasette-launch.gif
</code></pre>
<p>Original: 3.7MB file:</p>
<p><a href="https://user-images.githubusercontent.com/9599/133148193-89a01999-7fb4-407c-bb02-2bc79a70bd44.gif" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/133148193-89a01999-7fb4-407c-bb02-2bc79a70bd44.gif" alt="datasette-launch" style="max-width:100%;"></a></p>
<p>Compressed 613KB file:</p>
<p><a href="https://user-images.githubusercontent.com/9599/133148197-e52db60d-442d-4db3-bf7c-28d5579e3b8a.gif" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/133148197-e52db60d-442d-4db3-bf7c-28d5579e3b8a.gif" alt="datasette-launch-smaller" style="max-width:100%;"></a></p>
<p>The reduced colours there were a bit too much for me, especially for the purple gradient buttons at the end. So I tried this instead:</p>
<pre><code>gifsicle -O3 --colors 128 --lossy -o datasette-launch-smaller-2.gif datasette-launch.gif
</code></pre>
<p>Which gave me a 723KB file which I think looks good enough for my purposes:</p>
<p><a href="https://user-images.githubusercontent.com/9599/133148592-b98d5e78-f7fa-49e5-84d3-7c257c0bff17.gif" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/133148592-b98d5e78-f7fa-49e5-84d3-7c257c0bff17.gif" alt="datasette-launch-smaller-2" style="max-width:100%;"></a></p>
<h2>
<a id="user-content-using-imagemagick-mogrify" class="anchor" href="#using-imagemagick-mogrify" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using ImageMagick mogrify</h2>
<p>Found this tip <a href="https://stackoverflow.com/a/47343340/6083" rel="nofollow">on Stack Overflow</a>: to reduce the site of an animated GIF, you can use the <code>mogrify</code> tool like this:</p>
<pre><code>mogrify -layers 'optimize' -fuzz 7%  sqlite-convert-demo.gif
</code></pre>
<p>This saves over the original, so make a copy of it first.</p>
<p>I ran this against this 1.3MB animated GIF:</p>
<p>The result was this 401KB GIF:</p>
<p>The <code>-fuzz 7%</code> option is <a href="https://imagemagick.org/script/command-line-options.php#fuzz" rel="nofollow">documented here</a> - it treats similar colours as the same colour:</p>
<blockquote>
<p>The distance can be in absolute intensity units or, by appending % as a percentage of the maximum possible intensity (255, 65535, or 4294967295).</p>
</blockquote>


path: cloudrun_tailing-cloud-run-request-logs.md
topic: cloudrun
title: Tailing Google Cloud Run request logs and importing them into SQLite
url: https://github.com/simonw/til/blob/main/cloudrun/tailing-cloud-run-request-logs.md
created: 2021-08-09T11:32:01-07:00
created_utc: 2021-08-09T18:32:01+00:00
updated: 2021-08-13T22:07:23-07:00
updated_utc: 2021-08-14T05:07:23+00:00
shot_hash: eba49d224d98a67308c137bcc3f0e777
slug: tailing-cloud-run-request-logs
body:
The `gcloud` CLI tool has [the alpha ability to tail log files](https://cloud.google.com/logging/docs/reference/tools/gcloud-logging#live-tailing) - but it's a bit of a pain to setup.

You have to install two extras for it. First, this:

    gcloud alpha logging tail

That installs the functionality, but as the documentation will tell you:

> To use `gcloud alpha logging tail`, you need to have Python 3 and the `grpcio` Python package installed.

Assuming you have Python 3, the problem you have to solve is *which Python* is the `gcloud` tool using to run. After digging around in the source code using `cat $(which gcloud)` I spotted the following:

    CLOUDSDK_PYTHON=$(order_python python3 python2 python2.7 python)

So it looks like (on macOS at least) it prefers to use the `python3` binary if it can find it.

So this works to install `grpcio` somewhere it can see it:

    python3 -m pip install grpcio

Having done that, you can start running commands. `gcloud logging logs list` shows a list of logs:

```
~ % gcloud logging logs list
NAME
projects/datasette-222320/logs/cloudaudit.googleapis.com%2Factivity
projects/datasette-222320/logs/cloudaudit.googleapis.com%2Fdata_access
projects/datasette-222320/logs/cloudaudit.googleapis.com%2Fsystem_event
projects/datasette-222320/logs/cloudbuild
projects/datasette-222320/logs/clouderrorreporting.googleapis.com%2Finsights
projects/datasette-222320/logs/cloudtrace.googleapis.com%2FTraceLatencyShiftDetected
projects/datasette-222320/logs/run.googleapis.com%2Frequests
projects/datasette-222320/logs/run.googleapis.com%2Fstderr
projects/datasette-222320/logs/run.googleapis.com%2Fstdout
projects/datasette-222320/logs/run.googleapis.com%2Fvarlog%2Fsystem
```
Then you can use `gcloud alpha logging tail projects/datasette-222320/logs/run.googleapis.com%2Frequests` to start logging. Only you also need a `CLOUDSDK_PYTHON_SITEPACKAGES=1` environment variable so that `gcloud` knows to look for the `grpcio` dependency.

    CLOUDSDK_PYTHON_SITEPACKAGES=1 \
      gcloud alpha logging tail projects/datasette-222320/logs/run.googleapis.com%2Frequests

The default format is verbose YAML. A log entry looks like this:

```yaml
httpRequest:
  latency: 0.123684963s
  remoteIp: 66.249.69.240
  requestMethod: GET
  requestSize: '510'
  requestUrl: https://www.niche-museums.com/browse/museums.json?_facet_size=max&country=United+States&_facet=osm_city&_facet=updated&_facet=osm_suburb&_facet=osm_footway&osm_city=Santa+Cruz
  responseSize: '6403'
  serverIp: 142.250.125.121
  status: 200
  userAgent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
insertId: 611171fe000a38a469d59595
labels:
  instanceId: 00bf4bf02dab164592dbbb9220b56c3ce64cb0f1c1f37812d1d61e851a931e9964ba539c2ede42886773c82662cc28aa858749d2697f537ff7a61e7b
  service: niche-museums
logName: projects/datasette-222320/logs/run.googleapis.com%2Frequests
receiveTimestamp: '2021-08-09T18:20:46.935658405Z'
resource:
  labels:
    configuration_name: niche-museums
    location: us-central1
    project_id: datasette-222320
    revision_name: niche-museums-00039-sur
    service_name: niche-museums
  type: cloud_run_revision
severity: INFO
timestamp: '2021-08-09T18:20:46.669860Z'
trace: projects/datasette-222320/traces/306a0d6e7e055ba66172003a74c926c2
```

I decided to import into a SQLite database so I could use [Datasette](https://datasette.io/) to analyze the log files (hooray for facets).

Adding `--format json` switches the output to JSON - but it's a pretty-printed array of JSON objects, something like this:

```json
[
  {
    "httpRequest": {
      "latency": "0.112114537s",
      "remoteIp": "40.77.167.88",
      "requestMethod": "GET",
      "requestSize": "534",
      "requestUrl": "https://datasette.io/content/repos?forks=0&_facet=homepage&_facet=size&_facet=open_issues&open_issues=3&size=564&_sort=readme_html",
      "responseSize": "72757",
      "serverIp": "216.239.38.21",
      "status": 200,
      "userAgent": "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
    },
    "insertId": "6111722f000b5b4c4d4071e2",
    "labels": {
      "instanceId": "00bf4bf02d1d7fe4402c3aff8a34688d9a910e6ee6d2545ceebc1edefb99461481e6d9f9ae8de4e907e3d18b98ea9c7f57b2abb527c8857d9163ed193db766c349a1ee",
      "service": "datasette-io"
    },
    "logName": "projects/datasette-222320/logs/run.googleapis.com%2Frequests",
    "receiveTimestamp": "2021-08-09T18:21:36.061693305Z",
    "resource": {
      "labels": {
        "configuration_name": "datasette-io",
        "location": "us-central1",
        "project_id": "datasette-222320",
        "revision_name": "datasette-io-00416-coy",
        "service_name": "datasette-io"
      },
      "type": "cloud_run_revision"
    },
    "severity": "INFO",
    "timestamp": "2021-08-09T18:21:35.744268Z",
    "trace": "projects/datasette-222320/traces/016d640caf845fbf8709486bc8dff9c7"
  }
]
```

I want to stream the logs into `sqlite-utils` using [newline-delimited JSON](https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-newline-delimited-json) since that can insert while the data is still being tailed.

I ended up using two new `jq` recipes:

    cat example.json | jq -cn --stream 'fromstream(1|truncate_stream(inputs))'

This turns an `[{"array": "of objects"}, {"like": "this one"}]` into a stream of newline-delimited objects. I [found the recipe here](https://github.com/stedolan/jq/issues/1984#issuecomment-568918146) - I don't understand it.

As you can see above, the objects are nested. I want them as flat objects so that `sqlite-utils insert` will create a separate column for each nested value. I used [this recipe](https://til.simonwillison.net/jq/flatten-nested-json-objects-jq) for that.

The end result was this:

```
CLOUDSDK_PYTHON_SITEPACKAGES=1 gcloud alpha logging tail \
  projects/datasette-222320/logs/run.googleapis.com%2Frequests \
  --format json \
| jq -cn --stream 'fromstream(1|truncate_stream(inputs))' \
| jq -c '[leaf_paths as $path | {
  "key": $path | join("_"), "value": getpath($path)
}] | from_entries' \
| sqlite-utils insert /tmp/logs.db logs - --nl --alter --batch-size 1
```
That last line inserts the data into the `/tmp/logs.db` database file. `--nl` means "expect newline-delimited JSON", `--alter` means "add new columns if they are missing", `--batch-size 1` means "commit after every record" (so I can see them in Datasette while they are streaming in).

**UPDATE:** [sqlite-utils 3.15](https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-15) added a `--flatten` option which you can use instead of that second `jq` recipe, so this should work instead:

```
CLOUDSDK_PYTHON_SITEPACKAGES=1 gcloud alpha logging tail \
  projects/datasette-222320/logs/run.googleapis.com%2Frequests \
  --format json \
| jq -cn --stream 'fromstream(1|truncate_stream(inputs))' \
| sqlite-utils insert /tmp/logs.db logs - --nl --alter --batch-size 1 --flatten
```

The resulting schema looks like this (via `sqlite-utils schema /tmp/logs.db`):
```sql
CREATE TABLE [logs] (
   [httpRequest_latency] TEXT,
   [httpRequest_remoteIp] TEXT,
   [httpRequest_requestMethod] TEXT,
   [httpRequest_requestSize] TEXT,
   [httpRequest_requestUrl] TEXT,
   [httpRequest_responseSize] TEXT,
   [httpRequest_serverIp] TEXT,
   [httpRequest_status] INTEGER,
   [httpRequest_userAgent] TEXT,
   [insertId] TEXT,
   [labels_instanceId] TEXT,
   [labels_service] TEXT,
   [logName] TEXT,
   [receiveTimestamp] TEXT,
   [resource_labels_configuration_name] TEXT,
   [resource_labels_location] TEXT,
   [resource_labels_project_id] TEXT,
   [resource_labels_revision_name] TEXT,
   [resource_labels_service_name] TEXT,
   [resource_type] TEXT,
   [severity] TEXT,
   [timestamp] TEXT,
   [trace] TEXT,
   [httpRequest_referer] TEXT
);
```

Then I ran `datasette /tmp/logs.db` to start exploring the logs. Faceting by `resource_labels_service_name` was particularly useful.

<img width="1340" alt="Screenshot of logs in Datasette" src="https://user-images.githubusercontent.com/9599/128755995-fab7e478-82a5-4d80-a959-f89f7dd39209.png">

The `httpRequest_latency` column contains text data that looks like `0.012572683s` - thankfully if you cast it to a `float` the trailing `s` will be ignored. Here's an example query showing the services with the highest average latency:

```sql
select
  resource_labels_service_name,
  avg(cast(httpRequest_latency as float)) as avg_latency,
  count(*)
from
  logs
group by
  resource_labels_service_name
order by
  avg_latency desc
```

## Using the Logs explorer

Alternatively, you can use the Google Cloud logs explorer! It has pretty decent faceted search built in.

Here's a query showing results from that log file:
```
resource.type="cloud_run_revision"
log_name="projects/datasette-222320/logs/run.googleapis.com%2Frequests"
```
Run that at https://console.cloud.google.com/logs/query - or here's a link I can use to execute that directly (for the last 7 days): https://console.cloud.google.com/logs/query;query=resource.type%3D%22cloud_run_revision%22%0Alog_name%3D%22projects%2Fdatasette-222320%2Flogs%2Frun.googleapis.com%252Frequests%22;timeRange=P7D;?project=datasette-222320

html:
<p>The <code>gcloud</code> CLI tool has <a href="https://cloud.google.com/logging/docs/reference/tools/gcloud-logging#live-tailing" rel="nofollow">the alpha ability to tail log files</a> - but it's a bit of a pain to setup.</p>
<p>You have to install two extras for it. First, this:</p>
<pre><code>gcloud alpha logging tail
</code></pre>
<p>That installs the functionality, but as the documentation will tell you:</p>
<blockquote>
<p>To use <code>gcloud alpha logging tail</code>, you need to have Python 3 and the <code>grpcio</code> Python package installed.</p>
</blockquote>
<p>Assuming you have Python 3, the problem you have to solve is <em>which Python</em> is the <code>gcloud</code> tool using to run. After digging around in the source code using <code>cat $(which gcloud)</code> I spotted the following:</p>
<pre><code>CLOUDSDK_PYTHON=$(order_python python3 python2 python2.7 python)
</code></pre>
<p>So it looks like (on macOS at least) it prefers to use the <code>python3</code> binary if it can find it.</p>
<p>So this works to install <code>grpcio</code> somewhere it can see it:</p>
<pre><code>python3 -m pip install grpcio
</code></pre>
<p>Having done that, you can start running commands. <code>gcloud logging logs list</code> shows a list of logs:</p>
<pre><code>~ % gcloud logging logs list
NAME
projects/datasette-222320/logs/cloudaudit.googleapis.com%2Factivity
projects/datasette-222320/logs/cloudaudit.googleapis.com%2Fdata_access
projects/datasette-222320/logs/cloudaudit.googleapis.com%2Fsystem_event
projects/datasette-222320/logs/cloudbuild
projects/datasette-222320/logs/clouderrorreporting.googleapis.com%2Finsights
projects/datasette-222320/logs/cloudtrace.googleapis.com%2FTraceLatencyShiftDetected
projects/datasette-222320/logs/run.googleapis.com%2Frequests
projects/datasette-222320/logs/run.googleapis.com%2Fstderr
projects/datasette-222320/logs/run.googleapis.com%2Fstdout
projects/datasette-222320/logs/run.googleapis.com%2Fvarlog%2Fsystem
</code></pre>
<p>Then you can use <code>gcloud alpha logging tail projects/datasette-222320/logs/run.googleapis.com%2Frequests</code> to start logging. Only you also need a <code>CLOUDSDK_PYTHON_SITEPACKAGES=1</code> environment variable so that <code>gcloud</code> knows to look for the <code>grpcio</code> dependency.</p>
<pre><code>CLOUDSDK_PYTHON_SITEPACKAGES=1 \
  gcloud alpha logging tail projects/datasette-222320/logs/run.googleapis.com%2Frequests
</code></pre>
<p>The default format is verbose YAML. A log entry looks like this:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">httpRequest</span>:
  <span class="pl-ent">latency</span>: <span class="pl-s">0.123684963s</span>
  <span class="pl-ent">remoteIp</span>: <span class="pl-s">66.249.69.240</span>
  <span class="pl-ent">requestMethod</span>: <span class="pl-s">GET</span>
  <span class="pl-ent">requestSize</span>: <span class="pl-s"><span class="pl-pds">'</span>510<span class="pl-pds">'</span></span>
  <span class="pl-ent">requestUrl</span>: <span class="pl-s">https://www.niche-museums.com/browse/museums.json?_facet_size=max&amp;country=United+States&amp;_facet=osm_city&amp;_facet=updated&amp;_facet=osm_suburb&amp;_facet=osm_footway&amp;osm_city=Santa+Cruz</span>
  <span class="pl-ent">responseSize</span>: <span class="pl-s"><span class="pl-pds">'</span>6403<span class="pl-pds">'</span></span>
  <span class="pl-ent">serverIp</span>: <span class="pl-s">142.250.125.121</span>
  <span class="pl-ent">status</span>: <span class="pl-c1">200</span>
  <span class="pl-ent">userAgent</span>: <span class="pl-s">Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)</span>
<span class="pl-ent">insertId</span>: <span class="pl-s">611171fe000a38a469d59595</span>
<span class="pl-ent">labels</span>:
  <span class="pl-ent">instanceId</span>: <span class="pl-s">00bf4bf02dab164592dbbb9220b56c3ce64cb0f1c1f37812d1d61e851a931e9964ba539c2ede42886773c82662cc28aa858749d2697f537ff7a61e7b</span>
  <span class="pl-ent">service</span>: <span class="pl-s">niche-museums</span>
<span class="pl-ent">logName</span>: <span class="pl-s">projects/datasette-222320/logs/run.googleapis.com%2Frequests</span>
<span class="pl-ent">receiveTimestamp</span>: <span class="pl-s"><span class="pl-pds">'</span>2021-08-09T18:20:46.935658405Z<span class="pl-pds">'</span></span>
<span class="pl-ent">resource</span>:
  <span class="pl-ent">labels</span>:
    <span class="pl-ent">configuration_name</span>: <span class="pl-s">niche-museums</span>
    <span class="pl-ent">location</span>: <span class="pl-s">us-central1</span>
    <span class="pl-ent">project_id</span>: <span class="pl-s">datasette-222320</span>
    <span class="pl-ent">revision_name</span>: <span class="pl-s">niche-museums-00039-sur</span>
    <span class="pl-ent">service_name</span>: <span class="pl-s">niche-museums</span>
  <span class="pl-ent">type</span>: <span class="pl-s">cloud_run_revision</span>
<span class="pl-ent">severity</span>: <span class="pl-s">INFO</span>
<span class="pl-ent">timestamp</span>: <span class="pl-s"><span class="pl-pds">'</span>2021-08-09T18:20:46.669860Z<span class="pl-pds">'</span></span>
<span class="pl-ent">trace</span>: <span class="pl-s">projects/datasette-222320/traces/306a0d6e7e055ba66172003a74c926c2</span></pre></div>
<p>I decided to import into a SQLite database so I could use <a href="https://datasette.io/" rel="nofollow">Datasette</a> to analyze the log files (hooray for facets).</p>
<p>Adding <code>--format json</code> switches the output to JSON - but it's a pretty-printed array of JSON objects, something like this:</p>
<div class="highlight highlight-source-json"><pre>[
  {
    <span class="pl-s"><span class="pl-pds">"</span>httpRequest<span class="pl-pds">"</span></span>: {
      <span class="pl-s"><span class="pl-pds">"</span>latency<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>0.112114537s<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>remoteIp<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>40.77.167.88<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>requestMethod<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>GET<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>requestSize<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>534<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>requestUrl<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://datasette.io/content/repos?forks=0&amp;_facet=homepage&amp;_facet=size&amp;_facet=open_issues&amp;open_issues=3&amp;size=564&amp;_sort=readme_html<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>responseSize<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>72757<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>serverIp<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>216.239.38.21<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>status<span class="pl-pds">"</span></span>: <span class="pl-c1">200</span>,
      <span class="pl-s"><span class="pl-pds">"</span>userAgent<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)<span class="pl-pds">"</span></span>
    },
    <span class="pl-s"><span class="pl-pds">"</span>insertId<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>6111722f000b5b4c4d4071e2<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>labels<span class="pl-pds">"</span></span>: {
      <span class="pl-s"><span class="pl-pds">"</span>instanceId<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>00bf4bf02d1d7fe4402c3aff8a34688d9a910e6ee6d2545ceebc1edefb99461481e6d9f9ae8de4e907e3d18b98ea9c7f57b2abb527c8857d9163ed193db766c349a1ee<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>service<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>datasette-io<span class="pl-pds">"</span></span>
    },
    <span class="pl-s"><span class="pl-pds">"</span>logName<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>projects/datasette-222320/logs/run.googleapis.com%2Frequests<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>receiveTimestamp<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-08-09T18:21:36.061693305Z<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>resource<span class="pl-pds">"</span></span>: {
      <span class="pl-s"><span class="pl-pds">"</span>labels<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>configuration_name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>datasette-io<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>location<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>us-central1<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>project_id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>datasette-222320<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>revision_name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>datasette-io-00416-coy<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>service_name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>datasette-io<span class="pl-pds">"</span></span>
      },
      <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>cloud_run_revision<span class="pl-pds">"</span></span>
    },
    <span class="pl-s"><span class="pl-pds">"</span>severity<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>INFO<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>timestamp<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-08-09T18:21:35.744268Z<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>trace<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>projects/datasette-222320/traces/016d640caf845fbf8709486bc8dff9c7<span class="pl-pds">"</span></span>
  }
]</pre></div>
<p>I want to stream the logs into <code>sqlite-utils</code> using <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#inserting-newline-delimited-json" rel="nofollow">newline-delimited JSON</a> since that can insert while the data is still being tailed.</p>
<p>I ended up using two new <code>jq</code> recipes:</p>
<pre><code>cat example.json | jq -cn --stream 'fromstream(1|truncate_stream(inputs))'
</code></pre>
<p>This turns an <code>[{"array": "of objects"}, {"like": "this one"}]</code> into a stream of newline-delimited objects. I <a href="https://github.com/stedolan/jq/issues/1984#issuecomment-568918146">found the recipe here</a> - I don't understand it.</p>
<p>As you can see above, the objects are nested. I want them as flat objects so that <code>sqlite-utils insert</code> will create a separate column for each nested value. I used <a href="https://til.simonwillison.net/jq/flatten-nested-json-objects-jq" rel="nofollow">this recipe</a> for that.</p>
<p>The end result was this:</p>
<pre><code>CLOUDSDK_PYTHON_SITEPACKAGES=1 gcloud alpha logging tail \
  projects/datasette-222320/logs/run.googleapis.com%2Frequests \
  --format json \
| jq -cn --stream 'fromstream(1|truncate_stream(inputs))' \
| jq -c '[leaf_paths as $path | {
  "key": $path | join("_"), "value": getpath($path)
}] | from_entries' \
| sqlite-utils insert /tmp/logs.db logs - --nl --alter --batch-size 1
</code></pre>
<p>That last line inserts the data into the <code>/tmp/logs.db</code> database file. <code>--nl</code> means "expect newline-delimited JSON", <code>--alter</code> means "add new columns if they are missing", <code>--batch-size 1</code> means "commit after every record" (so I can see them in Datasette while they are streaming in).</p>
<p><strong>UPDATE:</strong> <a href="https://sqlite-utils.datasette.io/en/stable/changelog.html#v3-15" rel="nofollow">sqlite-utils 3.15</a> added a <code>--flatten</code> option which you can use instead of that second <code>jq</code> recipe, so this should work instead:</p>
<pre><code>CLOUDSDK_PYTHON_SITEPACKAGES=1 gcloud alpha logging tail \
  projects/datasette-222320/logs/run.googleapis.com%2Frequests \
  --format json \
| jq -cn --stream 'fromstream(1|truncate_stream(inputs))' \
| sqlite-utils insert /tmp/logs.db logs - --nl --alter --batch-size 1 --flatten
</code></pre>
<p>The resulting schema looks like this (via <code>sqlite-utils schema /tmp/logs.db</code>):</p>
<div class="highlight highlight-source-sql"><pre>CREATE TABLE [logs] (
   [httpRequest_latency] <span class="pl-k">TEXT</span>,
   [httpRequest_remoteIp] <span class="pl-k">TEXT</span>,
   [httpRequest_requestMethod] <span class="pl-k">TEXT</span>,
   [httpRequest_requestSize] <span class="pl-k">TEXT</span>,
   [httpRequest_requestUrl] <span class="pl-k">TEXT</span>,
   [httpRequest_responseSize] <span class="pl-k">TEXT</span>,
   [httpRequest_serverIp] <span class="pl-k">TEXT</span>,
   [httpRequest_status] <span class="pl-k">INTEGER</span>,
   [httpRequest_userAgent] <span class="pl-k">TEXT</span>,
   [insertId] <span class="pl-k">TEXT</span>,
   [labels_instanceId] <span class="pl-k">TEXT</span>,
   [labels_service] <span class="pl-k">TEXT</span>,
   [logName] <span class="pl-k">TEXT</span>,
   [receiveTimestamp] <span class="pl-k">TEXT</span>,
   [resource_labels_configuration_name] <span class="pl-k">TEXT</span>,
   [resource_labels_location] <span class="pl-k">TEXT</span>,
   [resource_labels_project_id] <span class="pl-k">TEXT</span>,
   [resource_labels_revision_name] <span class="pl-k">TEXT</span>,
   [resource_labels_service_name] <span class="pl-k">TEXT</span>,
   [resource_type] <span class="pl-k">TEXT</span>,
   [severity] <span class="pl-k">TEXT</span>,
   [<span class="pl-k">timestamp</span>] <span class="pl-k">TEXT</span>,
   [trace] <span class="pl-k">TEXT</span>,
   [httpRequest_referer] <span class="pl-k">TEXT</span>
);</pre></div>
<p>Then I ran <code>datasette /tmp/logs.db</code> to start exploring the logs. Faceting by <code>resource_labels_service_name</code> was particularly useful.</p>
<p><a href="https://user-images.githubusercontent.com/9599/128755995-fab7e478-82a5-4d80-a959-f89f7dd39209.png" target="_blank" rel="nofollow"><img width="1340" alt="Screenshot of logs in Datasette" src="https://user-images.githubusercontent.com/9599/128755995-fab7e478-82a5-4d80-a959-f89f7dd39209.png" style="max-width:100%;"></a></p>
<p>The <code>httpRequest_latency</code> column contains text data that looks like <code>0.012572683s</code> - thankfully if you cast it to a <code>float</code> the trailing <code>s</code> will be ignored. Here's an example query showing the services with the highest average latency:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  resource_labels_service_name,
  <span class="pl-c1">avg</span>(cast(httpRequest_latency <span class="pl-k">as</span> float)) <span class="pl-k">as</span> avg_latency,
  <span class="pl-c1">count</span>(<span class="pl-k">*</span>)
<span class="pl-k">from</span>
  logs
<span class="pl-k">group by</span>
  resource_labels_service_name
<span class="pl-k">order by</span>
  avg_latency <span class="pl-k">desc</span></pre></div>
<h2>
<a id="user-content-using-the-logs-explorer" class="anchor" href="#using-the-logs-explorer" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using the Logs explorer</h2>
<p>Alternatively, you can use the Google Cloud logs explorer! It has pretty decent faceted search built in.</p>
<p>Here's a query showing results from that log file:</p>
<pre><code>resource.type="cloud_run_revision"
log_name="projects/datasette-222320/logs/run.googleapis.com%2Frequests"
</code></pre>
<p>Run that at <a href="https://console.cloud.google.com/logs/query" rel="nofollow">https://console.cloud.google.com/logs/query</a> - or here's a link I can use to execute that directly (for the last 7 days): <a href="https://console.cloud.google.com/logs/query;query=resource.type%3D%22cloud_run_revision%22%0Alog_name%3D%22projects%2Fdatasette-222320%2Flogs%2Frun.googleapis.com%252Frequests%22;timeRange=P7D;?project=datasette-222320" rel="nofollow">https://console.cloud.google.com/logs/query;query=resource.type%3D%22cloud_run_revision%22%0Alog_name%3D%22projects%2Fdatasette-222320%2Flogs%2Frun.googleapis.com%252Frequests%22;timeRange=P7D;?project=datasette-222320</a></p>


path: python_find-local-variables-in-exception-traceback.md
topic: python
title: Find local variables in the traceback for an exception
url: https://github.com/simonw/til/blob/main/python/find-local-variables-in-exception-traceback.md
created: 2021-08-09T15:31:48-07:00
created_utc: 2021-08-09T22:31:48+00:00
updated: 2021-08-09T15:31:48-07:00
updated_utc: 2021-08-09T22:31:48+00:00
shot_hash: 272786bc5a008d779dd0361b32d304f5
slug: find-local-variables-in-exception-traceback
body:
For [sqlite-utils issue #309](https://github.com/simonw/sqlite-utils/issues/309) I had an error that looked like this:
```
OverflowError: Python int too large to convert to SQLite INTEGER
Traceback (most recent call last):
  File "/home/sean/.local/bin/sqlite-utils", line 8, in <module>
    sys.exit(cli())
  [...]
  File "/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/db.py", line 257, in execute
    return self.conn.execute(sql, parameters)
```
And I wanted to display the values of the `sql` and `parameters` variables as part of a custom error message.

It turns out Python exceptions have a `e.__traceback__` property which provides access to a traceback - and the local variables in that traceback are avialable on the `tb.tb_frame.f_locals` dictionary - or if not that one, the `tb.tb_next.tb_frame.f_locals` property and so on up to the top of the stack.

I wrote this function to find them:

```python
def _find_variables(tb, vars):
    to_find = list(vars)
    found = {}
    for var in to_find:
        if var in tb.tb_frame.f_locals:
            vars.remove(var)
            found[var] = tb.tb_frame.f_locals[var]
    if vars and tb.tb_next:
        found.update(_find_variables(tb.tb_next, vars))
    return found
```

Use it like this:

```python
    try:
        db[table].insert_all(
            docs, pk=pk, batch_size=batch_size, alter=alter, **extra_kwargs
        )
    except Exception as e:
        # If we can find sql= and params= arguments, show those
        variables = _find_variables(e.__traceback__, ["sql", "params"])
        if "sql" in variables and "params" in variables:
            raise click.ClickException(
                "{}\n\nsql = {}\nparams={}".format(
                    str(e), variables["sql"], variables["params"]
                )
            )
        else:
            raise
```
[Full commit](https://github.com/simonw/sqlite-utils/commit/14f643d9e91f5557d5e46251dadac481f4b41021) add this to `sqlite-utils`.

html:
<p>For <a href="https://github.com/simonw/sqlite-utils/issues/309">sqlite-utils issue #309</a> I had an error that looked like this:</p>
<pre><code>OverflowError: Python int too large to convert to SQLite INTEGER
Traceback (most recent call last):
  File "/home/sean/.local/bin/sqlite-utils", line 8, in &lt;module&gt;
    sys.exit(cli())
  [...]
  File "/home/sean/.local/lib/python3.8/site-packages/sqlite_utils/db.py", line 257, in execute
    return self.conn.execute(sql, parameters)
</code></pre>
<p>And I wanted to display the values of the <code>sql</code> and <code>parameters</code> variables as part of a custom error message.</p>
<p>It turns out Python exceptions have a <code>e.__traceback__</code> property which provides access to a traceback - and the local variables in that traceback are avialable on the <code>tb.tb_frame.f_locals</code> dictionary - or if not that one, the <code>tb.tb_next.tb_frame.f_locals</code> property and so on up to the top of the stack.</p>
<p>I wrote this function to find them:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">_find_variables</span>(<span class="pl-s1">tb</span>, <span class="pl-s1">vars</span>):
    <span class="pl-s1">to_find</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-s1">vars</span>)
    <span class="pl-s1">found</span> <span class="pl-c1">=</span> {}
    <span class="pl-k">for</span> <span class="pl-s1">var</span> <span class="pl-c1">in</span> <span class="pl-s1">to_find</span>:
        <span class="pl-k">if</span> <span class="pl-s1">var</span> <span class="pl-c1">in</span> <span class="pl-s1">tb</span>.<span class="pl-s1">tb_frame</span>.<span class="pl-s1">f_locals</span>:
            <span class="pl-s1">vars</span>.<span class="pl-en">remove</span>(<span class="pl-s1">var</span>)
            <span class="pl-s1">found</span>[<span class="pl-s1">var</span>] <span class="pl-c1">=</span> <span class="pl-s1">tb</span>.<span class="pl-s1">tb_frame</span>.<span class="pl-s1">f_locals</span>[<span class="pl-s1">var</span>]
    <span class="pl-k">if</span> <span class="pl-s1">vars</span> <span class="pl-c1">and</span> <span class="pl-s1">tb</span>.<span class="pl-s1">tb_next</span>:
        <span class="pl-s1">found</span>.<span class="pl-en">update</span>(<span class="pl-en">_find_variables</span>(<span class="pl-s1">tb</span>.<span class="pl-s1">tb_next</span>, <span class="pl-s1">vars</span>))
    <span class="pl-k">return</span> <span class="pl-s1">found</span></pre></div>
<p>Use it like this:</p>
<div class="highlight highlight-source-python"><pre>    <span class="pl-k">try</span>:
        <span class="pl-s1">db</span>[<span class="pl-s1">table</span>].<span class="pl-en">insert_all</span>(
            <span class="pl-s1">docs</span>, <span class="pl-s1">pk</span><span class="pl-c1">=</span><span class="pl-s1">pk</span>, <span class="pl-s1">batch_size</span><span class="pl-c1">=</span><span class="pl-s1">batch_size</span>, <span class="pl-s1">alter</span><span class="pl-c1">=</span><span class="pl-s1">alter</span>, <span class="pl-c1">**</span><span class="pl-s1">extra_kwargs</span>
        )
    <span class="pl-k">except</span> <span class="pl-v">Exception</span> <span class="pl-k">as</span> <span class="pl-s1">e</span>:
        <span class="pl-c"># If we can find sql= and params= arguments, show those</span>
        <span class="pl-s1">variables</span> <span class="pl-c1">=</span> <span class="pl-en">_find_variables</span>(<span class="pl-s1">e</span>.<span class="pl-s1">__traceback__</span>, [<span class="pl-s">"sql"</span>, <span class="pl-s">"params"</span>])
        <span class="pl-k">if</span> <span class="pl-s">"sql"</span> <span class="pl-c1">in</span> <span class="pl-s1">variables</span> <span class="pl-c1">and</span> <span class="pl-s">"params"</span> <span class="pl-c1">in</span> <span class="pl-s1">variables</span>:
            <span class="pl-k">raise</span> <span class="pl-s1">click</span>.<span class="pl-v">ClickException</span>(
                <span class="pl-s">"{}<span class="pl-cce">\n</span><span class="pl-cce">\n</span>sql = {}<span class="pl-cce">\n</span>params={}"</span>.<span class="pl-en">format</span>(
                    <span class="pl-en">str</span>(<span class="pl-s1">e</span>), <span class="pl-s1">variables</span>[<span class="pl-s">"sql"</span>], <span class="pl-s1">variables</span>[<span class="pl-s">"params"</span>]
                )
            )
        <span class="pl-k">else</span>:
            <span class="pl-k">raise</span></pre></div>
<p><a href="https://github.com/simonw/sqlite-utils/commit/14f643d9e91f5557d5e46251dadac481f4b41021">Full commit</a> add this to <code>sqlite-utils</code>.</p>


path: sphinx_sphinx-autodoc.md
topic: sphinx
title: Adding Sphinx autodoc to a project, and configuring Read The Docs to build it
url: https://github.com/simonw/til/blob/main/sphinx/sphinx-autodoc.md
created: 2021-08-10T16:31:15-07:00
created_utc: 2021-08-10T23:31:15+00:00
updated: 2022-03-11T10:06:59-08:00
updated_utc: 2022-03-11T18:06:59+00:00
shot_hash: c06bc3c2fde89ecf62873c1e41ab672e
slug: sphinx-autodoc
body:
I built a [new API reference page](https://sqlite-utils.datasette.io/en/latest/reference.html) today for `sqlite-utils`, using the Sphinx [autodoc extension](https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html) to extract docstrings from the code and use them to build a full class reference.

I've avoided this kind of documentation in the past because I think narrative prose is a *much* better way of providing documentation - but ``sqlite-utils`` already has [detailed narrative prose](https://sqlite-utils.datasette.io/en/stable/python-api.html), so I felt that adding reference documentation powered by docstrings could enhance that project - while also providing better inline document for tools such as Visual Studio Code and Jupyter.

<img width="500" alt="Screenshot of the new rendered documentation" src="https://user-images.githubusercontent.com/9599/128947691-403e0332-5bab-4851-a9aa-c29e759e944c.png">

## Setting up autodoc

Getting autodoc working in my local documentation environment took a few steps.

First, I added `"sphinx.ext.autodoc"` to the `extensions = []` list in my `docs/conf.py` file.

Then I created a new page in ``docs/reference.rst`` and added the following:

```rst
===============
 API Reference
===============

.. _reference_db_database:

sqlite_utils.db.Database
========================

.. autoclass:: sqlite_utils.db.Database
    :members:
    :undoc-members:
```
This was enough to start generating documentation for the `sqlite_utils.db.Database` class.

`:members:` means "show documentation for everything in the class that has a docstring".

`:undoc-members:` means "also include class members that don't have a docstring yet".

I added my own rST headings and reference links - this ensured that the classes would automatically show up in the table of contents for my documentation.

## Live updates while editing docstrings

I use [sphinx-autobuild](https://pypi.org/project/sphinx-autobuild/) to automatically rebuild and reload the browser while I'm editing documentation.

This has [a known limitation](https://github.com/executablebooks/sphinx-autobuild#relevant-sphinx-bugs) that it won't pick up on edits made to the `*.py` files that should cause a reload.

I found a workaround: add `-a` to disable incremental builds (rebuilding everything) and `--watch ../sqlite_utils` to trigger a new build when any file in that folder was updated.

I added this to [the Makefile](https://github.com/simonw/sqlite-utils/blob/6155da72c8939b5d9bdacb7853e5e8d1767ce1d5/docs/Makefile) is the `docs/` directory:
```
livehtml:
    sphinx-autobuild -a -b html "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(0) --watch ../sqlite_utils
```
Then running `make livehtml` started a web server for the documentation that would refresh the browser any time I edited a docstring.

## More autoclass options

I ended up using the following:

```rst
.. autoclass:: sqlite_utils.db.Database
    :members:
    :undoc-members:
    :show-inheritance:
    :special-members: __getitem__
    :exclude-members: use_counts_table, execute_returning_dicts, resolve_foreign_keys
```
`:show-inheritance:` adds a note about the inheritance hierarchy for the class.

By default `__special__` methods like `__getitem__` are not included. `:special-members:` fixes this for specified methods.

`:exclude-members;` accepts a comma-separated list of methods that shouldn't be shown in the documentation. I used this to remove some internal methods.

I could have renamed these to `_use_counts_table` but that would be a breaking API change, even though these methods are deliberately undocumented and shouldn't be used by anyone.

## Source, not alphabetical order

The class methods default to being shown in alphabetical order - adding ` autodoc_member_order = "bysource"` to `docs/conf.py` changed that to source order instead.

## reStructuredText in the docstrings

Adding reStructuredText to the docstrings works for formatting the generated API documentation. You can also use `:ref:reference-links`. Here's an example of a docstring with more complex formatting:

```python
    def rows_where(
        self,
        where: str = None,
        where_args: Optional[Union[Iterable, dict]] = None,
        order_by: str = None,
        select: str = "*",
        limit: int = None,
        offset: int = None,
    ) -> Generator[dict, None, None]:
        """
        Iterate over every row in this table or view that matches the specified where clause.
        Returns each row as a dictionary. See :ref:`python_api_rows` for more details.

        :param where: SQL where fragment to use, for example ``id > ?``
        :param where_args: Parameters to use with that fragment - an iterable for ``id > ?``
          parameters, or a dictionary for ``id > :id``
        :param order_by: Column or fragment of SQL to order by
        :param select: Comma-separated list of columns to select - defaults to ``*``
        :param limit: Integer number of rows to limit to
        :param offset: Integer for SQL offset
        """
        if not self.exists():
            # ...
```
If you add `autodoc_typehints = "description"` to the `conf.py` configuration the `:param name: Description` lines will be shown in a list of parameters that also includes the type hints:

<img width="674" alt="Screenshot showing the type hints in the list of parameters" src="https://user-images.githubusercontent.com/9599/157924734-dcf56d26-69dd-4201-b81a-988eb4414a8b.png">

I learned about this in [sqlite-utils/issues/413](https://github.com/simonw/sqlite-utils/issues/413) while trying to improve the display of complex type annotations.

## Getting it working on Read The Docs

Read The Docs has the option to deploy branches - you can set that up in  the "Versions" tab. I was working in an `autodoc` branch - I set that branch to be "Active" and "Hidden", so it would be built by Read The Docs but not linked to from anywhere.

My first attempt failed to render the generated documentation, because Read The Docs did not know where to import my code from.

I solved that by adding a `.readthedocs.yaml` configuration file to my repository containing the following:

```yaml

version: 2

sphinx:
  configuration: docs/conf.py

python:
  version: "3.8"
  install:
  - method: pip
    path: .
    extra_requirements:
    - docs
```

The key thing here is that `install` block, which tells Read The Docs to run the equivalent of this before building the documentation:

    pip install -e '.[docs]'

For my repository that installs the `docs` optional dependencies from the `setup.py`, but also installs the `sqlite_utils` Python package itself. That way Sphinx can find and import `sqlite_utils.db.Database` in order to generate the documentation.

## The end result

The new page of documentation is now live at [en/latest/reference.html](https://sqlite-utils.datasette.io/en/latest/reference.html). The pull request in which I figured this all out is [sqlite-utils/pull/312](https://github.com/simonw/sqlite-utils/pull/312).

html:
<p>I built a <a href="https://sqlite-utils.datasette.io/en/latest/reference.html" rel="nofollow">new API reference page</a> today for <code>sqlite-utils</code>, using the Sphinx <a href="https://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html" rel="nofollow">autodoc extension</a> to extract docstrings from the code and use them to build a full class reference.</p>
<p>I've avoided this kind of documentation in the past because I think narrative prose is a <em>much</em> better way of providing documentation - but <code>sqlite-utils</code> already has <a href="https://sqlite-utils.datasette.io/en/stable/python-api.html" rel="nofollow">detailed narrative prose</a>, so I felt that adding reference documentation powered by docstrings could enhance that project - while also providing better inline document for tools such as Visual Studio Code and Jupyter.</p>
<p><a href="https://user-images.githubusercontent.com/9599/128947691-403e0332-5bab-4851-a9aa-c29e759e944c.png" target="_blank" rel="nofollow"><img width="500" alt="Screenshot of the new rendered documentation" src="https://user-images.githubusercontent.com/9599/128947691-403e0332-5bab-4851-a9aa-c29e759e944c.png" style="max-width:100%;"></a></p>
<h2>
<a id="user-content-setting-up-autodoc" class="anchor" href="#setting-up-autodoc" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Setting up autodoc</h2>
<p>Getting autodoc working in my local documentation environment took a few steps.</p>
<p>First, I added <code>"sphinx.ext.autodoc"</code> to the <code>extensions = []</code> list in my <code>docs/conf.py</code> file.</p>
<p>Then I created a new page in <code>docs/reference.rst</code> and added the following:</p>
<div class="highlight highlight-text-restructuredtext"><pre><span class="pl-mh">===============</span>
 API Reference
<span class="pl-mh">===============</span>

.. <span class="pl-pds">_</span><span class="pl-corl">reference_db_database</span>:

sqlite_utils.db.Database
<span class="pl-mh">========================</span>

.. <span class="pl-c1">autoclass</span>:: <span class="pl-en">sqlite_utils.db.Database</span>
    <span class="pl-ent">:members:</span>
    <span class="pl-ent">:undoc-members:</span></pre></div>
<p>This was enough to start generating documentation for the <code>sqlite_utils.db.Database</code> class.</p>
<p><code>:members:</code> means "show documentation for everything in the class that has a docstring".</p>
<p><code>:undoc-members:</code> means "also include class members that don't have a docstring yet".</p>
<p>I added my own rST headings and reference links - this ensured that the classes would automatically show up in the table of contents for my documentation.</p>
<h2>
<a id="user-content-live-updates-while-editing-docstrings" class="anchor" href="#live-updates-while-editing-docstrings" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Live updates while editing docstrings</h2>
<p>I use <a href="https://pypi.org/project/sphinx-autobuild/" rel="nofollow">sphinx-autobuild</a> to automatically rebuild and reload the browser while I'm editing documentation.</p>
<p>This has <a href="https://github.com/executablebooks/sphinx-autobuild#relevant-sphinx-bugs">a known limitation</a> that it won't pick up on edits made to the <code>*.py</code> files that should cause a reload.</p>
<p>I found a workaround: add <code>-a</code> to disable incremental builds (rebuilding everything) and <code>--watch ../sqlite_utils</code> to trigger a new build when any file in that folder was updated.</p>
<p>I added this to <a href="https://github.com/simonw/sqlite-utils/blob/6155da72c8939b5d9bdacb7853e5e8d1767ce1d5/docs/Makefile">the Makefile</a> is the <code>docs/</code> directory:</p>
<pre><code>livehtml:
    sphinx-autobuild -a -b html "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(0) --watch ../sqlite_utils
</code></pre>
<p>Then running <code>make livehtml</code> started a web server for the documentation that would refresh the browser any time I edited a docstring.</p>
<h2>
<a id="user-content-more-autoclass-options" class="anchor" href="#more-autoclass-options" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>More autoclass options</h2>
<p>I ended up using the following:</p>
<div class="highlight highlight-text-restructuredtext"><pre>.. <span class="pl-c1">autoclass</span>:: <span class="pl-en">sqlite_utils.db.Database</span>
    <span class="pl-ent">:members:</span>
    <span class="pl-ent">:undoc-members:</span>
    <span class="pl-ent">:show-inheritance:</span>
    <span class="pl-ent">:special-members:</span> <span class="pl-corl">__getitem_</span>_
    <span class="pl-ent">:exclude-members:</span> use_counts_table, execute_returning_dicts, resolve_foreign_keys</pre></div>
<p><code>:show-inheritance:</code> adds a note about the inheritance hierarchy for the class.</p>
<p>By default <code>__special__</code> methods like <code>__getitem__</code> are not included. <code>:special-members:</code> fixes this for specified methods.</p>
<p><code>:exclude-members;</code> accepts a comma-separated list of methods that shouldn't be shown in the documentation. I used this to remove some internal methods.</p>
<p>I could have renamed these to <code>_use_counts_table</code> but that would be a breaking API change, even though these methods are deliberately undocumented and shouldn't be used by anyone.</p>
<h2>
<a id="user-content-source-not-alphabetical-order" class="anchor" href="#source-not-alphabetical-order" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Source, not alphabetical order</h2>
<p>The class methods default to being shown in alphabetical order - adding <code> autodoc_member_order = "bysource"</code> to <code>docs/conf.py</code> changed that to source order instead.</p>
<h2>
<a id="user-content-restructuredtext-in-the-docstrings" class="anchor" href="#restructuredtext-in-the-docstrings" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>reStructuredText in the docstrings</h2>
<p>Adding reStructuredText to the docstrings works for formatting the generated API documentation. You can also use <code>:ref:reference-links</code>. Here's an example of a docstring with more complex formatting:</p>
<div class="highlight highlight-source-python"><pre>    <span class="pl-k">def</span> <span class="pl-en">rows_where</span>(
        <span class="pl-s1">self</span>,
        <span class="pl-s1">where</span>: <span class="pl-s1">str</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>,
        <span class="pl-s1">where_args</span>: <span class="pl-v">Optional</span>[<span class="pl-v">Union</span>[<span class="pl-v">Iterable</span>, <span class="pl-s1">dict</span>]] <span class="pl-c1">=</span> <span class="pl-c1">None</span>,
        <span class="pl-s1">order_by</span>: <span class="pl-s1">str</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>,
        <span class="pl-s1">select</span>: <span class="pl-s1">str</span> <span class="pl-c1">=</span> <span class="pl-s">"*"</span>,
        <span class="pl-s1">limit</span>: <span class="pl-s1">int</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>,
        <span class="pl-s1">offset</span>: <span class="pl-s1">int</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>,
    ) <span class="pl-c1">-&gt;</span> <span class="pl-v">Generator</span>[<span class="pl-s1">dict</span>, <span class="pl-c1">None</span>, <span class="pl-c1">None</span>]:
        <span class="pl-s">"""</span>
<span class="pl-s">        Iterate over every row in this table or view that matches the specified where clause.</span>
<span class="pl-s">        Returns each row as a dictionary. See :ref:`python_api_rows` for more details.</span>
<span class="pl-s"></span>
<span class="pl-s">        :param where: SQL where fragment to use, for example ``id &gt; ?``</span>
<span class="pl-s">        :param where_args: Parameters to use with that fragment - an iterable for ``id &gt; ?``</span>
<span class="pl-s">          parameters, or a dictionary for ``id &gt; :id``</span>
<span class="pl-s">        :param order_by: Column or fragment of SQL to order by</span>
<span class="pl-s">        :param select: Comma-separated list of columns to select - defaults to ``*``</span>
<span class="pl-s">        :param limit: Integer number of rows to limit to</span>
<span class="pl-s">        :param offset: Integer for SQL offset</span>
<span class="pl-s">        """</span>
        <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">self</span>.<span class="pl-en">exists</span>():
            <span class="pl-c"># ...</span></pre></div>
<p>If you add <code>autodoc_typehints = "description"</code> to the <code>conf.py</code> configuration the <code>:param name: Description</code> lines will be shown in a list of parameters that also includes the type hints:</p>
<p><a href="https://user-images.githubusercontent.com/9599/157924734-dcf56d26-69dd-4201-b81a-988eb4414a8b.png" target="_blank" rel="nofollow"><img width="674" alt="Screenshot showing the type hints in the list of parameters" src="https://user-images.githubusercontent.com/9599/157924734-dcf56d26-69dd-4201-b81a-988eb4414a8b.png" style="max-width:100%;"></a></p>
<p>I learned about this in <a href="https://github.com/simonw/sqlite-utils/issues/413">sqlite-utils/issues/413</a> while trying to improve the display of complex type annotations.</p>
<h2>
<a id="user-content-getting-it-working-on-read-the-docs" class="anchor" href="#getting-it-working-on-read-the-docs" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Getting it working on Read The Docs</h2>
<p>Read The Docs has the option to deploy branches - you can set that up in  the "Versions" tab. I was working in an <code>autodoc</code> branch - I set that branch to be "Active" and "Hidden", so it would be built by Read The Docs but not linked to from anywhere.</p>
<p>My first attempt failed to render the generated documentation, because Read The Docs did not know where to import my code from.</p>
<p>I solved that by adding a <code>.readthedocs.yaml</code> configuration file to my repository containing the following:</p>
<div class="highlight highlight-source-yaml"><pre>
<span class="pl-ent">version</span>: <span class="pl-c1">2</span>

<span class="pl-ent">sphinx</span>:
  <span class="pl-ent">configuration</span>: <span class="pl-s">docs/conf.py</span>

<span class="pl-ent">python</span>:
  <span class="pl-ent">version</span>: <span class="pl-s"><span class="pl-pds">"</span>3.8<span class="pl-pds">"</span></span>
  <span class="pl-ent">install</span>:
  - <span class="pl-ent">method</span>: <span class="pl-s">pip</span>
    <span class="pl-ent">path</span>: <span class="pl-s">.</span>
    <span class="pl-ent">extra_requirements</span>:
    - <span class="pl-s">docs</span></pre></div>
<p>The key thing here is that <code>install</code> block, which tells Read The Docs to run the equivalent of this before building the documentation:</p>
<pre><code>pip install -e '.[docs]'
</code></pre>
<p>For my repository that installs the <code>docs</code> optional dependencies from the <code>setup.py</code>, but also installs the <code>sqlite_utils</code> Python package itself. That way Sphinx can find and import <code>sqlite_utils.db.Database</code> in order to generate the documentation.</p>
<h2>
<a id="user-content-the-end-result" class="anchor" href="#the-end-result" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>The end result</h2>
<p>The new page of documentation is now live at <a href="https://sqlite-utils.datasette.io/en/latest/reference.html" rel="nofollow">en/latest/reference.html</a>. The pull request in which I figured this all out is <a href="https://github.com/simonw/sqlite-utils/pull/312">sqlite-utils/pull/312</a>.</p>


path: sqlite_build-specific-sqlite-pysqlite-macos.md
topic: sqlite
title: Building a specific version of SQLite with pysqlite on macOS/Linux
url: https://github.com/simonw/til/blob/main/sqlite/build-specific-sqlite-pysqlite-macos.md
created: 2021-08-14T14:29:40-07:00
created_utc: 2021-08-14T21:29:40+00:00
updated: 2021-08-15T11:34:42-07:00
updated_utc: 2021-08-15T18:34:42+00:00
shot_hash: e559fdfff99c09300ade271647b4aaf0
slug: build-specific-sqlite-pysqlite-macos
body:
I wanted the ability to test my Python software against specific version of SQLite on macOS. I found a way to do that using [pysqlite3](https://github.com/coleifer/pysqlite3).

First, clone the GitHub mirror of SQLite (so I don't have to learn how to use Fossil):

    cd /tmp
    git clone https://github.com/sqlite/sqlite

Check out the version tag you want to try:

    cd /tmp/sqlite
    git checkout version-3.17.0

The SQLite [build docs](https://github.com/sqlite/sqlite/tree/version-3.36.0#compiling-for-unix-like-systems) suggest using a `bld` directory like this:

    mkdir /tmp/bld
    cd /tmp/bld
    ../sqlite/configure
    make sqlite3.c

This will have constructed the amalgamation source needed by `pysqlite3`.

Now build that:

    cd /tmp
    git clone https://github.com/coleifer/pysqlite3
    cd pysqlite3
    cp /tmp/bld/sqlite3.c .
    cp /tmp/bld/sqlite3.h .
    python3 setup.py build_static build

The end result sits in  a `pysqlite3` folder in, on my machine, `/tmp/pysqlite3/build/lib.macosx-10.15-x86_64-3.9` - test it like this:

    cd /tmp/pysqlite3/build/lib.macosx-10.15-x86_64-3.9
    python3
    Python 3.9.6 (default, Jun 29 2021, 06:20:32) 
    [Clang 12.0.0 (clang-1200.0.32.29)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import pysqlite3
    >>> pysqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0]
    '3.17.0'

## Turn that into a wheel

Running the following produces a wheel for the current platform that bundles the compiled `.so` file:

    python3 setup.py bdist_wheel

On my machine that created `/tmp/pysqlite3/dist/pysqlite3-0.4.6-cp39-cp39-macosx_10_15_x86_64.whl`.

Having created that file, running this in any virtual enviroment installed my custom build of `pysqlite3` with the correct SQLite version:

    pip install /tmp/pysqlite3/dist/pysqlite3-0.4.6-cp39-cp39-macosx_10_15_x86_64.whl

This exact same process works on Linux too (tested inside a default GitHub Actions Linux worker).

html:
<p>I wanted the ability to test my Python software against specific version of SQLite on macOS. I found a way to do that using <a href="https://github.com/coleifer/pysqlite3">pysqlite3</a>.</p>
<p>First, clone the GitHub mirror of SQLite (so I don't have to learn how to use Fossil):</p>
<pre><code>cd /tmp
git clone https://github.com/sqlite/sqlite
</code></pre>
<p>Check out the version tag you want to try:</p>
<pre><code>cd /tmp/sqlite
git checkout version-3.17.0
</code></pre>
<p>The SQLite <a href="https://github.com/sqlite/sqlite/tree/version-3.36.0#compiling-for-unix-like-systems">build docs</a> suggest using a <code>bld</code> directory like this:</p>
<pre><code>mkdir /tmp/bld
cd /tmp/bld
../sqlite/configure
make sqlite3.c
</code></pre>
<p>This will have constructed the amalgamation source needed by <code>pysqlite3</code>.</p>
<p>Now build that:</p>
<pre><code>cd /tmp
git clone https://github.com/coleifer/pysqlite3
cd pysqlite3
cp /tmp/bld/sqlite3.c .
cp /tmp/bld/sqlite3.h .
python3 setup.py build_static build
</code></pre>
<p>The end result sits in  a <code>pysqlite3</code> folder in, on my machine, <code>/tmp/pysqlite3/build/lib.macosx-10.15-x86_64-3.9</code> - test it like this:</p>
<pre><code>cd /tmp/pysqlite3/build/lib.macosx-10.15-x86_64-3.9
python3
Python 3.9.6 (default, Jun 29 2021, 06:20:32) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import pysqlite3
&gt;&gt;&gt; pysqlite3.connect(":memory:").execute("select sqlite_version()").fetchone()[0]
'3.17.0'
</code></pre>
<h2>
<a id="user-content-turn-that-into-a-wheel" class="anchor" href="#turn-that-into-a-wheel" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Turn that into a wheel</h2>
<p>Running the following produces a wheel for the current platform that bundles the compiled <code>.so</code> file:</p>
<pre><code>python3 setup.py bdist_wheel
</code></pre>
<p>On my machine that created <code>/tmp/pysqlite3/dist/pysqlite3-0.4.6-cp39-cp39-macosx_10_15_x86_64.whl</code>.</p>
<p>Having created that file, running this in any virtual enviroment installed my custom build of <code>pysqlite3</code> with the correct SQLite version:</p>
<pre><code>pip install /tmp/pysqlite3/dist/pysqlite3-0.4.6-cp39-cp39-macosx_10_15_x86_64.whl
</code></pre>
<p>This exact same process works on Linux too (tested inside a default GitHub Actions Linux worker).</p>


path: sqlite_track-timestamped-changes-to-a-table.md
topic: sqlite
title: Track timestamped changes to a SQLite table using triggers
url: https://github.com/simonw/til/blob/main/sqlite/track-timestamped-changes-to-a-table.md
created: 2021-08-19T11:55:56-07:00
created_utc: 2021-08-19T18:55:56+00:00
updated: 2021-08-19T12:08:20-07:00
updated_utc: 2021-08-19T19:08:20+00:00
shot_hash: 066645facef94d67ef71c44593afc15e
slug: track-timestamped-changes-to-a-table
body:
This is more of a "today I figured out" than a TIL.

I have an idea to implement the [Denormalized Query Engine design pattern](https://2017.djangocon.us/talks/the-denormalized-query-engine-design-pattern/) using SQLite triggers.

My goal is that any time an insert or update occurs on a table, a matching row will be written to a `_dqe_changes` table recording the rowid of the affected row and the unix timestamp (in milliseconds) that the change occurred.

If a row is deleted, a record will be written into tha table with a `1` in the `deleted` column.

This will allow subscribing scripts to poll that table for changes made since their last poll, based on the milliseconds timestamp field. They can then apply those changes to external data stores, such as an Elasticsearch index.

Here's the recipe I came up with:

```sql
-- This table exists just for table -> integer lookups,
-- to save space in the _dqe_changes table
CREATE TABLE _dqe_tables(
  id integer primary key,
  [table] text unique
);
-- This table records the timestamped changes
CREATE TABLE _dqe_changes(
  table_id integer,
  rowid integer,
  deleted integer, -- treated as a null or 1 boolean
  updated_ms integer,
  PRIMARY KEY (table_id, rowid),
  FOREIGN KEY (table_id) REFERENCES _dqe_tables(id)
);
-- Create an index for polling against
CREATE INDEX _dqe_changes_updated_ms ON _dqe_changes(updated_ms);

-- An example table:
CREATE TABLE foo (name text);

-- Each table needs to have these triggers created on it:
CREATE TRIGGER IF NOT EXISTS [foo_dqe_insert] AFTER INSERT ON [foo]
BEGIN
  INSERT OR IGNORE INTO _dqe_tables([table]) VALUES ('foo');
  INSERT OR REPLACE INTO _dqe_changes(table_id, rowid, updated_ms)
     VALUES (
        (select id from _dqe_tables where [table] = 'foo'),
        new.rowid,
        -- This is a recipe for timestamp in milliseconds, from
        -- https://stackoverflow.com/a/56895050/6083 
        strftime('%s','now') || substr(strftime('%f','now'),4)
    );
END;
CREATE TRIGGER IF NOT EXISTS [foo_dqe_update] AFTER UPDATE ON [foo]
BEGIN
  INSERT OR IGNORE INTO _dqe_tables([table]) VALUES ('foo');
  INSERT OR REPLACE INTO _dqe_changes(table_id, rowid, updated_ms)
     VALUES (
        (select id from _dqe_tables where [table] = 'foo'),
        new.rowid,
        strftime('%s','now') || substr(strftime('%f','now'),4)
    );
END;
CREATE TRIGGER IF NOT EXISTS [foo_dqe_delete] AFTER DELETE ON [foo]
BEGIN
  INSERT OR IGNORE INTO _dqe_tables([table]) VALUES ('foo');
  INSERT OR REPLACE INTO _dqe_changes(table_id, rowid, deleted, updated_ms)
     VALUES (
        (select id from _dqe_tables where [table] = 'foo'),
        old.rowid,
        1,
        strftime('%s','now') || substr(strftime('%f','now'),4)
    );
END;

INSERT INTO foo VALUES ('hello');
INSERT INTO foo VALUES ('hello2');
INSERT INTO foo VALUES ('hello3');
INSERT INTO foo VALUES ('hello4');
```
To test this I ran `sqlite3` (with no arguments, which provides an in-memory database to play with), pasted in the above and then ran this:
```
sqlite> .headers on
sqlite> select rowid, * from foo;
rowid|name
1|hello
2|hello2
3|hello3
4|hello4
sqlite> select * from _dqe_changes;
table_id|rowid|deleted|updated_ms
1|1||1629399072866
1|2||1629399072867
1|3||1629399072867
1|4||1629399072974
sqlite> select * from _dqe_tables;
id|table
1|foo
sqlite> delete from foo where name = 'hello3';
sqlite> select * from _dqe_changes;
table_id|rowid|deleted|updated_ms
1|1||1629399072866
1|2||1629399072867
1|4||1629399072974
1|3|1|1629399103169
sqlite> update foo set name = 'hello6' where name = 'hello';
sqlite> select * from _dqe_changes;
table_id|rowid|deleted|updated_ms
1|2||1629399072867
1|4||1629399072974
1|3|1|1629399103169
1|1||1629399115886
sqlite> select rowid, * from foo;
rowid|name
1|hello6
2|hello2
4|hello4
```
There's one catch with the above recipe: if you delete ALL of the rows from a table (`delete * from foo`) SQLite defaults to reusing rowids in that table, starting again from 1. This means that your accompanying records in the `_dqe_changes` table will have stale rowids, which could lead to surprising behaviour.

If tables have a `id integer primary key` column SQLite does NOT reuse rowids, [as explained here](https://www.sqlite.org/autoinc.html).

## Initializing for a new table

The above recipe expects the triggers to be installed on the table before any rows are written to it.

If applying this recipe to an already-populated table, the following SQL should initialize `_dqe_changes` with a row for every row:

```sql
INSERT OR IGNORE INTO _dqe_tables([table]) VALUES ('foo');
INSERT OR REPLACE INTO _dqe_changes(table_id, rowid, deleted, updated_ms)
     with table_id as (select id from _dqe_tables where [table] = 'foo')
     select
        table_id.id,
        foo.rowid,
        null as deleted,
        strftime('%s','now') || substr(strftime('%f','now'),4) as updated_ms
    from foo, table_id;
```

html:
<p>This is more of a "today I figured out" than a TIL.</p>
<p>I have an idea to implement the <a href="https://2017.djangocon.us/talks/the-denormalized-query-engine-design-pattern/" rel="nofollow">Denormalized Query Engine design pattern</a> using SQLite triggers.</p>
<p>My goal is that any time an insert or update occurs on a table, a matching row will be written to a <code>_dqe_changes</code> table recording the rowid of the affected row and the unix timestamp (in milliseconds) that the change occurred.</p>
<p>If a row is deleted, a record will be written into tha table with a <code>1</code> in the <code>deleted</code> column.</p>
<p>This will allow subscribing scripts to poll that table for changes made since their last poll, based on the milliseconds timestamp field. They can then apply those changes to external data stores, such as an Elasticsearch index.</p>
<p>Here's the recipe I came up with:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-c"><span class="pl-c">--</span> This table exists just for table -&gt; integer lookups,</span>
<span class="pl-c"><span class="pl-c">--</span> to save space in the _dqe_changes table</span>
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> <span class="pl-en">_dqe_tables</span>(
  id <span class="pl-k">integer</span> <span class="pl-k">primary key</span>,
  [table] <span class="pl-k">text</span> unique
);
<span class="pl-c"><span class="pl-c">--</span> This table records the timestamped changes</span>
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> <span class="pl-en">_dqe_changes</span>(
  table_id <span class="pl-k">integer</span>,
  rowid <span class="pl-k">integer</span>,
  deleted <span class="pl-k">integer</span>, <span class="pl-c"><span class="pl-c">--</span> treated as a null or 1 boolean</span>
  updated_ms <span class="pl-k">integer</span>,
  <span class="pl-k">PRIMARY KEY</span> (table_id, rowid),
  <span class="pl-k">FOREIGN KEY</span> (table_id) <span class="pl-k">REFERENCES</span> _dqe_tables(id)
);
<span class="pl-c"><span class="pl-c">--</span> Create an index for polling against</span>
<span class="pl-k">CREATE</span> <span class="pl-k">INDEX</span> <span class="pl-en">_dqe_changes_updated_ms</span> <span class="pl-k">ON</span> _dqe_changes(updated_ms);

<span class="pl-c"><span class="pl-c">--</span> An example table:</span>
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> <span class="pl-en">foo</span> (name <span class="pl-k">text</span>);

<span class="pl-c"><span class="pl-c">--</span> Each table needs to have these triggers created on it:</span>
<span class="pl-k">CREATE</span> <span class="pl-k">TRIGGER</span> <span class="pl-en">IF</span> NOT EXISTS [foo_dqe_insert] AFTER INSERT <span class="pl-k">ON</span> [foo]
<span class="pl-k">BEGIN</span>
  INSERT <span class="pl-k">OR</span> IGNORE INTO _dqe_tables([table]) <span class="pl-k">VALUES</span> (<span class="pl-s"><span class="pl-pds">'</span>foo<span class="pl-pds">'</span></span>);
  INSERT <span class="pl-k">OR</span> REPLACE INTO _dqe_changes(table_id, rowid, updated_ms)
     <span class="pl-k">VALUES</span> (
        (<span class="pl-k">select</span> id <span class="pl-k">from</span> _dqe_tables <span class="pl-k">where</span> [table] <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">'</span>foo<span class="pl-pds">'</span></span>),
        <span class="pl-c1">new</span>.<span class="pl-c1">rowid</span>,
        <span class="pl-c"><span class="pl-c">--</span> This is a recipe for timestamp in milliseconds, from</span>
        <span class="pl-c"><span class="pl-c">--</span> https://stackoverflow.com/a/56895050/6083 </span>
        strftime(<span class="pl-s"><span class="pl-pds">'</span>%s<span class="pl-pds">'</span></span>,<span class="pl-s"><span class="pl-pds">'</span>now<span class="pl-pds">'</span></span>) <span class="pl-k">||</span> substr(strftime(<span class="pl-s"><span class="pl-pds">'</span>%f<span class="pl-pds">'</span></span>,<span class="pl-s"><span class="pl-pds">'</span>now<span class="pl-pds">'</span></span>),<span class="pl-c1">4</span>)
    );
END;
<span class="pl-k">CREATE</span> <span class="pl-k">TRIGGER</span> <span class="pl-en">IF</span> NOT EXISTS [foo_dqe_update] AFTER <span class="pl-k">UPDATE</span> <span class="pl-k">ON</span> [foo]
<span class="pl-k">BEGIN</span>
  INSERT <span class="pl-k">OR</span> IGNORE INTO _dqe_tables([table]) <span class="pl-k">VALUES</span> (<span class="pl-s"><span class="pl-pds">'</span>foo<span class="pl-pds">'</span></span>);
  INSERT <span class="pl-k">OR</span> REPLACE INTO _dqe_changes(table_id, rowid, updated_ms)
     <span class="pl-k">VALUES</span> (
        (<span class="pl-k">select</span> id <span class="pl-k">from</span> _dqe_tables <span class="pl-k">where</span> [table] <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">'</span>foo<span class="pl-pds">'</span></span>),
        <span class="pl-c1">new</span>.<span class="pl-c1">rowid</span>,
        strftime(<span class="pl-s"><span class="pl-pds">'</span>%s<span class="pl-pds">'</span></span>,<span class="pl-s"><span class="pl-pds">'</span>now<span class="pl-pds">'</span></span>) <span class="pl-k">||</span> substr(strftime(<span class="pl-s"><span class="pl-pds">'</span>%f<span class="pl-pds">'</span></span>,<span class="pl-s"><span class="pl-pds">'</span>now<span class="pl-pds">'</span></span>),<span class="pl-c1">4</span>)
    );
END;
<span class="pl-k">CREATE</span> <span class="pl-k">TRIGGER</span> <span class="pl-en">IF</span> NOT EXISTS [foo_dqe_delete] AFTER <span class="pl-k">DELETE</span> <span class="pl-k">ON</span> [foo]
<span class="pl-k">BEGIN</span>
  INSERT <span class="pl-k">OR</span> IGNORE INTO _dqe_tables([table]) <span class="pl-k">VALUES</span> (<span class="pl-s"><span class="pl-pds">'</span>foo<span class="pl-pds">'</span></span>);
  INSERT <span class="pl-k">OR</span> REPLACE INTO _dqe_changes(table_id, rowid, deleted, updated_ms)
     <span class="pl-k">VALUES</span> (
        (<span class="pl-k">select</span> id <span class="pl-k">from</span> _dqe_tables <span class="pl-k">where</span> [table] <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">'</span>foo<span class="pl-pds">'</span></span>),
        <span class="pl-c1">old</span>.<span class="pl-c1">rowid</span>,
        <span class="pl-c1">1</span>,
        strftime(<span class="pl-s"><span class="pl-pds">'</span>%s<span class="pl-pds">'</span></span>,<span class="pl-s"><span class="pl-pds">'</span>now<span class="pl-pds">'</span></span>) <span class="pl-k">||</span> substr(strftime(<span class="pl-s"><span class="pl-pds">'</span>%f<span class="pl-pds">'</span></span>,<span class="pl-s"><span class="pl-pds">'</span>now<span class="pl-pds">'</span></span>),<span class="pl-c1">4</span>)
    );
END;

<span class="pl-k">INSERT INTO</span> foo <span class="pl-k">VALUES</span> (<span class="pl-s"><span class="pl-pds">'</span>hello<span class="pl-pds">'</span></span>);
<span class="pl-k">INSERT INTO</span> foo <span class="pl-k">VALUES</span> (<span class="pl-s"><span class="pl-pds">'</span>hello2<span class="pl-pds">'</span></span>);
<span class="pl-k">INSERT INTO</span> foo <span class="pl-k">VALUES</span> (<span class="pl-s"><span class="pl-pds">'</span>hello3<span class="pl-pds">'</span></span>);
<span class="pl-k">INSERT INTO</span> foo <span class="pl-k">VALUES</span> (<span class="pl-s"><span class="pl-pds">'</span>hello4<span class="pl-pds">'</span></span>);</pre></div>
<p>To test this I ran <code>sqlite3</code> (with no arguments, which provides an in-memory database to play with), pasted in the above and then ran this:</p>
<pre><code>sqlite&gt; .headers on
sqlite&gt; select rowid, * from foo;
rowid|name
1|hello
2|hello2
3|hello3
4|hello4
sqlite&gt; select * from _dqe_changes;
table_id|rowid|deleted|updated_ms
1|1||1629399072866
1|2||1629399072867
1|3||1629399072867
1|4||1629399072974
sqlite&gt; select * from _dqe_tables;
id|table
1|foo
sqlite&gt; delete from foo where name = 'hello3';
sqlite&gt; select * from _dqe_changes;
table_id|rowid|deleted|updated_ms
1|1||1629399072866
1|2||1629399072867
1|4||1629399072974
1|3|1|1629399103169
sqlite&gt; update foo set name = 'hello6' where name = 'hello';
sqlite&gt; select * from _dqe_changes;
table_id|rowid|deleted|updated_ms
1|2||1629399072867
1|4||1629399072974
1|3|1|1629399103169
1|1||1629399115886
sqlite&gt; select rowid, * from foo;
rowid|name
1|hello6
2|hello2
4|hello4
</code></pre>
<p>There's one catch with the above recipe: if you delete ALL of the rows from a table (<code>delete * from foo</code>) SQLite defaults to reusing rowids in that table, starting again from 1. This means that your accompanying records in the <code>_dqe_changes</code> table will have stale rowids, which could lead to surprising behaviour.</p>
<p>If tables have a <code>id integer primary key</code> column SQLite does NOT reuse rowids, <a href="https://www.sqlite.org/autoinc.html" rel="nofollow">as explained here</a>.</p>
<h2>
<a id="user-content-initializing-for-a-new-table" class="anchor" href="#initializing-for-a-new-table" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Initializing for a new table</h2>
<p>The above recipe expects the triggers to be installed on the table before any rows are written to it.</p>
<p>If applying this recipe to an already-populated table, the following SQL should initialize <code>_dqe_changes</code> with a row for every row:</p>
<div class="highlight highlight-source-sql"><pre>INSERT <span class="pl-k">OR</span> IGNORE INTO _dqe_tables([table]) <span class="pl-k">VALUES</span> (<span class="pl-s"><span class="pl-pds">'</span>foo<span class="pl-pds">'</span></span>);
INSERT <span class="pl-k">OR</span> REPLACE INTO _dqe_changes(table_id, rowid, deleted, updated_ms)
     with table_id <span class="pl-k">as</span> (<span class="pl-k">select</span> id <span class="pl-k">from</span> _dqe_tables <span class="pl-k">where</span> [table] <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">'</span>foo<span class="pl-pds">'</span></span>)
     <span class="pl-k">select</span>
        <span class="pl-c1">table_id</span>.<span class="pl-c1">id</span>,
        <span class="pl-c1">foo</span>.<span class="pl-c1">rowid</span>,
        <span class="pl-k">null</span> <span class="pl-k">as</span> deleted,
        strftime(<span class="pl-s"><span class="pl-pds">'</span>%s<span class="pl-pds">'</span></span>,<span class="pl-s"><span class="pl-pds">'</span>now<span class="pl-pds">'</span></span>) <span class="pl-k">||</span> substr(strftime(<span class="pl-s"><span class="pl-pds">'</span>%f<span class="pl-pds">'</span></span>,<span class="pl-s"><span class="pl-pds">'</span>now<span class="pl-pds">'</span></span>),<span class="pl-c1">4</span>) <span class="pl-k">as</span> updated_ms
    <span class="pl-k">from</span> foo, table_id;</pre></div>


path: observable-plot_histogram-with-tooltips.md
topic: observable-plot
title: Histogram with tooltips in Observable Plot
url: https://github.com/simonw/til/blob/main/observable-plot/histogram-with-tooltips.md
created: 2021-08-21T15:47:27-07:00
created_utc: 2021-08-21T22:47:27+00:00
updated: 2022-01-27T14:40:31-08:00
updated_utc: 2022-01-27T22:40:31+00:00
shot_hash: a009be70a435385014e51eeaab5e4ab7
slug: histogram-with-tooltips
body:
Given an array of datetime objects, I wanted to plot a histogram. But I wanted to automatically pick a bucket size for that histogram that resulted in something interesting, no matter what range of time the individual points covered.

I figured out that [d3](https://d3js.org/) has a mechanism for this: `d3.bin().thresholds()` ([docs here](https://github.com/d3/d3-array#bin-thresholds)). This defaults to using [Sturges' formula](https://en.wikipedia.org/wiki/Histogram#Sturges'_formula) but has various other options.

This [d3 Histogram](https://observablehq.com/@d3/histogram) notebook helped me figure this out:

![Animation showing what happens when I pass different values to the .thresholds() method](https://user-images.githubusercontent.com/9599/130336516-608f8d37-70e3-4506-8b16-f8ef00f9ec85.gif)

## Implementing this in Observable Plot

The higher level [Observable Plot](https://observablehq.com/@observablehq/plot) library provides binning. Here's the recipe I figured out for generating a histogram with that, in my [Histogram of my tweets over time](https://observablehq.com/@simonw/my-tweets-over-time) notebook:

```javascript
Plot.plot({
  y: {
    grid: true
  },
  marks: [Plot.rectY(tweets, Plot.binX({ y: "count" }, { x: "created_at" }))]
})
```

<img width="669" alt="My histogram of tweets" src="https://user-images.githubusercontent.com/9599/130336556-6f91a1e5-8efe-448d-83f3-a9748cb11aac.png">

It turns out that while d3 uses Sturges' formula Observable Plot instead uses Scott's rule:

> D3 chose Sturges’ formula because it was the popular choice (at the time) but Observable Plot defaults to Scott’s rule, with a max of 200 bins, which tends to perform better. Related: <a href="https://robjhyndman.com/papers/sturges.pdf">https://robjhyndman.com/papers/sturges.pdf</a></p>
> 
> [Mike Bostock](https://twitter.com/mbostock/status/1429281697854464002) - @mbostock

## Adding tooltips

I wanted to add tooltips to the above chart. This was the hardest part to figure out - it turns out that second argument to `.binX()` can take a `title` option, which can be a function that accepts a `bin` array of items and returns a title.

**UPDATE 27th January 2022:** The below code example no longer works - the `bin` argument passed to the `title:` function used to be a list of items in that bin, but now appears to just be a single item.

This is what I ended up using:

```javascript
Plot.plot({
  y: {
    grid: true
  },
  marks: [
    Plot.rectY(
      tweets,
      Plot.binX(
        { y: "count" },
        {
          x: "created_at",
          title: bin => {
            let sorted = [...Array.from(bin).map(t => t.created_at)].sort();
            let min = sorted[0];
            let max = sorted.slice(-1)[0];
            let count = sorted.length;
            return `${sorted.length} item${
              sorted.length == 0 ? '' : 's'
            }\nFrom: ${min}\nTo: ${max}`;
          }
        }
      )
    )
  ]
})
```
- `[...bin.map(t => t.created_at)].sort()` is a recipe for creating a sorted copy of an array of values
- `max = sorted.slice(-1)[0]` gets the last item in that array

Then I compose and return the string.

<img width="664" alt="Screnshoot showing the generated tooltip" src="https://user-images.githubusercontent.com/9599/130336609-c2ad97ff-93f3-49b2-a420-edb9ff558ae6.png">

Here's [my Twitter thread](https://twitter.com/simonw/status/1429194217482424324) where I figured this out.

html:
<p>Given an array of datetime objects, I wanted to plot a histogram. But I wanted to automatically pick a bucket size for that histogram that resulted in something interesting, no matter what range of time the individual points covered.</p>
<p>I figured out that <a href="https://d3js.org/" rel="nofollow">d3</a> has a mechanism for this: <code>d3.bin().thresholds()</code> (<a href="https://github.com/d3/d3-array#bin-thresholds">docs here</a>). This defaults to using <a href="https://en.wikipedia.org/wiki/Histogram#Sturges'_formula" rel="nofollow">Sturges' formula</a> but has various other options.</p>
<p>This <a href="https://observablehq.com/@d3/histogram" rel="nofollow">d3 Histogram</a> notebook helped me figure this out:</p>
<p><a href="https://user-images.githubusercontent.com/9599/130336516-608f8d37-70e3-4506-8b16-f8ef00f9ec85.gif" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/130336516-608f8d37-70e3-4506-8b16-f8ef00f9ec85.gif" alt="Animation showing what happens when I pass different values to the .thresholds() method" style="max-width:100%;"></a></p>
<h2>
<a id="user-content-implementing-this-in-observable-plot" class="anchor" href="#implementing-this-in-observable-plot" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Implementing this in Observable Plot</h2>
<p>The higher level <a href="https://observablehq.com/@observablehq/plot" rel="nofollow">Observable Plot</a> library provides binning. Here's the recipe I figured out for generating a histogram with that, in my <a href="https://observablehq.com/@simonw/my-tweets-over-time" rel="nofollow">Histogram of my tweets over time</a> notebook:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-v">Plot</span><span class="pl-kos">.</span><span class="pl-en">plot</span><span class="pl-kos">(</span><span class="pl-kos">{</span>
  <span class="pl-c1">y</span>: <span class="pl-kos">{</span>
    <span class="pl-c1">grid</span>: <span class="pl-c1">true</span>
  <span class="pl-kos">}</span><span class="pl-kos">,</span>
  <span class="pl-c1">marks</span>: <span class="pl-kos">[</span><span class="pl-v">Plot</span><span class="pl-kos">.</span><span class="pl-en">rectY</span><span class="pl-kos">(</span><span class="pl-s1">tweets</span><span class="pl-kos">,</span> <span class="pl-v">Plot</span><span class="pl-kos">.</span><span class="pl-en">binX</span><span class="pl-kos">(</span><span class="pl-kos">{</span> <span class="pl-c1">y</span>: <span class="pl-s">"count"</span> <span class="pl-kos">}</span><span class="pl-kos">,</span> <span class="pl-kos">{</span> <span class="pl-c1">x</span>: <span class="pl-s">"created_at"</span> <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">]</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span></pre></div>
<p><a href="https://user-images.githubusercontent.com/9599/130336556-6f91a1e5-8efe-448d-83f3-a9748cb11aac.png" target="_blank" rel="nofollow"><img width="669" alt="My histogram of tweets" src="https://user-images.githubusercontent.com/9599/130336556-6f91a1e5-8efe-448d-83f3-a9748cb11aac.png" style="max-width:100%;"></a></p>
<p>It turns out that while d3 uses Sturges' formula Observable Plot instead uses Scott's rule:</p>
<blockquote>
<p>D3 chose Sturges’ formula because it was the popular choice (at the time) but Observable Plot defaults to Scott’s rule, with a max of 200 bins, which tends to perform better. Related: <a href="https://robjhyndman.com/papers/sturges.pdf" rel="nofollow"></a><a href="https://robjhyndman.com/papers/sturges.pdf" rel="nofollow">https://robjhyndman.com/papers/sturges.pdf</a></p>
<p><a href="https://twitter.com/mbostock/status/1429281697854464002" rel="nofollow">Mike Bostock</a> - @mbostock</p>
</blockquote>
<h2>
<a id="user-content-adding-tooltips" class="anchor" href="#adding-tooltips" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Adding tooltips</h2>
<p>I wanted to add tooltips to the above chart. This was the hardest part to figure out - it turns out that second argument to <code>.binX()</code> can take a <code>title</code> option, which can be a function that accepts a <code>bin</code> array of items and returns a title.</p>
<p><strong>UPDATE 27th January 2022:</strong> The below code example no longer works - the <code>bin</code> argument passed to the <code>title:</code> function used to be a list of items in that bin, but now appears to just be a single item.</p>
<p>This is what I ended up using:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-v">Plot</span><span class="pl-kos">.</span><span class="pl-en">plot</span><span class="pl-kos">(</span><span class="pl-kos">{</span>
  <span class="pl-c1">y</span>: <span class="pl-kos">{</span>
    <span class="pl-c1">grid</span>: <span class="pl-c1">true</span>
  <span class="pl-kos">}</span><span class="pl-kos">,</span>
  <span class="pl-c1">marks</span>: <span class="pl-kos">[</span>
    <span class="pl-v">Plot</span><span class="pl-kos">.</span><span class="pl-en">rectY</span><span class="pl-kos">(</span>
      <span class="pl-s1">tweets</span><span class="pl-kos">,</span>
      <span class="pl-v">Plot</span><span class="pl-kos">.</span><span class="pl-en">binX</span><span class="pl-kos">(</span>
        <span class="pl-kos">{</span> <span class="pl-c1">y</span>: <span class="pl-s">"count"</span> <span class="pl-kos">}</span><span class="pl-kos">,</span>
        <span class="pl-kos">{</span>
          <span class="pl-c1">x</span>: <span class="pl-s">"created_at"</span><span class="pl-kos">,</span>
          <span class="pl-en">title</span>: <span class="pl-s1">bin</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
            <span class="pl-k">let</span> <span class="pl-s1">sorted</span> <span class="pl-c1">=</span> <span class="pl-kos">[</span>...<span class="pl-v">Array</span><span class="pl-kos">.</span><span class="pl-en">from</span><span class="pl-kos">(</span><span class="pl-s1">bin</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">map</span><span class="pl-kos">(</span><span class="pl-s1">t</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">t</span><span class="pl-kos">.</span><span class="pl-c1">created_at</span><span class="pl-kos">)</span><span class="pl-kos">]</span><span class="pl-kos">.</span><span class="pl-en">sort</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
            <span class="pl-k">let</span> <span class="pl-s1">min</span> <span class="pl-c1">=</span> <span class="pl-s1">sorted</span><span class="pl-kos">[</span><span class="pl-c1">0</span><span class="pl-kos">]</span><span class="pl-kos">;</span>
            <span class="pl-k">let</span> <span class="pl-s1">max</span> <span class="pl-c1">=</span> <span class="pl-s1">sorted</span><span class="pl-kos">.</span><span class="pl-en">slice</span><span class="pl-kos">(</span><span class="pl-c1">-</span><span class="pl-c1">1</span><span class="pl-kos">)</span><span class="pl-kos">[</span><span class="pl-c1">0</span><span class="pl-kos">]</span><span class="pl-kos">;</span>
            <span class="pl-k">let</span> <span class="pl-s1">count</span> <span class="pl-c1">=</span> <span class="pl-s1">sorted</span><span class="pl-kos">.</span><span class="pl-c1">length</span><span class="pl-kos">;</span>
            <span class="pl-k">return</span> <span class="pl-s">`<span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">sorted</span><span class="pl-kos">.</span><span class="pl-c1">length</span><span class="pl-kos">}</span></span> item<span class="pl-s1"><span class="pl-kos">${</span></span></span>
<span class="pl-s"><span class="pl-s1">              <span class="pl-s1">sorted</span><span class="pl-kos">.</span><span class="pl-c1">length</span> <span class="pl-c1">==</span> <span class="pl-c1">0</span> ? <span class="pl-s">''</span> : <span class="pl-s">'s'</span></span></span>
<span class="pl-s"><span class="pl-s1">            <span class="pl-kos">}</span></span>\nFrom: <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">min</span><span class="pl-kos">}</span></span>\nTo: <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">max</span><span class="pl-kos">}</span></span>`</span><span class="pl-kos">;</span>
          <span class="pl-kos">}</span>
        <span class="pl-kos">}</span>
      <span class="pl-kos">)</span>
    <span class="pl-kos">)</span>
  <span class="pl-kos">]</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span></pre></div>
<ul>
<li>
<code>[...bin.map(t =&gt; t.created_at)].sort()</code> is a recipe for creating a sorted copy of an array of values</li>
<li>
<code>max = sorted.slice(-1)[0]</code> gets the last item in that array</li>
</ul>
<p>Then I compose and return the string.</p>
<p><a href="https://user-images.githubusercontent.com/9599/130336609-c2ad97ff-93f3-49b2-a420-edb9ff558ae6.png" target="_blank" rel="nofollow"><img width="664" alt="Screnshoot showing the generated tooltip" src="https://user-images.githubusercontent.com/9599/130336609-c2ad97ff-93f3-49b2-a420-edb9ff558ae6.png" style="max-width:100%;"></a></p>
<p>Here's <a href="https://twitter.com/simonw/status/1429194217482424324" rel="nofollow">my Twitter thread</a> where I figured this out.</p>


path: datasette_search-all-columns-trick.md
topic: datasette
title: Searching all columns of a table in Datasette
url: https://github.com/simonw/til/blob/main/datasette/search-all-columns-trick.md
created: 2021-08-23T11:48:22-07:00
created_utc: 2021-08-23T18:48:22+00:00
updated: 2021-08-23T12:05:47-07:00
updated_utc: 2021-08-23T19:05:47+00:00
shot_hash: e24510529260e6312f52947cb05723d0
slug: search-all-columns-trick
body:
I came up with this trick today, when I wanted to run a `LIKE` search against every column in a table.

The trick is to generate a SQL query that does a `LIKE` search against every column of a table. We can generate that query using another query:

```sql
select
  'select * from "' || :table || '" where ' || group_concat(
    '"' || name || '" like ''%'' || :search || ''%''',
    ' or '
  )
from
  pragma_table_info(:table)
```
Here's what you get when you [run that query](https://fivethirtyeight.datasettes.com/fivethirtyeight?sql=select%0D%0A++%27select+*+from+%22%27+%7C%7C+%3Atable+%7C%7C+%27%22+where+%27+%7C%7C+group_concat%28%0D%0A++++%27%22%27+%7C%7C+name+%7C%7C+%27%22+like+%27%27%25%27%27+%7C%7C+%3Asearch+%7C%7C+%27%27%25%27%27%27%2C%0D%0A++++%27+or+%27%0D%0A++%29%0D%0Afrom%0D%0A++pragma_table_info%28%3Atable%29&table=avengers%2Favengers) against the [avengers example table](https://fivethirtyeight.datasettes.com/fivethirtyeight/avengers%2Favengers) from FiveThirtyEight (pretty-printed):

```sql
select
  *
from
  "avengers/avengers"
where
  "URL" like '%' || :search || '%'
  or "Name/Alias" like '%' || :search || '%'
  or "Appearances" like '%' || :search || '%'
  or "Current?" like '%' || :search || '%'
  or "Gender" like '%' || :search || '%'
  or "Probationary Introl" like '%' || :search || '%'
  or "Full/Reserve Avengers Intro" like '%' || :search || '%'
  or "Year" like '%' || :search || '%'
  or "Years since joining" like '%' || :search || '%'
  or "Honorary" like '%' || :search || '%'
  or "Death1" like '%' || :search || '%'
  or "Return1" like '%' || :search || '%'
  or "Death2" like '%' || :search || '%'
  or "Return2" like '%' || :search || '%'
  or "Death3" like '%' || :search || '%'
  or "Return3" like '%' || :search || '%'
  or "Death4" like '%' || :search || '%'
  or "Return4" like '%' || :search || '%'
  or "Death5" like '%' || :search || '%'
  or "Return5" like '%' || :search || '%'
  or "Notes" like '%' || :search || '%'
```
Here's [an example search](https://fivethirtyeight.datasettes.com/fivethirtyeight?sql=select%0D%0A++*%0D%0Afrom%0D%0A++%22avengers%2Favengers%22%0D%0Awhere%0D%0A++%22URL%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Name%2FAlias%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Appearances%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Current%3F%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Gender%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Probationary+Introl%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Full%2FReserve+Avengers+Intro%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Year%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Years+since+joining%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Honorary%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Death1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Return1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Death2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Return2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Death3%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Return3%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Death4%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Return4%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Death5%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Return5%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Notes%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27&search=Grim+Reaper) using that generated query.

## Same trick for the entire database

Here's a query that generates a query that searches every column in every table in the database!

```sql
with tables as (
  select
    name as table_name
  from
    sqlite_master
  where
    type = 'table'
),
queries as (
  select
    'select ''' || tables.table_name || ''' as _table, rowid from "' || tables.table_name || '" where ' || group_concat(
      '"' || name || '" like ''%'' || :search || ''%''',
      ' or '
    ) as query
  from
    pragma_table_info(tables.table_name),
    tables
  group by
    tables.table_name
)
select
  group_concat(query, ' union all ')
from
  queries
```
I tried this against the FiveThirtyEight database and the query it produced was way beyond the URL length limit for Cloud Run.

Here's the result if [run against latest.datasette.io/fixtures](https://latest.datasette.io/fixtures?sql=with+tables+as+%28%0D%0A++select%0D%0A++++name+as+table_name%0D%0A++from%0D%0A++++sqlite_master%0D%0A++where%0D%0A++++type+%3D+%27table%27%0D%0A%29%2C%0D%0Aqueries+as+%28%0D%0A++select%0D%0A++++%27select+%27%27%27+%7C%7C+tables.table_name+%7C%7C+%27%27%27+as+_table%2C+rowid+from+%22%27+%7C%7C+tables.table_name+%7C%7C+%27%22+where+%27+%7C%7C+group_concat%28%0D%0A++++++%27%22%27+%7C%7C+name+%7C%7C+%27%22+like+%27%27%25%27%27+%7C%7C+%3Asearch+%7C%7C+%27%27%25%27%27%27%2C%0D%0A++++++%27+or+%27%0D%0A++++%29+as+query%0D%0A++from%0D%0A++++pragma_table_info%28tables.table_name%29%2C%0D%0A++++tables%0D%0A++group+by%0D%0A++++tables.table_name%0D%0A%29%0D%0Aselect%0D%0A++group_concat%28query%2C+%27+union+all+%27%29%0D%0Afrom%0D%0A++queries):

```sql
select
  '123_starts_with_digits' as _table,
  rowid
from
  "123_starts_with_digits"
where
  "content" like '%' || :search || '%'
union all
select
  'Table With Space In Name' as _table,
  rowid
from
  "Table With Space In Name"
where
  "pk" like '%' || :search || '%'
  or "content" like '%' || :search || '%'
union all
select
  'attraction_characteristic' as _table,
  rowid
from
  "attraction_characteristic"
where
  "pk" like '%' || :search || '%'
  or "name" like '%' || :search || '%'
union all
select
  'binary_data' as _table,
  rowid
from
  "binary_data"
where
  "data" like '%' || :search || '%'
union all
select
  'complex_foreign_keys' as _table,
  rowid
from
  "complex_foreign_keys"
where
  "pk" like '%' || :search || '%'
  or "f1" like '%' || :search || '%'
  or "f2" like '%' || :search || '%'
  or "f3" like '%' || :search || '%'
union all
select
  'compound_primary_key' as _table,
  rowid
from
  "compound_primary_key"
where
  "pk1" like '%' || :search || '%'
  or "pk2" like '%' || :search || '%'
  or "content" like '%' || :search || '%'
union all
select
  'compound_three_primary_keys' as _table,
  rowid
from
  "compound_three_primary_keys"
where
  "pk1" like '%' || :search || '%'
  or "pk2" like '%' || :search || '%'
  or "pk3" like '%' || :search || '%'
  or "content" like '%' || :search || '%'
union all
select
  'custom_foreign_key_label' as _table,
  rowid
from
  "custom_foreign_key_label"
where
  "pk" like '%' || :search || '%'
  or "foreign_key_with_custom_label" like '%' || :search || '%'
union all
select
  'facet_cities' as _table,
  rowid
from
  "facet_cities"
where
  "id" like '%' || :search || '%'
  or "name" like '%' || :search || '%'
union all
select
  'facetable' as _table,
  rowid
from
  "facetable"
where
  "pk" like '%' || :search || '%'
  or "created" like '%' || :search || '%'
  or "planet_int" like '%' || :search || '%'
  or "on_earth" like '%' || :search || '%'
  or "state" like '%' || :search || '%'
  or "city_id" like '%' || :search || '%'
  or "neighborhood" like '%' || :search || '%'
  or "tags" like '%' || :search || '%'
  or "complex_array" like '%' || :search || '%'
  or "distinct_some_null" like '%' || :search || '%'
union all
select
  'foreign_key_references' as _table,
  rowid
from
  "foreign_key_references"
where
  "pk" like '%' || :search || '%'
  or "foreign_key_with_label" like '%' || :search || '%'
  or "foreign_key_with_blank_label" like '%' || :search || '%'
  or "foreign_key_with_no_label" like '%' || :search || '%'
  or "foreign_key_compound_pk1" like '%' || :search || '%'
  or "foreign_key_compound_pk2" like '%' || :search || '%'
union all
select
  'infinity' as _table,
  rowid
from
  "infinity"
where
  "value" like '%' || :search || '%'
union all
select
  'no_primary_key' as _table,
  rowid
from
  "no_primary_key"
where
  "content" like '%' || :search || '%'
  or "a" like '%' || :search || '%'
  or "b" like '%' || :search || '%'
  or "c" like '%' || :search || '%'
union all
select
  'primary_key_multiple_columns' as _table,
  rowid
from
  "primary_key_multiple_columns"
where
  "id" like '%' || :search || '%'
  or "content" like '%' || :search || '%'
  or "content2" like '%' || :search || '%'
union all
select
  'primary_key_multiple_columns_explicit_label' as _table,
  rowid
from
  "primary_key_multiple_columns_explicit_label"
where
  "id" like '%' || :search || '%'
  or "content" like '%' || :search || '%'
  or "content2" like '%' || :search || '%'
union all
select
  'roadside_attraction_characteristics' as _table,
  rowid
from
  "roadside_attraction_characteristics"
where
  "attraction_id" like '%' || :search || '%'
  or "characteristic_id" like '%' || :search || '%'
union all
select
  'roadside_attractions' as _table,
  rowid
from
  "roadside_attractions"
where
  "pk" like '%' || :search || '%'
  or "name" like '%' || :search || '%'
  or "address" like '%' || :search || '%'
  or "latitude" like '%' || :search || '%'
  or "longitude" like '%' || :search || '%'
union all
select
  'searchable' as _table,
  rowid
from
  "searchable"
where
  "pk" like '%' || :search || '%'
  or "text1" like '%' || :search || '%'
  or "text2" like '%' || :search || '%'
  or "name with . and spaces" like '%' || :search || '%'
union all
select
  'searchable_fts' as _table,
  rowid
from
  "searchable_fts"
where
  "text1" like '%' || :search || '%'
  or "text2" like '%' || :search || '%'
  or "name with . and spaces" like '%' || :search || '%'
union all
select
  'searchable_fts_docsize' as _table,
  rowid
from
  "searchable_fts_docsize"
where
  "docid" like '%' || :search || '%'
  or "size" like '%' || :search || '%'
union all
select
  'searchable_fts_segdir' as _table,
  rowid
from
  "searchable_fts_segdir"
where
  "level" like '%' || :search || '%'
  or "idx" like '%' || :search || '%'
  or "start_block" like '%' || :search || '%'
  or "leaves_end_block" like '%' || :search || '%'
  or "end_block" like '%' || :search || '%'
  or "root" like '%' || :search || '%'
union all
select
  'searchable_fts_segments' as _table,
  rowid
from
  "searchable_fts_segments"
where
  "blockid" like '%' || :search || '%'
  or "block" like '%' || :search || '%'
union all
select
  'searchable_fts_stat' as _table,
  rowid
from
  "searchable_fts_stat"
where
  "id" like '%' || :search || '%'
  or "value" like '%' || :search || '%'
union all
select
  'searchable_tags' as _table,
  rowid
from
  "searchable_tags"
where
  "searchable_id" like '%' || :search || '%'
  or "tag" like '%' || :search || '%'
union all
select
  'select' as _table,
  rowid
from
  "select"
where
  "group" like '%' || :search || '%'
  or "having" like '%' || :search || '%'
  or "and" like '%' || :search || '%'
  or "json" like '%' || :search || '%'
union all
select
  'simple_primary_key' as _table,
  rowid
from
  "simple_primary_key"
where
  "id" like '%' || :search || '%'
  or "content" like '%' || :search || '%'
union all
select
  'sortable' as _table,
  rowid
from
  "sortable"
where
  "pk1" like '%' || :search || '%'
  or "pk2" like '%' || :search || '%'
  or "content" like '%' || :search || '%'
  or "sortable" like '%' || :search || '%'
  or "sortable_with_nulls" like '%' || :search || '%'
  or "sortable_with_nulls_2" like '%' || :search || '%'
  or "text" like '%' || :search || '%'
union all
select
  'table/with/slashes.csv' as _table,
  rowid
from
  "table/with/slashes.csv"
where
  "pk" like '%' || :search || '%'
  or "content" like '%' || :search || '%'
union all
select
  'tags' as _table,
  rowid
from
  "tags"
where
  "tag" like '%' || :search || '%'
union all
select
  'units' as _table,
  rowid
from
  "units"
where
  "pk" like '%' || :search || '%'
  or "distance" like '%' || :search || '%'
  or "frequency" like '%' || :search || '%'
```
[It works!](https://latest.datasette.io/fixtures?sql=select%0D%0A++%27123_starts_with_digits%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22123_starts_with_digits%22%0D%0Awhere%0D%0A++%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27Table+With+Space+In+Name%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22Table+With+Space+In+Name%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27attraction_characteristic%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22attraction_characteristic%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22name%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27binary_data%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22binary_data%22%0D%0Awhere%0D%0A++%22data%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27complex_foreign_keys%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22complex_foreign_keys%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22f1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22f2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22f3%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27compound_primary_key%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22compound_primary_key%22%0D%0Awhere%0D%0A++%22pk1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22pk2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27compound_three_primary_keys%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22compound_three_primary_keys%22%0D%0Awhere%0D%0A++%22pk1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22pk2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22pk3%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27custom_foreign_key_label%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22custom_foreign_key_label%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22foreign_key_with_custom_label%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27facet_cities%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22facet_cities%22%0D%0Awhere%0D%0A++%22id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22name%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27facetable%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22facetable%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22created%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22planet_int%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22on_earth%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22state%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22city_id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22neighborhood%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22tags%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22complex_array%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22distinct_some_null%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27foreign_key_references%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22foreign_key_references%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22foreign_key_with_label%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22foreign_key_with_blank_label%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22foreign_key_with_no_label%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22foreign_key_compound_pk1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22foreign_key_compound_pk2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27infinity%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22infinity%22%0D%0Awhere%0D%0A++%22value%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27no_primary_key%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22no_primary_key%22%0D%0Awhere%0D%0A++%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22a%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22b%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22c%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27primary_key_multiple_columns%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22primary_key_multiple_columns%22%0D%0Awhere%0D%0A++%22id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27primary_key_multiple_columns_explicit_label%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22primary_key_multiple_columns_explicit_label%22%0D%0Awhere%0D%0A++%22id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27roadside_attraction_characteristics%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22roadside_attraction_characteristics%22%0D%0Awhere%0D%0A++%22attraction_id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22characteristic_id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27roadside_attractions%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22roadside_attractions%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22name%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22address%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22latitude%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22longitude%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22text1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22text2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22name+with+.+and+spaces%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable_fts%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable_fts%22%0D%0Awhere%0D%0A++%22text1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22text2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22name+with+.+and+spaces%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable_fts_docsize%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable_fts_docsize%22%0D%0Awhere%0D%0A++%22docid%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22size%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable_fts_segdir%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable_fts_segdir%22%0D%0Awhere%0D%0A++%22level%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22idx%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22start_block%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22leaves_end_block%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22end_block%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22root%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable_fts_segments%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable_fts_segments%22%0D%0Awhere%0D%0A++%22blockid%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22block%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable_fts_stat%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable_fts_stat%22%0D%0Awhere%0D%0A++%22id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22value%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable_tags%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable_tags%22%0D%0Awhere%0D%0A++%22searchable_id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22tag%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27select%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22select%22%0D%0Awhere%0D%0A++%22group%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22having%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22and%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22json%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27simple_primary_key%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22simple_primary_key%22%0D%0Awhere%0D%0A++%22id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27sortable%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22sortable%22%0D%0Awhere%0D%0A++%22pk1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22pk2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22sortable%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22sortable_with_nulls%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22sortable_with_nulls_2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22text%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27table%2Fwith%2Fslashes.csv%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22table%2Fwith%2Fslashes.csv%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27tags%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22tags%22%0D%0Awhere%0D%0A++%22tag%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27units%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22units%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22distance%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22frequency%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27&search=museum&_hide_sql=1)

html:
<p>I came up with this trick today, when I wanted to run a <code>LIKE</code> search against every column in a table.</p>
<p>The trick is to generate a SQL query that does a <code>LIKE</code> search against every column of a table. We can generate that query using another query:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>select * from "<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :table <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>" where <span class="pl-pds">'</span></span> <span class="pl-k">||</span> group_concat(
    <span class="pl-s"><span class="pl-pds">'</span>"<span class="pl-pds">'</span></span> <span class="pl-k">||</span> name <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>" like <span class="pl-pds">'</span><span class="pl-pds">'</span>%<span class="pl-pds">'</span><span class="pl-pds">'</span> || :search || <span class="pl-pds">'</span><span class="pl-pds">'</span>%<span class="pl-pds">'</span><span class="pl-pds">'</span><span class="pl-pds">'</span></span>,
    <span class="pl-s"><span class="pl-pds">'</span> or <span class="pl-pds">'</span></span>
  )
<span class="pl-k">from</span>
  pragma_table_info(:table)</pre></div>
<p>Here's what you get when you <a href="https://fivethirtyeight.datasettes.com/fivethirtyeight?sql=select%0D%0A++%27select+*+from+%22%27+%7C%7C+%3Atable+%7C%7C+%27%22+where+%27+%7C%7C+group_concat%28%0D%0A++++%27%22%27+%7C%7C+name+%7C%7C+%27%22+like+%27%27%25%27%27+%7C%7C+%3Asearch+%7C%7C+%27%27%25%27%27%27%2C%0D%0A++++%27+or+%27%0D%0A++%29%0D%0Afrom%0D%0A++pragma_table_info%28%3Atable%29&amp;table=avengers%2Favengers" rel="nofollow">run that query</a> against the <a href="https://fivethirtyeight.datasettes.com/fivethirtyeight/avengers%2Favengers" rel="nofollow">avengers example table</a> from FiveThirtyEight (pretty-printed):</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  <span class="pl-k">*</span>
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>avengers/avengers<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>URL<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Name/Alias<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Appearances<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Current?<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Gender<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Probationary Introl<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Full/Reserve Avengers Intro<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Year<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Years since joining<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Honorary<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Death1<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Return1<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Death2<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Return2<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Death3<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Return3<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Death4<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Return4<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Death5<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Return5<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>Notes<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span></pre></div>
<p>Here's <a href="https://fivethirtyeight.datasettes.com/fivethirtyeight?sql=select%0D%0A++*%0D%0Afrom%0D%0A++%22avengers%2Favengers%22%0D%0Awhere%0D%0A++%22URL%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Name%2FAlias%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Appearances%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Current%3F%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Gender%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Probationary+Introl%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Full%2FReserve+Avengers+Intro%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Year%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Years+since+joining%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Honorary%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Death1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Return1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Death2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Return2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Death3%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Return3%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Death4%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Return4%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Death5%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Return5%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22Notes%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27&amp;search=Grim+Reaper" rel="nofollow">an example search</a> using that generated query.</p>
<h2>
<a id="user-content-same-trick-for-the-entire-database" class="anchor" href="#same-trick-for-the-entire-database" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Same trick for the entire database</h2>
<p>Here's a query that generates a query that searches every column in every table in the database!</p>
<div class="highlight highlight-source-sql"><pre>with tables <span class="pl-k">as</span> (
  <span class="pl-k">select</span>
    name <span class="pl-k">as</span> table_name
  <span class="pl-k">from</span>
    sqlite_master
  <span class="pl-k">where</span>
    type <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">'</span>table<span class="pl-pds">'</span></span>
),
queries <span class="pl-k">as</span> (
  <span class="pl-k">select</span>
    <span class="pl-s"><span class="pl-pds">'</span>select <span class="pl-pds">'</span><span class="pl-pds">'</span><span class="pl-pds">'</span></span> <span class="pl-k">||</span> <span class="pl-c1">tables</span>.<span class="pl-c1">table_name</span> <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span><span class="pl-pds">'</span> as _table, rowid from "<span class="pl-pds">'</span></span> <span class="pl-k">||</span> <span class="pl-c1">tables</span>.<span class="pl-c1">table_name</span> <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>" where <span class="pl-pds">'</span></span> <span class="pl-k">||</span> group_concat(
      <span class="pl-s"><span class="pl-pds">'</span>"<span class="pl-pds">'</span></span> <span class="pl-k">||</span> name <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>" like <span class="pl-pds">'</span><span class="pl-pds">'</span>%<span class="pl-pds">'</span><span class="pl-pds">'</span> || :search || <span class="pl-pds">'</span><span class="pl-pds">'</span>%<span class="pl-pds">'</span><span class="pl-pds">'</span><span class="pl-pds">'</span></span>,
      <span class="pl-s"><span class="pl-pds">'</span> or <span class="pl-pds">'</span></span>
    ) <span class="pl-k">as</span> query
  <span class="pl-k">from</span>
    pragma_table_info(<span class="pl-c1">tables</span>.<span class="pl-c1">table_name</span>),
    tables
  <span class="pl-k">group by</span>
    <span class="pl-c1">tables</span>.<span class="pl-c1">table_name</span>
)
<span class="pl-k">select</span>
  group_concat(query, <span class="pl-s"><span class="pl-pds">'</span> union all <span class="pl-pds">'</span></span>)
<span class="pl-k">from</span>
  queries</pre></div>
<p>I tried this against the FiveThirtyEight database and the query it produced was way beyond the URL length limit for Cloud Run.</p>
<p>Here's the result if <a href="https://latest.datasette.io/fixtures?sql=with+tables+as+%28%0D%0A++select%0D%0A++++name+as+table_name%0D%0A++from%0D%0A++++sqlite_master%0D%0A++where%0D%0A++++type+%3D+%27table%27%0D%0A%29%2C%0D%0Aqueries+as+%28%0D%0A++select%0D%0A++++%27select+%27%27%27+%7C%7C+tables.table_name+%7C%7C+%27%27%27+as+_table%2C+rowid+from+%22%27+%7C%7C+tables.table_name+%7C%7C+%27%22+where+%27+%7C%7C+group_concat%28%0D%0A++++++%27%22%27+%7C%7C+name+%7C%7C+%27%22+like+%27%27%25%27%27+%7C%7C+%3Asearch+%7C%7C+%27%27%25%27%27%27%2C%0D%0A++++++%27+or+%27%0D%0A++++%29+as+query%0D%0A++from%0D%0A++++pragma_table_info%28tables.table_name%29%2C%0D%0A++++tables%0D%0A++group+by%0D%0A++++tables.table_name%0D%0A%29%0D%0Aselect%0D%0A++group_concat%28query%2C+%27+union+all+%27%29%0D%0Afrom%0D%0A++queries" rel="nofollow">run against latest.datasette.io/fixtures</a>:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>123_starts_with_digits<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>123_starts_with_digits<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>content<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>Table With Space In Name<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>Table With Space In Name<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>pk<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>content<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>attraction_characteristic<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>attraction_characteristic<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>pk<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>binary_data<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>binary_data<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>data<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>complex_foreign_keys<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>complex_foreign_keys<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>pk<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>f1<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>f2<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>f3<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>compound_primary_key<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>compound_primary_key<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>pk1<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>pk2<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>content<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>compound_three_primary_keys<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>compound_three_primary_keys<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>pk1<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>pk2<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>pk3<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>content<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>custom_foreign_key_label<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>custom_foreign_key_label<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>pk<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>foreign_key_with_custom_label<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>facet_cities<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>facet_cities<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>facetable<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>facetable<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>pk<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>created<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>planet_int<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>on_earth<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>city_id<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>neighborhood<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>tags<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>complex_array<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>distinct_some_null<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>foreign_key_references<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>foreign_key_references<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>pk<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>foreign_key_with_label<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>foreign_key_with_blank_label<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>foreign_key_with_no_label<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>foreign_key_compound_pk1<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>foreign_key_compound_pk2<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>infinity<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>infinity<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>no_primary_key<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>no_primary_key<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>content<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>a<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>b<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>c<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>primary_key_multiple_columns<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>primary_key_multiple_columns<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>content<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>content2<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>primary_key_multiple_columns_explicit_label<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>primary_key_multiple_columns_explicit_label<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>content<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>content2<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>roadside_attraction_characteristics<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>roadside_attraction_characteristics<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>attraction_id<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>characteristic_id<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>roadside_attractions<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>roadside_attractions<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>pk<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>address<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>latitude<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>longitude<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>searchable<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>searchable<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>pk<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>text1<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>text2<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>name with . and spaces<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>searchable_fts<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>searchable_fts<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>text1<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>text2<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>name with . and spaces<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>searchable_fts_docsize<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>searchable_fts_docsize<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>docid<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>size<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>searchable_fts_segdir<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>searchable_fts_segdir<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>level<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>idx<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>start_block<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>leaves_end_block<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>end_block<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>root<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>searchable_fts_segments<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>searchable_fts_segments<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>blockid<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>block<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>searchable_fts_stat<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>searchable_fts_stat<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>searchable_tags<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>searchable_tags<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>searchable_id<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>tag<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>select<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>select<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>group<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>having<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>and<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>json<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>simple_primary_key<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>simple_primary_key<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>content<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>sortable<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>sortable<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>pk1<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>pk2<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>content<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>sortable<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>sortable_with_nulls<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>sortable_with_nulls_2<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>text<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>table/with/slashes.csv<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>table/with/slashes.csv<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>pk<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>content<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>tags<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>tags<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>tag<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
<span class="pl-k">union all</span>
<span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>units<span class="pl-pds">'</span></span> <span class="pl-k">as</span> _table,
  rowid
<span class="pl-k">from</span>
  <span class="pl-s"><span class="pl-pds">"</span>units<span class="pl-pds">"</span></span>
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>pk<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>distance<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span>
  <span class="pl-k">or</span> <span class="pl-s"><span class="pl-pds">"</span>frequency<span class="pl-pds">"</span></span> <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :search <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>%<span class="pl-pds">'</span></span></pre></div>
<p><a href="https://latest.datasette.io/fixtures?sql=select%0D%0A++%27123_starts_with_digits%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22123_starts_with_digits%22%0D%0Awhere%0D%0A++%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27Table+With+Space+In+Name%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22Table+With+Space+In+Name%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27attraction_characteristic%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22attraction_characteristic%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22name%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27binary_data%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22binary_data%22%0D%0Awhere%0D%0A++%22data%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27complex_foreign_keys%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22complex_foreign_keys%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22f1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22f2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22f3%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27compound_primary_key%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22compound_primary_key%22%0D%0Awhere%0D%0A++%22pk1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22pk2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27compound_three_primary_keys%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22compound_three_primary_keys%22%0D%0Awhere%0D%0A++%22pk1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22pk2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22pk3%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27custom_foreign_key_label%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22custom_foreign_key_label%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22foreign_key_with_custom_label%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27facet_cities%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22facet_cities%22%0D%0Awhere%0D%0A++%22id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22name%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27facetable%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22facetable%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22created%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22planet_int%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22on_earth%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22state%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22city_id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22neighborhood%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22tags%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22complex_array%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22distinct_some_null%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27foreign_key_references%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22foreign_key_references%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22foreign_key_with_label%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22foreign_key_with_blank_label%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22foreign_key_with_no_label%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22foreign_key_compound_pk1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22foreign_key_compound_pk2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27infinity%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22infinity%22%0D%0Awhere%0D%0A++%22value%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27no_primary_key%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22no_primary_key%22%0D%0Awhere%0D%0A++%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22a%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22b%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22c%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27primary_key_multiple_columns%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22primary_key_multiple_columns%22%0D%0Awhere%0D%0A++%22id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27primary_key_multiple_columns_explicit_label%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22primary_key_multiple_columns_explicit_label%22%0D%0Awhere%0D%0A++%22id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27roadside_attraction_characteristics%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22roadside_attraction_characteristics%22%0D%0Awhere%0D%0A++%22attraction_id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22characteristic_id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27roadside_attractions%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22roadside_attractions%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22name%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22address%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22latitude%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22longitude%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22text1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22text2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22name+with+.+and+spaces%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable_fts%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable_fts%22%0D%0Awhere%0D%0A++%22text1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22text2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22name+with+.+and+spaces%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable_fts_docsize%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable_fts_docsize%22%0D%0Awhere%0D%0A++%22docid%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22size%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable_fts_segdir%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable_fts_segdir%22%0D%0Awhere%0D%0A++%22level%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22idx%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22start_block%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22leaves_end_block%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22end_block%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22root%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable_fts_segments%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable_fts_segments%22%0D%0Awhere%0D%0A++%22blockid%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22block%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable_fts_stat%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable_fts_stat%22%0D%0Awhere%0D%0A++%22id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22value%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27searchable_tags%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22searchable_tags%22%0D%0Awhere%0D%0A++%22searchable_id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22tag%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27select%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22select%22%0D%0Awhere%0D%0A++%22group%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22having%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22and%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22json%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27simple_primary_key%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22simple_primary_key%22%0D%0Awhere%0D%0A++%22id%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27sortable%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22sortable%22%0D%0Awhere%0D%0A++%22pk1%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22pk2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22sortable%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22sortable_with_nulls%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22sortable_with_nulls_2%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22text%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27table%2Fwith%2Fslashes.csv%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22table%2Fwith%2Fslashes.csv%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22content%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27tags%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22tags%22%0D%0Awhere%0D%0A++%22tag%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0Aunion+all%0D%0Aselect%0D%0A++%27units%27+as+_table%2C%0D%0A++rowid%0D%0Afrom%0D%0A++%22units%22%0D%0Awhere%0D%0A++%22pk%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22distance%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27%0D%0A++or+%22frequency%22+like+%27%25%27+%7C%7C+%3Asearch+%7C%7C+%27%25%27&amp;search=museum&amp;_hide_sql=1" rel="nofollow">It works!</a></p>


path: purpleair_purple-air-aqi.md
topic: purpleair
title: Calculating the AQI based on the Purple Air API for a sensor
url: https://github.com/simonw/til/blob/main/purpleair/purple-air-aqi.md
created: 2021-08-31T12:46:29-07:00
created_utc: 2021-08-31T19:46:29+00:00
updated: 2021-08-31T12:46:29-07:00
updated_utc: 2021-08-31T19:46:29+00:00
shot_hash: 5544b8baf8534b9fb01ff2a53e1b9f1a
slug: purple-air-aqi
body:
[Purple Air](https://www.purpleair.com/) sensors have an API at `https://www.purpleair.com/map.json?show=SENSOR-ID-HERE`, which returns JSON that looks something like this:

```json
{
  "mapVersion": "0.26",
  "baseVersion": "7",
  "mapVersionString": "",
  "results": [
    {
      "ID": 123,
      "Label": "Sensor label",
      "DEVICE_LOCATIONTYPE": "outside",
      "THINGSPEAK_PRIMARY_ID": "123",
      "THINGSPEAK_PRIMARY_ID_READ_KEY": "xxx",
      "THINGSPEAK_SECONDARY_ID": "1234",
      "THINGSPEAK_SECONDARY_ID_READ_KEY": "xxx",
      "Lat": 37.5,
      "Lon": -122.4,
      "PM2_5Value": "8.75",
      "LastSeen": 1630438756,
      "Type": "PMS5003+PMS5003+BME280",
      "Hidden": "false",
      "Flag": 1,
      "DEVICE_BRIGHTNESS": "15",
      "DEVICE_HARDWAREDISCOVERED": "2.0+OPENLOG+15476 MB+DS3231+BME280+PMSX003-B+PMSX003-A",
      "DEVICE_FIRMWAREVERSION": "6.01",
      "Version": "6.01",
      "LastUpdateCheck": 1630436115,
      "Created": 1622588142,
      "Uptime": "2701447",
      "RSSI": "-56",
      "Adc": "0.01",
      "p_0_3_um": "849.53",
      "p_0_5_um": "251.67",
      "p_1_0_um": "78.49",
      "p_2_5_um": "17.79",
      "p_5_0_um": "6.95",
      "p_10_0_um": "5.54",
      "pm1_0_cf_1": "3.58",
      "pm2_5_cf_1": "8.75",
      "pm10_0_cf_1": "14.04",
      "pm1_0_atm": "3.58",
      "pm2_5_atm": "8.75",
      "pm10_0_atm": "14.04",
      "isOwner": 0,
      "humidity": "48",
      "temp_f": "74",
      "pressure": "1005.56",
      "AGE": 1,
      "Stats": "{\"v\":8.75,\"v1\":9.4,\"v2\":10.22,\"v3\":10.96,\"v4\":14.17,\"v5\":15.51,\"v6\":12.53,\"pm\":8.75,\"lastModified\":1630438756184,\"timeSinceModified\":120110}"
    }
  ]
}
```
There's just one problem with this: it doesn't give you the AQI number that is displayed on their site! Instead it gives you the raw numbers that can be used to calculate that AQI number.

I figured someone must have solved this, so I ran a GitHub code search for [purpleair.com map json](https://github.com/search?q=purpleair.com+map+json&type=code) and found [zakj/scriptable](https://github.com/zakj/scriptable) with [code for decoding that](https://github.com/zakj/scriptable/blob/d42a2d7727a2642ad6ee880c15ae65dca4232fe4/purpleAir.js). I adapted it to the following:

```javascript
// Adapted from https://github.com/zakj/scriptable by Zak Johnson
async function fetchAqi(sensorId) {
  const response = await fetch(
    `https://www.purpleair.com/json?show=${sensorId}`
  );
  const json = await response.json();
  const stats = json.results
    .filter((r) => !(r.Flag || r.A_H))
    .map((r) => JSON.parse(r.Stats));
  const pm2_5 = stats.reduce((acc, { v }) => acc + v, 0) / stats.length;
  const trend = stats[0].v1 - stats[0].v3;
  return {
    current: aqiFromPm(pm2_5),
    trend: Math.abs(trend) > 5 ? trend : 0,
    details: json,
  };
}

function aqiFromPm(pm) {
  const table = [
    [0.0, 12.0, 0, 50],
    [12.1, 35.4, 51, 100],
    [35.5, 55.4, 101, 150],
    [55.5, 150.4, 151, 200],
    [150.5, 250.4, 201, 300],
    [250.5, 500.4, 301, 500],
  ];
  const computeAqi = (concI, [concLo, concHi, aqiLo, aqiHi]) =>
    Math.round(
      ((concI - concLo) / (concHi - concLo)) * (aqiHi - aqiLo) + aqiLo
    );
  const values = table.find(([concLo, concHi, aqiLo, aqiHi]) => pm <= concHi);
  return values ? computeAqi(pm, values) : 500;
}
```
I used this to build a simple demo at https://til.simonwillison.net/tools/aqi

html:
<p><a href="https://www.purpleair.com/" rel="nofollow">Purple Air</a> sensors have an API at <code>https://www.purpleair.com/map.json?show=SENSOR-ID-HERE</code>, which returns JSON that looks something like this:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-s"><span class="pl-pds">"</span>mapVersion<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>0.26<span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>baseVersion<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>7<span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>mapVersionString<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span><span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>results<span class="pl-pds">"</span></span>: [
    {
      <span class="pl-s"><span class="pl-pds">"</span>ID<span class="pl-pds">"</span></span>: <span class="pl-c1">123</span>,
      <span class="pl-s"><span class="pl-pds">"</span>Label<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Sensor label<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>DEVICE_LOCATIONTYPE<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>outside<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>THINGSPEAK_PRIMARY_ID<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>123<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>THINGSPEAK_PRIMARY_ID_READ_KEY<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>xxx<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>THINGSPEAK_SECONDARY_ID<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>1234<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>THINGSPEAK_SECONDARY_ID_READ_KEY<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>xxx<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>Lat<span class="pl-pds">"</span></span>: <span class="pl-c1">37.5</span>,
      <span class="pl-s"><span class="pl-pds">"</span>Lon<span class="pl-pds">"</span></span>: <span class="pl-c1">-122.4</span>,
      <span class="pl-s"><span class="pl-pds">"</span>PM2_5Value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>8.75<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>LastSeen<span class="pl-pds">"</span></span>: <span class="pl-c1">1630438756</span>,
      <span class="pl-s"><span class="pl-pds">"</span>Type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>PMS5003+PMS5003+BME280<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>Hidden<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>false<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>Flag<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>,
      <span class="pl-s"><span class="pl-pds">"</span>DEVICE_BRIGHTNESS<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>15<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>DEVICE_HARDWAREDISCOVERED<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2.0+OPENLOG+15476 MB+DS3231+BME280+PMSX003-B+PMSX003-A<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>DEVICE_FIRMWAREVERSION<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>6.01<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>Version<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>6.01<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>LastUpdateCheck<span class="pl-pds">"</span></span>: <span class="pl-c1">1630436115</span>,
      <span class="pl-s"><span class="pl-pds">"</span>Created<span class="pl-pds">"</span></span>: <span class="pl-c1">1622588142</span>,
      <span class="pl-s"><span class="pl-pds">"</span>Uptime<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2701447<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>RSSI<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>-56<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>Adc<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>0.01<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>p_0_3_um<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>849.53<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>p_0_5_um<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>251.67<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>p_1_0_um<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>78.49<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>p_2_5_um<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>17.79<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>p_5_0_um<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>6.95<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>p_10_0_um<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>5.54<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>pm1_0_cf_1<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>3.58<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>pm2_5_cf_1<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>8.75<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>pm10_0_cf_1<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>14.04<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>pm1_0_atm<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>3.58<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>pm2_5_atm<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>8.75<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>pm10_0_atm<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>14.04<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>isOwner<span class="pl-pds">"</span></span>: <span class="pl-c1">0</span>,
      <span class="pl-s"><span class="pl-pds">"</span>humidity<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>48<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>temp_f<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>74<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>pressure<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>1005.56<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>AGE<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>,
      <span class="pl-s"><span class="pl-pds">"</span>Stats<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>{<span class="pl-cce">\"</span>v<span class="pl-cce">\"</span>:8.75,<span class="pl-cce">\"</span>v1<span class="pl-cce">\"</span>:9.4,<span class="pl-cce">\"</span>v2<span class="pl-cce">\"</span>:10.22,<span class="pl-cce">\"</span>v3<span class="pl-cce">\"</span>:10.96,<span class="pl-cce">\"</span>v4<span class="pl-cce">\"</span>:14.17,<span class="pl-cce">\"</span>v5<span class="pl-cce">\"</span>:15.51,<span class="pl-cce">\"</span>v6<span class="pl-cce">\"</span>:12.53,<span class="pl-cce">\"</span>pm<span class="pl-cce">\"</span>:8.75,<span class="pl-cce">\"</span>lastModified<span class="pl-cce">\"</span>:1630438756184,<span class="pl-cce">\"</span>timeSinceModified<span class="pl-cce">\"</span>:120110}<span class="pl-pds">"</span></span>
    }
  ]
}</pre></div>
<p>There's just one problem with this: it doesn't give you the AQI number that is displayed on their site! Instead it gives you the raw numbers that can be used to calculate that AQI number.</p>
<p>I figured someone must have solved this, so I ran a GitHub code search for <a href="https://github.com/search?q=purpleair.com+map+json&amp;type=code">purpleair.com map json</a> and found <a href="https://github.com/zakj/scriptable">zakj/scriptable</a> with <a href="https://github.com/zakj/scriptable/blob/d42a2d7727a2642ad6ee880c15ae65dca4232fe4/purpleAir.js">code for decoding that</a>. I adapted it to the following:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-c">// Adapted from https://github.com/zakj/scriptable by Zak Johnson</span>
<span class="pl-k">async</span> <span class="pl-k">function</span> <span class="pl-en">fetchAqi</span><span class="pl-kos">(</span><span class="pl-s1">sensorId</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
  <span class="pl-k">const</span> <span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-en">fetch</span><span class="pl-kos">(</span>
    <span class="pl-s">`https://www.purpleair.com/json?show=<span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">sensorId</span><span class="pl-kos">}</span></span>`</span>
  <span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">json</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">response</span><span class="pl-kos">.</span><span class="pl-en">json</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">stats</span> <span class="pl-c1">=</span> <span class="pl-s1">json</span><span class="pl-kos">.</span><span class="pl-c1">results</span>
    <span class="pl-kos">.</span><span class="pl-en">filter</span><span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-s1">r</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-c1">!</span><span class="pl-kos">(</span><span class="pl-s1">r</span><span class="pl-kos">.</span><span class="pl-c1">Flag</span> <span class="pl-c1">||</span> <span class="pl-s1">r</span><span class="pl-kos">.</span><span class="pl-c1">A_H</span><span class="pl-kos">)</span><span class="pl-kos">)</span>
    <span class="pl-kos">.</span><span class="pl-en">map</span><span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-s1">r</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-c1">JSON</span><span class="pl-kos">.</span><span class="pl-en">parse</span><span class="pl-kos">(</span><span class="pl-s1">r</span><span class="pl-kos">.</span><span class="pl-c1">Stats</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">pm2_5</span> <span class="pl-c1">=</span> <span class="pl-s1">stats</span><span class="pl-kos">.</span><span class="pl-en">reduce</span><span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-s1">acc</span><span class="pl-kos">,</span> <span class="pl-kos">{</span> v <span class="pl-kos">}</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">acc</span> <span class="pl-c1">+</span> <span class="pl-s1">v</span><span class="pl-kos">,</span> <span class="pl-c1">0</span><span class="pl-kos">)</span> <span class="pl-c1">/</span> <span class="pl-s1">stats</span><span class="pl-kos">.</span><span class="pl-c1">length</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">trend</span> <span class="pl-c1">=</span> <span class="pl-s1">stats</span><span class="pl-kos">[</span><span class="pl-c1">0</span><span class="pl-kos">]</span><span class="pl-kos">.</span><span class="pl-c1">v1</span> <span class="pl-c1">-</span> <span class="pl-s1">stats</span><span class="pl-kos">[</span><span class="pl-c1">0</span><span class="pl-kos">]</span><span class="pl-kos">.</span><span class="pl-c1">v3</span><span class="pl-kos">;</span>
  <span class="pl-k">return</span> <span class="pl-kos">{</span>
    <span class="pl-c1">current</span>: <span class="pl-en">aqiFromPm</span><span class="pl-kos">(</span><span class="pl-s1">pm2_5</span><span class="pl-kos">)</span><span class="pl-kos">,</span>
    <span class="pl-c1">trend</span>: <span class="pl-v">Math</span><span class="pl-kos">.</span><span class="pl-en">abs</span><span class="pl-kos">(</span><span class="pl-s1">trend</span><span class="pl-kos">)</span> <span class="pl-c1">&gt;</span> <span class="pl-c1">5</span> ? <span class="pl-s1">trend</span> : <span class="pl-c1">0</span><span class="pl-kos">,</span>
    <span class="pl-c1">details</span>: <span class="pl-s1">json</span><span class="pl-kos">,</span>
  <span class="pl-kos">}</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span>

<span class="pl-k">function</span> <span class="pl-en">aqiFromPm</span><span class="pl-kos">(</span><span class="pl-s1">pm</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
  <span class="pl-k">const</span> <span class="pl-s1">table</span> <span class="pl-c1">=</span> <span class="pl-kos">[</span>
    <span class="pl-kos">[</span><span class="pl-c1">0.0</span><span class="pl-kos">,</span> <span class="pl-c1">12.0</span><span class="pl-kos">,</span> <span class="pl-c1">0</span><span class="pl-kos">,</span> <span class="pl-c1">50</span><span class="pl-kos">]</span><span class="pl-kos">,</span>
    <span class="pl-kos">[</span><span class="pl-c1">12.1</span><span class="pl-kos">,</span> <span class="pl-c1">35.4</span><span class="pl-kos">,</span> <span class="pl-c1">51</span><span class="pl-kos">,</span> <span class="pl-c1">100</span><span class="pl-kos">]</span><span class="pl-kos">,</span>
    <span class="pl-kos">[</span><span class="pl-c1">35.5</span><span class="pl-kos">,</span> <span class="pl-c1">55.4</span><span class="pl-kos">,</span> <span class="pl-c1">101</span><span class="pl-kos">,</span> <span class="pl-c1">150</span><span class="pl-kos">]</span><span class="pl-kos">,</span>
    <span class="pl-kos">[</span><span class="pl-c1">55.5</span><span class="pl-kos">,</span> <span class="pl-c1">150.4</span><span class="pl-kos">,</span> <span class="pl-c1">151</span><span class="pl-kos">,</span> <span class="pl-c1">200</span><span class="pl-kos">]</span><span class="pl-kos">,</span>
    <span class="pl-kos">[</span><span class="pl-c1">150.5</span><span class="pl-kos">,</span> <span class="pl-c1">250.4</span><span class="pl-kos">,</span> <span class="pl-c1">201</span><span class="pl-kos">,</span> <span class="pl-c1">300</span><span class="pl-kos">]</span><span class="pl-kos">,</span>
    <span class="pl-kos">[</span><span class="pl-c1">250.5</span><span class="pl-kos">,</span> <span class="pl-c1">500.4</span><span class="pl-kos">,</span> <span class="pl-c1">301</span><span class="pl-kos">,</span> <span class="pl-c1">500</span><span class="pl-kos">]</span><span class="pl-kos">,</span>
  <span class="pl-kos">]</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-en">computeAqi</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-s1">concI</span><span class="pl-kos">,</span> <span class="pl-kos">[</span><span class="pl-s1">concLo</span><span class="pl-kos">,</span> <span class="pl-s1">concHi</span><span class="pl-kos">,</span> <span class="pl-s1">aqiLo</span><span class="pl-kos">,</span> <span class="pl-s1">aqiHi</span><span class="pl-kos">]</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span>
    <span class="pl-v">Math</span><span class="pl-kos">.</span><span class="pl-en">round</span><span class="pl-kos">(</span>
      <span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-s1">concI</span> <span class="pl-c1">-</span> <span class="pl-s1">concLo</span><span class="pl-kos">)</span> <span class="pl-c1">/</span> <span class="pl-kos">(</span><span class="pl-s1">concHi</span> <span class="pl-c1">-</span> <span class="pl-s1">concLo</span><span class="pl-kos">)</span><span class="pl-kos">)</span> <span class="pl-c1">*</span> <span class="pl-kos">(</span><span class="pl-s1">aqiHi</span> <span class="pl-c1">-</span> <span class="pl-s1">aqiLo</span><span class="pl-kos">)</span> <span class="pl-c1">+</span> <span class="pl-s1">aqiLo</span>
    <span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">values</span> <span class="pl-c1">=</span> <span class="pl-s1">table</span><span class="pl-kos">.</span><span class="pl-en">find</span><span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-kos">[</span><span class="pl-s1">concLo</span><span class="pl-kos">,</span> <span class="pl-s1">concHi</span><span class="pl-kos">,</span> <span class="pl-s1">aqiLo</span><span class="pl-kos">,</span> <span class="pl-s1">aqiHi</span><span class="pl-kos">]</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">pm</span> <span class="pl-c1">&lt;=</span> <span class="pl-s1">concHi</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">return</span> <span class="pl-s1">values</span> ? <span class="pl-en">computeAqi</span><span class="pl-kos">(</span><span class="pl-s1">pm</span><span class="pl-kos">,</span> <span class="pl-s1">values</span><span class="pl-kos">)</span> : <span class="pl-c1">500</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span></pre></div>
<p>I used this to build a simple demo at <a href="https://til.simonwillison.net/tools/aqi" rel="nofollow">https://til.simonwillison.net/tools/aqi</a></p>


path: electron_electron-debugger-console.md
topic: electron
title: Using the Chrome DevTools console as a REPL for an Electron app
url: https://github.com/simonw/til/blob/main/electron/electron-debugger-console.md
created: 2021-08-31T14:09:41-07:00
created_utc: 2021-08-31T21:09:41+00:00
updated: 2021-08-31T14:09:41-07:00
updated_utc: 2021-08-31T21:09:41+00:00
shot_hash: a7c80b899e1517f7958dcac1820cbeca
slug: electron-debugger-console
body:
I figured out how to use the Chrome DevTools to execute JavaScript interactively inside the Electron main process. I always like having a REPL for exploring APIs, and this means I can explore the Electron and Node.js APIs interactively.

<img width="945" alt="Simon_Willison’s_Weblog_and_DevTools_-_Node_js_and_Inspect_with_Chrome_Developer_Tools" src="https://user-images.githubusercontent.com/9599/131575749-a509c528-6746-42b0-8efd-03cd77f6dc2d.png">

https://www.electronjs.org/docs/tutorial/debugging-main-process#--inspectport says you need to run:

    electron --inspect=5858 your/app

I start Electron by running `npm start`, so I modified my `package.json` to include this:

```json
  "scripts": {
    "start": "electron --inspect=5858 ."
```
Then I ran `npm start`.

To connect the debugger, open Google Chrome and visit `chrome://inspect/` - then click the "Open dedicated DevTools for Node" link.

In that window, select the "Connection" tab and add a connection to `localhost:5858`:

<img width="901" alt="8_31_21__2_08_PM" src="https://user-images.githubusercontent.com/9599/131576143-03b28fd7-fab4-495a-8060-662b0247eabd.png">

Switch back to the "Console" tab and you can start interacting with the Electron environment.

I tried this and it worked:

```javascript
const { app, Menu, BrowserWindow, dialog } = require("electron");
new BrowserWindow({height: 100, width: 100}).loadURL("https://simonwillison.net/");
```

html:
<p>I figured out how to use the Chrome DevTools to execute JavaScript interactively inside the Electron main process. I always like having a REPL for exploring APIs, and this means I can explore the Electron and Node.js APIs interactively.</p>
<p><a href="https://user-images.githubusercontent.com/9599/131575749-a509c528-6746-42b0-8efd-03cd77f6dc2d.png" target="_blank" rel="nofollow"><img width="945" alt="Simon_Willison’s_Weblog_and_DevTools_-_Node_js_and_Inspect_with_Chrome_Developer_Tools" src="https://user-images.githubusercontent.com/9599/131575749-a509c528-6746-42b0-8efd-03cd77f6dc2d.png" style="max-width:100%;"></a></p>
<p><a href="https://www.electronjs.org/docs/tutorial/debugging-main-process#--inspectport" rel="nofollow">https://www.electronjs.org/docs/tutorial/debugging-main-process#--inspectport</a> says you need to run:</p>
<pre><code>electron --inspect=5858 your/app
</code></pre>
<p>I start Electron by running <code>npm start</code>, so I modified my <code>package.json</code> to include this:</p>
<div class="highlight highlight-source-json"><pre>  <span class="pl-s"><span class="pl-pds">"</span>scripts<span class="pl-pds">"</span></span>: {
    <span class="pl-s"><span class="pl-pds">"</span>start<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>electron --inspect=5858 .<span class="pl-pds">"</span></span></pre></div>
<p>Then I ran <code>npm start</code>.</p>
<p>To connect the debugger, open Google Chrome and visit <code>chrome://inspect/</code> - then click the "Open dedicated DevTools for Node" link.</p>
<p>In that window, select the "Connection" tab and add a connection to <code>localhost:5858</code>:</p>
<p><a href="https://user-images.githubusercontent.com/9599/131576143-03b28fd7-fab4-495a-8060-662b0247eabd.png" target="_blank" rel="nofollow"><img width="901" alt="8_31_21__2_08_PM" src="https://user-images.githubusercontent.com/9599/131576143-03b28fd7-fab4-495a-8060-662b0247eabd.png" style="max-width:100%;"></a></p>
<p>Switch back to the "Console" tab and you can start interacting with the Electron environment.</p>
<p>I tried this and it worked:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">const</span> <span class="pl-kos">{</span> app<span class="pl-kos">,</span> Menu<span class="pl-kos">,</span> BrowserWindow<span class="pl-kos">,</span> dialog <span class="pl-kos">}</span> <span class="pl-c1">=</span> <span class="pl-en">require</span><span class="pl-kos">(</span><span class="pl-s">"electron"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-k">new</span> <span class="pl-v">BrowserWindow</span><span class="pl-kos">(</span><span class="pl-kos">{</span><span class="pl-c1">height</span>: <span class="pl-c1">100</span><span class="pl-kos">,</span> <span class="pl-c1">width</span>: <span class="pl-c1">100</span><span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">loadURL</span><span class="pl-kos">(</span><span class="pl-s">"https://simonwillison.net/"</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>


path: electron_electron-external-links-system-browser.md
topic: electron
title: Open external links in an Electron app using the system browser
url: https://github.com/simonw/til/blob/main/electron/electron-external-links-system-browser.md
created: 2021-09-02T14:15:19-07:00
created_utc: 2021-09-02T21:15:19+00:00
updated: 2021-09-02T14:15:19-07:00
updated_utc: 2021-09-02T21:15:19+00:00
shot_hash: 5d14045ddef5563541eefa461db1f283
slug: electron-external-links-system-browser
body:
For [Datasette.app](https://github.com/simonw/datasette-app) I wanted to ensure that links to external URLs would [open in the system browser](https://github.com/simonw/datasette-app/issues/34).

This recipe works:

```javascript
function postConfigure(window) {
  window.webContents.on("will-navigate", function (event, reqUrl) {
    let requestedHost = new URL(reqUrl).host;
    let currentHost = new URL(window.webContents.getURL()).host;
    if (requestedHost && requestedHost != currentHost) {
      event.preventDefault();
      shell.openExternal(reqUrl);
    }
  });
}
```
The `will-navigate` event fires before any in-browser navigations, which means they can be intercepted and cancelled if necessary.

I use the `URL()` class to extract the `.host` so I can check if the host being navigated to differs from the host that the application is running against (which is probably `localhost:$port`).

Initially I was using `require('url').URL` for this but that doesn't appear to be necessary - Node.js ships with `URL` as a top-level class these days.

`event.preventDefault()` cancels the navigation and `shell.openExternal(reqUrl)` opens the URL using the system default browsner.

I call this function on any new window I create using `new BrowserWindow` - for example:

```javascript
mainWindow = new BrowserWindow({
  width: 800,
  height: 600,
  show: false,
});
mainWindow.loadFile("loading.html");
mainWindow.once("ready-to-show", () => {
  mainWindow.show();
});
postConfigure(mainWindow);
```

html:
<p>For <a href="https://github.com/simonw/datasette-app">Datasette.app</a> I wanted to ensure that links to external URLs would <a href="https://github.com/simonw/datasette-app/issues/34">open in the system browser</a>.</p>
<p>This recipe works:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">function</span> <span class="pl-en">postConfigure</span><span class="pl-kos">(</span><span class="pl-s1">window</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
  <span class="pl-s1">window</span><span class="pl-kos">.</span><span class="pl-c1">webContents</span><span class="pl-kos">.</span><span class="pl-en">on</span><span class="pl-kos">(</span><span class="pl-s">"will-navigate"</span><span class="pl-kos">,</span> <span class="pl-k">function</span> <span class="pl-kos">(</span><span class="pl-s1">event</span><span class="pl-kos">,</span> <span class="pl-s1">reqUrl</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-k">let</span> <span class="pl-s1">requestedHost</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-c1">URL</span><span class="pl-kos">(</span><span class="pl-s1">reqUrl</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-c1">host</span><span class="pl-kos">;</span>
    <span class="pl-k">let</span> <span class="pl-s1">currentHost</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-c1">URL</span><span class="pl-kos">(</span><span class="pl-s1">window</span><span class="pl-kos">.</span><span class="pl-c1">webContents</span><span class="pl-kos">.</span><span class="pl-en">getURL</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-c1">host</span><span class="pl-kos">;</span>
    <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">requestedHost</span> <span class="pl-c1">&amp;&amp;</span> <span class="pl-s1">requestedHost</span> <span class="pl-c1">!=</span> <span class="pl-s1">currentHost</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
      <span class="pl-s1">event</span><span class="pl-kos">.</span><span class="pl-en">preventDefault</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-s1">shell</span><span class="pl-kos">.</span><span class="pl-en">openExternal</span><span class="pl-kos">(</span><span class="pl-s1">reqUrl</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span>
  <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span></pre></div>
<p>The <code>will-navigate</code> event fires before any in-browser navigations, which means they can be intercepted and cancelled if necessary.</p>
<p>I use the <code>URL()</code> class to extract the <code>.host</code> so I can check if the host being navigated to differs from the host that the application is running against (which is probably <code>localhost:$port</code>).</p>
<p>Initially I was using <code>require('url').URL</code> for this but that doesn't appear to be necessary - Node.js ships with <code>URL</code> as a top-level class these days.</p>
<p><code>event.preventDefault()</code> cancels the navigation and <code>shell.openExternal(reqUrl)</code> opens the URL using the system default browsner.</p>
<p>I call this function on any new window I create using <code>new BrowserWindow</code> - for example:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-s1">mainWindow</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">BrowserWindow</span><span class="pl-kos">(</span><span class="pl-kos">{</span>
  <span class="pl-c1">width</span>: <span class="pl-c1">800</span><span class="pl-kos">,</span>
  <span class="pl-c1">height</span>: <span class="pl-c1">600</span><span class="pl-kos">,</span>
  <span class="pl-c1">show</span>: <span class="pl-c1">false</span><span class="pl-kos">,</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-s1">mainWindow</span><span class="pl-kos">.</span><span class="pl-en">loadFile</span><span class="pl-kos">(</span><span class="pl-s">"loading.html"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-s1">mainWindow</span><span class="pl-kos">.</span><span class="pl-en">once</span><span class="pl-kos">(</span><span class="pl-s">"ready-to-show"</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
  <span class="pl-s1">mainWindow</span><span class="pl-kos">.</span><span class="pl-en">show</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-en">postConfigure</span><span class="pl-kos">(</span><span class="pl-s1">mainWindow</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>


path: github-actions_attach-generated-file-to-release.md
topic: github-actions
title: Attaching a generated file to a GitHub release using Actions
url: https://github.com/simonw/til/blob/main/github-actions/attach-generated-file-to-release.md
created: 2021-09-07T22:04:28-07:00
created_utc: 2021-09-08T05:04:28+00:00
updated: 2021-09-07T22:04:28-07:00
updated_utc: 2021-09-08T05:04:28+00:00
shot_hash: d3d35e3e7c1982434b617edf7ebb8060
slug: attach-generated-file-to-release
body:
For [Datasette Desktop](https://github.com/simonw/datasette-app) I wanted to run an action which, when I created a release, would build an asset for that release and then upload and attach it.

I triggered my action on the creation of a new release, like so:

```yaml
on:
  release:
    types: [created]
```

Assuming previous steps that create a file called `app.zip` in the root of the checkout, here's the final action step which worked for me:

```yaml
      - name: Upload release attachment
        uses: actions/github-script@v4
        with:
          script: |
            const fs = require('fs');
            const tag = context.ref.replace("refs/tags/", "");
            // Get release for this tag
            const release = await github.repos.getReleaseByTag({
              owner: context.repo.owner,
              repo: context.repo.repo,
              tag
            });
            // Upload the release asset
            await github.repos.uploadReleaseAsset({
              owner: context.repo.owner,
              repo: context.repo.repo,
              release_id: release.data.id,
              name: "app.zip",
              data: await fs.readFileSync("app.zip")
            });
```
It uses [actions/github-script](https://github.com/actions/github-script) which provides a pre-configured [octokit/rest.js](https://octokit.github.io/rest.js/) client object.

The `uploadReleaseAsset()` method needs the `owner`, `repo`, `release_id`, `name` (filename) and the file data.

These are mostly available, with the exception of `release_id`. That can be derived for the current release based on the `context.ref` value - strip that down to just the tag, then use `getReleaseByTag()` to get a release object. `release.data.id` will then be the numeric release ID.

My full workflow is at https://github.com/simonw/datasette-app/blob/0.1.0/.github/workflows/release.yml

html:
<p>For <a href="https://github.com/simonw/datasette-app">Datasette Desktop</a> I wanted to run an action which, when I created a release, would build an asset for that release and then upload and attach it.</p>
<p>I triggered my action on the creation of a new release, like so:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">on</span>:
  <span class="pl-ent">release</span>:
    <span class="pl-ent">types</span>: <span class="pl-s">[created]</span></pre></div>
<p>Assuming previous steps that create a file called <code>app.zip</code> in the root of the checkout, here's the final action step which worked for me:</p>
<div class="highlight highlight-source-yaml"><pre>      - <span class="pl-ent">name</span>: <span class="pl-s">Upload release attachment</span>
        <span class="pl-ent">uses</span>: <span class="pl-s">actions/github-script@v4</span>
        <span class="pl-ent">with</span>:
          <span class="pl-ent">script</span>: <span class="pl-s">|</span>
<span class="pl-s">            const fs = require('fs');</span>
<span class="pl-s">            const tag = context.ref.replace("refs/tags/", "");</span>
<span class="pl-s">            // Get release for this tag</span>
<span class="pl-s">            const release = await github.repos.getReleaseByTag({</span>
<span class="pl-s">              owner: context.repo.owner,</span>
<span class="pl-s">              repo: context.repo.repo,</span>
<span class="pl-s">              tag</span>
<span class="pl-s">            });</span>
<span class="pl-s">            // Upload the release asset</span>
<span class="pl-s">            await github.repos.uploadReleaseAsset({</span>
<span class="pl-s">              owner: context.repo.owner,</span>
<span class="pl-s">              repo: context.repo.repo,</span>
<span class="pl-s">              release_id: release.data.id,</span>
<span class="pl-s">              name: "app.zip",</span>
<span class="pl-s">              data: await fs.readFileSync("app.zip")</span>
<span class="pl-s">            });</span></pre></div>
<p>It uses <a href="https://github.com/actions/github-script">actions/github-script</a> which provides a pre-configured <a href="https://octokit.github.io/rest.js/" rel="nofollow">octokit/rest.js</a> client object.</p>
<p>The <code>uploadReleaseAsset()</code> method needs the <code>owner</code>, <code>repo</code>, <code>release_id</code>, <code>name</code> (filename) and the file data.</p>
<p>These are mostly available, with the exception of <code>release_id</code>. That can be derived for the current release based on the <code>context.ref</code> value - strip that down to just the tag, then use <code>getReleaseByTag()</code> to get a release object. <code>release.data.id</code> will then be the numeric release ID.</p>
<p>My full workflow is at <a href="https://github.com/simonw/datasette-app/blob/0.1.0/.github/workflows/release.yml">https://github.com/simonw/datasette-app/blob/0.1.0/.github/workflows/release.yml</a></p>


path: electron_sign-notarize-electron-macos.md
topic: electron
title: Signing and notarizing an Electron app for distribution using GitHub Actions
url: https://github.com/simonw/til/blob/main/electron/sign-notarize-electron-macos.md
created: 2021-09-08T10:41:46-07:00
created_utc: 2021-09-08T17:41:46+00:00
updated: 2021-09-08T10:41:46-07:00
updated_utc: 2021-09-08T17:41:46+00:00
shot_hash: 6882184d2acaa5b137e3e52a7f9feda2
slug: sign-notarize-electron-macos
body:
I had to figure this out for [Datasette Desktop](https://github.com/simonw/datasette-app).

## Pay for an Apple Developer account

First step is to pay $99/year for an [Apple Developer](https://developer.apple.com/) account.

I had a previous (expired) account with a UK address, and changing to a USA address required a support ticket - so instead I created a brand new Apple ID specifically for the developer account.

Since a later stage here involves storing the account password in a GitHub repository secret, I think this is a better way to go: I don't like the idea of my personal Apple ID account password being needed by anyone else who should be able to sign my application.

## Generate a Certificate Signing Request

First you need to generate a Certificate Signing Request using Keychain Access on a Mac - I was unable to figure out how to do this on the command-line.

Quoting https://help.apple.com/developer-account/#/devbfa00fef7:

> 1.  Launch Keychain Access located in `/Applications/Utilities`.
> 2.  Choose Keychain Access > Certificate Assistant > Request a Certificate from a Certificate Authority.
> 3.  In the Certificate Assistant dialog, enter an email address in the User Email Address field.
> 4.  In the Common Name field, enter a name for the key (for example, Gita Kumar Dev Key).
> 5.  Leave the CA Email Address field empty.
> 6.  Choose "Saved to disk", and click Continue.

This produces a `CertificateSigningRequest.certSigningRequest` file. Save that somewhere sensible.

## Creating a Developer ID Application certificate

The certificate needed is for a "Developer ID Application" - so select that option from the list of options on https://developer.apple.com/account/resources/certificates/add

Upload the `CertificateSigningRequest.certSigningRequest` file, and Apple should provide you a `developerID_application.cer` to download.

## Export it as a .p12 file

The final signing step requires a `.p12` file. It took me quite a while to figure out how to create this - in the end what worked for me was this:

1. Double-click the `developerID_application.cer` file and import it into my login keychain
2. In Keychain Access open the "My Certificates" pane
3. Select the "Developer ID Application: ..." certificate and the Private Key below it (created when generating the certificate signing request)
4. Right click and select "Export 2 items..."

![Screenshot of the Keynote export interface](https://user-images.githubusercontent.com/9599/132558174-c90410a7-8548-4642-a717-0b470788a5ea.png)

I saved the resulting file as `Developer-ID-Application-Certificates.p12`. It asked me to set a password, so I generated and saved a random one in 1Password.

## Building a signed copy of the application

At this point I turned to [electron-builder](https://www.electron.build/) to do the rest of the work. I installed it with:

    npm install electron-builder --save-dev

I added `"dist": "electron-builder --publish never"` to my `"scripts"` block in `package.json`.

Then I ran the following:

    CSC_KEY_PASSWORD=... \
      CSC_LINK=$(openssl base64 -in Developer-ID-Application-Certificates.p12) \
    npm run dist

The `CSC_KEY_PASSWORD` was the password I set earlier when I exported the certificate.

That `CSC_LINK` variable is set to the base64 encoded version of the certificate file. You can also pass the file itself, but I would need the base64 option later to work with GitHub actions.

This worked! It generated a signed `Datasette.app` package.

... which wasn't quite enough. It still wouldn't open without complaints on another machine until I had got it notarized.

## Notarizing the application

Notarizing involves uploading the application bundle to Apple's servers, where they run some automatic scans against it before returning a notarization ticket that can be "stapled" to the binary.

Thankfully [electron-notarize](https://github.com/electron/electron-notarize) does most of the work here, so I installed that:

    npm install electron-notarize --save-dev

I then went through an iteration cycle of trying out different combinations of settings until it finally worked.

I'll describe my finished configuration.

I have a file in `build/entitlements.mac.plist` containing the following:

```xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>com.apple.security.cs.allow-dyld-environment-variables</key>
    <true/>
    <key>com.apple.security.cs.disable-library-validation</key>
    <true/>
    <key>com.apple.security.cs.allow-jit</key>
    <true/>
    <key>com.apple.security.cs.allow-unsigned-executable-memory</key>
    <true/>
    <key>com.apple.security.cs.debugger</key>
    <true/>
    <key>com.apple.security.network.client</key>
    <true/>
    <key>com.apple.security.network.server</key>
    <true/>
    <key>com.apple.security.files.user-selected.read-only</key>
    <true/>
    <key>com.apple.security.inherit</key>
    <true/>
    <key>com.apple.security.automation.apple-events</key>
    <true/>
  </dict>
</plist>
```
The possible entitlements are [documented here](
https://developer.apple.com/library/archive/documentation/Miscellaneous/Reference/EntitlementKeyReference/Chapters/EnablingAppSandbox.html). I don't fully understand these ones, but they are what I got to after multiple rounds of experimentation.

I have a `scripts/notarize.js` file containing this (based on [Notarizing your Electron application](https://kilianvalkhof.com/2019/electron/notarizing-your-electron-application/) by 
Kilian Valkhof):

```javascript
/* Based on https://kilianvalkhof.com/2019/electron/notarizing-your-electron-application/ */

const { notarize } = require("electron-notarize");

exports.default = async function notarizing(context) {
  const { electronPlatformName, appOutDir } = context;
  if (electronPlatformName !== "darwin") {
    return;
  }

  const appName = context.packager.appInfo.productFilename;

  return await notarize({
    appBundleId: "io.datasette.app",
    appPath: `${appOutDir}/${appName}.app`,
    appleId: process.env.APPLEID,
    appleIdPassword: process.env.APPLEIDPASS,
  });
};
```
The `"build"` section of my `package.json` looks like this:

```json
  "build": {
    "appId": "io.datasette.app",
    "mac": {
      "category": "public.app-category.developer-tools",
      "hardenedRuntime" : true,
      "gatekeeperAssess": false,
      "entitlements": "build/entitlements.mac.plist",
      "entitlementsInherit": "build/entitlements.mac.plist",
      "binaries": [
        "./dist/mac/Datasette.app/Contents/Resources/python/bin/python3.9",
        "./dist/mac/Datasette.app/Contents/Resources/python/lib/python3.9/lib-dynload/xxlimited.cpython-39-darwin.so",
        "./dist/mac/Datasette.app/Contents/Resources/python/lib/python3.9/lib-dynload/_testcapi.cpython-39-darwin.so"
      ]
    },
    "afterSign": "scripts/notarize.js",
    "extraResources": [
      {
        "from": "python",
        "to": "python",
        "filter": [
          "**/*"
        ]
      }
    ]
  }
```
Again, I got here through a process of iteration - in particular, my application bundles a full copy of Python so I had to specify some additional binaries and `extraResources` - most applications will not need to do that.

Note that the `scripts/notarize.js` file uses two extra environment variables: `APPLEID` and `APPLEIDPASS`. These are the account credentials for my Apple Developer account's Apple ID.

(I also encountered an error `xcrun: error: unable to find utility "altool", not a developer tool or in PATH` - I resolved that by running `sudo xcode-select --reset`.)

## Creating an app-specific password

Another error I encountered was this one:

> Please sign in with an app-specific password. You can create one at appleid.apple.com

These can be created in the "Security" section of https://appleid.apple.com/account/home - I created one called "Notarize Apps" which I set as the `APPLEIDPASS` environment variable.

## Creating a signed and notarized build

With all of the above in place, creating a build on my laptop looked like this:
```
APPLEID=my-dedicated-appleid \
   APPLEIDPASS=app-specific-password \
   CSC_KEY_PASSWORD=key-password \
   CSC_LINK=$(openssl base64 -in Developer-ID-Application-Certificates.p12) \
   npm run dist
```
This worked! It produced a `Datasette.app` package which I could zip up, distribute to another machine, unzip and install - and it then opened without the terrifying security warning.

## Automating it all with GitHub Actions

I decided to build and notarize on _every push_ to my repository, so I could save the resulting build as an artifact and install any in-progress work on a computer to test it.

Apple [limit you to 75 notarizations a day](https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/customizing_the_notarization_workflow#3561440) so I think this is OK for my projects.

My full [test.yml](https://github.com/simonw/datasette-app/blob/0.1.0/.github/workflows/test.yml) looks like this:

```yaml
name: Test

on: push

jobs:
  test:
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v2
      - name: Configure Node caching
        uses: actions/cache@v2
        env:
          cache-name: cache-node-modules
        with:
          path: ~/.npm
          key: ${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-build-${{ env.cache-name }}-
            ${{ runner.os }}-build-
            ${{ runner.os }}-
      - uses: actions/cache@v2
        name: Configure pip caching
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
          restore-keys: |
            ${{ runner.os }}-pip-
      - name: Install Node dependencies
        run: npm install
      - name: Download standalone Python
        run: |
          ./download-python.sh
      - name: Run tests
        run: npm test
        timeout-minutes: 5
      - name: Build distribution
        env:
          CSC_KEY_PASSWORD: ${{ secrets.CSC_KEY_PASSWORD }}
          CSC_LINK: ${{ secrets.CSC_LINK }}
          APPLEID: ${{ secrets.APPLEID }}
          APPLEIDPASS: ${{ secrets.APPLEIDPASS }}
        run: npm run dist
      - name: Create zip file
        run: |
          cd dist/mac
          ditto -c -k --keepParent Datasette.app Datasette.app.zip
      - name: And a README (to work around GitHub double-zips)
        run: |
          echo "More information: https://datasette.io" > dist/mac/README.txt
      - name: Upload artifact
        uses: actions/upload-artifact@v2
        with:
          name: Datasette-macOS
          path: |
            dist/mac/Datasette.app.zip
            dist/mac/README.txt
```
The key stuff here is the "Build distribution" step. It sets four values that I have saved on the repository as secrets: `CSC_KEY_PASSWORD`, `CSC_LINK`, `APPLEID` and `APPLEIDPASS`.

The `CSC_LINK` variable is the base64-encoded contents of my `Developer-ID-Application-Certificates.p12` file. I generated that like so:

    openssl base64 -in developerID_application.cer

I have [a separate release.yml](https://github.com/simonw/datasette-app/blob/0.1.0/.github/workflows/release.yml) for building tagged releases, described in [this TIL](https://til.simonwillison.net/github-actions/attach-generated-file-to-release).

## The finished configuration

You can browse the code in [my 0.1.0 tag](https://github.com/simonw/datasette-app/tree/0.1.0) to see all of these parts in their final configuration, as used to create the 0.1.0 initial release of my application.

The original issue threads in which I figured this stuff out are:

- [Get an Apple developer certificate #45](https://github.com/simonw/datasette-app/issues/45)
- [Work out how to notarize the macOS application #50](https://github.com/simonw/datasette-app/issues/50)
- [GitHub Actions workflow for creating packages for releases #51](https://github.com/simonw/datasette-app/issues/51)

html:
<p>I had to figure this out for <a href="https://github.com/simonw/datasette-app">Datasette Desktop</a>.</p>
<h2>
<a id="user-content-pay-for-an-apple-developer-account" class="anchor" href="#pay-for-an-apple-developer-account" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Pay for an Apple Developer account</h2>
<p>First step is to pay $99/year for an <a href="https://developer.apple.com/" rel="nofollow">Apple Developer</a> account.</p>
<p>I had a previous (expired) account with a UK address, and changing to a USA address required a support ticket - so instead I created a brand new Apple ID specifically for the developer account.</p>
<p>Since a later stage here involves storing the account password in a GitHub repository secret, I think this is a better way to go: I don't like the idea of my personal Apple ID account password being needed by anyone else who should be able to sign my application.</p>
<h2>
<a id="user-content-generate-a-certificate-signing-request" class="anchor" href="#generate-a-certificate-signing-request" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Generate a Certificate Signing Request</h2>
<p>First you need to generate a Certificate Signing Request using Keychain Access on a Mac - I was unable to figure out how to do this on the command-line.</p>
<p>Quoting <a href="https://help.apple.com/developer-account/#/devbfa00fef7" rel="nofollow">https://help.apple.com/developer-account/#/devbfa00fef7</a>:</p>
<blockquote>
<ol>
<li>Launch Keychain Access located in <code>/Applications/Utilities</code>.</li>
<li>Choose Keychain Access &gt; Certificate Assistant &gt; Request a Certificate from a Certificate Authority.</li>
<li>In the Certificate Assistant dialog, enter an email address in the User Email Address field.</li>
<li>In the Common Name field, enter a name for the key (for example, Gita Kumar Dev Key).</li>
<li>Leave the CA Email Address field empty.</li>
<li>Choose "Saved to disk", and click Continue.</li>
</ol>
</blockquote>
<p>This produces a <code>CertificateSigningRequest.certSigningRequest</code> file. Save that somewhere sensible.</p>
<h2>
<a id="user-content-creating-a-developer-id-application-certificate" class="anchor" href="#creating-a-developer-id-application-certificate" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating a Developer ID Application certificate</h2>
<p>The certificate needed is for a "Developer ID Application" - so select that option from the list of options on <a href="https://developer.apple.com/account/resources/certificates/add" rel="nofollow">https://developer.apple.com/account/resources/certificates/add</a></p>
<p>Upload the <code>CertificateSigningRequest.certSigningRequest</code> file, and Apple should provide you a <code>developerID_application.cer</code> to download.</p>
<h2>
<a id="user-content-export-it-as-a-p12-file" class="anchor" href="#export-it-as-a-p12-file" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Export it as a .p12 file</h2>
<p>The final signing step requires a <code>.p12</code> file. It took me quite a while to figure out how to create this - in the end what worked for me was this:</p>
<ol>
<li>Double-click the <code>developerID_application.cer</code> file and import it into my login keychain</li>
<li>In Keychain Access open the "My Certificates" pane</li>
<li>Select the "Developer ID Application: ..." certificate and the Private Key below it (created when generating the certificate signing request)</li>
<li>Right click and select "Export 2 items..."</li>
</ol>
<p><a href="https://user-images.githubusercontent.com/9599/132558174-c90410a7-8548-4642-a717-0b470788a5ea.png" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/132558174-c90410a7-8548-4642-a717-0b470788a5ea.png" alt="Screenshot of the Keynote export interface" style="max-width:100%;"></a></p>
<p>I saved the resulting file as <code>Developer-ID-Application-Certificates.p12</code>. It asked me to set a password, so I generated and saved a random one in 1Password.</p>
<h2>
<a id="user-content-building-a-signed-copy-of-the-application" class="anchor" href="#building-a-signed-copy-of-the-application" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Building a signed copy of the application</h2>
<p>At this point I turned to <a href="https://www.electron.build/" rel="nofollow">electron-builder</a> to do the rest of the work. I installed it with:</p>
<pre><code>npm install electron-builder --save-dev
</code></pre>
<p>I added <code>"dist": "electron-builder --publish never"</code> to my <code>"scripts"</code> block in <code>package.json</code>.</p>
<p>Then I ran the following:</p>
<pre><code>CSC_KEY_PASSWORD=... \
  CSC_LINK=$(openssl base64 -in Developer-ID-Application-Certificates.p12) \
npm run dist
</code></pre>
<p>The <code>CSC_KEY_PASSWORD</code> was the password I set earlier when I exported the certificate.</p>
<p>That <code>CSC_LINK</code> variable is set to the base64 encoded version of the certificate file. You can also pass the file itself, but I would need the base64 option later to work with GitHub actions.</p>
<p>This worked! It generated a signed <code>Datasette.app</code> package.</p>
<p>... which wasn't quite enough. It still wouldn't open without complaints on another machine until I had got it notarized.</p>
<h2>
<a id="user-content-notarizing-the-application" class="anchor" href="#notarizing-the-application" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Notarizing the application</h2>
<p>Notarizing involves uploading the application bundle to Apple's servers, where they run some automatic scans against it before returning a notarization ticket that can be "stapled" to the binary.</p>
<p>Thankfully <a href="https://github.com/electron/electron-notarize">electron-notarize</a> does most of the work here, so I installed that:</p>
<pre><code>npm install electron-notarize --save-dev
</code></pre>
<p>I then went through an iteration cycle of trying out different combinations of settings until it finally worked.</p>
<p>I'll describe my finished configuration.</p>
<p>I have a file in <code>build/entitlements.mac.plist</code> containing the following:</p>
<div class="highlight highlight-text-xml"><pre>&lt;?<span class="pl-ent">xml</span><span class="pl-e"> version</span>=<span class="pl-s"><span class="pl-pds">"</span>1.0<span class="pl-pds">"</span></span><span class="pl-e"> encoding</span>=<span class="pl-s"><span class="pl-pds">"</span>UTF-8<span class="pl-pds">"</span></span>?&gt;
&lt;!<span class="pl-ent">DOCTYPE</span> <span class="pl-e">plist</span> PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"&gt;
&lt;<span class="pl-ent">plist</span> <span class="pl-e">version</span>=<span class="pl-s"><span class="pl-pds">"</span>1.0<span class="pl-pds">"</span></span>&gt;
  &lt;<span class="pl-ent">dict</span>&gt;
    &lt;<span class="pl-ent">key</span>&gt;com.apple.security.cs.allow-dyld-environment-variables&lt;/<span class="pl-ent">key</span>&gt;
    &lt;<span class="pl-ent">true</span>/&gt;
    &lt;<span class="pl-ent">key</span>&gt;com.apple.security.cs.disable-library-validation&lt;/<span class="pl-ent">key</span>&gt;
    &lt;<span class="pl-ent">true</span>/&gt;
    &lt;<span class="pl-ent">key</span>&gt;com.apple.security.cs.allow-jit&lt;/<span class="pl-ent">key</span>&gt;
    &lt;<span class="pl-ent">true</span>/&gt;
    &lt;<span class="pl-ent">key</span>&gt;com.apple.security.cs.allow-unsigned-executable-memory&lt;/<span class="pl-ent">key</span>&gt;
    &lt;<span class="pl-ent">true</span>/&gt;
    &lt;<span class="pl-ent">key</span>&gt;com.apple.security.cs.debugger&lt;/<span class="pl-ent">key</span>&gt;
    &lt;<span class="pl-ent">true</span>/&gt;
    &lt;<span class="pl-ent">key</span>&gt;com.apple.security.network.client&lt;/<span class="pl-ent">key</span>&gt;
    &lt;<span class="pl-ent">true</span>/&gt;
    &lt;<span class="pl-ent">key</span>&gt;com.apple.security.network.server&lt;/<span class="pl-ent">key</span>&gt;
    &lt;<span class="pl-ent">true</span>/&gt;
    &lt;<span class="pl-ent">key</span>&gt;com.apple.security.files.user-selected.read-only&lt;/<span class="pl-ent">key</span>&gt;
    &lt;<span class="pl-ent">true</span>/&gt;
    &lt;<span class="pl-ent">key</span>&gt;com.apple.security.inherit&lt;/<span class="pl-ent">key</span>&gt;
    &lt;<span class="pl-ent">true</span>/&gt;
    &lt;<span class="pl-ent">key</span>&gt;com.apple.security.automation.apple-events&lt;/<span class="pl-ent">key</span>&gt;
    &lt;<span class="pl-ent">true</span>/&gt;
  &lt;/<span class="pl-ent">dict</span>&gt;
&lt;/<span class="pl-ent">plist</span>&gt;</pre></div>
<p>The possible entitlements are <a href="https://developer.apple.com/library/archive/documentation/Miscellaneous/Reference/EntitlementKeyReference/Chapters/EnablingAppSandbox.html" rel="nofollow">documented here</a>. I don't fully understand these ones, but they are what I got to after multiple rounds of experimentation.</p>
<p>I have a <code>scripts/notarize.js</code> file containing this (based on <a href="https://kilianvalkhof.com/2019/electron/notarizing-your-electron-application/" rel="nofollow">Notarizing your Electron application</a> by
Kilian Valkhof):</p>
<div class="highlight highlight-source-js"><pre><span class="pl-c">/* Based on https://kilianvalkhof.com/2019/electron/notarizing-your-electron-application/ */</span>

<span class="pl-k">const</span> <span class="pl-kos">{</span> notarize <span class="pl-kos">}</span> <span class="pl-c1">=</span> <span class="pl-en">require</span><span class="pl-kos">(</span><span class="pl-s">"electron-notarize"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

<span class="pl-s1">exports</span><span class="pl-kos">.</span><span class="pl-en">default</span> <span class="pl-c1">=</span> <span class="pl-k">async</span> <span class="pl-k">function</span> <span class="pl-en">notarizing</span><span class="pl-kos">(</span><span class="pl-s1">context</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
  <span class="pl-k">const</span> <span class="pl-kos">{</span> electronPlatformName<span class="pl-kos">,</span> appOutDir <span class="pl-kos">}</span> <span class="pl-c1">=</span> <span class="pl-s1">context</span><span class="pl-kos">;</span>
  <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">electronPlatformName</span> <span class="pl-c1">!==</span> <span class="pl-s">"darwin"</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-k">return</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span>

  <span class="pl-k">const</span> <span class="pl-s1">appName</span> <span class="pl-c1">=</span> <span class="pl-s1">context</span><span class="pl-kos">.</span><span class="pl-c1">packager</span><span class="pl-kos">.</span><span class="pl-c1">appInfo</span><span class="pl-kos">.</span><span class="pl-c1">productFilename</span><span class="pl-kos">;</span>

  <span class="pl-k">return</span> <span class="pl-k">await</span> <span class="pl-en">notarize</span><span class="pl-kos">(</span><span class="pl-kos">{</span>
    <span class="pl-c1">appBundleId</span>: <span class="pl-s">"io.datasette.app"</span><span class="pl-kos">,</span>
    <span class="pl-c1">appPath</span>: <span class="pl-s">`<span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">appOutDir</span><span class="pl-kos">}</span></span>/<span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">appName</span><span class="pl-kos">}</span></span>.app`</span><span class="pl-kos">,</span>
    <span class="pl-c1">appleId</span>: <span class="pl-s1">process</span><span class="pl-kos">.</span><span class="pl-c1">env</span><span class="pl-kos">.</span><span class="pl-c1">APPLEID</span><span class="pl-kos">,</span>
    <span class="pl-c1">appleIdPassword</span>: <span class="pl-s1">process</span><span class="pl-kos">.</span><span class="pl-c1">env</span><span class="pl-kos">.</span><span class="pl-c1">APPLEIDPASS</span><span class="pl-kos">,</span>
  <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">;</span></pre></div>
<p>The <code>"build"</code> section of my <code>package.json</code> looks like this:</p>
<div class="highlight highlight-source-json"><pre>  <span class="pl-s"><span class="pl-pds">"</span>build<span class="pl-pds">"</span></span>: {
    <span class="pl-s"><span class="pl-pds">"</span>appId<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>io.datasette.app<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>mac<span class="pl-pds">"</span></span>: {
      <span class="pl-s"><span class="pl-pds">"</span>category<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>public.app-category.developer-tools<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>hardenedRuntime<span class="pl-pds">"</span></span> : <span class="pl-c1">true</span>,
      <span class="pl-s"><span class="pl-pds">"</span>gatekeeperAssess<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
      <span class="pl-s"><span class="pl-pds">"</span>entitlements<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>build/entitlements.mac.plist<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>entitlementsInherit<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>build/entitlements.mac.plist<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>binaries<span class="pl-pds">"</span></span>: [
        <span class="pl-s"><span class="pl-pds">"</span>./dist/mac/Datasette.app/Contents/Resources/python/bin/python3.9<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>./dist/mac/Datasette.app/Contents/Resources/python/lib/python3.9/lib-dynload/xxlimited.cpython-39-darwin.so<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>./dist/mac/Datasette.app/Contents/Resources/python/lib/python3.9/lib-dynload/_testcapi.cpython-39-darwin.so<span class="pl-pds">"</span></span>
      ]
    },
    <span class="pl-s"><span class="pl-pds">"</span>afterSign<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>scripts/notarize.js<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>extraResources<span class="pl-pds">"</span></span>: [
      {
        <span class="pl-s"><span class="pl-pds">"</span>from<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>python<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>to<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>python<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>filter<span class="pl-pds">"</span></span>: [
          <span class="pl-s"><span class="pl-pds">"</span>**/*<span class="pl-pds">"</span></span>
        ]
      }
    ]
  }</pre></div>
<p>Again, I got here through a process of iteration - in particular, my application bundles a full copy of Python so I had to specify some additional binaries and <code>extraResources</code> - most applications will not need to do that.</p>
<p>Note that the <code>scripts/notarize.js</code> file uses two extra environment variables: <code>APPLEID</code> and <code>APPLEIDPASS</code>. These are the account credentials for my Apple Developer account's Apple ID.</p>
<p>(I also encountered an error <code>xcrun: error: unable to find utility "altool", not a developer tool or in PATH</code> - I resolved that by running <code>sudo xcode-select --reset</code>.)</p>
<h2>
<a id="user-content-creating-an-app-specific-password" class="anchor" href="#creating-an-app-specific-password" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating an app-specific password</h2>
<p>Another error I encountered was this one:</p>
<blockquote>
<p>Please sign in with an app-specific password. You can create one at appleid.apple.com</p>
</blockquote>
<p>These can be created in the "Security" section of <a href="https://appleid.apple.com/account/home" rel="nofollow">https://appleid.apple.com/account/home</a> - I created one called "Notarize Apps" which I set as the <code>APPLEIDPASS</code> environment variable.</p>
<h2>
<a id="user-content-creating-a-signed-and-notarized-build" class="anchor" href="#creating-a-signed-and-notarized-build" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating a signed and notarized build</h2>
<p>With all of the above in place, creating a build on my laptop looked like this:</p>
<pre><code>APPLEID=my-dedicated-appleid \
   APPLEIDPASS=app-specific-password \
   CSC_KEY_PASSWORD=key-password \
   CSC_LINK=$(openssl base64 -in Developer-ID-Application-Certificates.p12) \
   npm run dist
</code></pre>
<p>This worked! It produced a <code>Datasette.app</code> package which I could zip up, distribute to another machine, unzip and install - and it then opened without the terrifying security warning.</p>
<h2>
<a id="user-content-automating-it-all-with-github-actions" class="anchor" href="#automating-it-all-with-github-actions" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Automating it all with GitHub Actions</h2>
<p>I decided to build and notarize on <em>every push</em> to my repository, so I could save the resulting build as an artifact and install any in-progress work on a computer to test it.</p>
<p>Apple <a href="https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/customizing_the_notarization_workflow#3561440" rel="nofollow">limit you to 75 notarizations a day</a> so I think this is OK for my projects.</p>
<p>My full <a href="https://github.com/simonw/datasette-app/blob/0.1.0/.github/workflows/test.yml">test.yml</a> looks like this:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Test</span>

<span class="pl-ent">on</span>: <span class="pl-s">push</span>

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">test</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">macos-latest</span>
    <span class="pl-ent">steps</span>:
      - <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v2</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Configure Node caching</span>
        <span class="pl-ent">uses</span>: <span class="pl-s">actions/cache@v2</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">cache-name</span>: <span class="pl-s">cache-node-modules</span>
        <span class="pl-ent">with</span>:
          <span class="pl-ent">path</span>: <span class="pl-s">~/.npm</span>
          <span class="pl-ent">key</span>: <span class="pl-s">${{ runner.os }}-build-${{ env.cache-name }}-${{ hashFiles('**/package-lock.json') }}</span>
          <span class="pl-ent">restore-keys</span>: <span class="pl-s">|</span>
<span class="pl-s">            ${{ runner.os }}-build-${{ env.cache-name }}-</span>
<span class="pl-s">            ${{ runner.os }}-build-</span>
<span class="pl-s">            ${{ runner.os }}-</span>
<span class="pl-s"></span>      - <span class="pl-ent">uses</span>: <span class="pl-s">actions/cache@v2</span>
        <span class="pl-ent">name</span>: <span class="pl-s">Configure pip caching</span>
        <span class="pl-ent">with</span>:
          <span class="pl-ent">path</span>: <span class="pl-s">~/.cache/pip</span>
          <span class="pl-ent">key</span>: <span class="pl-s">${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}</span>
          <span class="pl-ent">restore-keys</span>: <span class="pl-s">|</span>
<span class="pl-s">            ${{ runner.os }}-pip-</span>
<span class="pl-s"></span>      - <span class="pl-ent">name</span>: <span class="pl-s">Install Node dependencies</span>
        <span class="pl-ent">run</span>: <span class="pl-s">npm install</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Download standalone Python</span>
        <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">          ./download-python.sh</span>
<span class="pl-s"></span>      - <span class="pl-ent">name</span>: <span class="pl-s">Run tests</span>
        <span class="pl-ent">run</span>: <span class="pl-s">npm test</span>
        <span class="pl-ent">timeout-minutes</span>: <span class="pl-c1">5</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Build distribution</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">CSC_KEY_PASSWORD</span>: <span class="pl-s">${{ secrets.CSC_KEY_PASSWORD }}</span>
          <span class="pl-ent">CSC_LINK</span>: <span class="pl-s">${{ secrets.CSC_LINK }}</span>
          <span class="pl-ent">APPLEID</span>: <span class="pl-s">${{ secrets.APPLEID }}</span>
          <span class="pl-ent">APPLEIDPASS</span>: <span class="pl-s">${{ secrets.APPLEIDPASS }}</span>
        <span class="pl-ent">run</span>: <span class="pl-s">npm run dist</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Create zip file</span>
        <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">          cd dist/mac</span>
<span class="pl-s">          ditto -c -k --keepParent Datasette.app Datasette.app.zip</span>
<span class="pl-s"></span>      - <span class="pl-ent">name</span>: <span class="pl-s">And a README (to work around GitHub double-zips)</span>
        <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">          echo "More information: https://datasette.io" &gt; dist/mac/README.txt</span>
<span class="pl-s"></span>      - <span class="pl-ent">name</span>: <span class="pl-s">Upload artifact</span>
        <span class="pl-ent">uses</span>: <span class="pl-s">actions/upload-artifact@v2</span>
        <span class="pl-ent">with</span>:
          <span class="pl-ent">name</span>: <span class="pl-s">Datasette-macOS</span>
          <span class="pl-ent">path</span>: <span class="pl-s">|</span>
<span class="pl-s">            dist/mac/Datasette.app.zip</span>
<span class="pl-s">            dist/mac/README.txt</span></pre></div>
<p>The key stuff here is the "Build distribution" step. It sets four values that I have saved on the repository as secrets: <code>CSC_KEY_PASSWORD</code>, <code>CSC_LINK</code>, <code>APPLEID</code> and <code>APPLEIDPASS</code>.</p>
<p>The <code>CSC_LINK</code> variable is the base64-encoded contents of my <code>Developer-ID-Application-Certificates.p12</code> file. I generated that like so:</p>
<pre><code>openssl base64 -in developerID_application.cer
</code></pre>
<p>I have <a href="https://github.com/simonw/datasette-app/blob/0.1.0/.github/workflows/release.yml">a separate release.yml</a> for building tagged releases, described in <a href="https://til.simonwillison.net/github-actions/attach-generated-file-to-release" rel="nofollow">this TIL</a>.</p>
<h2>
<a id="user-content-the-finished-configuration" class="anchor" href="#the-finished-configuration" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>The finished configuration</h2>
<p>You can browse the code in <a href="https://github.com/simonw/datasette-app/tree/0.1.0">my 0.1.0 tag</a> to see all of these parts in their final configuration, as used to create the 0.1.0 initial release of my application.</p>
<p>The original issue threads in which I figured this stuff out are:</p>
<ul>
<li><a href="https://github.com/simonw/datasette-app/issues/45">Get an Apple developer certificate #45</a></li>
<li><a href="https://github.com/simonw/datasette-app/issues/50">Work out how to notarize the macOS application #50</a></li>
<li><a href="https://github.com/simonw/datasette-app/issues/51">GitHub Actions workflow for creating packages for releases #51</a></li>
</ul>


path: electron_python-inside-electron.md
topic: electron
title: Bundling Python inside an Electron app
url: https://github.com/simonw/til/blob/main/electron/python-inside-electron.md
created: 2021-09-08T16:38:57-07:00
created_utc: 2021-09-08T23:38:57+00:00
updated: 2021-09-08T16:38:57-07:00
updated_utc: 2021-09-08T23:38:57+00:00
shot_hash: 8490359447794f9b8a23fb242946a61c
slug: python-inside-electron
body:
For [Datasette Desktop](https://datasette.io/desktop) I chose to bundle a full version of Python 3.9 inside my `Datasette.app` application. I did this in order to support installation of plugins via `pip install` - you can read more about my reasoning in [Datasette Desktop—a macOS desktop application for Datasette](https://simonwillison.net/2021/Sep/8/datasette-desktop/).

I used [python-build-standalone](https://github.com/indygreg/python-build-standalone) for this, which provides a version of Python that is designed for easy of bundling - it's also used by [PyOxidize](https://github.com/indygreg/PyOxidizer). Both projects are created and maintained by Gregory Szorc.

## In development mode

In my Electron app's root folder I ran the following:
```
wget https://github.com/indygreg/python-build-standalone/releases/download/20210724/cpython-3.9.6-x86_64-apple-darwin-install_only-20210724T1424.tar.gz
tar -xzvf cpython-3.9.6-x86_64-apple-darwin-install_only-20210724T1424.tar.gz                                                                          
```
This gave me a `python/` subfolder containing a full standalone Python, ready to run on my Mac.

Running `python/bin/python3.9 --version` confirms that this is working correctly.

## Calling Python from Electron

I used the Node.js `child_process.execFile()` function to execute Python scripts from inside Electron, like this:

```javascript
const cp = require("child_process");
const util = require("util");
const execFile = util.promisify(cp.execFile);

await execFile(path_to_python, ["-m", "random"]);
```
`path_to_python` needs to be the path to that `python3.9` executable. I wrote a `findPython()` function to find that, like so:

```javascript
const fs = require("fs");

function findPython() {
  const possibilities = [
    // In packaged app
    path.join(process.resourcesPath, "python", "bin", "python3.9"),
    // In development
    path.join(__dirname, "python", "bin", "python3.9"),
  ];
  for (const path of possibilities) {
    if (fs.existsSync(path)) {
      return path;
    }
  }
  console.log("Could not find python3, checked", possibilities);
  app.quit();
}
```
The `resourcesPath` bit here works for the packaged and deployed application.

## Packaging the app

I needed that `python` folder to be bundled up as part of the `Datasette.app` application bundle. I achieved that by adding the following to my `"build"` section in `package.json`:

```json
    "extraResources": [
      {
        "from": "python",
        "to": "python",
        "filter": [
          "**/*"
        ]
      }
    ]
```
This causes `electron-builder` to copy the contents of that `python` folder into `Datasette.app/Contents/Resources/python` in the packaged application.

My `findPython()` function earlier knows to look for it there by creating a path to it starting with `process.resourcesPath`.

## Signing and notarizing

I wrote extensive notes on signing and notarizing in [Signing and notarizing an Electron app for distribution using GitHub Actions](https://til.simonwillison.net/electron/sign-notarize-electron-macos), which includes details on how I signed the Python binaries that ended up included in the package due to this pattern.

html:
<p>For <a href="https://datasette.io/desktop" rel="nofollow">Datasette Desktop</a> I chose to bundle a full version of Python 3.9 inside my <code>Datasette.app</code> application. I did this in order to support installation of plugins via <code>pip install</code> - you can read more about my reasoning in <a href="https://simonwillison.net/2021/Sep/8/datasette-desktop/" rel="nofollow">Datasette Desktop—a macOS desktop application for Datasette</a>.</p>
<p>I used <a href="https://github.com/indygreg/python-build-standalone">python-build-standalone</a> for this, which provides a version of Python that is designed for easy of bundling - it's also used by <a href="https://github.com/indygreg/PyOxidizer">PyOxidize</a>. Both projects are created and maintained by Gregory Szorc.</p>
<h2>
<a id="user-content-in-development-mode" class="anchor" href="#in-development-mode" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>In development mode</h2>
<p>In my Electron app's root folder I ran the following:</p>
<pre><code>wget https://github.com/indygreg/python-build-standalone/releases/download/20210724/cpython-3.9.6-x86_64-apple-darwin-install_only-20210724T1424.tar.gz
tar -xzvf cpython-3.9.6-x86_64-apple-darwin-install_only-20210724T1424.tar.gz                                                                          
</code></pre>
<p>This gave me a <code>python/</code> subfolder containing a full standalone Python, ready to run on my Mac.</p>
<p>Running <code>python/bin/python3.9 --version</code> confirms that this is working correctly.</p>
<h2>
<a id="user-content-calling-python-from-electron" class="anchor" href="#calling-python-from-electron" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Calling Python from Electron</h2>
<p>I used the Node.js <code>child_process.execFile()</code> function to execute Python scripts from inside Electron, like this:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">const</span> <span class="pl-s1">cp</span> <span class="pl-c1">=</span> <span class="pl-en">require</span><span class="pl-kos">(</span><span class="pl-s">"child_process"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-k">const</span> <span class="pl-s1">util</span> <span class="pl-c1">=</span> <span class="pl-en">require</span><span class="pl-kos">(</span><span class="pl-s">"util"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-k">const</span> <span class="pl-s1">execFile</span> <span class="pl-c1">=</span> <span class="pl-s1">util</span><span class="pl-kos">.</span><span class="pl-en">promisify</span><span class="pl-kos">(</span><span class="pl-s1">cp</span><span class="pl-kos">.</span><span class="pl-c1">execFile</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

<span class="pl-k">await</span> <span class="pl-s1">execFile</span><span class="pl-kos">(</span><span class="pl-s1">path_to_python</span><span class="pl-kos">,</span> <span class="pl-kos">[</span><span class="pl-s">"-m"</span><span class="pl-kos">,</span> <span class="pl-s">"random"</span><span class="pl-kos">]</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<p><code>path_to_python</code> needs to be the path to that <code>python3.9</code> executable. I wrote a <code>findPython()</code> function to find that, like so:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">const</span> <span class="pl-s1">fs</span> <span class="pl-c1">=</span> <span class="pl-en">require</span><span class="pl-kos">(</span><span class="pl-s">"fs"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

<span class="pl-k">function</span> <span class="pl-en">findPython</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
  <span class="pl-k">const</span> <span class="pl-s1">possibilities</span> <span class="pl-c1">=</span> <span class="pl-kos">[</span>
    <span class="pl-c">// In packaged app</span>
    <span class="pl-s1">path</span><span class="pl-kos">.</span><span class="pl-en">join</span><span class="pl-kos">(</span><span class="pl-s1">process</span><span class="pl-kos">.</span><span class="pl-c1">resourcesPath</span><span class="pl-kos">,</span> <span class="pl-s">"python"</span><span class="pl-kos">,</span> <span class="pl-s">"bin"</span><span class="pl-kos">,</span> <span class="pl-s">"python3.9"</span><span class="pl-kos">)</span><span class="pl-kos">,</span>
    <span class="pl-c">// In development</span>
    <span class="pl-s1">path</span><span class="pl-kos">.</span><span class="pl-en">join</span><span class="pl-kos">(</span><span class="pl-s1">__dirname</span><span class="pl-kos">,</span> <span class="pl-s">"python"</span><span class="pl-kos">,</span> <span class="pl-s">"bin"</span><span class="pl-kos">,</span> <span class="pl-s">"python3.9"</span><span class="pl-kos">)</span><span class="pl-kos">,</span>
  <span class="pl-kos">]</span><span class="pl-kos">;</span>
  <span class="pl-k">for</span> <span class="pl-kos">(</span><span class="pl-k">const</span> <span class="pl-s1">path</span> <span class="pl-k">of</span> <span class="pl-s1">possibilities</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">fs</span><span class="pl-kos">.</span><span class="pl-en">existsSync</span><span class="pl-kos">(</span><span class="pl-s1">path</span><span class="pl-kos">)</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
      <span class="pl-k">return</span> <span class="pl-s1">path</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span>
  <span class="pl-kos">}</span>
  <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s">"Could not find python3, checked"</span><span class="pl-kos">,</span> <span class="pl-s1">possibilities</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-s1">app</span><span class="pl-kos">.</span><span class="pl-en">quit</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span></pre></div>
<p>The <code>resourcesPath</code> bit here works for the packaged and deployed application.</p>
<h2>
<a id="user-content-packaging-the-app" class="anchor" href="#packaging-the-app" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Packaging the app</h2>
<p>I needed that <code>python</code> folder to be bundled up as part of the <code>Datasette.app</code> application bundle. I achieved that by adding the following to my <code>"build"</code> section in <code>package.json</code>:</p>
<div class="highlight highlight-source-json"><pre>    <span class="pl-s"><span class="pl-pds">"</span>extraResources<span class="pl-pds">"</span></span>: [
      {
        <span class="pl-s"><span class="pl-pds">"</span>from<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>python<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>to<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>python<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>filter<span class="pl-pds">"</span></span>: [
          <span class="pl-s"><span class="pl-pds">"</span>**/*<span class="pl-pds">"</span></span>
        ]
      }
    ]</pre></div>
<p>This causes <code>electron-builder</code> to copy the contents of that <code>python</code> folder into <code>Datasette.app/Contents/Resources/python</code> in the packaged application.</p>
<p>My <code>findPython()</code> function earlier knows to look for it there by creating a path to it starting with <code>process.resourcesPath</code>.</p>
<h2>
<a id="user-content-signing-and-notarizing" class="anchor" href="#signing-and-notarizing" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Signing and notarizing</h2>
<p>I wrote extensive notes on signing and notarizing in <a href="https://til.simonwillison.net/electron/sign-notarize-electron-macos" rel="nofollow">Signing and notarizing an Electron app for distribution using GitHub Actions</a>, which includes details on how I signed the Python binaries that ended up included in the package due to this pattern.</p>


