# til

path: bash_escaping-sql-for-curl-to-datasette.md
topic: bash
title: Escaping a SQL query to use with curl and Datasette
url: https://github.com/simonw/til/blob/main/bash/escaping-sql-for-curl-to-datasette.md
created: 2020-12-08T11:05:59-08:00
created_utc: 2020-12-08T19:05:59+00:00
updated: 2020-12-08T11:05:59-08:00
updated_utc: 2020-12-08T19:05:59+00:00
shot_hash: 9af124acc855d09f35908a7eaed9be9f
slug: escaping-sql-for-curl-to-datasette
body:
I used this pattern to pass a SQL query to Datasette's CSV export via curl and output the results, stripping off the first row (the header row) using `tail -n +2`.

SQL queries need to be URL-encoded - I did that be echoing the SQL query and piping it to a Python one-liner that calls the `urllib.parse.quote()` function.

```bash
curl -s "https://github-to-sqlite.dogsheep.net/github.csv?sql=$(echo '
select
  full_name
from
  repos
where
  rowid in (
    select
      repos.rowid
    from
      repos,
      json_each(repos.topics) j
    where
      j.value = "datasette-io"
  )
  and rowid in (
    select
      repos.rowid
    from
      repos,
      json_each(repos.topics) j
    where
      j.value = "datasette-plugin"
  )
order by
  updated_at desc
' | python3 -c \
  'import sys; import urllib.parse; print(urllib.parse.quote(sys.stdin.read()))')" \
  | tail -n +2
```
Here's [that SQL query](https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++full_name%0D%0Afrom%0D%0A++repos%0D%0Awhere%0D%0A++rowid+in+%28%0D%0A++++select%0D%0A++++++repos.rowid%0D%0A++++from%0D%0A++++++repos%2C%0D%0A++++++json_each%28repos.topics%29+j%0D%0A++++where%0D%0A++++++j.value+%3D+%22datasette-io%22%0D%0A++%29%0D%0A++and+rowid+in+%28%0D%0A++++select%0D%0A++++++repos.rowid%0D%0A++++from%0D%0A++++++repos%2C%0D%0A++++++json_each%28repos.topics%29+j%0D%0A++++where%0D%0A++++++j.value+%3D+%22datasette-plugin%22%0D%0A++%29%0D%0Aorder+by%0D%0A++updated_at+desc) in the Datasette web UI.

The output from the bash one-liner looks like this:
```
simonw/datasette-edit-schema
simonw/datasette-import-table
simonw/datasette-dateutil
simonw/datasette-seaborn
simonw/datasette-backup
simonw/datasette-yaml
simonw/datasette-schema-versions
simonw/datasette-graphql
simonw/datasette-insert
simonw/datasette-copyable
simonw/datasette-auth-passwords
simonw/datasette-glitch
simonw/datasette-block-robots
simonw/datasette-saved-queries
simonw/datasette-psutil
simonw/datasette-auth-tokens
simonw/datasette-permissions-sql
simonw/datasette-media
simonw/datasette-atom
simonw/datasette-vega
simonw/datasette-jellyfish
simonw/datasette-leaflet-geojson
simonw/datasette-template-sql
simonw/datasette-render-markdown
simonw/datasette-auth-github
simonw/datasette-mask-columns
simonw/datasette-jq
simonw/datasette-cluster-map
simonw/datasette-upload-csvs
simonw/datasette-publish-fly
simonw/datasette-render-images
simonw/datasette-render-timestamps
simonw/datasette-configure-fts
simonw/datasette-search-all
simonw/datasette-render-html
simonw/datasette-show-errors
simonw/datasette-column-inspect
simonw/datasette-ics
simonw/datasette-json-html
simonw/datasette-pretty-json
simonw/datasette-sqlite-fts4
simonw/datasette-bplist
simonw/datasette-render-binary
simonw/datasette-rure
simonw/datasette-haversine
```

html:
<p>I used this pattern to pass a SQL query to Datasette's CSV export via curl and output the results, stripping off the first row (the header row) using <code>tail -n +2</code>.</p>
<p>SQL queries need to be URL-encoded - I did that be echoing the SQL query and piping it to a Python one-liner that calls the <code>urllib.parse.quote()</code> function.</p>
<div class="highlight highlight-source-shell"><pre>curl -s <span class="pl-s"><span class="pl-pds">"</span>https://github-to-sqlite.dogsheep.net/github.csv?sql=<span class="pl-s"><span class="pl-pds">$(</span>echo <span class="pl-s"><span class="pl-pds">'</span></span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">select</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">  full_name</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">from</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">  repos</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">where</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">  rowid in (</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">    select</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">      repos.rowid</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">    from</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">      repos,</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">      json_each(repos.topics) j</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">    where</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">      j.value = "datasette-io"</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">  )</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">  and rowid in (</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">    select</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">      repos.rowid</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">    from</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">      repos,</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">      json_each(repos.topics) j</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">    where</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">      j.value = "datasette-plugin"</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">  )</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">order by</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s">  updated_at desc</span></span></span>
<span class="pl-s"><span class="pl-s"><span class="pl-s"><span class="pl-pds">'</span></span> <span class="pl-k">|</span> python3 -c \</span></span>
<span class="pl-s"><span class="pl-s">  <span class="pl-s"><span class="pl-pds">'</span>import sys; import urllib.parse; print(urllib.parse.quote(sys.stdin.read()))<span class="pl-pds">'</span></span><span class="pl-pds">)</span></span><span class="pl-pds">"</span></span> \
  <span class="pl-k">|</span> tail -n +2</pre></div>
<p>Here's <a href="https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++full_name%0D%0Afrom%0D%0A++repos%0D%0Awhere%0D%0A++rowid+in+%28%0D%0A++++select%0D%0A++++++repos.rowid%0D%0A++++from%0D%0A++++++repos%2C%0D%0A++++++json_each%28repos.topics%29+j%0D%0A++++where%0D%0A++++++j.value+%3D+%22datasette-io%22%0D%0A++%29%0D%0A++and+rowid+in+%28%0D%0A++++select%0D%0A++++++repos.rowid%0D%0A++++from%0D%0A++++++repos%2C%0D%0A++++++json_each%28repos.topics%29+j%0D%0A++++where%0D%0A++++++j.value+%3D+%22datasette-plugin%22%0D%0A++%29%0D%0Aorder+by%0D%0A++updated_at+desc" rel="nofollow">that SQL query</a> in the Datasette web UI.</p>
<p>The output from the bash one-liner looks like this:</p>
<pre><code>simonw/datasette-edit-schema
simonw/datasette-import-table
simonw/datasette-dateutil
simonw/datasette-seaborn
simonw/datasette-backup
simonw/datasette-yaml
simonw/datasette-schema-versions
simonw/datasette-graphql
simonw/datasette-insert
simonw/datasette-copyable
simonw/datasette-auth-passwords
simonw/datasette-glitch
simonw/datasette-block-robots
simonw/datasette-saved-queries
simonw/datasette-psutil
simonw/datasette-auth-tokens
simonw/datasette-permissions-sql
simonw/datasette-media
simonw/datasette-atom
simonw/datasette-vega
simonw/datasette-jellyfish
simonw/datasette-leaflet-geojson
simonw/datasette-template-sql
simonw/datasette-render-markdown
simonw/datasette-auth-github
simonw/datasette-mask-columns
simonw/datasette-jq
simonw/datasette-cluster-map
simonw/datasette-upload-csvs
simonw/datasette-publish-fly
simonw/datasette-render-images
simonw/datasette-render-timestamps
simonw/datasette-configure-fts
simonw/datasette-search-all
simonw/datasette-render-html
simonw/datasette-show-errors
simonw/datasette-column-inspect
simonw/datasette-ics
simonw/datasette-json-html
simonw/datasette-pretty-json
simonw/datasette-sqlite-fts4
simonw/datasette-bplist
simonw/datasette-render-binary
simonw/datasette-rure
simonw/datasette-haversine
</code></pre>


path: bash_skip-csv-rows-with-odd-numbers.md
topic: bash
title: Skipping CSV rows with odd numbers of quotes using ripgrep
url: https://github.com/simonw/til/blob/main/bash/skip-csv-rows-with-odd-numbers.md
created: 2020-12-11T19:50:58-08:00
created_utc: 2020-12-12T03:50:58+00:00
updated: 2021-01-18T17:27:54-08:00
updated_utc: 2021-01-19T01:27:54+00:00
shot_hash: 79abed69911556279dfa18b015588d8c
slug: skip-csv-rows-with-odd-numbers
body:
I'm working with several huge CSV files - over 5 million rows total - and I ran into a problem: it turned out there were a few lines in those files that imported incorrectly because they were not correctly escaped.

Here's an example of an invalid line:

    SAI Exempt,"Patty B"s Hats & Tees,LLC",,26 Broad St

The apostrophe in `Patty B's Hats & Tees` is incorrectly represented here as a double quote, and since that's in a double quoted string it breaks that line of CSV.

I decided to filter out any rows that had an odd number of quotation marks in them - saving those broken lines to try and clean up later.

## Finding rows with odd numbers of quotes

StackOverflow [offered this regular expression](https://stackoverflow.com/a/16863999) for finding lines with an odd number of quotation marks:

```
[^"]*"   # Match any number of non-quote characters, then a quote
(?:      # Now match an even number of quotes by matching:
 [^"]*"  #  any number of non-quote characters, then a quote
 [^"]*"  #  twice
)*       # and repeat any number of times.
[^"]*    # Finally, match any remaining non-quote characters
```

I translated this into a `ripgrep` expression, adding `^` to the beginning and `$` to the end in order to match whole strings.

This command counted the number of invalid lines:

    rg '^[^"]*"(?:[^"]*"[^"]*")*[^"]*$' --glob '*.csv' --count

    04.csv:52
    03.csv:42
    02.csv:24
    01.csv:29

Adding `--invert-match` showed me the count of lines that did NOT have an odd number of quotes:

    rg '^[^"]*"(?:[^"]*"[^"]*")*[^"]*$' --glob '*.csv' --count --invert-match

    05.csv:2829
    04.csv:812351
    03.csv:961311
    02.csv:994265
    01.csv:995404

This shows that the invalid lines are a tiny subset of the overall files.

Removing `--count` shows the actual content.

## Importing into SQLite with sqlite-utils

I used this for loop to import only the valid lines into a SQLite database:

```bash
for file in *.csv;
    do rg --invert-match '^[^"]*"(?:[^"]*"[^"]*")*[^"]*$' $file | \
    sqlite-utils insert my.db rows - --csv;
done;
```

## Saving the broken lines for later

To save the lines that contained odd numbers of double quotes I used this command:

```bash
rg '^[^"]*"(?:[^"]*"[^"]*")*[^"]*$' \
  --glob '*.csv' \
  --no-line-number \
  --no-filename > saved.txt
```
Since I don't actually care which file they lived in - all of these CSV files share the same structure - I used `--no-filename` to omit the filename from the results and `--no-line-number` to omit the line number. The result is a `saved.txt` file containing just the raw CSV data that I skipped from the import.

html:
<p>I'm working with several huge CSV files - over 5 million rows total - and I ran into a problem: it turned out there were a few lines in those files that imported incorrectly because they were not correctly escaped.</p>
<p>Here's an example of an invalid line:</p>
<pre><code>SAI Exempt,"Patty B"s Hats &amp; Tees,LLC",,26 Broad St
</code></pre>
<p>The apostrophe in <code>Patty B's Hats &amp; Tees</code> is incorrectly represented here as a double quote, and since that's in a double quoted string it breaks that line of CSV.</p>
<p>I decided to filter out any rows that had an odd number of quotation marks in them - saving those broken lines to try and clean up later.</p>
<h2>
<a id="user-content-finding-rows-with-odd-numbers-of-quotes" class="anchor" href="#finding-rows-with-odd-numbers-of-quotes" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Finding rows with odd numbers of quotes</h2>
<p>StackOverflow <a href="https://stackoverflow.com/a/16863999" rel="nofollow">offered this regular expression</a> for finding lines with an odd number of quotation marks:</p>
<pre><code>[^"]*"   # Match any number of non-quote characters, then a quote
(?:      # Now match an even number of quotes by matching:
 [^"]*"  #  any number of non-quote characters, then a quote
 [^"]*"  #  twice
)*       # and repeat any number of times.
[^"]*    # Finally, match any remaining non-quote characters
</code></pre>
<p>I translated this into a <code>ripgrep</code> expression, adding <code>^</code> to the beginning and <code>$</code> to the end in order to match whole strings.</p>
<p>This command counted the number of invalid lines:</p>
<pre><code>rg '^[^"]*"(?:[^"]*"[^"]*")*[^"]*$' --glob '*.csv' --count

04.csv:52
03.csv:42
02.csv:24
01.csv:29
</code></pre>
<p>Adding <code>--invert-match</code> showed me the count of lines that did NOT have an odd number of quotes:</p>
<pre><code>rg '^[^"]*"(?:[^"]*"[^"]*")*[^"]*$' --glob '*.csv' --count --invert-match

05.csv:2829
04.csv:812351
03.csv:961311
02.csv:994265
01.csv:995404
</code></pre>
<p>This shows that the invalid lines are a tiny subset of the overall files.</p>
<p>Removing <code>--count</code> shows the actual content.</p>
<h2>
<a id="user-content-importing-into-sqlite-with-sqlite-utils" class="anchor" href="#importing-into-sqlite-with-sqlite-utils" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Importing into SQLite with sqlite-utils</h2>
<p>I used this for loop to import only the valid lines into a SQLite database:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-k">for</span> <span class="pl-smi">file</span> <span class="pl-k">in</span> <span class="pl-k">*</span>.csv<span class="pl-k">;</span>
    <span class="pl-k">do</span> rg --invert-match <span class="pl-s"><span class="pl-pds">'</span>^[^"]*"(?:[^"]*"[^"]*")*[^"]*$<span class="pl-pds">'</span></span> <span class="pl-smi">$file</span> <span class="pl-k">|</span> \
    sqlite-utils insert my.db rows - --csv<span class="pl-k">;</span>
<span class="pl-k">done</span><span class="pl-k">;</span></pre></div>
<h2>
<a id="user-content-saving-the-broken-lines-for-later" class="anchor" href="#saving-the-broken-lines-for-later" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Saving the broken lines for later</h2>
<p>To save the lines that contained odd numbers of double quotes I used this command:</p>
<div class="highlight highlight-source-shell"><pre>rg <span class="pl-s"><span class="pl-pds">'</span>^[^"]*"(?:[^"]*"[^"]*")*[^"]*$<span class="pl-pds">'</span></span> \
  --glob <span class="pl-s"><span class="pl-pds">'</span>*.csv<span class="pl-pds">'</span></span> \
  --no-line-number \
  --no-filename <span class="pl-k">&gt;</span> saved.txt</pre></div>
<p>Since I don't actually care which file they lived in - all of these CSV files share the same structure - I used <code>--no-filename</code> to omit the filename from the results and <code>--no-line-number</code> to omit the line number. The result is a <code>saved.txt</code> file containing just the raw CSV data that I skipped from the import.</p>


path: sqlite_replicating-rqlite.md
topic: sqlite
title: Replicating SQLite with rqlite
url: https://github.com/simonw/til/blob/main/sqlite/replicating-rqlite.md
created: 2020-12-28T11:22:16-08:00
created_utc: 2020-12-28T19:22:16+00:00
updated: 2020-12-28T12:25:45-08:00
updated_utc: 2020-12-28T20:25:45+00:00
shot_hash: 74d453a19019020f4302ab7251303c7d
slug: replicating-rqlite
body:
I tried out [rqlite](https://github.com/rqlite/rqlite) today, a "lightweight, distributed relational database, which uses SQLite as its storage engine". It uses the Raft consensus algorithm to allow multiple SQLite instances to elect a leader and replicate changes amongst themselves.

By default `rqlite` asks you to use its own custom HTTP API - but I wanted to try running it against Datasette. rqlite author Philip O'Toole confirmed that [this should work](https://twitter.com/general_order24/status/1343619601758908419) provided any writes go through the API - each node can be configured to write to an on-disk database file which Datasette can then read from (the default is to use in-memory databases and an on-disk Raft log).

Here's how I got that working on my macOS laptop. I used the latest macOS binary from https://github.com/rqlite/rqlite/releases (`rqlite` is written in Go and provides pre-complied binaries for different systems).

    cd /tmp
    curl -L https://github.com/rqlite/rqlite/releases/download/v5.7.0/rqlite-v5.7.0-darwin-amd64.tar.gz \
      -o rqlite-v5.7.0-darwin-amd64.tar.gz
    tar xvfz rqlite-v5.7.0-darwin-amd64.tar.gz
    cd rqlite-v5.7.0-darwin-amd64

I started the first node like this:

    ./rqlited -on-disk ~/node.1

This created a SQLite database file in `~/node.1/db.sqlite` which I could open with Datasette.

Note that the `-on-disk` option has to come before the `~/node.1` - if it comes afterwards it silently fails to take effect.

I created a table and inserted a record using the client binary like this:

    ./rqlite
    CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT)
    INSERT INTO foo(name) VALUES("fiona")

Running Datasette against `~/node.1/db.sqlite` confirmed the table had been created.

Then I started a second node like this:

    ./rqlited -on-disk -node-id 2 -http-addr localhost:4003 \
        -raft-addr localhost:4004 -join http://localhost:4001 \
        ~/node.2

Since it was running on the same machine I had to pick different ports for it, then tell it to join the existing cluster at `http://localhost:4001`

This created a database file at `~/node.2/db.sqlite`.

I confirmed that this database had the `foo` table in it, then inserted another row and watched as both of my database files were updated with the new record.

html:
<p>I tried out <a href="https://github.com/rqlite/rqlite">rqlite</a> today, a "lightweight, distributed relational database, which uses SQLite as its storage engine". It uses the Raft consensus algorithm to allow multiple SQLite instances to elect a leader and replicate changes amongst themselves.</p>
<p>By default <code>rqlite</code> asks you to use its own custom HTTP API - but I wanted to try running it against Datasette. rqlite author Philip O'Toole confirmed that <a href="https://twitter.com/general_order24/status/1343619601758908419" rel="nofollow">this should work</a> provided any writes go through the API - each node can be configured to write to an on-disk database file which Datasette can then read from (the default is to use in-memory databases and an on-disk Raft log).</p>
<p>Here's how I got that working on my macOS laptop. I used the latest macOS binary from <a href="https://github.com/rqlite/rqlite/releases">https://github.com/rqlite/rqlite/releases</a> (<code>rqlite</code> is written in Go and provides pre-complied binaries for different systems).</p>
<pre><code>cd /tmp
curl -L https://github.com/rqlite/rqlite/releases/download/v5.7.0/rqlite-v5.7.0-darwin-amd64.tar.gz \
  -o rqlite-v5.7.0-darwin-amd64.tar.gz
tar xvfz rqlite-v5.7.0-darwin-amd64.tar.gz
cd rqlite-v5.7.0-darwin-amd64
</code></pre>
<p>I started the first node like this:</p>
<pre><code>./rqlited -on-disk ~/node.1
</code></pre>
<p>This created a SQLite database file in <code>~/node.1/db.sqlite</code> which I could open with Datasette.</p>
<p>Note that the <code>-on-disk</code> option has to come before the <code>~/node.1</code> - if it comes afterwards it silently fails to take effect.</p>
<p>I created a table and inserted a record using the client binary like this:</p>
<pre><code>./rqlite
CREATE TABLE foo (id INTEGER NOT NULL PRIMARY KEY, name TEXT)
INSERT INTO foo(name) VALUES("fiona")
</code></pre>
<p>Running Datasette against <code>~/node.1/db.sqlite</code> confirmed the table had been created.</p>
<p>Then I started a second node like this:</p>
<pre><code>./rqlited -on-disk -node-id 2 -http-addr localhost:4003 \
    -raft-addr localhost:4004 -join http://localhost:4001 \
    ~/node.2
</code></pre>
<p>Since it was running on the same machine I had to pick different ports for it, then tell it to join the existing cluster at <code>http://localhost:4001</code></p>
<p>This created a database file at <code>~/node.2/db.sqlite</code>.</p>
<p>I confirmed that this database had the <code>foo</code> table in it, then inserted another row and watched as both of my database files were updated with the new record.</p>


path: python_yielding-in-asyncio.md
topic: python
title: Relinquishing control in Python asyncio
url: https://github.com/simonw/til/blob/main/python/yielding-in-asyncio.md
created: 2020-12-29T11:25:14-08:00
created_utc: 2020-12-29T19:25:14+00:00
updated: 2020-12-29T11:25:14-08:00
updated_utc: 2020-12-29T19:25:14+00:00
shot_hash: 9aac6b6cc02d88b723851e4a958265d9
slug: yielding-in-asyncio
body:
`asyncio` in Python is a form of co-operative multitasking, where everything runs in a single thread but asynchronous tasks can yield to other tasks to allow them to execute.

Normally you do this with `await` - but I'm thinking through a problem at the moment which could involve long-running asyncio functions. To avoid blocking the event loop, I'd like them to periodically yield to see if there are any other tasks that need to spend some time with the CPU.

This doesn't seem to be covered in the Python `asyncio` documentation, but after some digging I came across this issue in the old `python/asyncio` repo: [Question: How to relinquishing control to the event loop in Python 3.5](https://github.com/python/asyncio/issues/284)

It turns out the supported, optimized idiom is this one:

```python
await asyncio.sleep(0)
```

html:
<p><code>asyncio</code> in Python is a form of co-operative multitasking, where everything runs in a single thread but asynchronous tasks can yield to other tasks to allow them to execute.</p>
<p>Normally you do this with <code>await</code> - but I'm thinking through a problem at the moment which could involve long-running asyncio functions. To avoid blocking the event loop, I'd like them to periodically yield to see if there are any other tasks that need to spend some time with the CPU.</p>
<p>This doesn't seem to be covered in the Python <code>asyncio</code> documentation, but after some digging I came across this issue in the old <code>python/asyncio</code> repo: <a href="https://github.com/python/asyncio/issues/284">Question: How to relinquishing control to the event loop in Python 3.5</a></p>
<p>It turns out the supported, optimized idiom is this one:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">await</span> <span class="pl-s1">asyncio</span>.<span class="pl-en">sleep</span>(<span class="pl-c1">0</span>)</pre></div>


path: javascript_jest-without-package-json.md
topic: javascript
title: Using Jest without a package.json
url: https://github.com/simonw/til/blob/main/javascript/jest-without-package-json.md
created: 2020-12-30T14:36:28-08:00
created_utc: 2020-12-30T22:36:28+00:00
updated: 2020-12-30T14:36:28-08:00
updated_utc: 2020-12-30T22:36:28+00:00
shot_hash: f42d147b954728780b470149918186ea
slug: jest-without-package-json
body:
I wanted to try out [Jest](https://jestjs.io/) for writing JavaScript unit tests, in a project that wasn't set up with `package.json` and other NPM related things.

Jest looks for `*.spec.js` tests in a `__tests__` directory. It expects to find configuration in a `package.json` file but it can be passed configuration using the `-c` option - which can be a path to a JSON configuration file or can be a JSON literal.

I created a file I wanted to test in `plugins.js` which looked like this. The `module.exports` at the bottom was required so Jest could later import the code:

```javascript
var datasette = datasette || {};
datasette.plugins = (() => {
    var registry = {};
    return {
        register: (hook, fn, parameters) => {
            if (!registry[hook]) {
                registry[hook] = [];
            }
            registry[hook].push([fn, parameters]);
        },
        call: (hook, args) => {
            args = args || {};
            var results = [];
            (registry[hook] || []).forEach(([fn, parameters]) => {
                /* Call with the correct arguments */
                var result = fn.apply(fn, parameters.map(parameter => args[parameter]));
                if (result !== undefined) {
                    results.push(result);
                }
            });
            return results;
        }
    };
})();

module.exports = datasette;
```

Then I created `__tests__/plugins.spec.js` with this:

```javascript
const datasette = require("../plugins.js");

describe("Datasette Plugins", () => {
  test("it should have datasette.plugins", () => {
    expect(!!datasette.plugins).toEqual(true);
  });
  test("registering a plugin should work", () => {
    datasette.plugins.register("numbers", (a, b) => a + b, ["a", "b"]);
    var result = datasette.plugins.call("numbers", { a: 1, b: 2 });
    expect(result).toEqual([3]);
    datasette.plugins.register("numbers", (a, b) => a * b, ["a", "b"]);
    var result2 = datasette.plugins.call("numbers", { a: 1, b: 2 });
    expect(result2).toEqual([3, 2]);
  });
});
```
Now I can run Jest in the same directory as `plugins.js` like this:
```
% npx jest -c '{}'
 PASS  __tests__/plugins.spec.js
  Datasette Plugins
    ✓ it should have datasette.plugins (3 ms)
    ✓ registering a plugin should work (1 ms)

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        1.163 s
Ran all test suites.
```

html:
<p>I wanted to try out <a href="https://jestjs.io/" rel="nofollow">Jest</a> for writing JavaScript unit tests, in a project that wasn't set up with <code>package.json</code> and other NPM related things.</p>
<p>Jest looks for <code>*.spec.js</code> tests in a <code>__tests__</code> directory. It expects to find configuration in a <code>package.json</code> file but it can be passed configuration using the <code>-c</code> option - which can be a path to a JSON configuration file or can be a JSON literal.</p>
<p>I created a file I wanted to test in <code>plugins.js</code> which looked like this. The <code>module.exports</code> at the bottom was required so Jest could later import the code:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">var</span> <span class="pl-s1">datasette</span> <span class="pl-c1">=</span> <span class="pl-s1">datasette</span> <span class="pl-c1">||</span> <span class="pl-kos">{</span><span class="pl-kos">}</span><span class="pl-kos">;</span>
<span class="pl-s1">datasette</span><span class="pl-kos">.</span><span class="pl-c1">plugins</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-k">var</span> <span class="pl-s1">registry</span> <span class="pl-c1">=</span> <span class="pl-kos">{</span><span class="pl-kos">}</span><span class="pl-kos">;</span>
    <span class="pl-k">return</span> <span class="pl-kos">{</span>
        <span class="pl-en">register</span>: <span class="pl-kos">(</span><span class="pl-s1">hook</span><span class="pl-kos">,</span> <span class="pl-s1">fn</span><span class="pl-kos">,</span> <span class="pl-s1">parameters</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
            <span class="pl-k">if</span> <span class="pl-kos">(</span>!<span class="pl-s1">registry</span><span class="pl-kos">[</span><span class="pl-s1">hook</span><span class="pl-kos">]</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
                <span class="pl-s1">registry</span><span class="pl-kos">[</span><span class="pl-s1">hook</span><span class="pl-kos">]</span> <span class="pl-c1">=</span> <span class="pl-kos">[</span><span class="pl-kos">]</span><span class="pl-kos">;</span>
            <span class="pl-kos">}</span>
            <span class="pl-s1">registry</span><span class="pl-kos">[</span><span class="pl-s1">hook</span><span class="pl-kos">]</span><span class="pl-kos">.</span><span class="pl-en">push</span><span class="pl-kos">(</span><span class="pl-kos">[</span><span class="pl-s1">fn</span><span class="pl-kos">,</span> <span class="pl-s1">parameters</span><span class="pl-kos">]</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
        <span class="pl-kos">}</span><span class="pl-kos">,</span>
        <span class="pl-en">call</span>: <span class="pl-kos">(</span><span class="pl-s1">hook</span><span class="pl-kos">,</span> <span class="pl-s1">args</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
            <span class="pl-s1">args</span> <span class="pl-c1">=</span> <span class="pl-s1">args</span> <span class="pl-c1">||</span> <span class="pl-kos">{</span><span class="pl-kos">}</span><span class="pl-kos">;</span>
            <span class="pl-k">var</span> <span class="pl-s1">results</span> <span class="pl-c1">=</span> <span class="pl-kos">[</span><span class="pl-kos">]</span><span class="pl-kos">;</span>
            <span class="pl-kos">(</span><span class="pl-s1">registry</span><span class="pl-kos">[</span><span class="pl-s1">hook</span><span class="pl-kos">]</span> <span class="pl-c1">||</span> <span class="pl-kos">[</span><span class="pl-kos">]</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">forEach</span><span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-kos">[</span><span class="pl-s1">fn</span><span class="pl-kos">,</span> <span class="pl-s1">parameters</span><span class="pl-kos">]</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
                <span class="pl-c">/* Call with the correct arguments */</span>
                <span class="pl-k">var</span> <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-s1">fn</span><span class="pl-kos">.</span><span class="pl-en">apply</span><span class="pl-kos">(</span><span class="pl-s1">fn</span><span class="pl-kos">,</span> <span class="pl-s1">parameters</span><span class="pl-kos">.</span><span class="pl-en">map</span><span class="pl-kos">(</span><span class="pl-s1">parameter</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">args</span><span class="pl-kos">[</span><span class="pl-s1">parameter</span><span class="pl-kos">]</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
                <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">result</span> !== <span class="pl-c1">undefined</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
                    <span class="pl-s1">results</span><span class="pl-kos">.</span><span class="pl-en">push</span><span class="pl-kos">(</span><span class="pl-s1">result</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
                <span class="pl-kos">}</span>
            <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
            <span class="pl-k">return</span> <span class="pl-s1">results</span><span class="pl-kos">;</span>
        <span class="pl-kos">}</span>
    <span class="pl-kos">}</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

<span class="pl-smi">module</span><span class="pl-kos">.</span><span class="pl-c1">exports</span> <span class="pl-c1">=</span> <span class="pl-s1">datasette</span><span class="pl-kos">;</span></pre></div>
<p>Then I created <code>__tests__/plugins.spec.js</code> with this:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">const</span> <span class="pl-s1">datasette</span> <span class="pl-c1">=</span> <span class="pl-en">require</span><span class="pl-kos">(</span><span class="pl-s">"../plugins.js"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

<span class="pl-en">describe</span><span class="pl-kos">(</span><span class="pl-s">"Datasette Plugins"</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
  <span class="pl-en">test</span><span class="pl-kos">(</span><span class="pl-s">"it should have datasette.plugins"</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-en">expect</span><span class="pl-kos">(</span>!!<span class="pl-s1">datasette</span><span class="pl-kos">.</span><span class="pl-c1">plugins</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">toEqual</span><span class="pl-kos">(</span><span class="pl-c1">true</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-en">test</span><span class="pl-kos">(</span><span class="pl-s">"registering a plugin should work"</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-s1">datasette</span><span class="pl-kos">.</span><span class="pl-c1">plugins</span><span class="pl-kos">.</span><span class="pl-en">register</span><span class="pl-kos">(</span><span class="pl-s">"numbers"</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-s1">a</span><span class="pl-kos">,</span> <span class="pl-s1">b</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">a</span> <span class="pl-c1">+</span> <span class="pl-s1">b</span><span class="pl-kos">,</span> <span class="pl-kos">[</span><span class="pl-s">"a"</span><span class="pl-kos">,</span> <span class="pl-s">"b"</span><span class="pl-kos">]</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-k">var</span> <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-s1">datasette</span><span class="pl-kos">.</span><span class="pl-c1">plugins</span><span class="pl-kos">.</span><span class="pl-en">call</span><span class="pl-kos">(</span><span class="pl-s">"numbers"</span><span class="pl-kos">,</span> <span class="pl-kos">{</span> <span class="pl-c1">a</span>: <span class="pl-c1">1</span><span class="pl-kos">,</span> <span class="pl-c1">b</span>: <span class="pl-c1">2</span> <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-en">expect</span><span class="pl-kos">(</span><span class="pl-s1">result</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">toEqual</span><span class="pl-kos">(</span><span class="pl-kos">[</span><span class="pl-c1">3</span><span class="pl-kos">]</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">datasette</span><span class="pl-kos">.</span><span class="pl-c1">plugins</span><span class="pl-kos">.</span><span class="pl-en">register</span><span class="pl-kos">(</span><span class="pl-s">"numbers"</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-s1">a</span><span class="pl-kos">,</span> <span class="pl-s1">b</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">a</span> * <span class="pl-s1">b</span><span class="pl-kos">,</span> <span class="pl-kos">[</span><span class="pl-s">"a"</span><span class="pl-kos">,</span> <span class="pl-s">"b"</span><span class="pl-kos">]</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-k">var</span> <span class="pl-s1">result2</span> <span class="pl-c1">=</span> <span class="pl-s1">datasette</span><span class="pl-kos">.</span><span class="pl-c1">plugins</span><span class="pl-kos">.</span><span class="pl-en">call</span><span class="pl-kos">(</span><span class="pl-s">"numbers"</span><span class="pl-kos">,</span> <span class="pl-kos">{</span> <span class="pl-c1">a</span>: <span class="pl-c1">1</span><span class="pl-kos">,</span> <span class="pl-c1">b</span>: <span class="pl-c1">2</span> <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-en">expect</span><span class="pl-kos">(</span><span class="pl-s1">result2</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">toEqual</span><span class="pl-kos">(</span><span class="pl-kos">[</span><span class="pl-c1">3</span><span class="pl-kos">,</span> <span class="pl-c1">2</span><span class="pl-kos">]</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<p>Now I can run Jest in the same directory as <code>plugins.js</code> like this:</p>
<pre><code>% npx jest -c '{}'
 PASS  __tests__/plugins.spec.js
  Datasette Plugins
    ✓ it should have datasette.plugins (3 ms)
    ✓ registering a plugin should work (1 ms)

Test Suites: 1 passed, 1 total
Tests:       2 passed, 2 total
Snapshots:   0 total
Time:        1.163 s
Ran all test suites.
</code></pre>


path: github-actions_prettier-github-actions.md
topic: github-actions
title: Using Prettier to check JavaScript code style in GitHub Actions
url: https://github.com/simonw/til/blob/main/github-actions/prettier-github-actions.md
created: 2020-12-31T13:55:53-08:00
created_utc: 2020-12-31T21:55:53+00:00
updated: 2021-01-03T10:24:55-08:00
updated_utc: 2021-01-03T18:24:55+00:00
shot_hash: f3e5aa46cc58f87ceffcfc8d43078768
slug: prettier-github-actions
body:
I [decided](https://github.com/simonw/datasette/issues/1166) to adopt [Prettier](https://prettier.io/) as the JavaScript code style for Datasette, based on my success with [Black](https://github.com/psf/black) for Python code.

I added `.prettierrc` to the root of the repo with the following settings:

```json
{
  "tabWidth": 2,
  "useTabs": false
}
```

Then I created the following `.github/workflows/prettier.yml` file:

```yaml
name: Check JavaScript for conformance with Prettier

on:
  push:
  pull_request:

jobs:
  prettier:
    runs-on: ubuntu-latest
    steps:
    - name: Check out repo
      uses: actions/checkout@v2
    - uses: actions/cache@v2
      name: Configure npm caching
      with:
        path: ~/.npm
        key: ${{ runner.os }}-npm-${{ hashFiles('**/workflows/prettier.yml') }}
        restore-keys: |
          ${{ runner.os }}-npm-
    - name: Run prettier
      run: |-
        npx prettier --check 'datasette/static/*[!.min].js'
```

The `npx prettier --check 'datasette/static/*[!.min].js'` line ensures that prettier is run in "check" mode (which fails the tests if a matching file does not conform to the formatting rules) - it checks any `.js` file in the `datasette/static` folder but excludes any `.min.js` minified files.

I'm using `npx` to run Prettier which installs it if it is missing - as far as I can tell `npx` respects the `.npm` cache so I'm using that to avoid downloading a new copy of Prettier every time. **UPDATE:** Apparently it doesn't, see [#1169](https://github.com/simonw/datasette/issues/1169)

I decided to use the hash of the `prettier.yml` file itself as the key for the cached data, so any time I change the workflow it should invalidate the cache.

html:
<p>I <a href="https://github.com/simonw/datasette/issues/1166">decided</a> to adopt <a href="https://prettier.io/" rel="nofollow">Prettier</a> as the JavaScript code style for Datasette, based on my success with <a href="https://github.com/psf/black">Black</a> for Python code.</p>
<p>I added <code>.prettierrc</code> to the root of the repo with the following settings:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-s"><span class="pl-pds">"</span>tabWidth<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>,
  <span class="pl-s"><span class="pl-pds">"</span>useTabs<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>
}</pre></div>
<p>Then I created the following <code>.github/workflows/prettier.yml</code> file:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Check JavaScript for conformance with Prettier</span>

<span class="pl-ent">on</span>:
  <span class="pl-ent">push</span>:
  <span class="pl-ent">pull_request</span>:

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">prettier</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">name</span>: <span class="pl-s">Check out repo</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v2</span>
    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/cache@v2</span>
      <span class="pl-ent">name</span>: <span class="pl-s">Configure npm caching</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">path</span>: <span class="pl-s">~/.npm</span>
        <span class="pl-ent">key</span>: <span class="pl-s">${{ runner.os }}-npm-${{ hashFiles('**/workflows/prettier.yml') }}</span>
        <span class="pl-ent">restore-keys</span>: <span class="pl-s">|</span>
<span class="pl-s">          ${{ runner.os }}-npm-</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Run prettier</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        npx prettier --check 'datasette/static/*[!.min].js'</span></pre></div>
<p>The <code>npx prettier --check 'datasette/static/*[!.min].js'</code> line ensures that prettier is run in "check" mode (which fails the tests if a matching file does not conform to the formatting rules) - it checks any <code>.js</code> file in the <code>datasette/static</code> folder but excludes any <code>.min.js</code> minified files.</p>
<p>I'm using <code>npx</code> to run Prettier which installs it if it is missing - as far as I can tell <code>npx</code> respects the <code>.npm</code> cache so I'm using that to avoid downloading a new copy of Prettier every time. <strong>UPDATE:</strong> Apparently it doesn't, see <a href="https://github.com/simonw/datasette/issues/1169">#1169</a></p>
<p>I decided to use the hash of the <code>prettier.yml</code> file itself as the key for the cached data, so any time I change the workflow it should invalidate the cache.</p>


path: python_packaging-pyinstaller.md
topic: python
title: Packaging a Python app as a standalone binary with PyInstaller
url: https://github.com/simonw/til/blob/main/python/packaging-pyinstaller.md
created: 2021-01-04T13:54:44-08:00
created_utc: 2021-01-04T21:54:44+00:00
updated: 2021-01-04T13:54:44-08:00
updated_utc: 2021-01-04T21:54:44+00:00
shot_hash: e13c87a31c97f22b2242f2eb517020a9
slug: packaging-pyinstaller
body:
[PyInstaller](https://www.pyinstaller.org/) can take a Python script and bundle it up as a standalone executable for macOS, Linux and apparently Windows too (I've not tried it on Windows yet).

Today I managed to build [Datasette](https://datasette.io/) executables for macOS and Linux using the following recipe:

```bash
export DATASETTE_BASE=$(python -c 'import os; print(os.path.dirname(__import__("datasette").__file__))') \
pyinstaller -F \
    --add-data "$DATASETTE_BASE/templates:datasette/templates" \
    --add-data "$DATASETTE_BASE/static:datasette/static" \
    --hidden-import datasette.publish \
    --hidden-import datasette.publish.heroku \
    --hidden-import datasette.publish.cloudrun \
    --hidden-import datasette.facets \
    --hidden-import datasette.sql_functions \
    --hidden-import datasette.actor_auth_cookie \
    --hidden-import datasette.default_permissions \
    --hidden-import datasette.default_magic_parameters \
    --hidden-import datasette.blob_renderer \
    --hidden-import datasette.default_menu_links \
    --hidden-import uvicorn \
    --hidden-import uvicorn.logging \
    --hidden-import uvicorn.loops \
    --hidden-import uvicorn.loops.auto \
    --hidden-import uvicorn.protocols \
    --hidden-import uvicorn.protocols.http \
    --hidden-import uvicorn.protocols.http.auto \
    --hidden-import uvicorn.protocols.websockets \
    --hidden-import uvicorn.protocols.websockets.auto \
    --hidden-import uvicorn.lifespan \
    --hidden-import uvicorn.lifespan.on \
    $(which datasette)
```

The `--hidden-import` lines are needed because PyInstaller attempts to follow the module import graph for a package, but is very easily confused. Datasette dynamically imports [a list of default plugins](https://github.com/simonw/datasette/blob/ab7767acbe021ed6ab0a8d4b56ec8b4af6ae9e86/datasette/plugins.py#L7-L17) so I had to explicitly list each of those. I don't know what's going on with `uvicorn` here - I kept on running the script and then running `dist/datasette` and getting errors like this one:

```
(pyinstaller-datasette) pyinstaller-datasette % dist/datasette ~/Dropbox/Development/datasette/fixtures.db
Traceback (most recent call last):
  File "datasette", line 8, in <module>
  File "click/core.py", line 829, in __call__
  File "click/core.py", line 782, in main
  File "click/core.py", line 1259, in invoke
  File "click/core.py", line 1066, in invoke
  File "click/core.py", line 610, in invoke
  File "datasette/cli.py", line 548, in serve
  File "uvicorn/main.py", line 386, in run
  File "uvicorn/server.py", line 48, in run
  File "asyncio/base_events.py", line 642, in run_until_complete
  File "uvicorn/server.py", line 55, in serve
  File "uvicorn/config.py", line 301, in load
  File "uvicorn/importer.py", line 23, in import_from_string
  File "uvicorn/importer.py", line 20, in import_from_string
  File "importlib/__init__.py", line 127, in import_module
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'uvicorn.protocols.websockets'
[32142] Failed to execute script datasette
```

I solved this by adding each `ModuleNotFoundError` module to `--hidden-import` until it worked.

I've tested this script (and the generated executables) on both macOS and Ubuntu Linux so far, and it's worked perfectly in both cases. See [issue 93](https://github.com/simonw/datasette/issues/93) for more details.

html:
<p><a href="https://www.pyinstaller.org/" rel="nofollow">PyInstaller</a> can take a Python script and bundle it up as a standalone executable for macOS, Linux and apparently Windows too (I've not tried it on Windows yet).</p>
<p>Today I managed to build <a href="https://datasette.io/" rel="nofollow">Datasette</a> executables for macOS and Linux using the following recipe:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-k">export</span> DATASETTE_BASE=<span class="pl-s"><span class="pl-pds">$(</span>python -c <span class="pl-s"><span class="pl-pds">'</span>import os; print(os.path.dirname(__import__("datasette").__file__))<span class="pl-pds">'</span></span><span class="pl-pds">)</span></span> \
pyinstaller -F \
    --add-data <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$DATASETTE_BASE</span>/templates:datasette/templates<span class="pl-pds">"</span></span> \
    --add-data <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$DATASETTE_BASE</span>/static:datasette/static<span class="pl-pds">"</span></span> \
    --hidden-import datasette.publish \
    --hidden-import datasette.publish.heroku \
    --hidden-import datasette.publish.cloudrun \
    --hidden-import datasette.facets \
    --hidden-import datasette.sql_functions \
    --hidden-import datasette.actor_auth_cookie \
    --hidden-import datasette.default_permissions \
    --hidden-import datasette.default_magic_parameters \
    --hidden-import datasette.blob_renderer \
    --hidden-import datasette.default_menu_links \
    --hidden-import uvicorn \
    --hidden-import uvicorn.logging \
    --hidden-import uvicorn.loops \
    --hidden-import uvicorn.loops.auto \
    --hidden-import uvicorn.protocols \
    --hidden-import uvicorn.protocols.http \
    --hidden-import uvicorn.protocols.http.auto \
    --hidden-import uvicorn.protocols.websockets \
    --hidden-import uvicorn.protocols.websockets.auto \
    --hidden-import uvicorn.lifespan \
    --hidden-import uvicorn.lifespan.on \
    <span class="pl-s"><span class="pl-pds">$(</span>which datasette<span class="pl-pds">)</span></span></pre></div>
<p>The <code>--hidden-import</code> lines are needed because PyInstaller attempts to follow the module import graph for a package, but is very easily confused. Datasette dynamically imports <a href="https://github.com/simonw/datasette/blob/ab7767acbe021ed6ab0a8d4b56ec8b4af6ae9e86/datasette/plugins.py#L7-L17">a list of default plugins</a> so I had to explicitly list each of those. I don't know what's going on with <code>uvicorn</code> here - I kept on running the script and then running <code>dist/datasette</code> and getting errors like this one:</p>
<pre><code>(pyinstaller-datasette) pyinstaller-datasette % dist/datasette ~/Dropbox/Development/datasette/fixtures.db
Traceback (most recent call last):
  File "datasette", line 8, in &lt;module&gt;
  File "click/core.py", line 829, in __call__
  File "click/core.py", line 782, in main
  File "click/core.py", line 1259, in invoke
  File "click/core.py", line 1066, in invoke
  File "click/core.py", line 610, in invoke
  File "datasette/cli.py", line 548, in serve
  File "uvicorn/main.py", line 386, in run
  File "uvicorn/server.py", line 48, in run
  File "asyncio/base_events.py", line 642, in run_until_complete
  File "uvicorn/server.py", line 55, in serve
  File "uvicorn/config.py", line 301, in load
  File "uvicorn/importer.py", line 23, in import_from_string
  File "uvicorn/importer.py", line 20, in import_from_string
  File "importlib/__init__.py", line 127, in import_module
  File "&lt;frozen importlib._bootstrap&gt;", line 1030, in _gcd_import
  File "&lt;frozen importlib._bootstrap&gt;", line 1007, in _find_and_load
  File "&lt;frozen importlib._bootstrap&gt;", line 972, in _find_and_load_unlocked
  File "&lt;frozen importlib._bootstrap&gt;", line 228, in _call_with_frames_removed
  File "&lt;frozen importlib._bootstrap&gt;", line 1030, in _gcd_import
  File "&lt;frozen importlib._bootstrap&gt;", line 1007, in _find_and_load
  File "&lt;frozen importlib._bootstrap&gt;", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'uvicorn.protocols.websockets'
[32142] Failed to execute script datasette
</code></pre>
<p>I solved this by adding each <code>ModuleNotFoundError</code> module to <code>--hidden-import</code> until it worked.</p>
<p>I've tested this script (and the generated executables) on both macOS and Ubuntu Linux so far, and it's worked perfectly in both cases. See <a href="https://github.com/simonw/datasette/issues/93">issue 93</a> for more details.</p>


path: sqlite_column-combinations.md
topic: sqlite
title: Identifying column combination patterns in a SQLite table
url: https://github.com/simonw/til/blob/main/sqlite/column-combinations.md
created: 2021-01-12T14:00:51-08:00
created_utc: 2021-01-12T22:00:51+00:00
updated: 2022-03-20T22:34:14-07:00
updated_utc: 2022-03-21T05:34:14+00:00
shot_hash: ffad8b4a9d53c870c1cabf6a56cbe68d
slug: column-combinations
body:
Given a large, heterogeneous table I wanted to identify patterns in the rows in terms of which columns were not null.

Imagine a table like this for example:

    id  field1  field2  field3  field4

I want to know how many records have values for (id, field1, field2) compared to the number of rows with values for (id, field3, field4).

I worked out the following query pattern for answering this question:

```sql
select
    case when [pk] is not null then 'pk, ' else '' end ||
    case when [created] is not null then 'created, ' else '' end ||
    case when [planet_int] is not null then 'planet_int, ' else '' end ||
    case when [on_earth] is not null then 'on_earth, ' else '' end ||
    case when [state] is not null then 'state, ' else '' end ||
    case when [_city_id] is not null then '_city_id, ' else '' end ||
    case when [_neighborhood] is not null then '_neighborhood, ' else '' end ||
    case when [tags] is not null then 'tags, ' else '' end ||
    case when [complex_array] is not null then 'complex_array, ' else '' end ||
    case when [distinct_some_null] is not null then 'distinct_some_null, ' else '' end
  as columns,
  count(*) as num_rows
from
  [facetable]
group by
  columns
order by
  num_rows desc
```
[Try that here](https://latest.datasette.io/fixtures?sql=select%0D%0A++++case+when+%5Bpk%5D+is+not+null+then+%27pk%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Bcreated%5D+is+not+null+then+%27created%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Bplanet_int%5D+is+not+null+then+%27planet_int%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Bon_earth%5D+is+not+null+then+%27on_earth%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Bstate%5D+is+not+null+then+%27state%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5B_city_id%5D+is+not+null+then+%27_city_id%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5B_neighborhood%5D+is+not+null+then+%27_neighborhood%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Btags%5D+is+not+null+then+%27tags%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Bcomplex_array%5D+is+not+null+then+%27complex_array%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Bdistinct_some_null%5D+is+not+null+then+%27distinct_some_null%2C+%27+else+%27%27+end%0D%0A++as+columns%2C%0D%0A++count%28*%29+as+num_rows%0D%0Afrom%0D%0A++%5Bfacetable%5D%0D%0Agroup+by%0D%0A++columns%0D%0Aorder+by%0D%0A++num_rows+desc).

This has the desired effect: it gives me back all of the combinations of not-null columns in the table, with a count for each one.

(Running this on a table with 1,000,000+ rows took about 40 seconds, so I had to use `datasette data.db --setting sql_time_limit_ms 100000` to bump up the default time limit in Datasette.)

One remaining problem: how to generate the above query for an arbitrary table. I came up with the following SQL query for generating a SQL query like the above:

```sql
select 'select
' || group_concat('    case when [' || name || '] is not null then ' || quote(name || ', ') || ' else '''' end', ' ||
') || '
  as columns,
  count(*) as num_rows
from
  [' || :table || ']
group by
  columns
order by
  num_rows desc' as query from pragma_table_info(:table)
```
[Try that out](https://latest-with-plugins.datasette.io/fixtures?sql=select+%27select%0D%0A%27+%7C%7C+group_concat%28%27++++case+when+%5B%27+%7C%7C+name+%7C%7C+%27%5D+is+not+null+then+%27+%7C%7C+quote%28name+%7C%7C+%27%2C+%27%29+%7C%7C+%27+else+%27%27%27%27+end%27%2C+%27+%7C%7C%0D%0A%27%29+%7C%7C+%27%0D%0A++as+columns%2C%0D%0A++count%28*%29+as+num_rows%0D%0Afrom%0D%0A++%5B%27+%7C%7C+%3Atable+%7C%7C+%27%5D%0D%0Agroup+by%0D%0A++columns%0D%0Aorder+by%0D%0A++num_rows+desc%27+as+query+from+pragma_table_info%28%3Atable%29&table=facetable) in a demo that includes the [datasette-query-links](https://datasette.io/plugins/datasette-query-links) plugin.
This takes `:table` as an input and generates SQL which can be used to generate column-combination counts.

html:
<p>Given a large, heterogeneous table I wanted to identify patterns in the rows in terms of which columns were not null.</p>
<p>Imagine a table like this for example:</p>
<pre><code>id  field1  field2  field3  field4
</code></pre>
<p>I want to know how many records have values for (id, field1, field2) compared to the number of rows with values for (id, field3, field4).</p>
<p>I worked out the following query pattern for answering this question:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
    case when [pk] <span class="pl-k">is not null</span> then <span class="pl-s"><span class="pl-pds">'</span>pk, <span class="pl-pds">'</span></span> else <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span> end <span class="pl-k">||</span>
    case when [created] <span class="pl-k">is not null</span> then <span class="pl-s"><span class="pl-pds">'</span>created, <span class="pl-pds">'</span></span> else <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span> end <span class="pl-k">||</span>
    case when [planet_int] <span class="pl-k">is not null</span> then <span class="pl-s"><span class="pl-pds">'</span>planet_int, <span class="pl-pds">'</span></span> else <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span> end <span class="pl-k">||</span>
    case when [on_earth] <span class="pl-k">is not null</span> then <span class="pl-s"><span class="pl-pds">'</span>on_earth, <span class="pl-pds">'</span></span> else <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span> end <span class="pl-k">||</span>
    case when [state] <span class="pl-k">is not null</span> then <span class="pl-s"><span class="pl-pds">'</span>state, <span class="pl-pds">'</span></span> else <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span> end <span class="pl-k">||</span>
    case when [_city_id] <span class="pl-k">is not null</span> then <span class="pl-s"><span class="pl-pds">'</span>_city_id, <span class="pl-pds">'</span></span> else <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span> end <span class="pl-k">||</span>
    case when [_neighborhood] <span class="pl-k">is not null</span> then <span class="pl-s"><span class="pl-pds">'</span>_neighborhood, <span class="pl-pds">'</span></span> else <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span> end <span class="pl-k">||</span>
    case when [tags] <span class="pl-k">is not null</span> then <span class="pl-s"><span class="pl-pds">'</span>tags, <span class="pl-pds">'</span></span> else <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span> end <span class="pl-k">||</span>
    case when [complex_array] <span class="pl-k">is not null</span> then <span class="pl-s"><span class="pl-pds">'</span>complex_array, <span class="pl-pds">'</span></span> else <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span> end <span class="pl-k">||</span>
    case when [distinct_some_null] <span class="pl-k">is not null</span> then <span class="pl-s"><span class="pl-pds">'</span>distinct_some_null, <span class="pl-pds">'</span></span> else <span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span> end
  <span class="pl-k">as</span> columns,
  <span class="pl-c1">count</span>(<span class="pl-k">*</span>) <span class="pl-k">as</span> num_rows
<span class="pl-k">from</span>
  [facetable]
<span class="pl-k">group by</span>
  columns
<span class="pl-k">order by</span>
  num_rows <span class="pl-k">desc</span></pre></div>
<p><a href="https://latest.datasette.io/fixtures?sql=select%0D%0A++++case+when+%5Bpk%5D+is+not+null+then+%27pk%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Bcreated%5D+is+not+null+then+%27created%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Bplanet_int%5D+is+not+null+then+%27planet_int%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Bon_earth%5D+is+not+null+then+%27on_earth%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Bstate%5D+is+not+null+then+%27state%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5B_city_id%5D+is+not+null+then+%27_city_id%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5B_neighborhood%5D+is+not+null+then+%27_neighborhood%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Btags%5D+is+not+null+then+%27tags%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Bcomplex_array%5D+is+not+null+then+%27complex_array%2C+%27+else+%27%27+end+%7C%7C%0D%0A++++case+when+%5Bdistinct_some_null%5D+is+not+null+then+%27distinct_some_null%2C+%27+else+%27%27+end%0D%0A++as+columns%2C%0D%0A++count%28*%29+as+num_rows%0D%0Afrom%0D%0A++%5Bfacetable%5D%0D%0Agroup+by%0D%0A++columns%0D%0Aorder+by%0D%0A++num_rows+desc" rel="nofollow">Try that here</a>.</p>
<p>This has the desired effect: it gives me back all of the combinations of not-null columns in the table, with a count for each one.</p>
<p>(Running this on a table with 1,000,000+ rows took about 40 seconds, so I had to use <code>datasette data.db --setting sql_time_limit_ms 100000</code> to bump up the default time limit in Datasette.)</p>
<p>One remaining problem: how to generate the above query for an arbitrary table. I came up with the following SQL query for generating a SQL query like the above:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>select</span>
<span class="pl-s"><span class="pl-pds">'</span></span> <span class="pl-k">||</span> group_concat(<span class="pl-s"><span class="pl-pds">'</span>    case when [<span class="pl-pds">'</span></span> <span class="pl-k">||</span> name <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>] is not null then <span class="pl-pds">'</span></span> <span class="pl-k">||</span> quote(name <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>, <span class="pl-pds">'</span></span>) <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span> else <span class="pl-pds">'</span><span class="pl-pds">'</span><span class="pl-pds">'</span><span class="pl-pds">'</span> end<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span> ||</span>
<span class="pl-s"><span class="pl-pds">'</span></span>) <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s">  as columns,</span>
<span class="pl-s">  count(*) as num_rows</span>
<span class="pl-s">from</span>
<span class="pl-s">  [<span class="pl-pds">'</span></span> <span class="pl-k">||</span> :table <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>]</span>
<span class="pl-s">group by</span>
<span class="pl-s">  columns</span>
<span class="pl-s">order by</span>
<span class="pl-s">  num_rows desc<span class="pl-pds">'</span></span> <span class="pl-k">as</span> query <span class="pl-k">from</span> pragma_table_info(:table)</pre></div>
<p><a href="https://latest-with-plugins.datasette.io/fixtures?sql=select+%27select%0D%0A%27+%7C%7C+group_concat%28%27++++case+when+%5B%27+%7C%7C+name+%7C%7C+%27%5D+is+not+null+then+%27+%7C%7C+quote%28name+%7C%7C+%27%2C+%27%29+%7C%7C+%27+else+%27%27%27%27+end%27%2C+%27+%7C%7C%0D%0A%27%29+%7C%7C+%27%0D%0A++as+columns%2C%0D%0A++count%28*%29+as+num_rows%0D%0Afrom%0D%0A++%5B%27+%7C%7C+%3Atable+%7C%7C+%27%5D%0D%0Agroup+by%0D%0A++columns%0D%0Aorder+by%0D%0A++num_rows+desc%27+as+query+from+pragma_table_info%28%3Atable%29&amp;table=facetable" rel="nofollow">Try that out</a> in a demo that includes the <a href="https://datasette.io/plugins/datasette-query-links" rel="nofollow">datasette-query-links</a> plugin.
This takes <code>:table</code> as an input and generates SQL which can be used to generate column-combination counts.</p>


path: aws_recovering-lightsail-data.md
topic: aws
title: Recovering data from AWS Lightsail using EC2
url: https://github.com/simonw/til/blob/main/aws/recovering-lightsail-data.md
created: 2021-01-16T13:07:24-08:00
created_utc: 2021-01-16T21:07:24+00:00
updated: 2021-01-16T13:07:24-08:00
updated_utc: 2021-01-16T21:07:24+00:00
shot_hash: cccb1e66be6677f244d6dba2918cd3a6
slug: recovering-lightsail-data
body:
I ran into problems with my AWS Lightsail instance: it exceeded the CPU burst quota for too long and was suspended, and I couldn't figure out how to un-suspend it.

I had a snapshot of the hard drive and I wanted to recover the data from it. This ended up taking far longer than I expected - I imagine there's a better way of doing this but here's how I solved it.

Short version: I migrated the snapshot to EC2, then launched an EC2 instance and mounted that snapshot as an EBS volume.

Long version (because I had to figure out a lot of steps along the way):

1. I activated the Lightsail "Export to Amazon EC2" option on the snapshot
2. I waited a while for the export to complete
3. This launched a new EC2 instance for me... but for some reason I couldn't SSH into that instance. So I terminated it.
4. I used the EC2 web console to figure out the AWS identifier for the EC2 copy of the Lightsail snapshot - something like `snap-02a530e12a34`
5. I created a brand new EC2 instance and on the "Add storage" panel I added an EBS volume for `/dev/sdb` with the snapshot identifier I found in the previous step. I started this instance with a keypair so I could SSH into it.
6. I mounted the EBS volume - see section below
7. ... I used `scp` (with the keypair) to copy off the data

## Mounting the EBS volume

I hadn't worked with EBS before so this took some figuring out. My instance was configured with `/dev/sdb` as an EBS volume. I confirmed that the data was accessible like so:

    [ec2-user@ip-172-31-26-179 dev]$ sudo file -s /dev/xvdb
    /dev/xvdb: x86 boot sector; partition 1: ID=0x83, active, starthead 32, startsector 2048, 167770079 sectors, code offset 0x63

Then I created a `/data` directory and mounted the volume:

    [ec2-user@ip-172-31-26-179 dev]$ sudo mkdir /data
    [ec2-user@ip-172-31-26-179 dev]$ sudo mount /dev/xvdb1 /data

I actually tried `sudo mount /dev/xvdb /data` first and got a `mount: /data: wrong fs type` error - [this StackOverflow answer](https://serverfault.com/questions/632905/cannot-mount-an-existing-ebs-on-aws/632906#632906) helped me solve that.

html:
<p>I ran into problems with my AWS Lightsail instance: it exceeded the CPU burst quota for too long and was suspended, and I couldn't figure out how to un-suspend it.</p>
<p>I had a snapshot of the hard drive and I wanted to recover the data from it. This ended up taking far longer than I expected - I imagine there's a better way of doing this but here's how I solved it.</p>
<p>Short version: I migrated the snapshot to EC2, then launched an EC2 instance and mounted that snapshot as an EBS volume.</p>
<p>Long version (because I had to figure out a lot of steps along the way):</p>
<ol>
<li>I activated the Lightsail "Export to Amazon EC2" option on the snapshot</li>
<li>I waited a while for the export to complete</li>
<li>This launched a new EC2 instance for me... but for some reason I couldn't SSH into that instance. So I terminated it.</li>
<li>I used the EC2 web console to figure out the AWS identifier for the EC2 copy of the Lightsail snapshot - something like <code>snap-02a530e12a34</code>
</li>
<li>I created a brand new EC2 instance and on the "Add storage" panel I added an EBS volume for <code>/dev/sdb</code> with the snapshot identifier I found in the previous step. I started this instance with a keypair so I could SSH into it.</li>
<li>I mounted the EBS volume - see section below</li>
<li>... I used <code>scp</code> (with the keypair) to copy off the data</li>
</ol>
<h2>
<a id="user-content-mounting-the-ebs-volume" class="anchor" href="#mounting-the-ebs-volume" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Mounting the EBS volume</h2>
<p>I hadn't worked with EBS before so this took some figuring out. My instance was configured with <code>/dev/sdb</code> as an EBS volume. I confirmed that the data was accessible like so:</p>
<pre><code>[ec2-user@ip-172-31-26-179 dev]$ sudo file -s /dev/xvdb
/dev/xvdb: x86 boot sector; partition 1: ID=0x83, active, starthead 32, startsector 2048, 167770079 sectors, code offset 0x63
</code></pre>
<p>Then I created a <code>/data</code> directory and mounted the volume:</p>
<pre><code>[ec2-user@ip-172-31-26-179 dev]$ sudo mkdir /data
[ec2-user@ip-172-31-26-179 dev]$ sudo mount /dev/xvdb1 /data
</code></pre>
<p>I actually tried <code>sudo mount /dev/xvdb /data</code> first and got a <code>mount: /data: wrong fs type</code> error - <a href="https://serverfault.com/questions/632905/cannot-mount-an-existing-ebs-on-aws/632906#632906" rel="nofollow">this StackOverflow answer</a> helped me solve that.</p>


path: github_bulk-repo-github-graphql.md
topic: github
title: Bulk fetching repository details with the GitHub GraphQL API
url: https://github.com/simonw/til/blob/main/github/bulk-repo-github-graphql.md
created: 2021-01-17T11:49:46-08:00
created_utc: 2021-01-17T19:49:46+00:00
updated: 2021-01-17T11:49:46-08:00
updated_utc: 2021-01-17T19:49:46+00:00
shot_hash: 6929353335fde1f3d56f0ff76fa7dda1
slug: bulk-repo-github-graphql
body:
I wanted to be able to fetch details of a list of different repositories from the GitHub GraphQL API by name in a single operation.

It turns out the `search()` operation can be used for this, given 100 repos at a time. The trick is to use the `repo:` search operator, e.g `repo:simonw/datasette repo:django/django` as demonstrated by [this search](https://github.com/search?q=repo%3Asimonw%2Fdatasette+repo:simonw/sqlite-utils&type=Repositories).

Here's the GraphQL query, tried out using https://docs.github.com/en/graphql/overview/explorer

```graphql
{
  search(type: REPOSITORY, query: "repo:simonw/datasette repo:django/django", first: 100) {
    nodes {
      ... on Repository {
        id
        nameWithOwner
        createdAt
        repositoryTopics(first: 100) {
          totalCount
          nodes {
            topic {
              name
            }
          }
        }
        openIssueCount: issues(states: [OPEN]) {
          totalCount
        }
        closedIssueCount: issues(states: [CLOSED]) {
          totalCount
        }
        releases(last: 1) {
          totalCount
          nodes {
            tagName
          }
        }
      }
    }
  }
}
```

html:
<p>I wanted to be able to fetch details of a list of different repositories from the GitHub GraphQL API by name in a single operation.</p>
<p>It turns out the <code>search()</code> operation can be used for this, given 100 repos at a time. The trick is to use the <code>repo:</code> search operator, e.g <code>repo:simonw/datasette repo:django/django</code> as demonstrated by <a href="https://github.com/search?q=repo%3Asimonw%2Fdatasette+repo:simonw/sqlite-utils&amp;type=Repositories">this search</a>.</p>
<p>Here's the GraphQL query, tried out using <a href="https://docs.github.com/en/graphql/overview/explorer">https://docs.github.com/en/graphql/overview/explorer</a></p>
<div class="highlight highlight-source-graphql"><pre>{
  <span class="pl-v">search</span>(<span class="pl-v">type</span>:<span class="pl-c1"> REPOSITORY</span>, <span class="pl-v">query</span>: <span class="pl-s"><span class="pl-pds">"</span>repo:simonw/datasette repo:django/django<span class="pl-pds">"</span></span>, <span class="pl-v">first</span>: <span class="pl-c1">100</span>) {
    <span class="pl-v">nodes</span> {
      <span class="pl-k">...</span> <span class="pl-k">on</span> <span class="pl-c1">Repository</span> {
        <span class="pl-v">id</span>
        <span class="pl-v">nameWithOwner</span>
        <span class="pl-v">createdAt</span>
        <span class="pl-v">repositoryTopics</span>(<span class="pl-v">first</span>: <span class="pl-c1">100</span>) {
          <span class="pl-v">totalCount</span>
          <span class="pl-v">nodes</span> {
            <span class="pl-v">topic</span> {
              <span class="pl-v">name</span>
            }
          }
        }
        <span class="pl-s">openIssueCount</span>: <span class="pl-v">issues</span>(<span class="pl-v">states</span>: [<span class="pl-c1">OPEN</span>]) {
          <span class="pl-v">totalCount</span>
        }
        <span class="pl-s">closedIssueCount</span>: <span class="pl-v">issues</span>(<span class="pl-v">states</span>: [<span class="pl-c1">CLOSED</span>]) {
          <span class="pl-v">totalCount</span>
        }
        <span class="pl-v">releases</span>(<span class="pl-v">last</span>: <span class="pl-c1">1</span>) {
          <span class="pl-v">totalCount</span>
          <span class="pl-v">nodes</span> {
            <span class="pl-v">tagName</span>
          }
        }
      }
    }
  }
}</pre></div>


path: sqlite_fixing-column-encoding-with-ftfy-and-sqlite-transform.md
topic: sqlite
title: Fixing broken text encodings with sqlite-transform and ftfy
created: 2021-01-18T12:44:30-08:00
created_utc: 2021-01-18T20:44:30+00:00
updated: 2021-01-18T17:27:54-08:00
updated_utc: 2021-01-19T01:27:54+00:00
shot_hash: 87b05e84b44eefabe9e538b89db73dc0
slug: fixing-column-encoding-with-ftfy-and-sqlite-transform
url:
https://github.com/simonw/til/blob/main/sqlite/fixing-column-encoding-with-ftfy-and-sqlite-transform.md

body:
I was working with a database table that included values that were clearly in the wrong character encoding - values like this:

    Rue LÃ©opold I

I used my [sqlite-transform](https://github.com/simonw/sqlite-transform) tool with the [ftfy Python library](https://pypi.org/project/ftfy/) to fix that by running the following:

```bash
sqlite-transform lambda chiens.db espaces-pour-chiens-et-espaces-interdits-aux-chiens namefr \
  --code 'ftfy.fix_encoding(value)' \
  --import ftfy
```

That's the database file, the table and the column, then `--code` and `--import` to specify the transformation.

Since I had installed `sqlite-transform` using `pipx install sqlite-transform` I needed to first install the `ftfy` library into the correct virtual environment. The recipe for doing that is:

```bash
pipx inject sqlite-transform ftfy
```

html:
<p>I was working with a database table that included values that were clearly in the wrong character encoding - values like this:</p>
<pre><code>Rue LÃ©opold I
</code></pre>
<p>I used my <a href="https://github.com/simonw/sqlite-transform">sqlite-transform</a> tool with the <a href="https://pypi.org/project/ftfy/" rel="nofollow">ftfy Python library</a> to fix that by running the following:</p>
<div class="highlight highlight-source-shell"><pre>sqlite-transform lambda chiens.db espaces-pour-chiens-et-espaces-interdits-aux-chiens namefr \
  --code <span class="pl-s"><span class="pl-pds">'</span>ftfy.fix_encoding(value)<span class="pl-pds">'</span></span> \
  --import ftfy</pre></div>
<p>That's the database file, the table and the column, then <code>--code</code> and <code>--import</code> to specify the transformation.</p>
<p>Since I had installed <code>sqlite-transform</code> using <code>pipx install sqlite-transform</code> I needed to first install the <code>ftfy</code> library into the correct virtual environment. The recipe for doing that is:</p>
<div class="highlight highlight-source-shell"><pre>pipx inject sqlite-transform ftfy</pre></div>


path: github_syntax-highlighting-python-console.md
topic: github
title: Syntax highlighting Python console examples with GFM
url: https://github.com/simonw/til/blob/main/github/syntax-highlighting-python-console.md
created: 2021-01-18T17:21:26-08:00
created_utc: 2021-01-19T01:21:26+00:00
updated: 2021-01-18T17:21:26-08:00
updated_utc: 2021-01-19T01:21:26+00:00
shot_hash: da72aecd0ac60de69935a07d7899a45a
slug: syntax-highlighting-python-console
body:
It turns out [GitHub Flavored Markdown](https://github.github.com/gfm/) can apply syntax highlighting to Python console examples, like this one:

```pycon
>>> import csv
>>> with open('eggs.csv', newline='') as csvfile:
...     spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
...     for row in spamreader:
...         print(', '.join(row))
Spam, Spam, Spam, Spam, Spam, Baked Beans
Spam, Lovely Spam, Wonderful Spam
```

The trick is to use the following:

````
```pycon
>>> import csv
>>> with open('eggs.csv', newline='') as csvfile:
...     spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
...     for row in spamreader:
...         print(', '.join(row))
Spam, Spam, Spam, Spam, Spam, Baked Beans
Spam, Lovely Spam, Wonderful Spam
```
````
I figured out the `pycon` code by scanning through the [languages.yml](https://github.com/github/linguist/blob/v7.12.2/lib/linguist/languages.yml#L4406-L4414) file for linguist, the library GitHub use for their syntax highlighting.

While writing this TIL I also learned how to embed triple-backticks in a code block - you surround the block with more-than-three backticks (thanks to [this tip](https://github.com/jonschlinkert/remarkable/issues/146#issuecomment-85539428)):


`````
````
```pycon
>>> import csv
>>> with open('eggs.csv', newline='') as csvfile:
...     spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
...     for row in spamreader:
...         print(', '.join(row))
Spam, Spam, Spam, Spam, Spam, Baked Beans
Spam, Lovely Spam, Wonderful Spam
```
````
`````

html:
<p>It turns out <a href="https://github.github.com/gfm/">GitHub Flavored Markdown</a> can apply syntax highlighting to Python console examples, like this one:</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; <span class="pl-k">import</span> csv
&gt;&gt;&gt; <span class="pl-k">with</span> <span class="pl-c1">open</span>(<span class="pl-s"><span class="pl-pds">'</span>eggs.csv<span class="pl-pds">'</span></span>, <span class="pl-v">newline</span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span><span class="pl-pds">'</span></span>) <span class="pl-k">as</span> csvfile:
...     spamreader <span class="pl-k">=</span> csv.reader(csvfile, <span class="pl-v">delimiter</span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span> <span class="pl-pds">'</span></span>, <span class="pl-v">quotechar</span><span class="pl-k">=</span><span class="pl-s"><span class="pl-pds">'</span>|<span class="pl-pds">'</span></span>)
...     <span class="pl-k">for</span> row <span class="pl-k">in</span> spamreader:
...         <span class="pl-c1">print</span>(<span class="pl-s"><span class="pl-pds">'</span>, <span class="pl-pds">'</span></span>.join(row))
Spam, Spam, Spam, Spam, Spam, Baked Beans
Spam, Lovely Spam, Wonderful Spam</pre></div>
<p>The trick is to use the following:</p>
<pre><code>```pycon
&gt;&gt;&gt; import csv
&gt;&gt;&gt; with open('eggs.csv', newline='') as csvfile:
...     spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
...     for row in spamreader:
...         print(', '.join(row))
Spam, Spam, Spam, Spam, Spam, Baked Beans
Spam, Lovely Spam, Wonderful Spam
```
</code></pre>
<p>I figured out the <code>pycon</code> code by scanning through the <a href="https://github.com/github/linguist/blob/v7.12.2/lib/linguist/languages.yml#L4406-L4414">languages.yml</a> file for linguist, the library GitHub use for their syntax highlighting.</p>
<p>While writing this TIL I also learned how to embed triple-backticks in a code block - you surround the block with more-than-three backticks (thanks to <a href="https://github.com/jonschlinkert/remarkable/issues/146#issuecomment-85539428">this tip</a>):</p>
<pre><code>````
```pycon
&gt;&gt;&gt; import csv
&gt;&gt;&gt; with open('eggs.csv', newline='') as csvfile:
...     spamreader = csv.reader(csvfile, delimiter=' ', quotechar='|')
...     for row in spamreader:
...         print(', '.join(row))
Spam, Spam, Spam, Spam, Spam, Baked Beans
Spam, Lovely Spam, Wonderful Spam
```
````
</code></pre>


path: jupyter_javascript-in-a-jupyter-notebook.md
topic: jupyter
title: Embedding JavaScript in a Jupyter notebook
url: https://github.com/simonw/til/blob/main/jupyter/javascript-in-a-jupyter-notebook.md
created: 2021-01-22T13:52:44-08:00
created_utc: 2021-01-22T21:52:44+00:00
updated: 2021-01-22T13:52:44-08:00
updated_utc: 2021-01-22T21:52:44+00:00
shot_hash: 49ede4cad3d6950905dc35277c4eb29f
slug: javascript-in-a-jupyter-notebook
body:
I recently found out modern browsers include a JavaScript API for creating public/private keys for cryptography.

I used this as an opportunity to figure out how to run JavaScript in a Jupyter notebook cell, using the [IPython.display.Javascript](https://ipython.readthedocs.io/en/stable/api/generated/IPython.display.html#IPython.display.Javascript) mechanism.

This mechanism allows you to execute JavaScript that will be used to render a cell. It includes a copy of jQuery and makes a `element` variable available corresponding to the cell your code is running in. It also provides a `IPython.notebook.kernel.execute()` method which can be used to execute Python code from JavaScript.

Here's what I came up with:

```python
from IPython.display import Javascript, display

display(Javascript("""
function generateKey() {
  window.crypto.subtle
    .generateKey(
      {
        name: "RSA-OAEP",
        modulusLength: 4096,
        publicExponent: new Uint8Array([1, 0, 1]),
        hash: "SHA-256",
      },
      true,
      ["encrypt", "decrypt"]
    )
    .then((keyPair) => {
      crypto.subtle.exportKey("jwk", keyPair.privateKey).then((v) => {
        element.find("textarea:first").val(JSON.stringify(v, null, 4));
        IPython.notebook.kernel.execute(
          "private_key = " + JSON.stringify(JSON.stringify(v, null, 4))
        );
      });
      crypto.subtle.exportKey("jwk", keyPair.publicKey).then((v) => {
        element.find("textarea:last").val(JSON.stringify(v, null, 4));
        IPython.notebook.kernel.execute(
          "public_key = " + JSON.stringify(JSON.stringify(v, null, 4))
        );
      });
    });
}
element.append(`
<h2>Generate a public/private key in JavaScript</h2>
<p><button>Generate key</button></p>
<h3>Private key</h3>
<textarea style="width: 100%; height: 20em"></textarea>
<h3>Public key</h3>
<textarea style="width: 100%; height: 20em"></textarea>
`);
element.find("button").click(generateKey);
"""))
```

This works! I get a new cell in the document containing the HTML from that backtick string at the end of the JavaScript code. When I click the button it runs my `generateKey` function which generates a new key, displays it in the two textareas and makes that key available to the Jupyter environment as the `public_key` and `private_key` variables. I can then use them in subsequent cells from Python like this:

```python
import json
private = json.loads(private_key)
public = json.loads(public_key)
```

html:
<p>I recently found out modern browsers include a JavaScript API for creating public/private keys for cryptography.</p>
<p>I used this as an opportunity to figure out how to run JavaScript in a Jupyter notebook cell, using the <a href="https://ipython.readthedocs.io/en/stable/api/generated/IPython.display.html#IPython.display.Javascript" rel="nofollow">IPython.display.Javascript</a> mechanism.</p>
<p>This mechanism allows you to execute JavaScript that will be used to render a cell. It includes a copy of jQuery and makes a <code>element</code> variable available corresponding to the cell your code is running in. It also provides a <code>IPython.notebook.kernel.execute()</code> method which can be used to execute Python code from JavaScript.</p>
<p>Here's what I came up with:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-v">IPython</span>.<span class="pl-s1">display</span> <span class="pl-k">import</span> <span class="pl-v">Javascript</span>, <span class="pl-s1">display</span>

<span class="pl-en">display</span>(<span class="pl-v">Javascript</span>(<span class="pl-s">"""</span>
<span class="pl-s">function generateKey() {</span>
<span class="pl-s">  window.crypto.subtle</span>
<span class="pl-s">    .generateKey(</span>
<span class="pl-s">      {</span>
<span class="pl-s">        name: "RSA-OAEP",</span>
<span class="pl-s">        modulusLength: 4096,</span>
<span class="pl-s">        publicExponent: new Uint8Array([1, 0, 1]),</span>
<span class="pl-s">        hash: "SHA-256",</span>
<span class="pl-s">      },</span>
<span class="pl-s">      true,</span>
<span class="pl-s">      ["encrypt", "decrypt"]</span>
<span class="pl-s">    )</span>
<span class="pl-s">    .then((keyPair) =&gt; {</span>
<span class="pl-s">      crypto.subtle.exportKey("jwk", keyPair.privateKey).then((v) =&gt; {</span>
<span class="pl-s">        element.find("textarea:first").val(JSON.stringify(v, null, 4));</span>
<span class="pl-s">        IPython.notebook.kernel.execute(</span>
<span class="pl-s">          "private_key = " + JSON.stringify(JSON.stringify(v, null, 4))</span>
<span class="pl-s">        );</span>
<span class="pl-s">      });</span>
<span class="pl-s">      crypto.subtle.exportKey("jwk", keyPair.publicKey).then((v) =&gt; {</span>
<span class="pl-s">        element.find("textarea:last").val(JSON.stringify(v, null, 4));</span>
<span class="pl-s">        IPython.notebook.kernel.execute(</span>
<span class="pl-s">          "public_key = " + JSON.stringify(JSON.stringify(v, null, 4))</span>
<span class="pl-s">        );</span>
<span class="pl-s">      });</span>
<span class="pl-s">    });</span>
<span class="pl-s">}</span>
<span class="pl-s">element.append(`</span>
<span class="pl-s">&lt;h2&gt;Generate a public/private key in JavaScript&lt;/h2&gt;</span>
<span class="pl-s">&lt;p&gt;&lt;button&gt;Generate key&lt;/button&gt;&lt;/p&gt;</span>
<span class="pl-s">&lt;h3&gt;Private key&lt;/h3&gt;</span>
<span class="pl-s">&lt;textarea style="width: 100%; height: 20em"&gt;&lt;/textarea&gt;</span>
<span class="pl-s">&lt;h3&gt;Public key&lt;/h3&gt;</span>
<span class="pl-s">&lt;textarea style="width: 100%; height: 20em"&gt;&lt;/textarea&gt;</span>
<span class="pl-s">`);</span>
<span class="pl-s">element.find("button").click(generateKey);</span>
<span class="pl-s">"""</span>))</pre></div>
<p>This works! I get a new cell in the document containing the HTML from that backtick string at the end of the JavaScript code. When I click the button it runs my <code>generateKey</code> function which generates a new key, displays it in the two textareas and makes that key available to the Jupyter environment as the <code>public_key</code> and <code>private_key</code> variables. I can then use them in subsequent cells from Python like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">json</span>
<span class="pl-s1">private</span> <span class="pl-c1">=</span> <span class="pl-s1">json</span>.<span class="pl-en">loads</span>(<span class="pl-s1">private_key</span>)
<span class="pl-s1">public</span> <span class="pl-c1">=</span> <span class="pl-s1">json</span>.<span class="pl-en">loads</span>(<span class="pl-s1">public_key</span>)</pre></div>


path: cookiecutter_pytest-for-cookiecutter.md
topic: cookiecutter
title: Testing cookiecutter templates with pytest
url: https://github.com/simonw/til/blob/main/cookiecutter/pytest-for-cookiecutter.md
created: 2021-01-27T15:50:02-08:00
created_utc: 2021-01-27T23:50:02+00:00
updated: 2021-01-27T15:58:29-08:00
updated_utc: 2021-01-27T23:58:29+00:00
shot_hash: d71fe0d87b578550b41660a4de61ee0f
slug: pytest-for-cookiecutter
body:
I added some unit tests to my [datasette-plugin](https://github.com/simonw/datasette-plugin) cookiecutter template today, since the latest features involved adding a `hooks/post_gen_project.py` script.

Here's [the full test script](https://github.com/simonw/datasette-plugin/blob/503e6fef8e1000ab70103a61571d47ce966064ba/tests/test_cookiecutter_template.py) I wrote. It lives in `tests/test_cookiecutter_template.py` in the root of the repository.

To run the tests I have to use `pytest tests` because running just `pytest` gets confused when it tries to run the templated tests that form part of the cookiecutter template.

The pattern I'm using looks like this:

```python
from cookiecutter.main import cookiecutter
import pathlib

TEMPLATE_DIRECTORY = str(pathlib.Path(__file__).parent.parent)


def test_static_and_templates(tmpdir):
    cookiecutter(
        template=TEMPLATE_DIRECTORY,
        output_dir=str(tmpdir),
        no_input=True,
        extra_context={
            "plugin_name": "foo",
            "description": "blah",
            "include_templates_directory": "y",
            "include_static_directory": "y",
        },
    )
    assert paths(tmpdir) == {
        "datasette-foo",
        "datasette-foo/.github",
        "datasette-foo/.github/workflows",
        "datasette-foo/.github/workflows/publish.yml",
        "datasette-foo/.github/workflows/test.yml",
        "datasette-foo/.gitignore",
        "datasette-foo/datasette_foo",
        "datasette-foo/datasette_foo/__init__.py",
        "datasette-foo/datasette_foo/static",
        "datasette-foo/datasette_foo/templates",
        "datasette-foo/README.md",
        "datasette-foo/setup.py",
        "datasette-foo/tests",
        "datasette-foo/tests/test_foo.py",
    }
    setup_py = (tmpdir / "datasette-foo" / "setup.py").read_text("utf-8")
    assert (
        'package_data={\n        "datasette_foo": ["static/*", "templates/*"]\n    }'
    ) in setup_py


def paths(directory):
    paths = list(pathlib.Path(directory).glob("**/*"))
    paths = [r.relative_to(directory) for r in paths]
    return {str(f) for f in paths if str(f) != "."}
```

html:
<p>I added some unit tests to my <a href="https://github.com/simonw/datasette-plugin">datasette-plugin</a> cookiecutter template today, since the latest features involved adding a <code>hooks/post_gen_project.py</code> script.</p>
<p>Here's <a href="https://github.com/simonw/datasette-plugin/blob/503e6fef8e1000ab70103a61571d47ce966064ba/tests/test_cookiecutter_template.py">the full test script</a> I wrote. It lives in <code>tests/test_cookiecutter_template.py</code> in the root of the repository.</p>
<p>To run the tests I have to use <code>pytest tests</code> because running just <code>pytest</code> gets confused when it tries to run the templated tests that form part of the cookiecutter template.</p>
<p>The pattern I'm using looks like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">cookiecutter</span>.<span class="pl-s1">main</span> <span class="pl-k">import</span> <span class="pl-s1">cookiecutter</span>
<span class="pl-k">import</span> <span class="pl-s1">pathlib</span>

<span class="pl-v">TEMPLATE_DIRECTORY</span> <span class="pl-c1">=</span> <span class="pl-en">str</span>(<span class="pl-s1">pathlib</span>.<span class="pl-v">Path</span>(<span class="pl-s1">__file__</span>).<span class="pl-s1">parent</span>.<span class="pl-s1">parent</span>)


<span class="pl-k">def</span> <span class="pl-en">test_static_and_templates</span>(<span class="pl-s1">tmpdir</span>):
    <span class="pl-en">cookiecutter</span>(
        <span class="pl-s1">template</span><span class="pl-c1">=</span><span class="pl-v">TEMPLATE_DIRECTORY</span>,
        <span class="pl-s1">output_dir</span><span class="pl-c1">=</span><span class="pl-en">str</span>(<span class="pl-s1">tmpdir</span>),
        <span class="pl-s1">no_input</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
        <span class="pl-s1">extra_context</span><span class="pl-c1">=</span>{
            <span class="pl-s">"plugin_name"</span>: <span class="pl-s">"foo"</span>,
            <span class="pl-s">"description"</span>: <span class="pl-s">"blah"</span>,
            <span class="pl-s">"include_templates_directory"</span>: <span class="pl-s">"y"</span>,
            <span class="pl-s">"include_static_directory"</span>: <span class="pl-s">"y"</span>,
        },
    )
    <span class="pl-k">assert</span> <span class="pl-en">paths</span>(<span class="pl-s1">tmpdir</span>) <span class="pl-c1">==</span> {
        <span class="pl-s">"datasette-foo"</span>,
        <span class="pl-s">"datasette-foo/.github"</span>,
        <span class="pl-s">"datasette-foo/.github/workflows"</span>,
        <span class="pl-s">"datasette-foo/.github/workflows/publish.yml"</span>,
        <span class="pl-s">"datasette-foo/.github/workflows/test.yml"</span>,
        <span class="pl-s">"datasette-foo/.gitignore"</span>,
        <span class="pl-s">"datasette-foo/datasette_foo"</span>,
        <span class="pl-s">"datasette-foo/datasette_foo/__init__.py"</span>,
        <span class="pl-s">"datasette-foo/datasette_foo/static"</span>,
        <span class="pl-s">"datasette-foo/datasette_foo/templates"</span>,
        <span class="pl-s">"datasette-foo/README.md"</span>,
        <span class="pl-s">"datasette-foo/setup.py"</span>,
        <span class="pl-s">"datasette-foo/tests"</span>,
        <span class="pl-s">"datasette-foo/tests/test_foo.py"</span>,
    }
    <span class="pl-s1">setup_py</span> <span class="pl-c1">=</span> (<span class="pl-s1">tmpdir</span> <span class="pl-c1">/</span> <span class="pl-s">"datasette-foo"</span> <span class="pl-c1">/</span> <span class="pl-s">"setup.py"</span>).<span class="pl-en">read_text</span>(<span class="pl-s">"utf-8"</span>)
    <span class="pl-k">assert</span> (
        <span class="pl-s">'package_data={<span class="pl-cce">\n</span>        "datasette_foo": ["static/*", "templates/*"]<span class="pl-cce">\n</span>    }'</span>
    ) <span class="pl-c1">in</span> <span class="pl-s1">setup_py</span>


<span class="pl-k">def</span> <span class="pl-en">paths</span>(<span class="pl-s1">directory</span>):
    <span class="pl-s1">paths</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-s1">pathlib</span>.<span class="pl-v">Path</span>(<span class="pl-s1">directory</span>).<span class="pl-en">glob</span>(<span class="pl-s">"**/*"</span>))
    <span class="pl-s1">paths</span> <span class="pl-c1">=</span> [<span class="pl-s1">r</span>.<span class="pl-en">relative_to</span>(<span class="pl-s1">directory</span>) <span class="pl-k">for</span> <span class="pl-s1">r</span> <span class="pl-c1">in</span> <span class="pl-s1">paths</span>]
    <span class="pl-k">return</span> {<span class="pl-en">str</span>(<span class="pl-s1">f</span>) <span class="pl-k">for</span> <span class="pl-s1">f</span> <span class="pl-c1">in</span> <span class="pl-s1">paths</span> <span class="pl-k">if</span> <span class="pl-en">str</span>(<span class="pl-s1">f</span>) <span class="pl-c1">!=</span> <span class="pl-s">"."</span>}</pre></div>


path: cookiecutter_conditionally-creating-directories.md
topic: cookiecutter
title: Conditionally creating directories in cookiecutter
url: https://github.com/simonw/til/blob/main/cookiecutter/conditionally-creating-directories.md
created: 2021-01-27T15:56:28-08:00
created_utc: 2021-01-27T23:56:28+00:00
updated: 2021-01-27T15:56:28-08:00
updated_utc: 2021-01-27T23:56:28+00:00
shot_hash: e0f45335a94143e5aac8b22e5820e564
slug: conditionally-creating-directories
body:
I wanted my [datasette-plugin](https://github.com/simonw/datasette-plugin) cookiecutter template to create empty `static` and `templates` directories if the user replied `y` to the `include_static_directory` and `include_templates_directory` prompts.

The solution was to add a `hooks/post_gen_project.py` script containing the following:

```python
import os
import shutil


include_static_directory = bool("{{ cookiecutter.include_static_directory }}")
include_templates_directory = bool("{{ cookiecutter.include_templates_directory }}")


if include_static_directory:
    os.makedirs(
        os.path.join(
            os.getcwd(),
            "datasette_{{ cookiecutter.underscored }}",
            "static",
        )
    )


if include_templates_directory:
    os.makedirs(
        os.path.join(
            os.getcwd(),
            "datasette_{{ cookiecutter.underscored }}",
            "templates",
        )
    )
```

Note that these scripts are run through the cookiecutter Jinja template system, so they can use `{{ }}` Jinja syntax to read cookiecutter inputs.

html:
<p>I wanted my <a href="https://github.com/simonw/datasette-plugin">datasette-plugin</a> cookiecutter template to create empty <code>static</code> and <code>templates</code> directories if the user replied <code>y</code> to the <code>include_static_directory</code> and <code>include_templates_directory</code> prompts.</p>
<p>The solution was to add a <code>hooks/post_gen_project.py</code> script containing the following:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">os</span>
<span class="pl-k">import</span> <span class="pl-s1">shutil</span>


<span class="pl-s1">include_static_directory</span> <span class="pl-c1">=</span> <span class="pl-en">bool</span>(<span class="pl-s">"{{ cookiecutter.include_static_directory }}"</span>)
<span class="pl-s1">include_templates_directory</span> <span class="pl-c1">=</span> <span class="pl-en">bool</span>(<span class="pl-s">"{{ cookiecutter.include_templates_directory }}"</span>)


<span class="pl-k">if</span> <span class="pl-s1">include_static_directory</span>:
    <span class="pl-s1">os</span>.<span class="pl-en">makedirs</span>(
        <span class="pl-s1">os</span>.<span class="pl-s1">path</span>.<span class="pl-en">join</span>(
            <span class="pl-s1">os</span>.<span class="pl-en">getcwd</span>(),
            <span class="pl-s">"datasette_{{ cookiecutter.underscored }}"</span>,
            <span class="pl-s">"static"</span>,
        )
    )


<span class="pl-k">if</span> <span class="pl-s1">include_templates_directory</span>:
    <span class="pl-s1">os</span>.<span class="pl-en">makedirs</span>(
        <span class="pl-s1">os</span>.<span class="pl-s1">path</span>.<span class="pl-en">join</span>(
            <span class="pl-s1">os</span>.<span class="pl-en">getcwd</span>(),
            <span class="pl-s">"datasette_{{ cookiecutter.underscored }}"</span>,
            <span class="pl-s">"templates"</span>,
        )
    )</pre></div>
<p>Note that these scripts are run through the cookiecutter Jinja template system, so they can use <code>{{ }}</code> Jinja syntax to read cookiecutter inputs.</p>


path: sqlite_splitting-commas-sqlite.md
topic: sqlite
title: Splitting on commas in SQLite
url: https://github.com/simonw/til/blob/main/sqlite/splitting-commas-sqlite.md
created: 2021-02-01T11:02:59-08:00
created_utc: 2021-02-01T19:02:59+00:00
updated: 2021-02-01T11:02:59-08:00
updated_utc: 2021-02-01T19:02:59+00:00
shot_hash: f78e24bf1cd7d69693908b4aa1fb85ea
slug: splitting-commas-sqlite
body:
I had an input string in `x,y,z` format and I needed to split it into three separate values in SQLite. I managed to do it using a confusing combination of the `instr()` and `substr()` functions.

Here's what I came up with:

```sql
with comma_locations as (
  select instr(:path, ',') as first_comma,
  instr(:path, ',') + instr(substr(:path, instr(:path, ',') + 1), ',') as second_comma
), variables as (
  select
    substr(:path, 0, first_comma) as first,
    substr(:path, first_comma + 1, second_comma - first_comma - 1) as second,
    substr(:path, second_comma + 1) as third
  from comma_locations
)
select * from variables
```

Against an input of `x12,y1234,z12345` it returns this:

| first | second | third |
| --- | --- | --- |
| x12 | y1234 | z12345 |

Here's [a live demo of the query](https://latest.datasette.io/fixtures?sql=with+comma_locations+as+%28%0D%0A++select+instr%28%3Apath%2C+%27%2C%27%29+as+first_comma%2C%0D%0A++instr%28%3Apath%2C+%27%2C%27%29+%2B+instr%28substr%28%3Apath%2C+instr%28%3Apath%2C+%27%2C%27%29+%2B+1%29%2C+%27%2C%27%29+as+second_comma%0D%0A%29%2C+variables+as+%28%0D%0A++select%0D%0A++++substr%28%3Apath%2C+0%2C+first_comma%29+as+first%2C%0D%0A++++substr%28%3Apath%2C+first_comma+%2B+1%2C+second_comma+-+first_comma+-+1%29+as+second%2C%0D%0A++++substr%28%3Apath%2C+second_comma+%2B+1%29+as+third%0D%0A++from+comma_locations%0D%0A%29%0D%0Aselect+*+from+variables&path=x12%2Cy1234%2Cz12345).

html:
<p>I had an input string in <code>x,y,z</code> format and I needed to split it into three separate values in SQLite. I managed to do it using a confusing combination of the <code>instr()</code> and <code>substr()</code> functions.</p>
<p>Here's what I came up with:</p>
<div class="highlight highlight-source-sql"><pre>with comma_locations <span class="pl-k">as</span> (
  <span class="pl-k">select</span> instr(:<span class="pl-k">path</span>, <span class="pl-s"><span class="pl-pds">'</span>,<span class="pl-pds">'</span></span>) <span class="pl-k">as</span> first_comma,
  instr(:<span class="pl-k">path</span>, <span class="pl-s"><span class="pl-pds">'</span>,<span class="pl-pds">'</span></span>) <span class="pl-k">+</span> instr(substr(:<span class="pl-k">path</span>, instr(:<span class="pl-k">path</span>, <span class="pl-s"><span class="pl-pds">'</span>,<span class="pl-pds">'</span></span>) <span class="pl-k">+</span> <span class="pl-c1">1</span>), <span class="pl-s"><span class="pl-pds">'</span>,<span class="pl-pds">'</span></span>) <span class="pl-k">as</span> second_comma
), variables <span class="pl-k">as</span> (
  <span class="pl-k">select</span>
    substr(:<span class="pl-k">path</span>, <span class="pl-c1">0</span>, first_comma) <span class="pl-k">as</span> first,
    substr(:<span class="pl-k">path</span>, first_comma <span class="pl-k">+</span> <span class="pl-c1">1</span>, second_comma <span class="pl-k">-</span> first_comma <span class="pl-k">-</span> <span class="pl-c1">1</span>) <span class="pl-k">as</span> second,
    substr(:<span class="pl-k">path</span>, second_comma <span class="pl-k">+</span> <span class="pl-c1">1</span>) <span class="pl-k">as</span> third
  <span class="pl-k">from</span> comma_locations
)
<span class="pl-k">select</span> <span class="pl-k">*</span> <span class="pl-k">from</span> variables</pre></div>
<p>Against an input of <code>x12,y1234,z12345</code> it returns this:</p>
<table>
<thead>
<tr>
<th>first</th>
<th>second</th>
<th>third</th>
</tr>
</thead>
<tbody>
<tr>
<td>x12</td>
<td>y1234</td>
<td>z12345</td>
</tr>
</tbody>
</table>
<p>Here's <a href="https://latest.datasette.io/fixtures?sql=with+comma_locations+as+%28%0D%0A++select+instr%28%3Apath%2C+%27%2C%27%29+as+first_comma%2C%0D%0A++instr%28%3Apath%2C+%27%2C%27%29+%2B+instr%28substr%28%3Apath%2C+instr%28%3Apath%2C+%27%2C%27%29+%2B+1%29%2C+%27%2C%27%29+as+second_comma%0D%0A%29%2C+variables+as+%28%0D%0A++select%0D%0A++++substr%28%3Apath%2C+0%2C+first_comma%29+as+first%2C%0D%0A++++substr%28%3Apath%2C+first_comma+%2B+1%2C+second_comma+-+first_comma+-+1%29+as+second%2C%0D%0A++++substr%28%3Apath%2C+second_comma+%2B+1%29+as+third%0D%0A++from+comma_locations%0D%0A%29%0D%0Aselect+*+from+variables&amp;path=x12%2Cy1234%2Cz12345" rel="nofollow">a live demo of the query</a>.</p>


path: datasette_serving-mbtiles.md
topic: datasette
title: Serving MBTiles with datasette-media
url: https://github.com/simonw/til/blob/main/datasette/serving-mbtiles.md
created: 2021-02-03T15:12:05-08:00
created_utc: 2021-02-03T23:12:05+00:00
updated: 2021-02-03T15:12:05-08:00
updated_utc: 2021-02-03T23:12:05+00:00
shot_hash: cc9a5cbba9f61f58837d25d2b323bcbc
slug: serving-mbtiles
body:
The [MBTiles](https://github.com/mapbox/mbtiles-spec) format uses SQLite to bundle map tiles for use with libraries such as Leaflet.

I figured out how to use the [datasette-media](https://datasette.io/plugins/datasette-media) to serve tiles from this MBTiles file containing two zoom levels of tiles for San Francisco: https://static.simonwillison.net/static/2021/San_Francisco.mbtiles

This TIL is now entirely obsolete: I used this prototype to build the new [datasette-tiles](https://datasette.io/plugins/datasette-tiles) plugin.

```yaml
plugins:
  datasette-cluster-map:
    tile_layer: "/-/media/tiles/{z},{x},{y}"
    tile_layer_options:
      attribution: "© OpenStreetMap contributors"
      tms: 1
      bounds: [[37.61746256103807, -122.57290320721465],[37.85395101481279, -122.27695899334748]]
      minZoom: 15
      maxZoom: 16
  datasette-media:
    tiles:
      database: San_Francisco
      sql:
        with comma_locations as (
          select instr(:key, ',') as first_comma,
          instr(:key, ',') + instr(substr(:key, instr(:key, ',') + 1), ',') as second_comma
        ),
        variables as (
          select
            substr(:key, 0, first_comma) as z,
            substr(:key, first_comma + 1, second_comma - first_comma - 1) as x,
            substr(:key, second_comma + 1) as y
          from comma_locations
        )
        select
          tile_data as content,
          'image/png' as content_type
        from
          tiles, variables
        where
          zoom_level = variables.z
          and tile_column = variables.x
          and tile_row = variables.y
```

html:
<p>The <a href="https://github.com/mapbox/mbtiles-spec">MBTiles</a> format uses SQLite to bundle map tiles for use with libraries such as Leaflet.</p>
<p>I figured out how to use the <a href="https://datasette.io/plugins/datasette-media" rel="nofollow">datasette-media</a> to serve tiles from this MBTiles file containing two zoom levels of tiles for San Francisco: <a href="https://static.simonwillison.net/static/2021/San_Francisco.mbtiles" rel="nofollow">https://static.simonwillison.net/static/2021/San_Francisco.mbtiles</a></p>
<p>This TIL is now entirely obsolete: I used this prototype to build the new <a href="https://datasette.io/plugins/datasette-tiles" rel="nofollow">datasette-tiles</a> plugin.</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">plugins</span>:
  <span class="pl-ent">datasette-cluster-map</span>:
    <span class="pl-ent">tile_layer</span>: <span class="pl-s"><span class="pl-pds">"</span>/-/media/tiles/{z},{x},{y}<span class="pl-pds">"</span></span>
    <span class="pl-ent">tile_layer_options</span>:
      <span class="pl-ent">attribution</span>: <span class="pl-s"><span class="pl-pds">"</span>© OpenStreetMap contributors<span class="pl-pds">"</span></span>
      <span class="pl-ent">tms</span>: <span class="pl-c1">1</span>
      <span class="pl-ent">bounds</span>: <span class="pl-s">[[37.61746256103807, -122.57290320721465],[37.85395101481279, -122.27695899334748]]</span>
      <span class="pl-ent">minZoom</span>: <span class="pl-c1">15</span>
      <span class="pl-ent">maxZoom</span>: <span class="pl-c1">16</span>
  <span class="pl-ent">datasette-media</span>:
    <span class="pl-ent">tiles</span>:
      <span class="pl-ent">database</span>: <span class="pl-s">San_Francisco</span>
      <span class="pl-ent">sql</span>:
        <span class="pl-s">with comma_locations as (</span>
          <span class="pl-s">select instr(:key, ',') as first_comma,</span>
          <span class="pl-s">instr(:key, ',') + instr(substr(:key, instr(:key, ',') + 1), ',') as second_comma</span>
        <span class="pl-s">),</span>
        <span class="pl-s">variables as (</span>
          <span class="pl-s">select</span>
            <span class="pl-s">substr(:key, 0, first_comma) as z,</span>
            <span class="pl-s">substr(:key, first_comma + 1, second_comma - first_comma - 1) as x,</span>
            <span class="pl-s">substr(:key, second_comma + 1) as y</span>
          <span class="pl-s">from comma_locations</span>
        <span class="pl-s">)</span>
        <span class="pl-s">select</span>
          <span class="pl-s">tile_data as content,</span>
          <span class="pl-s"><span class="pl-pds">'</span>image/png<span class="pl-pds">'</span></span> <span class="pl-s">as content_type</span>
        <span class="pl-s">from</span>
          <span class="pl-s">tiles, variables</span>
        <span class="pl-s">where</span>
          <span class="pl-s">zoom_level = variables.z</span>
          <span class="pl-s">and tile_column = variables.x</span>
          <span class="pl-s">and tile_row = variables.y</span></pre></div>


path: gis_mapzen-elevation-tiles.md
topic: gis
title: Downloading MapZen elevation tiles
url: https://github.com/simonw/til/blob/main/gis/mapzen-elevation-tiles.md
created: 2021-02-04T10:48:48-08:00
created_utc: 2021-02-04T18:48:48+00:00
updated: 2021-02-04T10:48:48-08:00
updated_utc: 2021-02-04T18:48:48+00:00
shot_hash: eac3531e08b6e6ded4c323148bf26b69
slug: mapzen-elevation-tiles
body:
[Via Tony Hirst](https://twitter.com/psychemedia/status/1357280624319553537) I found out about [MapZen's elevation tiles](https://www.mapzen.com/blog/terrain-tile-service/), which encode elevation data in PNG and other formats.

These days they live at https://registry.opendata.aws/terrain-tiles/

I managed to download a subset of them using [download-tiles](https://datasette.io/tools/download-tiles) like so:

```
download-tiles elevation.mbtiles -z 0-4 \
  --tiles-url='https://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png'
```
I'm worried I may have got the x and y the wrong way round though, see comments on https://github.com/simonw/datasette-tiles/issues/15

html:
<p><a href="https://twitter.com/psychemedia/status/1357280624319553537" rel="nofollow">Via Tony Hirst</a> I found out about <a href="https://www.mapzen.com/blog/terrain-tile-service/" rel="nofollow">MapZen's elevation tiles</a>, which encode elevation data in PNG and other formats.</p>
<p>These days they live at <a href="https://registry.opendata.aws/terrain-tiles/" rel="nofollow">https://registry.opendata.aws/terrain-tiles/</a></p>
<p>I managed to download a subset of them using <a href="https://datasette.io/tools/download-tiles" rel="nofollow">download-tiles</a> like so:</p>
<pre><code>download-tiles elevation.mbtiles -z 0-4 \
  --tiles-url='https://s3.amazonaws.com/elevation-tiles-prod/terrarium/{z}/{x}/{y}.png'
</code></pre>
<p>I'm worried I may have got the x and y the wrong way round though, see comments on <a href="https://github.com/simonw/datasette-tiles/issues/15">https://github.com/simonw/datasette-tiles/issues/15</a></p>


path: macos_shrinking-pngs-with-pngquant-and-oxipng.md
topic: macos
title: Shrinking PNG files with pngquant and oxipng
url: https://github.com/simonw/til/blob/main/macos/shrinking-pngs-with-pngquant-and-oxipng.md
created: 2021-02-07T11:57:17-08:00
created_utc: 2021-02-07T19:57:17+00:00
updated: 2021-02-07T11:57:17-08:00
updated_utc: 2021-02-07T19:57:17+00:00
shot_hash: c72662ce5c2e4c3591cd562b98b9f8fb
slug: shrinking-pngs-with-pngquant-and-oxipng
body:
I usually use [Squoosh.app](https://squoosh.app/) to reduce the size of my PNGs, but in this case I had a folder with nearly 50 images in it so I wanted to do it using the command-line.

[pngquant](https://pngquant.org/) can reduce the number of colours in a PNG image, which I find makes a huge differente to the file size (also possible using Squoosh).

[oxipng](https://github.com/shssoichiro/oxipng) is a performante lossless PNG compressor.

I got great results by running `pngquant` first, then `oxipng` on the results.

Both can be installed via Homebrew:

    brew install pngquant oxipng

Then I ran this command to reduce to a maximum of 50 colours per image:

    pngquant --quality 20-50 *.png

(I don't know if the lower bound of 20 is the right thing to do here, maybe `0-50` would be better?)

This creates a file called `x-fs8.png` for any file called `x.png`.

Then I ran this to apply `oxipng`:

    oxipng -o 3 -i 0 --strip safe *-fs8.png

The results were that a 383KB file dropped down to just 70KB. The images are visually different but the size savings are huge, which is particularly important if you plan to put 50+ images on a single web page.

html:
<p>I usually use <a href="https://squoosh.app/" rel="nofollow">Squoosh.app</a> to reduce the size of my PNGs, but in this case I had a folder with nearly 50 images in it so I wanted to do it using the command-line.</p>
<p><a href="https://pngquant.org/" rel="nofollow">pngquant</a> can reduce the number of colours in a PNG image, which I find makes a huge differente to the file size (also possible using Squoosh).</p>
<p><a href="https://github.com/shssoichiro/oxipng">oxipng</a> is a performante lossless PNG compressor.</p>
<p>I got great results by running <code>pngquant</code> first, then <code>oxipng</code> on the results.</p>
<p>Both can be installed via Homebrew:</p>
<pre><code>brew install pngquant oxipng
</code></pre>
<p>Then I ran this command to reduce to a maximum of 50 colours per image:</p>
<pre><code>pngquant --quality 20-50 *.png
</code></pre>
<p>(I don't know if the lower bound of 20 is the right thing to do here, maybe <code>0-50</code> would be better?)</p>
<p>This creates a file called <code>x-fs8.png</code> for any file called <code>x.png</code>.</p>
<p>Then I ran this to apply <code>oxipng</code>:</p>
<pre><code>oxipng -o 3 -i 0 --strip safe *-fs8.png
</code></pre>
<p>The results were that a 383KB file dropped down to just 70KB. The images are visually different but the size savings are huge, which is particularly important if you plan to put 50+ images on a single web page.</p>


path: python_csv-error-column-too-large.md
topic: python
title: Handling CSV files with wide columns in Python
url: https://github.com/simonw/til/blob/main/python/csv-error-column-too-large.md
created: 2021-02-15T11:00:59-08:00
created_utc: 2021-02-15T19:00:59+00:00
updated: 2021-02-15T11:00:59-08:00
updated_utc: 2021-02-15T19:00:59+00:00
shot_hash: eb29a63da51e7ba07ab2ffec696cbf48
slug: csv-error-column-too-large
body:
Users [were reporting](https://github.com/simonw/sqlite-utils/issues/229) the following error using `sqlite-utils` to import some CSV files:

    _csv.Error: field larger than field limit (131072)

It turns out the Python standard library CSV module enforces a default field size limit on columns, and anything with more than 128KB of text in a column will raise an error.

You can modify this error using the [csv.field_size_limit(new_limit)](https://docs.python.org/3/library/csv.html#csv.field_size_limit) function.

There's one catch: the method doesn't provide a way to say "no limit". And it can throw an error if you feed it a value that is larger than the C long integer size on your platform.

So how do you set it to the maximum possible value? There's [an extensive StackOverflow thread](https://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072) about this, with a number of different proposed solutions. Several of those use `ctypes` to find the correct value.

I didn't want to add a `ctypes` dependency out of paranoia that someone would try to use my library on a platform that didn't support it (I don't know if that paranoia has any basis at all). So I picked this pattern, [suggested by](https://stackoverflow.com/a/15063941) StackOverflow user **user1251007**:

```python
# Increase CSV field size limit to maximim possible
# https://stackoverflow.com/a/15063941
field_size_limit = sys.maxsize

while True:
    try:
        csv_std.field_size_limit(field_size_limit)
        break
    except OverflowError:
        field_size_limit = int(field_size_limit / 10)
```
This appears to work just fine. On macOS `sys.maxsize` works already, and on other platforms it should pick a field size limit that works.

html:
<p>Users <a href="https://github.com/simonw/sqlite-utils/issues/229">were reporting</a> the following error using <code>sqlite-utils</code> to import some CSV files:</p>
<pre><code>_csv.Error: field larger than field limit (131072)
</code></pre>
<p>It turns out the Python standard library CSV module enforces a default field size limit on columns, and anything with more than 128KB of text in a column will raise an error.</p>
<p>You can modify this error using the <a href="https://docs.python.org/3/library/csv.html#csv.field_size_limit" rel="nofollow">csv.field_size_limit(new_limit)</a> function.</p>
<p>There's one catch: the method doesn't provide a way to say "no limit". And it can throw an error if you feed it a value that is larger than the C long integer size on your platform.</p>
<p>So how do you set it to the maximum possible value? There's <a href="https://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072" rel="nofollow">an extensive StackOverflow thread</a> about this, with a number of different proposed solutions. Several of those use <code>ctypes</code> to find the correct value.</p>
<p>I didn't want to add a <code>ctypes</code> dependency out of paranoia that someone would try to use my library on a platform that didn't support it (I don't know if that paranoia has any basis at all). So I picked this pattern, <a href="https://stackoverflow.com/a/15063941" rel="nofollow">suggested by</a> StackOverflow user <strong>user1251007</strong>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-c"># Increase CSV field size limit to maximim possible</span>
<span class="pl-c"># https://stackoverflow.com/a/15063941</span>
<span class="pl-s1">field_size_limit</span> <span class="pl-c1">=</span> <span class="pl-s1">sys</span>.<span class="pl-s1">maxsize</span>

<span class="pl-k">while</span> <span class="pl-c1">True</span>:
    <span class="pl-k">try</span>:
        <span class="pl-s1">csv_std</span>.<span class="pl-en">field_size_limit</span>(<span class="pl-s1">field_size_limit</span>)
        <span class="pl-k">break</span>
    <span class="pl-k">except</span> <span class="pl-v">OverflowError</span>:
        <span class="pl-s1">field_size_limit</span> <span class="pl-c1">=</span> <span class="pl-en">int</span>(<span class="pl-s1">field_size_limit</span> <span class="pl-c1">/</span> <span class="pl-c1">10</span>)</pre></div>
<p>This appears to work just fine. On macOS <code>sys.maxsize</code> works already, and on other platforms it should pick a field size limit that works.</p>


path: python_io-bufferedreader.md
topic: python
title: Using io.BufferedReader to peek against a non-peekable stream
url: https://github.com/simonw/til/blob/main/python/io-bufferedreader.md
created: 2021-02-15T11:17:28-08:00
created_utc: 2021-02-15T19:17:28+00:00
updated: 2021-02-15T11:17:28-08:00
updated_utc: 2021-02-15T19:17:28+00:00
shot_hash: 5b239049684feada5a90ece879f1e874
slug: io-bufferedreader
body:
When building the [--sniff option](https://github.com/simonw/sqlite-utils/issues/230) for `sqlite-utils insert` (which attempts to detect the correct CSV delimiter and quote character by looking at the first 2048 bytes of a CSV file) I had the need to peek ahead in an incoming stream of data.

I use Click, and Click can automatically handle both files and standard input. The problem I had is that peeking ahead in a file is easy (you can call `.read()` and then `.seek(0)`, or use the `.peek()` method directly) but peaking ahead in standard input is not - anything you consume from that is not available to rewind to later on.

Since my code works by passing a file-like object to the `csv.reader()` function I needed a way to read the first 2048 bytes but then reset the stream ready for that function to consume it.

I figured out how to do that using the `io.BufferedReader` class. Here's the pattern:

```python
import io
import sys
import csv

# Get a file-like object in binary mode
fp = open("myfile.csv", "rb")
# Or from standard input (need to use .buffer here)
fp = sys.stdin.buffer

# Wrap it in a buffered reader with a 4096 byte buffer
buffered = io.BufferedReader(fp, buffer_size=4096)

# Wrap THAT in a text io wrapper that can decode to unicode
decoded = io.TextIOWrapper(buffered, encoding="utf-8")

# Now I can read the first 2048 bytes...
first_bytes = buffered.peek(2048)

# But I can still pass the "decoded" object to csv.reader
reader = csv.reader(decoded)
for row in reader:
    print(row)
```
My implementation is in [this commit](https://github.com/simonw/sqlite-utils/commit/99ff0a288c08ec2071139c6031eb880fa9c95310).

html:
<p>When building the <a href="https://github.com/simonw/sqlite-utils/issues/230">--sniff option</a> for <code>sqlite-utils insert</code> (which attempts to detect the correct CSV delimiter and quote character by looking at the first 2048 bytes of a CSV file) I had the need to peek ahead in an incoming stream of data.</p>
<p>I use Click, and Click can automatically handle both files and standard input. The problem I had is that peeking ahead in a file is easy (you can call <code>.read()</code> and then <code>.seek(0)</code>, or use the <code>.peek()</code> method directly) but peaking ahead in standard input is not - anything you consume from that is not available to rewind to later on.</p>
<p>Since my code works by passing a file-like object to the <code>csv.reader()</code> function I needed a way to read the first 2048 bytes but then reset the stream ready for that function to consume it.</p>
<p>I figured out how to do that using the <code>io.BufferedReader</code> class. Here's the pattern:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">io</span>
<span class="pl-k">import</span> <span class="pl-s1">sys</span>
<span class="pl-k">import</span> <span class="pl-s1">csv</span>

<span class="pl-c"># Get a file-like object in binary mode</span>
<span class="pl-s1">fp</span> <span class="pl-c1">=</span> <span class="pl-en">open</span>(<span class="pl-s">"myfile.csv"</span>, <span class="pl-s">"rb"</span>)
<span class="pl-c"># Or from standard input (need to use .buffer here)</span>
<span class="pl-s1">fp</span> <span class="pl-c1">=</span> <span class="pl-s1">sys</span>.<span class="pl-s1">stdin</span>.<span class="pl-s1">buffer</span>

<span class="pl-c"># Wrap it in a buffered reader with a 4096 byte buffer</span>
<span class="pl-s1">buffered</span> <span class="pl-c1">=</span> <span class="pl-s1">io</span>.<span class="pl-v">BufferedReader</span>(<span class="pl-s1">fp</span>, <span class="pl-s1">buffer_size</span><span class="pl-c1">=</span><span class="pl-c1">4096</span>)

<span class="pl-c"># Wrap THAT in a text io wrapper that can decode to unicode</span>
<span class="pl-s1">decoded</span> <span class="pl-c1">=</span> <span class="pl-s1">io</span>.<span class="pl-v">TextIOWrapper</span>(<span class="pl-s1">buffered</span>, <span class="pl-s1">encoding</span><span class="pl-c1">=</span><span class="pl-s">"utf-8"</span>)

<span class="pl-c"># Now I can read the first 2048 bytes...</span>
<span class="pl-s1">first_bytes</span> <span class="pl-c1">=</span> <span class="pl-s1">buffered</span>.<span class="pl-en">peek</span>(<span class="pl-c1">2048</span>)

<span class="pl-c"># But I can still pass the "decoded" object to csv.reader</span>
<span class="pl-s1">reader</span> <span class="pl-c1">=</span> <span class="pl-s1">csv</span>.<span class="pl-en">reader</span>(<span class="pl-s1">decoded</span>)
<span class="pl-k">for</span> <span class="pl-s1">row</span> <span class="pl-c1">in</span> <span class="pl-s1">reader</span>:
    <span class="pl-en">print</span>(<span class="pl-s1">row</span>)</pre></div>
<p>My implementation is in <a href="https://github.com/simonw/sqlite-utils/commit/99ff0a288c08ec2071139c6031eb880fa9c95310">this commit</a>.</p>


path: jq_radio-garden-jq.md
topic: jq
title: Loading radio.garden into SQLite using jq
url: https://github.com/simonw/til/blob/main/jq/radio-garden-jq.md
created: 2021-02-17T16:44:29-08:00
created_utc: 2021-02-18T00:44:29+00:00
updated: 2021-02-17T16:49:50-08:00
updated_utc: 2021-02-18T00:49:50+00:00
shot_hash: bda38b0e66ac41b9fb4f6b172164116c
slug: radio-garden-jq
body:
http://radio.garden/ is an amazing website which displays a 3D globe covered in radio stations and lets you listen to any of them.

I wanted to have a play around with their data. I fired up the Firefox Developer Tools, switched to the Network tab and sorted by largest first, which revealed this URL to a 1.4MB JSON file:

    http://radio.garden/api/ara/content/places

I ran that through `jq` to pretty print it and see what it looked like:

    % cat places | jq | head -n 20 
    {
      "apiVersion": 1,
      "version": "e56d3ed3",
      "data": {
        "list": [
          {
            "id": "GBy0N9TE",
            "title": "Sukhumi",
            "country": "Abkhazia, Georgia",
            "url": "/visit/sukhumi/GBy0N9TE",
            "size": 1,
            "boost": true,
            "geo": [
              41.023,
              43.001
            ]
          },
          {
            "id": "PbqG2Mmi",
            "title": "Ghazni",

I need a flat JSON list to load this into `sqlite-utils`. I used `jq` to reshape the data like so:

    % cat places | jq '[.data.list[] | {
        id: .id,
        title: .title,
        country: .country,
        url: .url,
        size: .size,
        latitude: .geo[1],
        longitude: .geo[0]
    }]' | head -n 20
    [
      {
        "id": "GBy0N9TE",
        "title": "Sukhumi",
        "country": "Abkhazia, Georgia",
        "url": "/visit/sukhumi/GBy0N9TE",
        "size": 1,
        "latitude": 43.001,
        "longitude": 41.023
      },
      {
        "id": "PbqG2Mmi",
        "title": "Ghazni",
        "country": "Afghanistan",
        "url": "/visit/ghazni/PbqG2Mmi",
        "size": 1,
        "latitude": 33.545,
        "longitude": 68.417
      },
      {

Then I piped that result to `sqlite-utils`:

    % cat places | jq '[.data.list[] | {
        id: .id,
        title: .title,
        country: .country,
        url: .url,
        size: .size,
        latitude: .geo[1],
        longitude: .geo[0]
    }]' | sqlite-utils insert radio.db stations - --pk=id

Here's the whole process as a one-liner:

    curl http://radio.garden/api/ara/content/places | jq '[.data.list[] | {
        id: .id,
        title: .title,
        country: .country,
        url: .url,
        size: .size,
        latitude: .geo[1],
        longitude: .geo[0]
    }]' | sqlite-utils insert radio.db stations - --pk=id

I then opened it in Datasette so I could do things like see it on a map and facet by country:

    datasette install datasette-cluster-map
    datasette radio.db -o

html:
<p><a href="http://radio.garden/" rel="nofollow">http://radio.garden/</a> is an amazing website which displays a 3D globe covered in radio stations and lets you listen to any of them.</p>
<p>I wanted to have a play around with their data. I fired up the Firefox Developer Tools, switched to the Network tab and sorted by largest first, which revealed this URL to a 1.4MB JSON file:</p>
<pre><code>http://radio.garden/api/ara/content/places
</code></pre>
<p>I ran that through <code>jq</code> to pretty print it and see what it looked like:</p>
<pre><code>% cat places | jq | head -n 20 
{
  "apiVersion": 1,
  "version": "e56d3ed3",
  "data": {
    "list": [
      {
        "id": "GBy0N9TE",
        "title": "Sukhumi",
        "country": "Abkhazia, Georgia",
        "url": "/visit/sukhumi/GBy0N9TE",
        "size": 1,
        "boost": true,
        "geo": [
          41.023,
          43.001
        ]
      },
      {
        "id": "PbqG2Mmi",
        "title": "Ghazni",
</code></pre>
<p>I need a flat JSON list to load this into <code>sqlite-utils</code>. I used <code>jq</code> to reshape the data like so:</p>
<pre><code>% cat places | jq '[.data.list[] | {
    id: .id,
    title: .title,
    country: .country,
    url: .url,
    size: .size,
    latitude: .geo[1],
    longitude: .geo[0]
}]' | head -n 20
[
  {
    "id": "GBy0N9TE",
    "title": "Sukhumi",
    "country": "Abkhazia, Georgia",
    "url": "/visit/sukhumi/GBy0N9TE",
    "size": 1,
    "latitude": 43.001,
    "longitude": 41.023
  },
  {
    "id": "PbqG2Mmi",
    "title": "Ghazni",
    "country": "Afghanistan",
    "url": "/visit/ghazni/PbqG2Mmi",
    "size": 1,
    "latitude": 33.545,
    "longitude": 68.417
  },
  {
</code></pre>
<p>Then I piped that result to <code>sqlite-utils</code>:</p>
<pre><code>% cat places | jq '[.data.list[] | {
    id: .id,
    title: .title,
    country: .country,
    url: .url,
    size: .size,
    latitude: .geo[1],
    longitude: .geo[0]
}]' | sqlite-utils insert radio.db stations - --pk=id
</code></pre>
<p>Here's the whole process as a one-liner:</p>
<pre><code>curl http://radio.garden/api/ara/content/places | jq '[.data.list[] | {
    id: .id,
    title: .title,
    country: .country,
    url: .url,
    size: .size,
    latitude: .geo[1],
    longitude: .geo[0]
}]' | sqlite-utils insert radio.db stations - --pk=id
</code></pre>
<p>I then opened it in Datasette so I could do things like see it on a map and facet by country:</p>
<pre><code>datasette install datasette-cluster-map
datasette radio.db -o
</code></pre>


path: sphinx_sphinx-ext-extlinks.md
topic: sphinx
title: Using sphinx.ext.extlinks for issue links
url: https://github.com/simonw/til/blob/main/sphinx/sphinx-ext-extlinks.md
created: 2021-02-17T17:27:04-08:00
created_utc: 2021-02-18T01:27:04+00:00
updated: 2021-02-17T17:27:04-08:00
updated_utc: 2021-02-18T01:27:04+00:00
shot_hash: 3ac4a2fdbf2ba631c4c6e0aeb215b270
slug: sphinx-ext-extlinks
body:
Datasette's [release notes](https://github.com/simonw/datasette/blob/main/docs/changelog.rst) are formatted using Sphinx. Almost every bullet point links to the corresponding GitHub issue, so they were full of lines that look like this:

``` - Fixed a bug where ``?_search=`` and ``?_sort=`` parameters were incorrectly duplicated when the filter form on the table page was re-submitted. (`#1214 <https://github.com/simonw/datasette/issues/1214>`__) ```

I noticed that [the aspw documentation](https://github.com/simonw/datasette/issues/1227) was using `sphinx.ext.extlinks` to define a macro for this: `` :issue:`268` `` - so I decided to configure that for Datasette.

I added the following to the `conf.py` for my documentation:

```python
extensions = ["sphinx.ext.extlinks"]

extlinks = {
    "issue": ("https://github.com/simonw/datasette/issues/%s", "#"),
}
```

Then in Visual Studio Code I opened the "Edit -> Replace in Files" tool. I used this search pattern:

    `#(\d+) <https://github.com/simonw/datasette/issues/\1>`__

And this as the replacement value:

    :issue:`$1`

Note that the search pattern uses `\1` as the group reference for the captured number, but in the replacement value you use `$1` to re-use that value.

Here's [the commit](https://github.com/simonw/datasette/commit/d2d53a5559f3014cccba2cba7e1eab1e5854c759) where I applied that change to Datasette's existing documentation.

html:
<p>Datasette's <a href="https://github.com/simonw/datasette/blob/main/docs/changelog.rst">release notes</a> are formatted using Sphinx. Almost every bullet point links to the corresponding GitHub issue, so they were full of lines that look like this:</p>
<p><code>- Fixed a bug where ``?_search=`` and ``?_sort=`` parameters were incorrectly duplicated when the filter form on the table page was re-submitted. (`#1214 &lt;https://github.com/simonw/datasette/issues/1214&gt;`__)</code></p>
<p>I noticed that <a href="https://github.com/simonw/datasette/issues/1227">the aspw documentation</a> was using <code>sphinx.ext.extlinks</code> to define a macro for this: <code>:issue:`268`</code> - so I decided to configure that for Datasette.</p>
<p>I added the following to the <code>conf.py</code> for my documentation:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">extensions</span> <span class="pl-c1">=</span> [<span class="pl-s">"sphinx.ext.extlinks"</span>]

<span class="pl-s1">extlinks</span> <span class="pl-c1">=</span> {
    <span class="pl-s">"issue"</span>: (<span class="pl-s">"https://github.com/simonw/datasette/issues/%s"</span>, <span class="pl-s">"#"</span>),
}</pre></div>
<p>Then in Visual Studio Code I opened the "Edit -&gt; Replace in Files" tool. I used this search pattern:</p>
<pre><code>`#(\d+) &lt;https://github.com/simonw/datasette/issues/\1&gt;`__
</code></pre>
<p>And this as the replacement value:</p>
<pre><code>:issue:`$1`
</code></pre>
<p>Note that the search pattern uses <code>\1</code> as the group reference for the captured number, but in the replacement value you use <code>$1</code> to re-use that value.</p>
<p>Here's <a href="https://github.com/simonw/datasette/commit/d2d53a5559f3014cccba2cba7e1eab1e5854c759">the commit</a> where I applied that change to Datasette's existing documentation.</p>


path: postgresql_show-schema.md
topic: postgresql
title: Show the SQL schema for a PostgreSQL database
url: https://github.com/simonw/til/blob/main/postgresql/show-schema.md
created: 2021-02-23T19:03:04-08:00
created_utc: 2021-02-24T03:03:04+00:00
updated: 2021-02-23T19:03:04-08:00
updated_utc: 2021-02-24T03:03:04+00:00
shot_hash: 4fd7cb44c62146740f00eb4218283e1d
slug: show-schema
body:
This took me longer to figure out than I care to admit.

    pg_dump -s name-of-database

Surprisingly there doesn't seem to be an easy way to do this using a `SELECT` statement within PostgreSQL itself.

StackOverflow [offers up](https://stackoverflow.com/a/16154183) a terrifying 72 line SQL monstrosity that attempts to recreate the `CREATE TABLE` for a given table.

html:
<p>This took me longer to figure out than I care to admit.</p>
<pre><code>pg_dump -s name-of-database
</code></pre>
<p>Surprisingly there doesn't seem to be an easy way to do this using a <code>SELECT</code> statement within PostgreSQL itself.</p>
<p>StackOverflow <a href="https://stackoverflow.com/a/16154183" rel="nofollow">offers up</a> a terrifying 72 line SQL monstrosity that attempts to recreate the <code>CREATE TABLE</code> for a given table.</p>


path: github-actions_postgresq-service-container.md
topic: github-actions
title: Running tests against PostgreSQL in a service container
url: https://github.com/simonw/til/blob/main/github-actions/postgresq-service-container.md
created: 2021-02-23T19:33:40-08:00
created_utc: 2021-02-24T03:33:40+00:00
updated: 2021-06-11T16:07:45-07:00
updated_utc: 2021-06-11T23:07:45+00:00
shot_hash: d993b9702b4103d0bb42935af055454d
slug: postgresq-service-container
body:
I wanted to run some Django tests - using `pytest-django` and with Django configured to pick up the `DATABASE_URL` environment variable via [dj-database-url](https://github.com/jacobian/dj-database-url) - against a PostgreSQL server running in GitHub Actions.

It took a while to figure out the right pattern. The trick was to define a `postgres:` service and then set the `DATABASE_URL` environment variable to the following:

    postgres://postgres:postgres@127.0.0.1:${{ job.services.postgres.ports['5432'] }}/dbname

Here's my full `.github/workflows/test.yml`:

```yaml
name: Run tests

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:12
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: postgres
          POSTGRES_DB: dbname
        options:
          --health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5
        ports:
        - 5432:5432
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python
      uses: actions/setup-python@v2
      with:
        python-version: 3.8
    - uses: actions/cache@v2
      name: Configure pip caching
      with:
        path: ~/.cache/pip
        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
        restore-keys: |
          ${{ runner.os }}-pip-
    - name: Install dependencies
      run: |
        pip install -r requirements.txt
    - name: Run tests
      env:
        DATABASE_URL: postgres://postgres:postgres@127.0.0.1:${{ job.services.postgres.ports['5432'] }}/dbname
      run: |
        cd myproject
        pytest
```

## And against MySQL

I had to figure this out against MySQL as well for `db-to-sqlite` - here's [the workflow test.yml file](https://github.com/simonw/db-to-sqlite/blob/1.4/.github/workflows/test.yml) I ended up with. Key extract here:

```yaml
    services:
      mysql:
        image: mysql
        env:
          MYSQL_ALLOW_EMPTY_PASSWORD: yes
          MYSQL_DATABASE: test_db_to_sqlite
        options: >-
          --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3
        ports:
          - 3306:3306
    # ...
    - name: Run tests
      env:
        MYSQL_TEST_DB_CONNECTION: mysql://root@127.0.0.1:${{ job.services.mysql.ports['3306'] }}/test_db_to_sqlite
      run: pytest -vv
```

html:
<p>I wanted to run some Django tests - using <code>pytest-django</code> and with Django configured to pick up the <code>DATABASE_URL</code> environment variable via <a href="https://github.com/jacobian/dj-database-url">dj-database-url</a> - against a PostgreSQL server running in GitHub Actions.</p>
<p>It took a while to figure out the right pattern. The trick was to define a <code>postgres:</code> service and then set the <code>DATABASE_URL</code> environment variable to the following:</p>
<pre><code>postgres://postgres:postgres@127.0.0.1:${{ job.services.postgres.ports['5432'] }}/dbname
</code></pre>
<p>Here's my full <code>.github/workflows/test.yml</code>:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Run tests</span>

<span class="pl-ent">on</span>: <span class="pl-s">[push]</span>

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">test</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">services</span>:
      <span class="pl-ent">postgres</span>:
        <span class="pl-ent">image</span>: <span class="pl-s">postgres:12</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">POSTGRES_USER</span>: <span class="pl-s">postgres</span>
          <span class="pl-ent">POSTGRES_PASSWORD</span>: <span class="pl-s">postgres</span>
          <span class="pl-ent">POSTGRES_DB</span>: <span class="pl-s">dbname</span>
        <span class="pl-ent">options</span>:
          <span class="pl-s">--health-cmd pg_isready --health-interval 10s --health-timeout 5s --health-retries 5</span>
        <span class="pl-ent">ports</span>:
        - <span class="pl-c1">5432:5432</span>
    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v2</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Set up Python</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">actions/setup-python@v2</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">python-version</span>: <span class="pl-c1">3.8</span>
    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/cache@v2</span>
      <span class="pl-ent">name</span>: <span class="pl-s">Configure pip caching</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">path</span>: <span class="pl-s">~/.cache/pip</span>
        <span class="pl-ent">key</span>: <span class="pl-s">${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}</span>
        <span class="pl-ent">restore-keys</span>: <span class="pl-s">|</span>
<span class="pl-s">          ${{ runner.os }}-pip-</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Install dependencies</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        pip install -r requirements.txt</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Run tests</span>
      <span class="pl-ent">env</span>:
        <span class="pl-ent">DATABASE_URL</span>: <span class="pl-s">postgres://postgres:postgres@127.0.0.1:${{ job.services.postgres.ports['5432'] }}/dbname</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        cd myproject</span>
<span class="pl-s">        pytest</span></pre></div>
<h2>
<a id="user-content-and-against-mysql" class="anchor" href="#and-against-mysql" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>And against MySQL</h2>
<p>I had to figure this out against MySQL as well for <code>db-to-sqlite</code> - here's <a href="https://github.com/simonw/db-to-sqlite/blob/1.4/.github/workflows/test.yml">the workflow test.yml file</a> I ended up with. Key extract here:</p>
<div class="highlight highlight-source-yaml"><pre>    <span class="pl-ent">services</span>:
      <span class="pl-ent">mysql</span>:
        <span class="pl-ent">image</span>: <span class="pl-s">mysql</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">MYSQL_ALLOW_EMPTY_PASSWORD</span>: <span class="pl-s">yes</span>
          <span class="pl-ent">MYSQL_DATABASE</span>: <span class="pl-s">test_db_to_sqlite</span>
        <span class="pl-ent">options</span>: <span class="pl-s">&gt;-</span>
<span class="pl-s">          --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3</span>
<span class="pl-s"></span>        <span class="pl-ent">ports</span>:
          - <span class="pl-c1">3306:3306</span>
    <span class="pl-c"><span class="pl-c">#</span> ...</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Run tests</span>
      <span class="pl-ent">env</span>:
        <span class="pl-ent">MYSQL_TEST_DB_CONNECTION</span>: <span class="pl-s">mysql://root@127.0.0.1:${{ job.services.mysql.ports['3306'] }}/test_db_to_sqlite</span>
      <span class="pl-ent">run</span>: <span class="pl-s">pytest -vv</span></pre></div>


path: django_extra-read-only-admin-information.md
topic: django
title: Adding extra read-only information to a Django admin change page
url: https://github.com/simonw/til/blob/main/django/extra-read-only-admin-information.md
created: 2021-02-25T17:49:17-08:00
created_utc: 2021-02-26T01:49:17+00:00
updated: 2021-02-27T12:34:46-08:00
updated_utc: 2021-02-27T20:34:46+00:00
shot_hash: 4f6dce09e12d1b504c5bcac65757c888
slug: extra-read-only-admin-information
body:
I figured out this pattern today for adding templated extra blocks of information to the Django admin change page for an object.

It's really simply and incredibly useful. I can see myself using this a lot in the future.

```python
from django.contrib import admin
from django.template.loader import render_to_string
from django.utils.safestring import mark_safe
from .models import Reporter


@admin.register(Reporter)
class ReporterAdmin(admin.ModelAdmin):
    # ...
    readonly_fields = ("recent_calls",)

    def recent_calls(self, instance):
        return mark_safe(
            render_to_string(
                "admin/_reporter_recent_calls.html",
                {
                    "reporter": instance,
                    "recent_calls": instance.call_reports.order_by("-created_at")[:20],
                    "call_count": instance.call_reports.count(),
                },
            )
        )
```

That's it! `recent_calls` is marked as a read-only field, then implemented as a method which returns HTML. That method passes the instance to a template using `render_to_string`. That template looks like this:

```html+jinja
<h2>{{ reporter }} has made {{ call_count }} call{{ call_count|pluralize }}</h2>

<p><strong>Recent calls</strong> (<a href="/admin/core/callreport/?reported_by__exact={{ reporter.id }}">view all</a>)</p>

{% for call in recent_calls %}
    <p><a href="/admin/core/location/{{ call.location.id }}/change/">{{ call.location }}</a> on {{ call.created_at }}</p>
{% endfor %}
```

html:
<p>I figured out this pattern today for adding templated extra blocks of information to the Django admin change page for an object.</p>
<p>It's really simply and incredibly useful. I can see myself using this a lot in the future.</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">contrib</span> <span class="pl-k">import</span> <span class="pl-s1">admin</span>
<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">template</span>.<span class="pl-s1">loader</span> <span class="pl-k">import</span> <span class="pl-s1">render_to_string</span>
<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">utils</span>.<span class="pl-s1">safestring</span> <span class="pl-k">import</span> <span class="pl-s1">mark_safe</span>
<span class="pl-k">from</span> .<span class="pl-s1">models</span> <span class="pl-k">import</span> <span class="pl-v">Reporter</span>


<span class="pl-en">@<span class="pl-s1">admin</span>.<span class="pl-s1">register</span>(<span class="pl-v">Reporter</span>)</span>
<span class="pl-k">class</span> <span class="pl-v">ReporterAdmin</span>(<span class="pl-s1">admin</span>.<span class="pl-v">ModelAdmin</span>):
    <span class="pl-c"># ...</span>
    <span class="pl-s1">readonly_fields</span> <span class="pl-c1">=</span> (<span class="pl-s">"recent_calls"</span>,)

    <span class="pl-k">def</span> <span class="pl-en">recent_calls</span>(<span class="pl-s1">self</span>, <span class="pl-s1">instance</span>):
        <span class="pl-k">return</span> <span class="pl-en">mark_safe</span>(
            <span class="pl-en">render_to_string</span>(
                <span class="pl-s">"admin/_reporter_recent_calls.html"</span>,
                {
                    <span class="pl-s">"reporter"</span>: <span class="pl-s1">instance</span>,
                    <span class="pl-s">"recent_calls"</span>: <span class="pl-s1">instance</span>.<span class="pl-s1">call_reports</span>.<span class="pl-en">order_by</span>(<span class="pl-s">"-created_at"</span>)[:<span class="pl-c1">20</span>],
                    <span class="pl-s">"call_count"</span>: <span class="pl-s1">instance</span>.<span class="pl-s1">call_reports</span>.<span class="pl-en">count</span>(),
                },
            )
        )</pre></div>
<p>That's it! <code>recent_calls</code> is marked as a read-only field, then implemented as a method which returns HTML. That method passes the instance to a template using <code>render_to_string</code>. That template looks like this:</p>
<div class="highlight highlight-text-html-django"><pre>&lt;<span class="pl-ent">h2</span>&gt;{{ reporter }} has made {{ call_count }} call{{ call_count|pluralize }}&lt;/<span class="pl-ent">h2</span>&gt;

&lt;<span class="pl-ent">p</span>&gt;&lt;<span class="pl-ent">strong</span>&gt;Recent calls&lt;/<span class="pl-ent">strong</span>&gt; (&lt;<span class="pl-ent">a</span> <span class="pl-e">href</span>=<span class="pl-s"><span class="pl-pds">"</span>/admin/core/callreport/?reported_by__exact={{ reporter.id }}<span class="pl-pds">"</span></span>&gt;view all&lt;/<span class="pl-ent">a</span>&gt;)&lt;/<span class="pl-ent">p</span>&gt;

<span class="pl-e">{%</span> <span class="pl-k">for</span> <span class="pl-s">call</span> <span class="pl-k">in</span> <span class="pl-s">recent_calls</span> <span class="pl-e">%}</span>
    &lt;<span class="pl-ent">p</span>&gt;&lt;<span class="pl-ent">a</span> <span class="pl-e">href</span>=<span class="pl-s"><span class="pl-pds">"</span>/admin/core/location/{{ call.location.id }}/change/<span class="pl-pds">"</span></span>&gt;{{ call.location }}&lt;/<span class="pl-ent">a</span>&gt; on {{ call.created_at }}&lt;/<span class="pl-ent">p</span>&gt;
<span class="pl-e">{%</span> <span class="pl-k">endfor</span> <span class="pl-e">%}</span></pre></div>


path: postgresql_read-only-postgresql-user.md
topic: postgresql
title: Granting a PostgreSQL user read-only access to some tables
url: https://github.com/simonw/til/blob/main/postgresql/read-only-postgresql-user.md
created: 2021-02-26T07:58:19-08:00
created_utc: 2021-02-26T15:58:19+00:00
updated: 2021-02-26T07:58:19-08:00
updated_utc: 2021-02-26T15:58:19+00:00
shot_hash: 1eb4f457d5561fa0923f300bf31fa8ee
slug: read-only-postgresql-user
body:
I wanted to grant a PostgreSQL user (or role) read-only access to a specific list of tables.

I created the role using the Heroku PostgreSQL web console. Having done that, it had the name `read-only-core-tables`.

I attached it to my Heroku application using the web console, then I ran the following to get myself a default-permission terminal session:

    % heroku pg:psql postgresql-metric-59331 --app myapp

As the default user I ran the following:

```sql
GRANT USAGE ON SCHEMA PUBLIC TO "read-only-core-tables";
```

This grants that user the ability to see what tables exist - after running this the `\dt` command for that user started showing a full list of tables.

But... `select * from table` returned permission denied for every table. To allow select access I ran this:

```sql
GRANT SELECT ON TABLE
    public.availability_tag,
    public.county,
    public.django_content_type,
    public.django_migrations
TO "read-only-core-tables";
```
That's all it took - my read-only user was then able to run `select * from county` and see the results - but attempts to select against tables not in that allow-list were denied.

html:
<p>I wanted to grant a PostgreSQL user (or role) read-only access to a specific list of tables.</p>
<p>I created the role using the Heroku PostgreSQL web console. Having done that, it had the name <code>read-only-core-tables</code>.</p>
<p>I attached it to my Heroku application using the web console, then I ran the following to get myself a default-permission terminal session:</p>
<pre><code>% heroku pg:psql postgresql-metric-59331 --app myapp
</code></pre>
<p>As the default user I ran the following:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">GRANT</span> USAGE <span class="pl-k">ON</span> SCHEMA PUBLIC TO <span class="pl-s"><span class="pl-pds">"</span>read-only-core-tables<span class="pl-pds">"</span></span>;</pre></div>
<p>This grants that user the ability to see what tables exist - after running this the <code>\dt</code> command for that user started showing a full list of tables.</p>
<p>But... <code>select * from table</code> returned permission denied for every table. To allow select access I ran this:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">GRANT</span> <span class="pl-k">SELECT</span> <span class="pl-k">ON</span> TABLE
    <span class="pl-c1">public</span>.<span class="pl-c1">availability_tag</span>,
    <span class="pl-c1">public</span>.<span class="pl-c1">county</span>,
    <span class="pl-c1">public</span>.<span class="pl-c1">django_content_type</span>,
    <span class="pl-c1">public</span>.<span class="pl-c1">django_migrations</span>
TO <span class="pl-s"><span class="pl-pds">"</span>read-only-core-tables<span class="pl-pds">"</span></span>;</pre></div>
<p>That's all it took - my read-only user was then able to run <code>select * from county</code> and see the results - but attempts to select against tables not in that allow-list were denied.</p>


path: django_testing-django-admin-with-pytest.md
topic: django
title: Writing tests for the Django admin with pytest-django
url: https://github.com/simonw/til/blob/main/django/testing-django-admin-with-pytest.md
created: 2021-03-02T13:08:34-08:00
created_utc: 2021-03-02T21:08:34+00:00
updated: 2021-03-02T23:37:18-08:00
updated_utc: 2021-03-03T07:37:18+00:00
shot_hash: 9b8d9be51081f4bffc50faf2d80e3d9e
slug: testing-django-admin-with-pytest
body:
I'm using [pytest-django](https://pytest-django.readthedocs.io/) on a project and I wanted to write a test for a Django admin create form submission. Here's the pattern I came up with:

```python
from .models import Location
import pytest


def test_admin_create_location_sets_public_id(client, admin_user):
    client.force_login(admin_user)
    assert Location.objects.count() == 0
    response = client.post(
        "/admin/core/location/add/",
        {
            "name": "hello",
            "state": "13",
            "location_type": "1",
            "latitude": "0",
            "longitude": "0",
            "_save": "Save",
        },
    )
    # 200 means the form is being re-displayed with errors
    assert response.status_code == 302
    location = Location.objects.order_by("-id")[0]
    assert location.name == "hello"
    assert location.public_id == "lc"
```
The trick here is to use the `client` and `admin_user` pytest-django fixtures ([documented here](https://pytest-django.readthedocs.io/en/latest/helpers.html#fixtures)) to get a configured test client and admin user object, then use `client.force_login(admin_user)` to obtain a session where that user is signed-in to the admin. Then write tests as normal.

## Using the admin_client fixture

Even better: use the `admin_client` fixture provided by `pytest-django 
` which is already signed into the admin:

```python
def test_admin_create_location_sets_public_id(admin_client):
    response = admin_client.post(
        "/admin/core/location/add/",
    # ...
```

Before finding out that this was included I implemented my own version of it:

```python
import pytest


@pytest.fixture()
def admin_client(client, admin_user):
    client.force_login(admin_user)
    return client

# Then write tests like this:
def test_admin_create_location_sets_public_id(admin_client):
    response = admin_client.post(
        "/admin/core/location/add/",
    # ...
```

html:
<p>I'm using <a href="https://pytest-django.readthedocs.io/" rel="nofollow">pytest-django</a> on a project and I wanted to write a test for a Django admin create form submission. Here's the pattern I came up with:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> .<span class="pl-s1">models</span> <span class="pl-k">import</span> <span class="pl-v">Location</span>
<span class="pl-k">import</span> <span class="pl-s1">pytest</span>


<span class="pl-k">def</span> <span class="pl-en">test_admin_create_location_sets_public_id</span>(<span class="pl-s1">client</span>, <span class="pl-s1">admin_user</span>):
    <span class="pl-s1">client</span>.<span class="pl-en">force_login</span>(<span class="pl-s1">admin_user</span>)
    <span class="pl-k">assert</span> <span class="pl-v">Location</span>.<span class="pl-s1">objects</span>.<span class="pl-en">count</span>() <span class="pl-c1">==</span> <span class="pl-c1">0</span>
    <span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">client</span>.<span class="pl-en">post</span>(
        <span class="pl-s">"/admin/core/location/add/"</span>,
        {
            <span class="pl-s">"name"</span>: <span class="pl-s">"hello"</span>,
            <span class="pl-s">"state"</span>: <span class="pl-s">"13"</span>,
            <span class="pl-s">"location_type"</span>: <span class="pl-s">"1"</span>,
            <span class="pl-s">"latitude"</span>: <span class="pl-s">"0"</span>,
            <span class="pl-s">"longitude"</span>: <span class="pl-s">"0"</span>,
            <span class="pl-s">"_save"</span>: <span class="pl-s">"Save"</span>,
        },
    )
    <span class="pl-c"># 200 means the form is being re-displayed with errors</span>
    <span class="pl-k">assert</span> <span class="pl-s1">response</span>.<span class="pl-s1">status_code</span> <span class="pl-c1">==</span> <span class="pl-c1">302</span>
    <span class="pl-s1">location</span> <span class="pl-c1">=</span> <span class="pl-v">Location</span>.<span class="pl-s1">objects</span>.<span class="pl-en">order_by</span>(<span class="pl-s">"-id"</span>)[<span class="pl-c1">0</span>]
    <span class="pl-k">assert</span> <span class="pl-s1">location</span>.<span class="pl-s1">name</span> <span class="pl-c1">==</span> <span class="pl-s">"hello"</span>
    <span class="pl-k">assert</span> <span class="pl-s1">location</span>.<span class="pl-s1">public_id</span> <span class="pl-c1">==</span> <span class="pl-s">"lc"</span></pre></div>
<p>The trick here is to use the <code>client</code> and <code>admin_user</code> pytest-django fixtures (<a href="https://pytest-django.readthedocs.io/en/latest/helpers.html#fixtures" rel="nofollow">documented here</a>) to get a configured test client and admin user object, then use <code>client.force_login(admin_user)</code> to obtain a session where that user is signed-in to the admin. Then write tests as normal.</p>
<h2>
<a id="user-content-using-the-admin_client-fixture" class="anchor" href="#using-the-admin_client-fixture" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using the admin_client fixture</h2>
<p>Even better: use the <code>admin_client</code> fixture provided by <code>pytest-django  </code> which is already signed into the admin:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-s1">test_admin_create_location_sets_public_id</span>(<span class="pl-s1">admin_client</span>):
    <span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">admin_client</span>.<span class="pl-s1">post</span>(
        <span class="pl-s">"/admin/core/location/add/"</span>,
    <span class="pl-c"># ...</span></pre></div>
<p>Before finding out that this was included I implemented my own version of it:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">pytest</span>


<span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-s1">fixture</span>()</span>
<span class="pl-k">def</span> <span class="pl-en">admin_client</span>(<span class="pl-s1">client</span>, <span class="pl-s1">admin_user</span>):
    <span class="pl-s1">client</span>.<span class="pl-en">force_login</span>(<span class="pl-s1">admin_user</span>)
    <span class="pl-k">return</span> <span class="pl-s1">client</span>

<span class="pl-c"># Then write tests like this:</span>
<span class="pl-k">def</span> <span class="pl-s1">test_admin_create_location_sets_public_id</span>(<span class="pl-s1">admin_client</span>):
    <span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">admin_client</span>.<span class="pl-s1">post</span>(
        <span class="pl-s">"/admin/core/location/add/"</span>,
    <span class="pl-c"># ...</span></pre></div>


path: django_show-timezone-in-django-admin.md
topic: django
title: Show the timezone for datetimes in the Django admin
url: https://github.com/simonw/til/blob/main/django/show-timezone-in-django-admin.md
created: 2021-03-02T21:17:45-08:00
created_utc: 2021-03-03T05:17:45+00:00
updated: 2021-03-02T21:17:45-08:00
updated_utc: 2021-03-03T05:17:45+00:00
shot_hash: 76244473ec3e74804200af70034ecce8
slug: show-timezone-in-django-admin
body:
Django supports storing dates in a database as UTC but displaying them in some other timezone - which is good. But... by default datetimes are shown in the Django admin interface without any clue as to what timezone they are being displayed in.

This is really confusing. A time may be stored as UTC in the database but in the admin interface it's displaying in PST, without any visual indication as to what is going on.

I found a pattern today for improving this. You can use `django.conf.locale.en.formats` to specify a custom date format for a specific locale (thanks, [Stack Overflow](https://stackoverflow.com/a/32355642)). Then you can use the `e` date format option to include a string indicating the timezone that is being displayed, as [documented here](https://docs.djangoproject.com/en/3.1/ref/templates/builtins/#date).

In `settings.py` do this:

```python
from django.conf.locale.en import formats as en_formats

en_formats.DATETIME_FORMAT = "jS M Y fA e"
```

I added a middleware to force the displayed timezone for every page on my site to `America/Los_Angeles` like so:

```python
from django.utils import timezone
import pytz

class TimezoneMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response

    def __call__(self, request):
        timezone.activate(pytz.timezone("America/Los_Angeles"))
        return self.get_response(request)
```
I put this in a file called `core/timezone_middleware.py` and added it to my `MIDDLEWARE` setting in `settings.py` like so:
```
MIDDLEWARE = [
    # ...
    "core.timezone_middleware.TimezoneMiddleware",
]
```
Now datetimes show up in my admin interface looking like this, with a `PST` suffix:

<img width="593" alt="Select_report_to_change___Django_site_admin" src="https://user-images.githubusercontent.com/9599/109755937-c4fd1600-7b9b-11eb-9c65-f84bbb84ed21.png">

## Showing UTC too

I decided I'd like to see the UTC time too, just to help me truly understand what had been stored. I did that by adding the following method to my Django model class:

```python
# Earlier
from django.utils import dateformat
import pytz

# Added to the model class:

    def created_at_utc(self):
        tz = pytz.UTC
        created_at_utc = timezone.localtime(self.created_at, tz)
        return (
            dateformat.format(created_at_utc, "jS M Y fA e")
        )
```
Then I added `created_at_utc` to both the `list_filter` and the `readonly_fields` tuples in the admin configuration for that model. This caused it to show up in the list view and also as a read-only field at the bottom of the edit view.

<img width="651" alt="Change_report___Django_site_admin" src="https://user-images.githubusercontent.com/9599/109756379-afd4b700-7b9c-11eb-8bc4-acb168c53943.png">

Note that I'm calling `dateformat.format()` in the method and returning a string - this ensures Django's automatic formatting doesn't get the chance to convert it back to PST again.

html:
<p>Django supports storing dates in a database as UTC but displaying them in some other timezone - which is good. But... by default datetimes are shown in the Django admin interface without any clue as to what timezone they are being displayed in.</p>
<p>This is really confusing. A time may be stored as UTC in the database but in the admin interface it's displaying in PST, without any visual indication as to what is going on.</p>
<p>I found a pattern today for improving this. You can use <code>django.conf.locale.en.formats</code> to specify a custom date format for a specific locale (thanks, <a href="https://stackoverflow.com/a/32355642" rel="nofollow">Stack Overflow</a>). Then you can use the <code>e</code> date format option to include a string indicating the timezone that is being displayed, as <a href="https://docs.djangoproject.com/en/3.1/ref/templates/builtins/#date" rel="nofollow">documented here</a>.</p>
<p>In <code>settings.py</code> do this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">conf</span>.<span class="pl-s1">locale</span>.<span class="pl-s1">en</span> <span class="pl-k">import</span> <span class="pl-s1">formats</span> <span class="pl-k">as</span> <span class="pl-s1">en_formats</span>

<span class="pl-s1">en_formats</span>.<span class="pl-v">DATETIME_FORMAT</span> <span class="pl-c1">=</span> <span class="pl-s">"jS M Y fA e"</span></pre></div>
<p>I added a middleware to force the displayed timezone for every page on my site to <code>America/Los_Angeles</code> like so:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">utils</span> <span class="pl-k">import</span> <span class="pl-s1">timezone</span>
<span class="pl-k">import</span> <span class="pl-s1">pytz</span>

<span class="pl-k">class</span> <span class="pl-v">TimezoneMiddleware</span>:
    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">get_response</span>):
        <span class="pl-s1">self</span>.<span class="pl-s1">get_response</span> <span class="pl-c1">=</span> <span class="pl-s1">get_response</span>

    <span class="pl-k">def</span> <span class="pl-en">__call__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">request</span>):
        <span class="pl-s1">timezone</span>.<span class="pl-en">activate</span>(<span class="pl-s1">pytz</span>.<span class="pl-en">timezone</span>(<span class="pl-s">"America/Los_Angeles"</span>))
        <span class="pl-k">return</span> <span class="pl-s1">self</span>.<span class="pl-en">get_response</span>(<span class="pl-s1">request</span>)</pre></div>
<p>I put this in a file called <code>core/timezone_middleware.py</code> and added it to my <code>MIDDLEWARE</code> setting in <code>settings.py</code> like so:</p>
<pre><code>MIDDLEWARE = [
    # ...
    "core.timezone_middleware.TimezoneMiddleware",
]
</code></pre>
<p>Now datetimes show up in my admin interface looking like this, with a <code>PST</code> suffix:</p>
<p><a href="https://user-images.githubusercontent.com/9599/109755937-c4fd1600-7b9b-11eb-9c65-f84bbb84ed21.png" target="_blank" rel="nofollow"><img width="593" alt="Select_report_to_change___Django_site_admin" src="https://user-images.githubusercontent.com/9599/109755937-c4fd1600-7b9b-11eb-9c65-f84bbb84ed21.png" style="max-width:100%;"></a></p>
<h2>
<a id="user-content-showing-utc-too" class="anchor" href="#showing-utc-too" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Showing UTC too</h2>
<p>I decided I'd like to see the UTC time too, just to help me truly understand what had been stored. I did that by adding the following method to my Django model class:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-c"># Earlier</span>
<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">utils</span> <span class="pl-k">import</span> <span class="pl-s1">dateformat</span>
<span class="pl-k">import</span> <span class="pl-s1">pytz</span>

<span class="pl-c"># Added to the model class:</span>

    <span class="pl-k">def</span> <span class="pl-en">created_at_utc</span>(<span class="pl-s1">self</span>):
        <span class="pl-s1">tz</span> <span class="pl-c1">=</span> <span class="pl-s1">pytz</span>.<span class="pl-v">UTC</span>
        <span class="pl-s1">created_at_utc</span> <span class="pl-c1">=</span> <span class="pl-s1">timezone</span>.<span class="pl-en">localtime</span>(<span class="pl-s1">self</span>.<span class="pl-s1">created_at</span>, <span class="pl-s1">tz</span>)
        <span class="pl-k">return</span> (
            <span class="pl-s1">dateformat</span>.<span class="pl-en">format</span>(<span class="pl-s1">created_at_utc</span>, <span class="pl-s">"jS M Y fA e"</span>)
        )</pre></div>
<p>Then I added <code>created_at_utc</code> to both the <code>list_filter</code> and the <code>readonly_fields</code> tuples in the admin configuration for that model. This caused it to show up in the list view and also as a read-only field at the bottom of the edit view.</p>
<p><a href="https://user-images.githubusercontent.com/9599/109756379-afd4b700-7b9c-11eb-8bc4-acb168c53943.png" target="_blank" rel="nofollow"><img width="651" alt="Change_report___Django_site_admin" src="https://user-images.githubusercontent.com/9599/109756379-afd4b700-7b9c-11eb-8bc4-acb168c53943.png" style="max-width:100%;"></a></p>
<p>Note that I'm calling <code>dateformat.format()</code> in the method and returning a string - this ensures Django's automatic formatting doesn't get the chance to convert it back to PST again.</p>


path: mediawiki_mediawiki-sqlite-macos.md
topic: mediawiki
title: How to run MediaWiki with SQLite on a macOS laptop
url: https://github.com/simonw/til/blob/main/mediawiki/mediawiki-sqlite-macos.md
created: 2021-03-06T21:02:38-08:00
created_utc: 2021-03-07T05:02:38+00:00
updated: 2021-03-06T21:12:51-08:00
updated_utc: 2021-03-07T05:12:51+00:00
shot_hash: b86375e87154fcb04b5463db3c80e0bc
slug: mediawiki-sqlite-macos
body:
Today I [got curious](https://twitter.com/simonw/status/1368414296888070146) about how MediaWiki records page history, so I started digging around and in the process figured out how to run it against a SQLite database on my macOS laptop!

## Requirements: PHP

To run MediaWiki you need PHP and the SQLite drivers. I was surprised to find out I had PHP already (if not I would have tried installing through Homebrew). It turns out it comes pre-installed on your Mac!

    ~ % php --version
    PHP 7.3.11 (cli) (built: Jun  5 2020 23:50:40) ( NTS )
    Copyright (c) 1997-2018 The PHP Group
    Zend Engine v3.3.11, Copyright (c) 1998-2018 Zend Technologies

To see version information, do this:

    % cd /tmp
    % echo '<? phpinfo(); ?>' > index.php
    % php -S localhost:8000
    PHP 7.3.11 Development Server started at Sat Mar  6 20:50:32 2021
    Listening on http://localhost:8000
    Document root is /private/tmp
    Press Ctrl-C to quit.

`php -S localhost:8000` runs PHP's built-in development server, saving you from having to configure Apache.

Then visit http://localhost:8000 to view the PHP info page.

I searched that page for `sqlite` and found that it had the `pdo_sqlite` driver installed already. So that should be everything I need to run MediaWiki.

## Downloading MediaWiki

I downloaded the latest version of MediaWiki from [their downloads page](https://www.mediawiki.org/wiki/Download), unzipped it, ran `php -S localhost:8000` and got this error message:

<img width="664" alt="MediaWiki_1_35" src="https://user-images.githubusercontent.com/9599/110229420-1d932280-7ebe-11eb-98f3-eb13fa7f07c7.png">

Turns out the latest MediaWiki requires PHP 7.3.19, but the version bundled with my laptop was 7.3.11.

I didn't want to mess around with upgrading PHP, so I used the [compatibility page](https://www.mediawiki.org/wiki/Compatibility#PHP) to figure out the most recent MediaWiki version that would work with PHP 7.3.11. I decided to try MediaWiki 1.31, which can be downloaded from <https://releases.wikimedia.org/mediawiki/1.31/?C=S;O=D>

Here's what worked for me:

    % mkdir ~/wiki
    % cd ~/wiki
    % wget https://releases.wikimedia.org/mediawiki/1.31/mediawiki-1.31.12.zip
    % unzip mediawiki-1.31.12.zip
    % cd mediawiki-1.31.12
    % php -S localhost:8000
    PHP 7.3.11 Development Server started at Sat Mar  6 20:58:21 2021
    Listening on http://localhost:8000
    Document root is /Users/simon/wiki/mediawiki-1.31.12

Now visiting http://localhost:8000 gave me the interactive setup tool.

I clicked through their wizard, selected SQLite for the database option and told it where I wanted the database to live:

<img width="856" alt="MediaWiki_1_31_12_installation" src="https://user-images.githubusercontent.com/9599/110229508-e4a77d80-7ebe-11eb-9a71-70ba16e25f92.png">

At the end of the wizard it gave me a `LocalSettings.php` file to drop into my root `~/wiki/mediawiki-1.31.12` directory... and I was done! Didn't even need to restart the PHP process (because PHP).

I created a page and then ran Datasette against the SQLite databases it was using like this:

    % datasette ~/wiki/data/*.sqlite

html:
<p>Today I <a href="https://twitter.com/simonw/status/1368414296888070146" rel="nofollow">got curious</a> about how MediaWiki records page history, so I started digging around and in the process figured out how to run it against a SQLite database on my macOS laptop!</p>
<h2>
<a id="user-content-requirements-php" class="anchor" href="#requirements-php" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Requirements: PHP</h2>
<p>To run MediaWiki you need PHP and the SQLite drivers. I was surprised to find out I had PHP already (if not I would have tried installing through Homebrew). It turns out it comes pre-installed on your Mac!</p>
<pre><code>~ % php --version
PHP 7.3.11 (cli) (built: Jun  5 2020 23:50:40) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.11, Copyright (c) 1998-2018 Zend Technologies
</code></pre>
<p>To see version information, do this:</p>
<pre><code>% cd /tmp
% echo '&lt;? phpinfo(); ?&gt;' &gt; index.php
% php -S localhost:8000
PHP 7.3.11 Development Server started at Sat Mar  6 20:50:32 2021
Listening on http://localhost:8000
Document root is /private/tmp
Press Ctrl-C to quit.
</code></pre>
<p><code>php -S localhost:8000</code> runs PHP's built-in development server, saving you from having to configure Apache.</p>
<p>Then visit <a href="http://localhost:8000" rel="nofollow">http://localhost:8000</a> to view the PHP info page.</p>
<p>I searched that page for <code>sqlite</code> and found that it had the <code>pdo_sqlite</code> driver installed already. So that should be everything I need to run MediaWiki.</p>
<h2>
<a id="user-content-downloading-mediawiki" class="anchor" href="#downloading-mediawiki" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Downloading MediaWiki</h2>
<p>I downloaded the latest version of MediaWiki from <a href="https://www.mediawiki.org/wiki/Download" rel="nofollow">their downloads page</a>, unzipped it, ran <code>php -S localhost:8000</code> and got this error message:</p>
<p><a href="https://user-images.githubusercontent.com/9599/110229420-1d932280-7ebe-11eb-98f3-eb13fa7f07c7.png" target="_blank" rel="nofollow"><img width="664" alt="MediaWiki_1_35" src="https://user-images.githubusercontent.com/9599/110229420-1d932280-7ebe-11eb-98f3-eb13fa7f07c7.png" style="max-width:100%;"></a></p>
<p>Turns out the latest MediaWiki requires PHP 7.3.19, but the version bundled with my laptop was 7.3.11.</p>
<p>I didn't want to mess around with upgrading PHP, so I used the <a href="https://www.mediawiki.org/wiki/Compatibility#PHP" rel="nofollow">compatibility page</a> to figure out the most recent MediaWiki version that would work with PHP 7.3.11. I decided to try MediaWiki 1.31, which can be downloaded from <a href="https://releases.wikimedia.org/mediawiki/1.31/?C=S;O=D" rel="nofollow">https://releases.wikimedia.org/mediawiki/1.31/?C=S;O=D</a></p>
<p>Here's what worked for me:</p>
<pre><code>% mkdir ~/wiki
% cd ~/wiki
% wget https://releases.wikimedia.org/mediawiki/1.31/mediawiki-1.31.12.zip
% unzip mediawiki-1.31.12.zip
% cd mediawiki-1.31.12
% php -S localhost:8000
PHP 7.3.11 Development Server started at Sat Mar  6 20:58:21 2021
Listening on http://localhost:8000
Document root is /Users/simon/wiki/mediawiki-1.31.12
</code></pre>
<p>Now visiting <a href="http://localhost:8000" rel="nofollow">http://localhost:8000</a> gave me the interactive setup tool.</p>
<p>I clicked through their wizard, selected SQLite for the database option and told it where I wanted the database to live:</p>
<p><a href="https://user-images.githubusercontent.com/9599/110229508-e4a77d80-7ebe-11eb-9a71-70ba16e25f92.png" target="_blank" rel="nofollow"><img width="856" alt="MediaWiki_1_31_12_installation" src="https://user-images.githubusercontent.com/9599/110229508-e4a77d80-7ebe-11eb-9a71-70ba16e25f92.png" style="max-width:100%;"></a></p>
<p>At the end of the wizard it gave me a <code>LocalSettings.php</code> file to drop into my root <code>~/wiki/mediawiki-1.31.12</code> directory... and I was done! Didn't even need to restart the PHP process (because PHP).</p>
<p>I created a page and then ran Datasette against the SQLite databases it was using like this:</p>
<pre><code>% datasette ~/wiki/data/*.sqlite
</code></pre>


path: django_pretty-print-json-admin.md
topic: django
title: Pretty-printing all read-only JSON in the Django admin
url: https://github.com/simonw/til/blob/main/django/pretty-print-json-admin.md
created: 2021-03-07T23:02:09-08:00
created_utc: 2021-03-08T07:02:09+00:00
updated: 2021-03-07T23:02:09-08:00
updated_utc: 2021-03-08T07:02:09+00:00
shot_hash: 1d4911dda034b51c8bcb108131578df4
slug: pretty-print-json-admin
body:
I have a bunch of models with JSON fields that are marked as read-only in the Django admin - usually because they're recording the raw JSON that was imported from an API somewhere to create an object, for debugging purposes.

Here's a pattern I found for pretty-printing ANY JSON value that is displayed in a read-only field in the admin. Create a template called `admin/change_form.html` and populate it with the following:

```html+django
{% extends "admin/change_form.html" %}
{% block admin_change_form_document_ready %}
{{ block.super }}
<script>
Array.from(document.querySelectorAll('div.readonly')).forEach(div => {
    let data;
    try {
        data = JSON.parse(div.innerText);
    } catch {
        // Not valid JSON
        return;
    }
    div.style.whiteSpace = 'pre-wrap';
    div.style.fontFamily = 'courier';
    div.style.fontSize = '0.9em';
    div.innerText = JSON.stringify(data, null, 2);
});
</script>
{% endblock %}
```
This JavaScript will execute on every Django change form page, scanning for `div.readonly`, checking to see if the div contains a valid JSON value and pretty-printing it using JavaScript if it does.

It's a cheap hack and it works great.

html:
<p>I have a bunch of models with JSON fields that are marked as read-only in the Django admin - usually because they're recording the raw JSON that was imported from an API somewhere to create an object, for debugging purposes.</p>
<p>Here's a pattern I found for pretty-printing ANY JSON value that is displayed in a read-only field in the admin. Create a template called <code>admin/change_form.html</code> and populate it with the following:</p>
<div class="highlight highlight-text-html-django"><pre><span class="pl-e">{%</span> <span class="pl-k">extends</span> <span class="pl-s">"admin/change_form.html"</span> <span class="pl-e">%}</span>
<span class="pl-e">{%</span> <span class="pl-k">block</span> <span class="pl-s">admin_change_form_document_ready</span> <span class="pl-e">%}</span>
{{ block.super }}
&lt;<span class="pl-ent">script</span>&gt;<span class="pl-s1"></span>
<span class="pl-s1"><span class="pl-c1">Array</span>.<span class="pl-en">from</span>(<span class="pl-c1">document</span>.<span class="pl-c1">querySelectorAll</span>(<span class="pl-s"><span class="pl-pds">'</span>div.readonly<span class="pl-pds">'</span></span>)).<span class="pl-c1">forEach</span>(<span class="pl-smi">div</span> <span class="pl-k">=&gt;</span> {</span>
<span class="pl-s1">    <span class="pl-k">let</span> data;</span>
<span class="pl-s1">    <span class="pl-k">try</span> {</span>
<span class="pl-s1">        data <span class="pl-k">=</span> <span class="pl-c1">JSON</span>.<span class="pl-c1">parse</span>(<span class="pl-smi">div</span>.<span class="pl-smi">innerText</span>);</span>
<span class="pl-s1">    } <span class="pl-k">catch</span> {</span>
<span class="pl-s1">        <span class="pl-c"><span class="pl-c">//</span> Not valid JSON</span></span>
<span class="pl-s1">        <span class="pl-k">return</span>;</span>
<span class="pl-s1">    }</span>
<span class="pl-s1">    <span class="pl-smi">div</span>.<span class="pl-c1">style</span>.<span class="pl-c1">whiteSpace</span> <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">'</span>pre-wrap<span class="pl-pds">'</span></span>;</span>
<span class="pl-s1">    <span class="pl-smi">div</span>.<span class="pl-c1">style</span>.<span class="pl-c1">fontFamily</span> <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">'</span>courier<span class="pl-pds">'</span></span>;</span>
<span class="pl-s1">    <span class="pl-smi">div</span>.<span class="pl-c1">style</span>.<span class="pl-c1">fontSize</span> <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">'</span>0.9em<span class="pl-pds">'</span></span>;</span>
<span class="pl-s1">    <span class="pl-smi">div</span>.<span class="pl-smi">innerText</span> <span class="pl-k">=</span> <span class="pl-c1">JSON</span>.<span class="pl-c1">stringify</span>(data, <span class="pl-c1">null</span>, <span class="pl-c1">2</span>);</span>
<span class="pl-s1">});</span>
<span class="pl-s1"></span>&lt;/<span class="pl-ent">script</span>&gt;
<span class="pl-e">{%</span> <span class="pl-k">endblock</span> <span class="pl-e">%}</span></pre></div>
<p>This JavaScript will execute on every Django change form page, scanning for <code>div.readonly</code>, checking to see if the div contains a valid JSON value and pretty-printing it using JavaScript if it does.</p>
<p>It's a cheap hack and it works great.</p>


path: jq_flatten-nested-json-objects-jq.md
topic: jq
title: Flattening nested JSON objects with jq
url: https://github.com/simonw/til/blob/main/jq/flatten-nested-json-objects-jq.md
created: 2021-03-11T07:54:22-08:00
created_utc: 2021-03-11T15:54:22+00:00
updated: 2021-03-11T07:54:22-08:00
updated_utc: 2021-03-11T15:54:22+00:00
shot_hash: 205547010ca561829f6551b36b015a96
slug: flatten-nested-json-objects-jq
body:
I wanted to take a nested set of JSON objects and import them into a SQLite database using `sqlite-utils insert` - but I wanted to "flatten" some of the nested rows.

Example data:
```json
{
  "status": "success",
  "data": {
    "generated": "2021-02-18T20:14:02.288Z",
    "sites": [
      {
        "id": "full_data",
        "name": "Moscone Center South (full data)",
        "active": true,
        "location": {
          "address": "747 Howard St, San Francisco, CA 94103",
          "url": "https://www.google.com/maps/place/Moscone+Center+South,+747+Howard+St,+San+Francisco,+CA+94103",
          "lng": -122.401253,
          "lat": 37.78392
        },
        "info": {
          "url": "https://sf.gov/location/moscone-center-south-covid-19-vaccine-site"
        },
        "booking": {
          "url": "https://myturn.ca.gov",
          "dropins": false,
          "info": null
        },
        "access": {
          "wheelchair": true,
          "languages": {
            "en": true,
            "es": true,
            "zh": true,
            "fil": false,
            "vi": false,
            "ru": false
          },
          "remote_translation": {
            "available": false,
            "info": null
          }
        },
        "access_mode": {
          "walk": true,
          "drive": false
        },
        "open_to": {
          "everyone": true,
          "text": "Open to the public"
        },
        "appointments": {
          "available": true,
          "last_updated": "2021-02-18T20:14:02.288Z"
        },
        "eligibility": {
          "65_and_over": true,
          "healthcare_workers": true,
          "education_and_childcare": false,
          "agriculture_and_food": false,
          "emergency_services": false
        }
      }
    ]
  }
}
```
I wanted to turn this into an array of non-nested objects, like this:
```json
[
  {
    "id": "full_data",
    "name": "Moscone Center South (full data)",
    "active": true,
    "location_address": "747 Howard St, San Francisco, CA 94103",
    "location_url": "https://www.google.com/maps/place/Moscone+Center+South,+747+Howard+St,+San+Francisco,+CA+94103",
    "location_lng": -122.401253,
    "location_lat": 37.78392
  }
]
```
Thanks to [this StackOverflow answer](https://stackoverflow.com/a/37555908) I found the following `jq` fragment:
```jq
[leaf_paths as $path | {
    "key": $path | join("_"), "value": getpath($path)
}] | from_entries
```
This fragment transforms a nested JSON object into a flat one with `"location_address"` style keys instead.

I like trying these things out in interactive tools - https://www.jqkungfu.com/ is my current favourite, which runs the original `jq` in your browser compiled to WebAssembly.

I pasted in the example from above and then used this `jq` query to confirm that it works - the `.data.sites[] | [ ... ]` pattern here pulls out the `["data"]["sites"]` array and applies the flatten transformation to every item within it: 

    .data.sites[] | [ [leaf_paths as $path | {"key": $path | join("_"), "value": getpath($path)}] | from_entries ]

It worked!

The final, full recipe I used to pull down the JSON, transform and flatten it and insert it into a SQLite database was this:

```bash
curl 'https://vaccination-site-microservice.vercel.app/api/v1/sites' | \
  jq .data.sites | jq '
    [.[] | 
    [leaf_paths as $path | {"key": $path | join("_"), "value": getpath($path)}]
    | from_entries]
  ' | \
  sqlite-utils insert /tmp/sf.db sites -
```

html:
<p>I wanted to take a nested set of JSON objects and import them into a SQLite database using <code>sqlite-utils insert</code> - but I wanted to "flatten" some of the nested rows.</p>
<p>Example data:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-s"><span class="pl-pds">"</span>status<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>success<span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>data<span class="pl-pds">"</span></span>: {
    <span class="pl-s"><span class="pl-pds">"</span>generated<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-02-18T20:14:02.288Z<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>sites<span class="pl-pds">"</span></span>: [
      {
        <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>full_data<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Moscone Center South (full data)<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>active<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
        <span class="pl-s"><span class="pl-pds">"</span>location<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>address<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>747 Howard St, San Francisco, CA 94103<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://www.google.com/maps/place/Moscone+Center+South,+747+Howard+St,+San+Francisco,+CA+94103<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>lng<span class="pl-pds">"</span></span>: <span class="pl-c1">-122.401253</span>,
          <span class="pl-s"><span class="pl-pds">"</span>lat<span class="pl-pds">"</span></span>: <span class="pl-c1">37.78392</span>
        },
        <span class="pl-s"><span class="pl-pds">"</span>info<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://sf.gov/location/moscone-center-south-covid-19-vaccine-site<span class="pl-pds">"</span></span>
        },
        <span class="pl-s"><span class="pl-pds">"</span>booking<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://myturn.ca.gov<span class="pl-pds">"</span></span>,
          <span class="pl-s"><span class="pl-pds">"</span>dropins<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
          <span class="pl-s"><span class="pl-pds">"</span>info<span class="pl-pds">"</span></span>: <span class="pl-c1">null</span>
        },
        <span class="pl-s"><span class="pl-pds">"</span>access<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>wheelchair<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
          <span class="pl-s"><span class="pl-pds">"</span>languages<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>en<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
            <span class="pl-s"><span class="pl-pds">"</span>es<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
            <span class="pl-s"><span class="pl-pds">"</span>zh<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
            <span class="pl-s"><span class="pl-pds">"</span>fil<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
            <span class="pl-s"><span class="pl-pds">"</span>vi<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
            <span class="pl-s"><span class="pl-pds">"</span>ru<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>
          },
          <span class="pl-s"><span class="pl-pds">"</span>remote_translation<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>available<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
            <span class="pl-s"><span class="pl-pds">"</span>info<span class="pl-pds">"</span></span>: <span class="pl-c1">null</span>
          }
        },
        <span class="pl-s"><span class="pl-pds">"</span>access_mode<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>walk<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
          <span class="pl-s"><span class="pl-pds">"</span>drive<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>
        },
        <span class="pl-s"><span class="pl-pds">"</span>open_to<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>everyone<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
          <span class="pl-s"><span class="pl-pds">"</span>text<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Open to the public<span class="pl-pds">"</span></span>
        },
        <span class="pl-s"><span class="pl-pds">"</span>appointments<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>available<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
          <span class="pl-s"><span class="pl-pds">"</span>last_updated<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-02-18T20:14:02.288Z<span class="pl-pds">"</span></span>
        },
        <span class="pl-s"><span class="pl-pds">"</span>eligibility<span class="pl-pds">"</span></span>: {
          <span class="pl-s"><span class="pl-pds">"</span>65_and_over<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
          <span class="pl-s"><span class="pl-pds">"</span>healthcare_workers<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
          <span class="pl-s"><span class="pl-pds">"</span>education_and_childcare<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
          <span class="pl-s"><span class="pl-pds">"</span>agriculture_and_food<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
          <span class="pl-s"><span class="pl-pds">"</span>emergency_services<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>
        }
      }
    ]
  }
}</pre></div>
<p>I wanted to turn this into an array of non-nested objects, like this:</p>
<div class="highlight highlight-source-json"><pre>[
  {
    <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>full_data<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Moscone Center South (full data)<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>active<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
    <span class="pl-s"><span class="pl-pds">"</span>location_address<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>747 Howard St, San Francisco, CA 94103<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>location_url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://www.google.com/maps/place/Moscone+Center+South,+747+Howard+St,+San+Francisco,+CA+94103<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>location_lng<span class="pl-pds">"</span></span>: <span class="pl-c1">-122.401253</span>,
    <span class="pl-s"><span class="pl-pds">"</span>location_lat<span class="pl-pds">"</span></span>: <span class="pl-c1">37.78392</span>
  }
]</pre></div>
<p>Thanks to <a href="https://stackoverflow.com/a/37555908" rel="nofollow">this StackOverflow answer</a> I found the following <code>jq</code> fragment:</p>
<div class="highlight highlight-source-jq"><pre>[leaf_paths <span class="pl-k">as</span> $path | {
    <span class="pl-s">"key"</span>: $path | <span class="pl-c1">join</span>(<span class="pl-s">"_"</span>), <span class="pl-s">"value"</span>: <span class="pl-c1">getpath</span>($path)
}] | from_entries</pre></div>
<p>This fragment transforms a nested JSON object into a flat one with <code>"location_address"</code> style keys instead.</p>
<p>I like trying these things out in interactive tools - <a href="https://www.jqkungfu.com/" rel="nofollow">https://www.jqkungfu.com/</a> is my current favourite, which runs the original <code>jq</code> in your browser compiled to WebAssembly.</p>
<p>I pasted in the example from above and then used this <code>jq</code> query to confirm that it works - the <code>.data.sites[] | [ ... ]</code> pattern here pulls out the <code>["data"]["sites"]</code> array and applies the flatten transformation to every item within it:</p>
<pre><code>.data.sites[] | [ [leaf_paths as $path | {"key": $path | join("_"), "value": getpath($path)}] | from_entries ]
</code></pre>
<p>It worked!</p>
<p>The final, full recipe I used to pull down the JSON, transform and flatten it and insert it into a SQLite database was this:</p>
<div class="highlight highlight-source-shell"><pre>curl <span class="pl-s"><span class="pl-pds">'</span>https://vaccination-site-microservice.vercel.app/api/v1/sites<span class="pl-pds">'</span></span> <span class="pl-k">|</span> \
  jq .data.sites <span class="pl-k">|</span> jq <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s">    [.[] | </span>
<span class="pl-s">    [leaf_paths as $path | {"key": $path | join("_"), "value": getpath($path)}]</span>
<span class="pl-s">    | from_entries]</span>
<span class="pl-s">  <span class="pl-pds">'</span></span> <span class="pl-k">|</span> \
  sqlite-utils insert /tmp/sf.db sites -</pre></div>


path: jq_convert-no-decimal-point-latitude-jq.md
topic: jq
title: Converting no-decimal-point latitudes and longitudes using jq
url: https://github.com/simonw/til/blob/main/jq/convert-no-decimal-point-latitude-jq.md
created: 2021-03-11T08:04:06-08:00
created_utc: 2021-03-11T16:04:06+00:00
updated: 2021-03-11T08:04:06-08:00
updated_utc: 2021-03-11T16:04:06+00:00
shot_hash: e375a856e92212c4d26bc12a4a77f746
slug: convert-no-decimal-point-latitude-jq
body:
I had some data with weird co-ordinates in it:

```json
{
    "Agency": "F",
    "Address": "S WAYSIDE DR",
    "CrossStreet": "BLK GULF FWY IB",
    "KeyMap": "534C",
    "XCoord": "-95314927",
    "YCoord": "29716075",
    "CombinedResponse": "F",
    "CallTimeOpened": "02/20/2021 22:29",
    "IncidentType": "Ems Event",
    "AlarmLevel": "0",
    "NumberOfUnits": "1",
    "Units": "A023;"
}
```
Note the `"XCoord": "-95314927"` and `"YCoord": "29716075"` in there.

I know that this data refers to somewhere in Houston, and [a few people](https://twitter.com/simonw/status/1369163089636036610) on Twitter pointed out that it's actually really simple: it's just missing the decimal point!

Here's the `jq` recipe I came up with to transform this:

```jq
{
    latitude: (.YCoord  | tonumber / 1000000.0),
    longitude: (.XCoord | tonumber / 1000000.0),
    date: (.CallTimeOpened | strptime("%m/%d/%Y %H:%M") | todate),
    key: (.CallTimeOpened + " " + .XCoord + " " + .YCoord )
} + .
```
The `+ .` at the end adds back on the original object.

Running this example on https://www.jqkungfu.com/ gives the following output:

```json
{
    "Address": "S WAYSIDE DR",
    "Agency": "F",
    "AlarmLevel": "0",
    "CallTimeOpened": "02/20/2021 22:29",
    "CombinedResponse": "F",
    "CrossStreet": "BLK GULF FWY IB",
    "IncidentType": "Ems Event",
    "KeyMap": "534C",
    "NumberOfUnits": "1",
    "Units": "A023;",
    "XCoord": "-95314927",
    "YCoord": "29716075",
    "date": "2021-02-20T22:29:00Z",
    "key": "02/20/2021 22:29 -95314927 29716075",
    "latitude": 29.716075,
    "longitude": -95.314927
}
```
Bonus is this example: I concatenated together a unique primary key for each record in `key`, and I figured out how to use the `strptime()` function to parse a date.

html:
<p>I had some data with weird co-ordinates in it:</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-s"><span class="pl-pds">"</span>Agency<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>F<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>Address<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>S WAYSIDE DR<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>CrossStreet<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>BLK GULF FWY IB<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>KeyMap<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>534C<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>XCoord<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>-95314927<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>YCoord<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>29716075<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>CombinedResponse<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>F<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>CallTimeOpened<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>02/20/2021 22:29<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>IncidentType<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Ems Event<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>AlarmLevel<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>0<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>NumberOfUnits<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>1<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>Units<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>A023;<span class="pl-pds">"</span></span>
}</pre></div>
<p>Note the <code>"XCoord": "-95314927"</code> and <code>"YCoord": "29716075"</code> in there.</p>
<p>I know that this data refers to somewhere in Houston, and <a href="https://twitter.com/simonw/status/1369163089636036610" rel="nofollow">a few people</a> on Twitter pointed out that it's actually really simple: it's just missing the decimal point!</p>
<p>Here's the <code>jq</code> recipe I came up with to transform this:</p>
<div class="highlight highlight-source-jq"><pre>{
    latitude: (.YCoord  | tonumber / <span class="pl-c1">1000000.0</span>),
    longitude: (.XCoord | tonumber / <span class="pl-c1">1000000.0</span>),
    date: (.CallTimeOpened | <span class="pl-c1">strptime</span>(<span class="pl-s">"%m/%d/%Y %H:%M"</span>) | todate),
    key: (.CallTimeOpened + <span class="pl-s">" "</span> + .XCoord + <span class="pl-s">" "</span> + .YCoord )
} + .</pre></div>
<p>The <code>+ .</code> at the end adds back on the original object.</p>
<p>Running this example on <a href="https://www.jqkungfu.com/" rel="nofollow">https://www.jqkungfu.com/</a> gives the following output:</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-s"><span class="pl-pds">"</span>Address<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>S WAYSIDE DR<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>Agency<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>F<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>AlarmLevel<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>0<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>CallTimeOpened<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>02/20/2021 22:29<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>CombinedResponse<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>F<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>CrossStreet<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>BLK GULF FWY IB<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>IncidentType<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Ems Event<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>KeyMap<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>534C<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>NumberOfUnits<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>1<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>Units<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>A023;<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>XCoord<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>-95314927<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>YCoord<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>29716075<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021-02-20T22:29:00Z<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>key<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>02/20/2021 22:29 -95314927 29716075<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>latitude<span class="pl-pds">"</span></span>: <span class="pl-c1">29.716075</span>,
    <span class="pl-s"><span class="pl-pds">"</span>longitude<span class="pl-pds">"</span></span>: <span class="pl-c1">-95.314927</span>
}</pre></div>
<p>Bonus is this example: I concatenated together a unique primary key for each record in <code>key</code>, and I figured out how to use the <code>strptime()</code> function to parse a date.</p>


path: django_almost-facet-counts-django-admin.md
topic: django
title: How to almost get facet counts in the Django admin
url: https://github.com/simonw/til/blob/main/django/almost-facet-counts-django-admin.md
created: 2021-03-11T14:50:25-08:00
created_utc: 2021-03-11T22:50:25+00:00
updated: 2021-03-11T14:50:25-08:00
updated_utc: 2021-03-11T22:50:25+00:00
shot_hash: 2dfba328de3d3ebec6519713a6c970a7
slug: almost-facet-counts-django-admin
body:
For a tantalizing moment today I thought I'd found a recipe for adding facet counts to the Django admin.

I love faceted browsing. I've implemented it at least a dozen times in my career, using everything from Solr and Elasticsearch to PostgreSQL (see [Implementing faceted search with Django and PostgreSQL](https://simonwillison.net/2017/Oct/5/django-postgresql-faceted-search/)) or SQLite (see [Datasette Facets](https://simonwillison.net/2018/May/20/datasette-facets/)).

The Django admin almost has facets out of the box, thanks to the `list_filter` interface. But they're missing the all-imprtant count values! Those are the thing that makes faceted search so valuable to me. Today I decided to try and add them.

## Almost facet counts

Here's my first attempt. This assumes a model has a `State` foreign key, and adds faceting by state:

```python
class StateCountFilter(admin.SimpleListFilter):
    title = 'State count'
    parameter_name = 'state_count'

    def lookups(self, request, model_admin):
        qs = model_admin.get_queryset(request)
        states_and_counts = qs.values_list(
            "state__abbreviation", "state__name"
        ).annotate(n = Count('state__abbreviation'))
        for abbreviation, name, count in states_and_counts:
            yield abbreviation, '{}: {:,}'.format(name, count)

    def queryset(self, request, queryset):
        state = self.value()
        if state:
            return queryset.filter(
                state__abbreviation=state
            )

# Then add this to the ModelAdmin:

@admin.register(Location)
class LocationAdmin(admin.ModelAdmin):
    list_filter = (
        StateCountFilter,
    )
```
I tried this out, and for a glorious moment I thought I had solved it! I added it to another column too, and started trying it out.

<img width="1217" alt="110856792-eda4a000-826c-11eb-8f99-2676c1030423" src="https://user-images.githubusercontent.com/9599/110865748-f4391480-8278-11eb-90b4-a12b42f3c5de.png">

Then I attempted to apply one of the filters:

<img width="1190" alt="broken" src="https://user-images.githubusercontent.com/9599/110865785-074be480-8279-11eb-8d8f-b87cb3ba025a.png">

This is broken. The whole point of facet counts is that they show you counts for your existing selection - so you can filter down to the state of Oregon and see how many locations of type Pharmacy you have in that state.

But that wasn't happening here, because I was calculating the counts using `model_admin.get_queryset(request)` - which returns the unfiltered root queryset.

## Trying to get filtered counts

The challenge here is to get access to the currently filtered selection from within that `lookups()` method.

Here's the closest I got:

```python
    def lookups(self, request, model_admin):
        changelist = model_admin.get_changelist_instance(request)
        qs = changelist.get_queryset(request)
        states_and_counts = qs.values_list(
            "state__abbreviation", "state__name"
        ).annotate(n = Count('state__abbreviation'))
```
I tried this out... and hit a `RecursionError`! It turns out that `.get_changelist_instance()` method itself calls the `.lookups()` method, presumably because it needs those lookups in order to construct the current filtered selection (or to validate the request arguments perhaps).

I added `traceback.print_stack(limit=10)` to my `lookups()` method to confirm that I was right about this - here's the traceback:

```
  File "../site-packages/django/contrib/admin/sites.py", line 233, in inner
    return view(request, *args, **kwargs)
  File "../site-packages/django/utils/decorators.py", line 43, in _wrapper
    return bound_method(*args, **kwargs)
  File "../site-packages/django/utils/decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "../site-packages/django/contrib/admin/options.py", line 1693, in changelist_view
    cl = self.get_changelist_instance(request)
  File "../site-packages/django/contrib/admin/options.py", line 735, in get_changelist_instance
    return ChangeList(
  File "../site-packages/django/contrib/admin/views/main.py", line 99, in __init__
    self.queryset = self.get_queryset(request)
  File "../site-packages/django/contrib/admin/views/main.py", line 450, in get_queryset
    ) = self.get_filters(request)
  File "../site-packages/django/contrib/admin/views/main.py", line 137, in get_filters
    spec = list_filter(request, lookup_params, self.model, self.model_admin)
  File "../site-packages/django/contrib/admin/filters.py", line 79, in __init__
    self.lookup_choices = list(lookup_choices)
  File "core/admin.py", line 104, in lookups
    traceback.print_stack(limit=10)
```
I tried searching around to see if anyone else had solved this problem, but the best I could find was [this StackOverflow comment](https://stackoverflow.com/questions/28257979/how-to-access-the-filtered-queryset-in-django-admin-simplelistfilter#comment113524600_28258062) which hit the same recursion blocker that I found.

So as far as I can tell it isn't currently possible to implement facet counts correctly in a Django `SimpleListFilter.lookups()` method.

## Possible alternative: do it in JavaScript

I _really_ want facet counts. One workaround I'm considering is to do it in JavaScript: load the standard admin page, then have some custom JavaScript that hits a custom API endpoint with the current set of querystring parameters, fetches back filter and facet counts and injects them into the correct place on the page.

I really wish Django could handle this for me natively though!

html:
<p>For a tantalizing moment today I thought I'd found a recipe for adding facet counts to the Django admin.</p>
<p>I love faceted browsing. I've implemented it at least a dozen times in my career, using everything from Solr and Elasticsearch to PostgreSQL (see <a href="https://simonwillison.net/2017/Oct/5/django-postgresql-faceted-search/" rel="nofollow">Implementing faceted search with Django and PostgreSQL</a>) or SQLite (see <a href="https://simonwillison.net/2018/May/20/datasette-facets/" rel="nofollow">Datasette Facets</a>).</p>
<p>The Django admin almost has facets out of the box, thanks to the <code>list_filter</code> interface. But they're missing the all-imprtant count values! Those are the thing that makes faceted search so valuable to me. Today I decided to try and add them.</p>
<h2>
<a id="user-content-almost-facet-counts" class="anchor" href="#almost-facet-counts" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Almost facet counts</h2>
<p>Here's my first attempt. This assumes a model has a <code>State</code> foreign key, and adds faceting by state:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">class</span> <span class="pl-v">StateCountFilter</span>(<span class="pl-s1">admin</span>.<span class="pl-v">SimpleListFilter</span>):
    <span class="pl-s1">title</span> <span class="pl-c1">=</span> <span class="pl-s">'State count'</span>
    <span class="pl-s1">parameter_name</span> <span class="pl-c1">=</span> <span class="pl-s">'state_count'</span>

    <span class="pl-k">def</span> <span class="pl-en">lookups</span>(<span class="pl-s1">self</span>, <span class="pl-s1">request</span>, <span class="pl-s1">model_admin</span>):
        <span class="pl-s1">qs</span> <span class="pl-c1">=</span> <span class="pl-s1">model_admin</span>.<span class="pl-en">get_queryset</span>(<span class="pl-s1">request</span>)
        <span class="pl-s1">states_and_counts</span> <span class="pl-c1">=</span> <span class="pl-s1">qs</span>.<span class="pl-en">values_list</span>(
            <span class="pl-s">"state__abbreviation"</span>, <span class="pl-s">"state__name"</span>
        ).<span class="pl-en">annotate</span>(<span class="pl-s1">n</span> <span class="pl-c1">=</span> <span class="pl-v">Count</span>(<span class="pl-s">'state__abbreviation'</span>))
        <span class="pl-k">for</span> <span class="pl-s1">abbreviation</span>, <span class="pl-s1">name</span>, <span class="pl-s1">count</span> <span class="pl-c1">in</span> <span class="pl-s1">states_and_counts</span>:
            <span class="pl-k">yield</span> <span class="pl-s1">abbreviation</span>, <span class="pl-s">'{}: {:,}'</span>.<span class="pl-en">format</span>(<span class="pl-s1">name</span>, <span class="pl-s1">count</span>)

    <span class="pl-k">def</span> <span class="pl-en">queryset</span>(<span class="pl-s1">self</span>, <span class="pl-s1">request</span>, <span class="pl-s1">queryset</span>):
        <span class="pl-s1">state</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">value</span>()
        <span class="pl-k">if</span> <span class="pl-s1">state</span>:
            <span class="pl-k">return</span> <span class="pl-s1">queryset</span>.<span class="pl-en">filter</span>(
                <span class="pl-s1">state__abbreviation</span><span class="pl-c1">=</span><span class="pl-s1">state</span>
            )

<span class="pl-c"># Then add this to the ModelAdmin:</span>

<span class="pl-en">@<span class="pl-s1">admin</span>.<span class="pl-en">register</span>(<span class="pl-v">Location</span>)</span>
<span class="pl-k">class</span> <span class="pl-v">LocationAdmin</span>(<span class="pl-s1">admin</span>.<span class="pl-v">ModelAdmin</span>):
    <span class="pl-s1">list_filter</span> <span class="pl-c1">=</span> (
        <span class="pl-v">StateCountFilter</span>,
    )</pre></div>
<p>I tried this out, and for a glorious moment I thought I had solved it! I added it to another column too, and started trying it out.</p>
<p><a href="https://user-images.githubusercontent.com/9599/110865748-f4391480-8278-11eb-90b4-a12b42f3c5de.png" target="_blank" rel="nofollow"><img width="1217" alt="110856792-eda4a000-826c-11eb-8f99-2676c1030423" src="https://user-images.githubusercontent.com/9599/110865748-f4391480-8278-11eb-90b4-a12b42f3c5de.png" style="max-width:100%;"></a></p>
<p>Then I attempted to apply one of the filters:</p>
<p><a href="https://user-images.githubusercontent.com/9599/110865785-074be480-8279-11eb-8d8f-b87cb3ba025a.png" target="_blank" rel="nofollow"><img width="1190" alt="broken" src="https://user-images.githubusercontent.com/9599/110865785-074be480-8279-11eb-8d8f-b87cb3ba025a.png" style="max-width:100%;"></a></p>
<p>This is broken. The whole point of facet counts is that they show you counts for your existing selection - so you can filter down to the state of Oregon and see how many locations of type Pharmacy you have in that state.</p>
<p>But that wasn't happening here, because I was calculating the counts using <code>model_admin.get_queryset(request)</code> - which returns the unfiltered root queryset.</p>
<h2>
<a id="user-content-trying-to-get-filtered-counts" class="anchor" href="#trying-to-get-filtered-counts" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Trying to get filtered counts</h2>
<p>The challenge here is to get access to the currently filtered selection from within that <code>lookups()</code> method.</p>
<p>Here's the closest I got:</p>
<div class="highlight highlight-source-python"><pre>    <span class="pl-k">def</span> <span class="pl-en">lookups</span>(<span class="pl-s1">self</span>, <span class="pl-s1">request</span>, <span class="pl-s1">model_admin</span>):
        <span class="pl-s1">changelist</span> <span class="pl-c1">=</span> <span class="pl-s1">model_admin</span>.<span class="pl-en">get_changelist_instance</span>(<span class="pl-s1">request</span>)
        <span class="pl-s1">qs</span> <span class="pl-c1">=</span> <span class="pl-s1">changelist</span>.<span class="pl-en">get_queryset</span>(<span class="pl-s1">request</span>)
        <span class="pl-s1">states_and_counts</span> <span class="pl-c1">=</span> <span class="pl-s1">qs</span>.<span class="pl-en">values_list</span>(
            <span class="pl-s">"state__abbreviation"</span>, <span class="pl-s">"state__name"</span>
        ).<span class="pl-en">annotate</span>(<span class="pl-s1">n</span> <span class="pl-c1">=</span> <span class="pl-v">Count</span>(<span class="pl-s">'state__abbreviation'</span>))</pre></div>
<p>I tried this out... and hit a <code>RecursionError</code>! It turns out that <code>.get_changelist_instance()</code> method itself calls the <code>.lookups()</code> method, presumably because it needs those lookups in order to construct the current filtered selection (or to validate the request arguments perhaps).</p>
<p>I added <code>traceback.print_stack(limit=10)</code> to my <code>lookups()</code> method to confirm that I was right about this - here's the traceback:</p>
<pre><code>  File "../site-packages/django/contrib/admin/sites.py", line 233, in inner
    return view(request, *args, **kwargs)
  File "../site-packages/django/utils/decorators.py", line 43, in _wrapper
    return bound_method(*args, **kwargs)
  File "../site-packages/django/utils/decorators.py", line 130, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "../site-packages/django/contrib/admin/options.py", line 1693, in changelist_view
    cl = self.get_changelist_instance(request)
  File "../site-packages/django/contrib/admin/options.py", line 735, in get_changelist_instance
    return ChangeList(
  File "../site-packages/django/contrib/admin/views/main.py", line 99, in __init__
    self.queryset = self.get_queryset(request)
  File "../site-packages/django/contrib/admin/views/main.py", line 450, in get_queryset
    ) = self.get_filters(request)
  File "../site-packages/django/contrib/admin/views/main.py", line 137, in get_filters
    spec = list_filter(request, lookup_params, self.model, self.model_admin)
  File "../site-packages/django/contrib/admin/filters.py", line 79, in __init__
    self.lookup_choices = list(lookup_choices)
  File "core/admin.py", line 104, in lookups
    traceback.print_stack(limit=10)
</code></pre>
<p>I tried searching around to see if anyone else had solved this problem, but the best I could find was <a href="https://stackoverflow.com/questions/28257979/how-to-access-the-filtered-queryset-in-django-admin-simplelistfilter#comment113524600_28258062" rel="nofollow">this StackOverflow comment</a> which hit the same recursion blocker that I found.</p>
<p>So as far as I can tell it isn't currently possible to implement facet counts correctly in a Django <code>SimpleListFilter.lookups()</code> method.</p>
<h2>
<a id="user-content-possible-alternative-do-it-in-javascript" class="anchor" href="#possible-alternative-do-it-in-javascript" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Possible alternative: do it in JavaScript</h2>
<p>I <em>really</em> want facet counts. One workaround I'm considering is to do it in JavaScript: load the standard admin page, then have some custom JavaScript that hits a custom API endpoint with the current set of querystring parameters, fetches back filter and facet counts and injects them into the correct place on the page.</p>
<p>I really wish Django could handle this for me natively though!</p>


path: datasette_issues-open-for-less-than-x-seconds.md
topic: datasette
title: Querying for GitHub issues open for less than 60 seconds
url: https://github.com/simonw/til/blob/main/datasette/issues-open-for-less-than-x-seconds.md
created: 2021-03-12T07:34:42-08:00
created_utc: 2021-03-12T15:34:42+00:00
updated: 2021-03-12T07:34:42-08:00
updated_utc: 2021-03-12T15:34:42+00:00
shot_hash: e0f132933840169839c18ddd06e78cac
slug: issues-open-for-less-than-x-seconds
body:
While [writing this thread](https://twitter.com/simonw/status/1370390336514658310) about my habit of opening issues and closing them a few seconds later just so I could link to them in a commit message I decided to answer the question "How many of my issues were open for less than 60 seconds?"

Thanks to [github-to-sqlite](https://datasette.io/tools/github-to-sqlite) I have an [issues database table](https://github-to-sqlite.dogsheep.net/github/issues) containing issues from all of my public projects.

I needed to figure out how to calculate the difference between `closed_at` and `created_at` in seconds. This works:

```sql
select strftime('%s',issues.closed_at) - strftime('%s',issues.created_at) as duration_open_in_seconds ...
```
I wanted to be able to input the number of seconds as a parameter. I used this:
```sql
duration_open_in_seconds < CAST(:max_duration_in_seconds AS INTEGER)
```
This is the full query - [try it out here](https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++json_object%28%0D%0A++++%27label%27%2C+repos.full_name+%7C%7C+%27+%23%27+%7C%7C+issues.number%2C%0D%0A++++%27href%27%2C+%27https%3A%2F%2Fgithub.com%2F%27+%7C%7C+repos.full_name+%7C%7C+%27%2Fissues%2F%27+%7C%7C+issues.number%0D%0A++%29+as+link%2C%0D%0A++strftime%28%27%25s%27%2Cissues.closed_at%29+-+strftime%28%27%25s%27%2Cissues.created_at%29+as+duration_open_in_seconds%2C%0D%0A++issues.number+as+issue_number%2C%0D%0A++issues.title%2C%0D%0A++users.login%2C%0D%0A++issues.closed_at%2C%0D%0A++issues.created_at%2C%0D%0A++issues.body%2C%0D%0A++issues.type%0D%0Afrom%0D%0A++issues+join+repos+on+issues.repo+%3D+repos.id%0D%0A++join+users+on+issues.user+%3D+users.id%0D%0A++where+issues.closed_at+is+not+null+and+duration_open_in_seconds+%3C+CAST%28%3Amax_duration_in_seconds+AS+INTEGER%29%0D%0Aorder+by%0D%0A++issues.closed_at+desc&max_duration_in_seconds=60):

```sql
select
  json_object(
    'label', repos.full_name || ' #' || issues.number,
    'href', 'https://github.com/' || repos.full_name || '/issues/' || issues.number
  ) as link,
  strftime('%s',issues.closed_at) - strftime('%s',issues.created_at) as duration_open_in_seconds,
  issues.number as issue_number,
  issues.title,
  users.login,
  issues.closed_at,
  issues.created_at,
  issues.body,
  issues.type
from
  issues join repos on issues.repo = repos.id
  join users on issues.user = users.id
  where issues.closed_at is not null and duration_open_in_seconds < CAST(:max_duration_in_seconds AS INTEGER)
order by
  issues.closed_at desc
```

html:
<p>While <a href="https://twitter.com/simonw/status/1370390336514658310" rel="nofollow">writing this thread</a> about my habit of opening issues and closing them a few seconds later just so I could link to them in a commit message I decided to answer the question "How many of my issues were open for less than 60 seconds?"</p>
<p>Thanks to <a href="https://datasette.io/tools/github-to-sqlite" rel="nofollow">github-to-sqlite</a> I have an <a href="https://github-to-sqlite.dogsheep.net/github/issues" rel="nofollow">issues database table</a> containing issues from all of my public projects.</p>
<p>I needed to figure out how to calculate the difference between <code>closed_at</code> and <code>created_at</code> in seconds. This works:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> strftime(<span class="pl-s"><span class="pl-pds">'</span>%s<span class="pl-pds">'</span></span>,<span class="pl-c1">issues</span>.<span class="pl-c1">closed_at</span>) <span class="pl-k">-</span> strftime(<span class="pl-s"><span class="pl-pds">'</span>%s<span class="pl-pds">'</span></span>,<span class="pl-c1">issues</span>.<span class="pl-c1">created_at</span>) <span class="pl-k">as</span> duration_open_in_seconds ...</pre></div>
<p>I wanted to be able to input the number of seconds as a parameter. I used this:</p>
<div class="highlight highlight-source-sql"><pre>duration_open_in_seconds <span class="pl-k">&lt;</span> CAST(:max_duration_in_seconds <span class="pl-k">AS</span> <span class="pl-k">INTEGER</span>)</pre></div>
<p>This is the full query - <a href="https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++json_object%28%0D%0A++++%27label%27%2C+repos.full_name+%7C%7C+%27+%23%27+%7C%7C+issues.number%2C%0D%0A++++%27href%27%2C+%27https%3A%2F%2Fgithub.com%2F%27+%7C%7C+repos.full_name+%7C%7C+%27%2Fissues%2F%27+%7C%7C+issues.number%0D%0A++%29+as+link%2C%0D%0A++strftime%28%27%25s%27%2Cissues.closed_at%29+-+strftime%28%27%25s%27%2Cissues.created_at%29+as+duration_open_in_seconds%2C%0D%0A++issues.number+as+issue_number%2C%0D%0A++issues.title%2C%0D%0A++users.login%2C%0D%0A++issues.closed_at%2C%0D%0A++issues.created_at%2C%0D%0A++issues.body%2C%0D%0A++issues.type%0D%0Afrom%0D%0A++issues+join+repos+on+issues.repo+%3D+repos.id%0D%0A++join+users+on+issues.user+%3D+users.id%0D%0A++where+issues.closed_at+is+not+null+and+duration_open_in_seconds+%3C+CAST%28%3Amax_duration_in_seconds+AS+INTEGER%29%0D%0Aorder+by%0D%0A++issues.closed_at+desc&amp;max_duration_in_seconds=60" rel="nofollow">try it out here</a>:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  json_object(
    <span class="pl-s"><span class="pl-pds">'</span>label<span class="pl-pds">'</span></span>, <span class="pl-c1">repos</span>.<span class="pl-c1">full_name</span> <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span> #<span class="pl-pds">'</span></span> <span class="pl-k">||</span> <span class="pl-c1">issues</span>.<span class="pl-c1">number</span>,
    <span class="pl-s"><span class="pl-pds">'</span>href<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>https://github.com/<span class="pl-pds">'</span></span> <span class="pl-k">||</span> <span class="pl-c1">repos</span>.<span class="pl-c1">full_name</span> <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>/issues/<span class="pl-pds">'</span></span> <span class="pl-k">||</span> <span class="pl-c1">issues</span>.<span class="pl-c1">number</span>
  ) <span class="pl-k">as</span> link,
  strftime(<span class="pl-s"><span class="pl-pds">'</span>%s<span class="pl-pds">'</span></span>,<span class="pl-c1">issues</span>.<span class="pl-c1">closed_at</span>) <span class="pl-k">-</span> strftime(<span class="pl-s"><span class="pl-pds">'</span>%s<span class="pl-pds">'</span></span>,<span class="pl-c1">issues</span>.<span class="pl-c1">created_at</span>) <span class="pl-k">as</span> duration_open_in_seconds,
  <span class="pl-c1">issues</span>.<span class="pl-c1">number</span> <span class="pl-k">as</span> issue_number,
  <span class="pl-c1">issues</span>.<span class="pl-c1">title</span>,
  <span class="pl-c1">users</span>.<span class="pl-c1">login</span>,
  <span class="pl-c1">issues</span>.<span class="pl-c1">closed_at</span>,
  <span class="pl-c1">issues</span>.<span class="pl-c1">created_at</span>,
  <span class="pl-c1">issues</span>.<span class="pl-c1">body</span>,
  <span class="pl-c1">issues</span>.<span class="pl-c1">type</span>
<span class="pl-k">from</span>
  issues <span class="pl-k">join</span> repos <span class="pl-k">on</span> <span class="pl-c1">issues</span>.<span class="pl-c1">repo</span> <span class="pl-k">=</span> <span class="pl-c1">repos</span>.<span class="pl-c1">id</span>
  <span class="pl-k">join</span> users <span class="pl-k">on</span> <span class="pl-c1">issues</span>.<span class="pl-c1">user</span> <span class="pl-k">=</span> <span class="pl-c1">users</span>.<span class="pl-c1">id</span>
  <span class="pl-k">where</span> <span class="pl-c1">issues</span>.<span class="pl-c1">closed_at</span> <span class="pl-k">is not null</span> <span class="pl-k">and</span> duration_open_in_seconds <span class="pl-k">&lt;</span> CAST(:max_duration_in_seconds <span class="pl-k">AS</span> <span class="pl-k">INTEGER</span>)
<span class="pl-k">order by</span>
  <span class="pl-c1">issues</span>.<span class="pl-c1">closed_at</span> <span class="pl-k">desc</span></pre></div>


path: sqlite_utc-items-on-thursday-in-pst.md
topic: sqlite
title: Querying for items stored in UTC that were created on a Thursday in PST
url: https://github.com/simonw/til/blob/main/sqlite/utc-items-on-thursday-in-pst.md
created: 2021-03-12T19:55:21-08:00
created_utc: 2021-03-13T03:55:21+00:00
updated: 2021-03-12T19:55:21-08:00
updated_utc: 2021-03-13T03:55:21+00:00
shot_hash: f27ad33d52211f50328ba59b0c5f372b
slug: utc-items-on-thursday-in-pst
body:
This came up as [a question](https://news.ycombinator.com/item?id=26443148) on Hacker News. How can you query a SQLite database for items that were created on a Thursday in PST, when the data is stored in UTC?

I have datetimes stored in UTC, so I first needed to convert them to PST by applying the 8 hour time difference, using `datetime(author_date, '-8 hours') as author_date_pst`.

Then I used `strftime('%w')` to get the day of week (as a number contained in a string).

Then I can filter for that equalling '4' for Thursday.

```sql
select
  author_date,
  datetime(author_date, '-8 hours') as author_date_pst,
  strftime('%w', datetime(author_date, '-8 hours')) as dayofweek_pst,
  *
from
  commits
where
  dayofweek_pst = '4' -- Thursday
```
[Try this query](https://github-to-sqlite.dogsheep.net/github?sql=select+author_date%2C%0D%0Adatetime%28author_date%2C+%27-8+hours%27%29+as+author_date_pst%2C+%0D%0Astrftime%28%27%25w%27%2C+datetime%28author_date%2C+%27-8+hours%27%29%29+as+dayofweek_pst%2C%0D%0A*+from+commits%0D%0Awhere+dayofweek_pst+%3D+%274%27+--+Thursday+).

SQLite documentation for date time functions is at https://sqlite.org/lang_datefunc.html

html:
<p>This came up as <a href="https://news.ycombinator.com/item?id=26443148" rel="nofollow">a question</a> on Hacker News. How can you query a SQLite database for items that were created on a Thursday in PST, when the data is stored in UTC?</p>
<p>I have datetimes stored in UTC, so I first needed to convert them to PST by applying the 8 hour time difference, using <code>datetime(author_date, '-8 hours') as author_date_pst</code>.</p>
<p>Then I used <code>strftime('%w')</code> to get the day of week (as a number contained in a string).</p>
<p>Then I can filter for that equalling '4' for Thursday.</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  author_date,
  datetime(author_date, <span class="pl-s"><span class="pl-pds">'</span>-8 hours<span class="pl-pds">'</span></span>) <span class="pl-k">as</span> author_date_pst,
  strftime(<span class="pl-s"><span class="pl-pds">'</span>%w<span class="pl-pds">'</span></span>, datetime(author_date, <span class="pl-s"><span class="pl-pds">'</span>-8 hours<span class="pl-pds">'</span></span>)) <span class="pl-k">as</span> dayofweek_pst,
  <span class="pl-k">*</span>
<span class="pl-k">from</span>
  commits
<span class="pl-k">where</span>
  dayofweek_pst <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">'</span>4<span class="pl-pds">'</span></span> <span class="pl-c"><span class="pl-c">--</span> Thursday</span></pre></div>
<p><a href="https://github-to-sqlite.dogsheep.net/github?sql=select+author_date%2C%0D%0Adatetime%28author_date%2C+%27-8+hours%27%29+as+author_date_pst%2C+%0D%0Astrftime%28%27%25w%27%2C+datetime%28author_date%2C+%27-8+hours%27%29%29+as+dayofweek_pst%2C%0D%0A*+from+commits%0D%0Awhere+dayofweek_pst+%3D+%274%27+--+Thursday+" rel="nofollow">Try this query</a>.</p>
<p>SQLite documentation for date time functions is at <a href="https://sqlite.org/lang_datefunc.html" rel="nofollow">https://sqlite.org/lang_datefunc.html</a></p>


path: docker_gdb-python-docker.md
topic: docker
title: Running gdb against a Python process in a running Docker container
url: https://github.com/simonw/til/blob/main/docker/gdb-python-docker.md
created: 2021-03-21T22:48:21-07:00
created_utc: 2021-03-22T05:48:21+00:00
updated: 2021-03-21T22:48:21-07:00
updated_utc: 2021-03-22T05:48:21+00:00
shot_hash: 7b9f40a0c261e3d630b151d72313cabc
slug: gdb-python-docker
body:
While investigating [Datasette issue #1268](https://github.com/simonw/datasette/issues/1268) I found myself with a Python process that was hanging, and I decided to try running `gdb` against it based on tips in [Debugging of CPython processes with gdb](https://www.podoliaka.org/2016/04/10/debugging-cpython-gdb/)

Here's the recipe that worked:

1. Find the Docker container ID using `docker ps` - in my case it was `16197781a7b5`
2. Attach a new bash shell to that process in privileged mode (needed to get `gdb` to work): `docker exec --privileged -it 16197781a7b5 bash`
3. Install `gdb` and the Python tooling for using it: `apt-get install gdb python3-dbg`
4. Use `top` to find the pid of the running Python process that was hanging. It was `20` for me.
5. Run `gdb /usr/bin/python3 -p 20` to launch `gdb` against that process
6. In the `(gdb)` prompt run `py-bt` to see a backtrace.

I'm sure there's lots more that can be done in `gdb` at this point, but that's how I got to a place where I could interact with the Python process that was running in the Docker container.

html:
<p>While investigating <a href="https://github.com/simonw/datasette/issues/1268">Datasette issue #1268</a> I found myself with a Python process that was hanging, and I decided to try running <code>gdb</code> against it based on tips in <a href="https://www.podoliaka.org/2016/04/10/debugging-cpython-gdb/" rel="nofollow">Debugging of CPython processes with gdb</a></p>
<p>Here's the recipe that worked:</p>
<ol>
<li>Find the Docker container ID using <code>docker ps</code> - in my case it was <code>16197781a7b5</code>
</li>
<li>Attach a new bash shell to that process in privileged mode (needed to get <code>gdb</code> to work): <code>docker exec --privileged -it 16197781a7b5 bash</code>
</li>
<li>Install <code>gdb</code> and the Python tooling for using it: <code>apt-get install gdb python3-dbg</code>
</li>
<li>Use <code>top</code> to find the pid of the running Python process that was hanging. It was <code>20</code> for me.</li>
<li>Run <code>gdb /usr/bin/python3 -p 20</code> to launch <code>gdb</code> against that process</li>
<li>In the <code>(gdb)</code> prompt run <code>py-bt</code> to see a backtrace.</li>
</ol>
<p>I'm sure there's lots more that can be done in <code>gdb</code> at this point, but that's how I got to a place where I could interact with the Python process that was running in the Docker container.</p>


path: python_tracing-every-statement.md
topic: python
title: Tracing every executed Python statement
url: https://github.com/simonw/til/blob/main/python/tracing-every-statement.md
created: 2021-03-21T22:51:17-07:00
created_utc: 2021-03-22T05:51:17+00:00
updated: 2021-03-21T22:51:17-07:00
updated_utc: 2021-03-22T05:51:17+00:00
shot_hash: ed8e960c729eb6d38947e07bcf67cc91
slug: tracing-every-statement
body:
Today I learned how to use the Python [trace module](https://docs.python.org/3/library/trace.html) to output every single executed line of Python code in a program - useful for figuring out exactly when a crash or infinite loop happens.

The basic format is to run:

    python3 -m trace --trace myscript.py

This will execute the script and print out every single line of code as it executes - which can be a LOT of output. It slows the program down to a crawl - just starting up Datasette took probably over a minute and churned through hundreds of thousands of lines of console output.

Since Datasette is a command-line application, I needed to use the following recipe to trace it:

    python3 -m trace --trace $(which datasette) fixtures.db -p 8002

html:
<p>Today I learned how to use the Python <a href="https://docs.python.org/3/library/trace.html" rel="nofollow">trace module</a> to output every single executed line of Python code in a program - useful for figuring out exactly when a crash or infinite loop happens.</p>
<p>The basic format is to run:</p>
<pre><code>python3 -m trace --trace myscript.py
</code></pre>
<p>This will execute the script and print out every single line of code as it executes - which can be a LOT of output. It slows the program down to a crawl - just starting up Datasette took probably over a minute and churned through hundreds of thousands of lines of console output.</p>
<p>Since Datasette is a command-line application, I needed to use the following recipe to trace it:</p>
<pre><code>python3 -m trace --trace $(which datasette) fixtures.db -p 8002
</code></pre>


path: docker_debian-unstable-packages.md
topic: docker
title: Installing packages from Debian unstable in a Docker image based on stable
url: https://github.com/simonw/til/blob/main/docker/debian-unstable-packages.md
created: 2021-03-22T14:42:43-07:00
created_utc: 2021-03-22T21:42:43+00:00
updated: 2021-03-22T14:42:43-07:00
updated_utc: 2021-03-22T21:42:43+00:00
shot_hash: 1fb78a85967e7d8fb6cab2c7bb53b67f
slug: debian-unstable-packages
body:
For [Datasette #1249](https://github.com/simonw/datasette/issues/1249) I wanted to build a Docker image from the `python:3.9.2-slim-buster` base image ("buster" is the current stable release of Debian) but include a single package from "sid", the unstable Debian distribution.

I needed to do this because the latest version of SpatiaLite, version 5, was available in `sid` but not in `buster` (which only has 4.3.0a):

https://packages.debian.org/search?keywords=spatialite

<img width="923" alt="Package libsqlite3-mod-spatialite&#13;&#13;stretch (oldstable) (libs): Geospatial extension for SQLite - loadable module&#13;    4.3.0a-5+b1: amd64 arm64 armel armhf i386 mips mips64el mipsel ppc64el s390x&#13;    buster (stable) (libs): Geospatial extension for SQLite - loadable module&#13;    4.3.0a-5+b2: amd64 arm64 armel armhf i386 mips mips64el mipsel ppc64el s390x&#13;    bullseye (testing) (libs): Geospatial extension for SQLite - loadable module&#13;    5.0.1-2: amd64 arm64 armel armhf i386 mips64el mipsel ppc64el s390x&#13;    sid (unstable) (libs): Geospatial extension for SQLite - loadable module&#13;    5.0.1-2: alpha amd64 arm64 armel armhf hppa i386 m68k mips64el mipsel ppc64 ppc64el riscv64 s390x sh4 sparc64 x32&#13;    experimental (libs): Geospatial extension for SQLite - loadable module&#13;    5.0.0~beta0-1~exp2 [debports]: powerpcspe" src="https://user-images.githubusercontent.com/9599/112061886-5cf77b00-8b1c-11eb-8f4c-91dce388dc33.png">

The recipe that ended up working for me was to install `software-properties-common` to get the `apt-get-repository` command, then use that to install a package from `sid`:

```dockerfile
RUN apt-get update && \
    apt-get -y --no-install-recommends install software-properties-common && \
    add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \
    apt-get update && \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite
```

Here's the full Dockerfile I used:

```dockerfile
FROM python:3.9.2-slim-buster as build

# software-properties-common provides add-apt-repository
RUN apt-get update && \
    apt-get -y --no-install-recommends install software-properties-common && \
    add-apt-repository "deb http://httpredir.debian.org/debian sid main" && \
    apt-get update && \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite && \
    apt clean && \
    rm -rf /var/lib/apt && \
    rm -rf /var/lib/dpkg

RUN pip install datasette && \
    find /usr/local/lib -name '__pycache__' | xargs rm -r && \
    rm -rf /root/.cache/pip

EXPOSE 8001
CMD ["datasette"]
```

html:
<p>For <a href="https://github.com/simonw/datasette/issues/1249">Datasette #1249</a> I wanted to build a Docker image from the <code>python:3.9.2-slim-buster</code> base image ("buster" is the current stable release of Debian) but include a single package from "sid", the unstable Debian distribution.</p>
<p>I needed to do this because the latest version of SpatiaLite, version 5, was available in <code>sid</code> but not in <code>buster</code> (which only has 4.3.0a):</p>
<p><a href="https://packages.debian.org/search?keywords=spatialite" rel="nofollow">https://packages.debian.org/search?keywords=spatialite</a></p>
<p><a href="https://user-images.githubusercontent.com/9599/112061886-5cf77b00-8b1c-11eb-8f4c-91dce388dc33.png" target="_blank" rel="nofollow"><img width="923" alt="Package libsqlite3-mod-spatialite

stretch (oldstable) (libs): Geospatial extension for SQLite - loadable module
    4.3.0a-5+b1: amd64 arm64 armel armhf i386 mips mips64el mipsel ppc64el s390x
    buster (stable) (libs): Geospatial extension for SQLite - loadable module
    4.3.0a-5+b2: amd64 arm64 armel armhf i386 mips mips64el mipsel ppc64el s390x
    bullseye (testing) (libs): Geospatial extension for SQLite - loadable module
    5.0.1-2: amd64 arm64 armel armhf i386 mips64el mipsel ppc64el s390x
    sid (unstable) (libs): Geospatial extension for SQLite - loadable module
    5.0.1-2: alpha amd64 arm64 armel armhf hppa i386 m68k mips64el mipsel ppc64 ppc64el riscv64 s390x sh4 sparc64 x32
    experimental (libs): Geospatial extension for SQLite - loadable module
    5.0.0~beta0-1~exp2 [debports]: powerpcspe" src="https://user-images.githubusercontent.com/9599/112061886-5cf77b00-8b1c-11eb-8f4c-91dce388dc33.png" style="max-width:100%;"></a></p>
<p>The recipe that ended up working for me was to install <code>software-properties-common</code> to get the <code>apt-get-repository</code> command, then use that to install a package from <code>sid</code>:</p>
<div class="highlight highlight-source-dockerfile"><pre><span class="pl-k">RUN</span> apt-get update &amp;&amp; \
    apt-get -y --no-install-recommends install software-properties-common &amp;&amp; \
    add-apt-repository <span class="pl-s">"deb http://httpredir.debian.org/debian sid main"</span> &amp;&amp; \
    apt-get update &amp;&amp; \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite</pre></div>
<p>Here's the full Dockerfile I used:</p>
<div class="highlight highlight-source-dockerfile"><pre><span class="pl-k">FROM</span> python:3.9.2-slim-buster as build

<span class="pl-c"><span class="pl-c">#</span> software-properties-common provides add-apt-repository</span>
<span class="pl-k">RUN</span> apt-get update &amp;&amp; \
    apt-get -y --no-install-recommends install software-properties-common &amp;&amp; \
    add-apt-repository <span class="pl-s">"deb http://httpredir.debian.org/debian sid main"</span> &amp;&amp; \
    apt-get update &amp;&amp; \
    apt-get -t sid install -y --no-install-recommends libsqlite3-mod-spatialite &amp;&amp; \
    apt clean &amp;&amp; \
    rm -rf /var/lib/apt &amp;&amp; \
    rm -rf /var/lib/dpkg

<span class="pl-k">RUN</span> pip install datasette &amp;&amp; \
    find /usr/local/lib -name <span class="pl-s">'__pycache__'</span> | xargs rm -r &amp;&amp; \
    rm -rf /root/.cache/pip

<span class="pl-k">EXPOSE</span> 8001
<span class="pl-k">CMD</span> [<span class="pl-s">"datasette"</span>]</pre></div>


path: postgresql_closest-locations-to-a-point.md
topic: postgresql
title: Closest locations to a point
url: https://github.com/simonw/til/blob/main/postgresql/closest-locations-to-a-point.md
created: 2021-03-22T22:19:31-07:00
created_utc: 2021-03-23T05:19:31+00:00
updated: 2021-03-23T13:00:03-07:00
updated_utc: 2021-03-23T20:00:03+00:00
shot_hash: aa428c2a3155ce1fb90a0f1b8c598ba0
slug: closest-locations-to-a-point
body:
Here's a PostgreSQL SQL query that returns the closest locations to a point, based on a brute-force approach where the database calculates the distance (in miles) to every single row and then sorts by that distance.

It's adapted from [this StackOverflow answer](https://gis.stackexchange.com/a/31629), which helpfully points out that if you want kilometers rather than miles you can swap the `3959` constant for `6371` instead.

There are much more efficient ways to do this if you are using PostGIS, described in this [Nearest-Neighbour Searching](https://postgis.net/workshops/postgis-intro/knn.html) tutorial - but if you're not using PostGIS this works pretty well.

I ran this against a table with over 9,000 rows and got results back in less than 20ms.

```sql
with locations_with_distance as (
  select
    *,
    (
      acos (
        cos (
          radians(%(latitude)s::float)
        ) * cos(
          radians(latitude)
        ) * cos(
          radians(longitude) - radians(%(longitude)s::float)
        ) + sin(
          radians(%(latitude)s::float)
        ) * sin(radians(latitude))
      ) * 3959
    ) as distance_miles
  from
    location
)
select
  *
from
  locations_with_distance
order by
  distance_miles
limit
  20
```
The `%(latitude)s` and `%(longitude)s` bits are named parameters when working with the Python [psycopg2](https://pypi.org/project/psycopg2/) library - they also work with [django-sql-dashboard](https://pypi.org/project/django-sql-dashboard/) which I used to prototype this query.

## Translated to Django

Here's that same formula using the Django ORM:

```python
from django.db.models import F
from django.db.models.functions import ACos, Cos, Radians, Sin

locations = Location.objects.annotate(
    distance_miles = ACos(
        Cos(
            Radians(input_latitude)
        ) * Cos(
            Radians(F('latitude'))
        ) * Cos(
            Radians(F('longitude')) - Radians(input_longitude)
        ) + Sin(
            Radians(input_latitude)
        ) * Sin(Radians(F('latitude')))
    ) * 3959
).order_by('distance_miles')[:10]
```

html:
<p>Here's a PostgreSQL SQL query that returns the closest locations to a point, based on a brute-force approach where the database calculates the distance (in miles) to every single row and then sorts by that distance.</p>
<p>It's adapted from <a href="https://gis.stackexchange.com/a/31629" rel="nofollow">this StackOverflow answer</a>, which helpfully points out that if you want kilometers rather than miles you can swap the <code>3959</code> constant for <code>6371</code> instead.</p>
<p>There are much more efficient ways to do this if you are using PostGIS, described in this <a href="https://postgis.net/workshops/postgis-intro/knn.html" rel="nofollow">Nearest-Neighbour Searching</a> tutorial - but if you're not using PostGIS this works pretty well.</p>
<p>I ran this against a table with over 9,000 rows and got results back in less than 20ms.</p>
<div class="highlight highlight-source-sql"><pre>with locations_with_distance <span class="pl-k">as</span> (
  <span class="pl-k">select</span>
    <span class="pl-k">*</span>,
    (
      acos (
        cos (
          radians(%(latitude)s::float)
        ) <span class="pl-k">*</span> cos(
          radians(latitude)
        ) <span class="pl-k">*</span> cos(
          radians(longitude) <span class="pl-k">-</span> radians(%(longitude)s::float)
        ) <span class="pl-k">+</span> sin(
          radians(%(latitude)s::float)
        ) <span class="pl-k">*</span> sin(radians(latitude))
      ) <span class="pl-k">*</span> <span class="pl-c1">3959</span>
    ) <span class="pl-k">as</span> distance_miles
  <span class="pl-k">from</span>
    location
)
<span class="pl-k">select</span>
  <span class="pl-k">*</span>
<span class="pl-k">from</span>
  locations_with_distance
<span class="pl-k">order by</span>
  distance_miles
<span class="pl-k">limit</span>
  <span class="pl-c1">20</span></pre></div>
<p>The <code>%(latitude)s</code> and <code>%(longitude)s</code> bits are named parameters when working with the Python <a href="https://pypi.org/project/psycopg2/" rel="nofollow">psycopg2</a> library - they also work with <a href="https://pypi.org/project/django-sql-dashboard/" rel="nofollow">django-sql-dashboard</a> which I used to prototype this query.</p>
<h2>
<a id="user-content-translated-to-django" class="anchor" href="#translated-to-django" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Translated to Django</h2>
<p>Here's that same formula using the Django ORM:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">db</span>.<span class="pl-s1">models</span> <span class="pl-k">import</span> <span class="pl-v">F</span>
<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">db</span>.<span class="pl-s1">models</span>.<span class="pl-s1">functions</span> <span class="pl-k">import</span> <span class="pl-v">ACos</span>, <span class="pl-v">Cos</span>, <span class="pl-v">Radians</span>, <span class="pl-v">Sin</span>

<span class="pl-s1">locations</span> <span class="pl-c1">=</span> <span class="pl-v">Location</span>.<span class="pl-s1">objects</span>.<span class="pl-en">annotate</span>(
    <span class="pl-s1">distance_miles</span> <span class="pl-c1">=</span> <span class="pl-v">ACos</span>(
        <span class="pl-v">Cos</span>(
            <span class="pl-v">Radians</span>(<span class="pl-s1">input_latitude</span>)
        ) <span class="pl-c1">*</span> <span class="pl-v">Cos</span>(
            <span class="pl-v">Radians</span>(<span class="pl-v">F</span>(<span class="pl-s">'latitude'</span>))
        ) <span class="pl-c1">*</span> <span class="pl-v">Cos</span>(
            <span class="pl-v">Radians</span>(<span class="pl-v">F</span>(<span class="pl-s">'longitude'</span>)) <span class="pl-c1">-</span> <span class="pl-v">Radians</span>(<span class="pl-s1">input_longitude</span>)
        ) <span class="pl-c1">+</span> <span class="pl-v">Sin</span>(
            <span class="pl-v">Radians</span>(<span class="pl-s1">input_latitude</span>)
        ) <span class="pl-c1">*</span> <span class="pl-v">Sin</span>(<span class="pl-v">Radians</span>(<span class="pl-v">F</span>(<span class="pl-s">'latitude'</span>)))
    ) <span class="pl-c1">*</span> <span class="pl-c1">3959</span>
).<span class="pl-en">order_by</span>(<span class="pl-s">'distance_miles'</span>)[:<span class="pl-c1">10</span>]</pre></div>


path: zeit-now_redirecting-all-paths-on-vercel.md
topic: zeit-now
title: Redirecting all paths on a Vercel instance
url: https://github.com/simonw/til/blob/main/zeit-now/redirecting-all-paths-on-vercel.md
created: 2021-03-27T08:45:13-07:00
created_utc: 2021-03-27T15:45:13+00:00
updated: 2021-03-27T08:45:13-07:00
updated_utc: 2021-03-27T15:45:13+00:00
shot_hash: 2d0e84cbd4dda00e05f7457fc214b013
slug: redirecting-all-paths-on-vercel
body:
I wanted to redirect all traffic to `https://something.vercel.app/` to a different domain - preserving the path and the querystring and serving a 301 status code.

The [Vercel redirects](https://vercel.com/docs/configuration#project/redirects) documentation doesn't mention that you can capture patterns from the source and place them in the destination. The example they give is:

```json
{
  "redirects": [
    { "source": "/me", "destination": "/profile.html" },
    { "source": "/user", "destination": "/api/user", "permanent": false },
    { "source": "/view-source", "destination": "https://github.com/vercel/vercel" }
  ]
}
```
It turns out you can use `(.*)` in the source and then `$1` in the destination. Here's the example that worked for me:
```json
{
    "redirects": [
        {
            "source": "/(.*)",
            "destination": "https://timezones-api.datasette.io/$1",
            "statusCode": 301
        }
    ]
}
```
Vercel use 307 and 308 HTTP status codes. I prefer 301 and 302 which is why I used `statusCode` above instead of their `"permanent": true`.

Here's how I deployed it:
```
~ % cd /tmp
/tmp % mkdir timezones-api
/tmp % cd timezones-api 
timezones-api % echo '{
    "redirects": [
        {
            "source": "/(.*)",
            "destination": "https://timezones-api.datasette.io/$1",
            "statusCode": 301
        }
    ]
}
' > vercel.json
timezones-api % vercel 
Vercel CLI 19.2.0
> NOTE: Deploying to Now 2.0 automatically. More: https://vercel.com/docs/version-detection
? Set up and deploy “/private/tmp/timezones-api”? [Y/n] y
? Which scope do you want to deploy to? simonw
? Found project “simonw/timezones-api”. Link to it? [Y/n] y
🔗  Linked to simonw/timezones-api (created .vercel and added it to .gitignore)
No framework detected. Default Project Settings:
- Build Command: `npm run vercel-build` or `npm run build`
- Output Directory: `public` if it exists, or `.`
- Development Command: None
? Want to override the settings? [y/N] y
? Which settings would you like to overwrite (select multiple)? None
🔍  Inspect: https://vercel.com/simonw/timezones-api-cn71cufah/simonw [1s]
✅  Production: https://timezones-api-simonw.vercel.app [copied to clipboard] [10s]
📝  Deployed to production. Run `vercel --prod` to overwrite later (https://vercel.link/2F).
💡  To change the domain or build command, go to https://vercel.com/simonw/timezones-api/settings
```

html:
<p>I wanted to redirect all traffic to <code>https://something.vercel.app/</code> to a different domain - preserving the path and the querystring and serving a 301 status code.</p>
<p>The <a href="https://vercel.com/docs/configuration#project/redirects" rel="nofollow">Vercel redirects</a> documentation doesn't mention that you can capture patterns from the source and place them in the destination. The example they give is:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-s"><span class="pl-pds">"</span>redirects<span class="pl-pds">"</span></span>: [
    { <span class="pl-s"><span class="pl-pds">"</span>source<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>/me<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>destination<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>/profile.html<span class="pl-pds">"</span></span> },
    { <span class="pl-s"><span class="pl-pds">"</span>source<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>/user<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>destination<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>/api/user<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>permanent<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span> },
    { <span class="pl-s"><span class="pl-pds">"</span>source<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>/view-source<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>destination<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://github.com/vercel/vercel<span class="pl-pds">"</span></span> }
  ]
}</pre></div>
<p>It turns out you can use <code>(.*)</code> in the source and then <code>$1</code> in the destination. Here's the example that worked for me:</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-s"><span class="pl-pds">"</span>redirects<span class="pl-pds">"</span></span>: [
        {
            <span class="pl-s"><span class="pl-pds">"</span>source<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>/(.*)<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>destination<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://timezones-api.datasette.io/$1<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>statusCode<span class="pl-pds">"</span></span>: <span class="pl-c1">301</span>
        }
    ]
}</pre></div>
<p>Vercel use 307 and 308 HTTP status codes. I prefer 301 and 302 which is why I used <code>statusCode</code> above instead of their <code>"permanent": true</code>.</p>
<p>Here's how I deployed it:</p>
<pre><code>~ % cd /tmp
/tmp % mkdir timezones-api
/tmp % cd timezones-api 
timezones-api % echo '{
    "redirects": [
        {
            "source": "/(.*)",
            "destination": "https://timezones-api.datasette.io/$1",
            "statusCode": 301
        }
    ]
}
' &gt; vercel.json
timezones-api % vercel 
Vercel CLI 19.2.0
&gt; NOTE: Deploying to Now 2.0 automatically. More: https://vercel.com/docs/version-detection
? Set up and deploy “/private/tmp/timezones-api”? [Y/n] y
? Which scope do you want to deploy to? simonw
? Found project “simonw/timezones-api”. Link to it? [Y/n] y
🔗  Linked to simonw/timezones-api (created .vercel and added it to .gitignore)
No framework detected. Default Project Settings:
- Build Command: `npm run vercel-build` or `npm run build`
- Output Directory: `public` if it exists, or `.`
- Development Command: None
? Want to override the settings? [y/N] y
? Which settings would you like to overwrite (select multiple)? None
🔍  Inspect: https://vercel.com/simonw/timezones-api-cn71cufah/simonw [1s]
✅  Production: https://timezones-api-simonw.vercel.app [copied to clipboard] [10s]
📝  Deployed to production. Run `vercel --prod` to overwrite later (https://vercel.link/2F).
💡  To change the domain or build command, go to https://vercel.com/simonw/timezones-api/settings
</code></pre>


path: azure_all-traffic-to-subdomain.md
topic: azure
title: Writing an Azure Function that serves all traffic to a subdomain
url: https://github.com/simonw/til/blob/main/azure/all-traffic-to-subdomain.md
created: 2021-03-27T20:49:56-07:00
created_utc: 2021-03-28T03:49:56+00:00
updated: 2021-03-27T20:49:56-07:00
updated_utc: 2021-03-28T03:49:56+00:00
shot_hash: 70d46c9c26f38e5813d281c2b7c56c1d
slug: all-traffic-to-subdomain
body:
[Azure Functions](https://docs.microsoft.com/en-us/azure/azure-functions/) default to serving traffic from a path like `/api/FunctionName` - for example `https://your-subdomain.azurewebsites.net/api/MyFunction`.

If you want to serve an entire website through a single function (e.g. using [Datasette](https://datasette.io/)) you need that function to we called for any traffic to that subdomain.

Here's how to do that - to capture all traffic to any path under `https://your-subdomain.azurewebsites.net/`.

First add the following section to your `host.json` file:

```
    "extensions": {
        "http": {
            "routePrefix": ""
        }
    }
```
Then add `"route": "{*route}"` to the `function.json` file for the function that you would like to serve all traffic. Mine ended up looking like this:
```json
{
    "scriptFile": "__init__.py",
    "bindings": [
        {
            "authLevel": "Anonymous",
            "type": "httpTrigger",
            "direction": "in",
            "name": "req",
            "route": "{*route}",
            "methods": [
                "get",
                "post"
            ]
        },
        {
            "type": "http",
            "direction": "out",
            "name": "$return"
        }
    ]
}
```
See https://github.com/simonw/azure-functions-datasette for an example that uses this pattern.

html:
<p><a href="https://docs.microsoft.com/en-us/azure/azure-functions/" rel="nofollow">Azure Functions</a> default to serving traffic from a path like <code>/api/FunctionName</code> - for example <code>https://your-subdomain.azurewebsites.net/api/MyFunction</code>.</p>
<p>If you want to serve an entire website through a single function (e.g. using <a href="https://datasette.io/" rel="nofollow">Datasette</a>) you need that function to we called for any traffic to that subdomain.</p>
<p>Here's how to do that - to capture all traffic to any path under <code>https://your-subdomain.azurewebsites.net/</code>.</p>
<p>First add the following section to your <code>host.json</code> file:</p>
<pre><code>    "extensions": {
        "http": {
            "routePrefix": ""
        }
    }
</code></pre>
<p>Then add <code>"route": "{*route}"</code> to the <code>function.json</code> file for the function that you would like to serve all traffic. Mine ended up looking like this:</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-s"><span class="pl-pds">"</span>scriptFile<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>__init__.py<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>bindings<span class="pl-pds">"</span></span>: [
        {
            <span class="pl-s"><span class="pl-pds">"</span>authLevel<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Anonymous<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>httpTrigger<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>direction<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>in<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>req<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>route<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>{*route}<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>methods<span class="pl-pds">"</span></span>: [
                <span class="pl-s"><span class="pl-pds">"</span>get<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>post<span class="pl-pds">"</span></span>
            ]
        },
        {
            <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>http<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>direction<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>out<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>$return<span class="pl-pds">"</span></span>
        }
    ]
}</pre></div>
<p>See <a href="https://github.com/simonw/azure-functions-datasette">https://github.com/simonw/azure-functions-datasette</a> for an example that uses this pattern.</p>


path: markdown_markdown-extensions-python.md
topic: markdown
title: Useful Markdown extensions in Python
url: https://github.com/simonw/til/blob/main/markdown/markdown-extensions-python.md
created: 2021-04-03T21:22:12-07:00
created_utc: 2021-04-04T04:22:12+00:00
updated: 2021-04-03T21:22:12-07:00
updated_utc: 2021-04-04T04:22:12+00:00
shot_hash: 6aa25750583c4adebedf824a2ea0dc15
slug: markdown-extensions-python
body:
I wanted to render some markdown in Python but with the following extra features:

- URLize any bare URLs in the content, like GFM
- Handle `` ```python `` at the start of a block without showing the `python` bit (even if no syntax highlight is applied)
- Show a table of contents

For URLizing the best solution I've found is [r0wb0t/markdown-urlize](https://github.com/r0wb0t/markdown-urlize) - it's not available on PyPI so you have to download the [mdx_urlize.py](https://github.com/r0wb0t/markdown-urlize/blob/master/mdx_urlize.py) file and import from it.

The other two features can be handled by the [toc](https://python-markdown.github.io/extensions/toc/) and [fenced_code](https://python-markdown.github.io/extensions/fenced_code_blocks/) extensions, both of which are included with Python Markdown and can be activated by passing their names as strings to the `extensions=` list.

Here's what I ended up doing (in a Django view):

```python
from mdx_urlize import UrlizeExtension
import markdown
import pathlib

def docs(request):
    content = (pathlib.Path(__file__).parent / "docs.md").read_text()
    md = markdown.Markdown(
        extensions=["toc", "fenced_code", UrlizeExtension()], output_format="html5"
    )
    html = md.convert(content)
    return render(
        request,
        "docs.html",
        {
            "content": html,
            "toc": md.toc,
        },
    )
```
And in the `docs.html` template:

```html+django
{% extends "base.html" %}

{% block title %}Documentation{% endblock %}

{% block content %}
<h1>Documentation</h1>
{{ toc|safe }}
{{ content|safe }}
{% endblock %}
```

html:
<p>I wanted to render some markdown in Python but with the following extra features:</p>
<ul>
<li>URLize any bare URLs in the content, like GFM</li>
<li>Handle <code>```python</code> at the start of a block without showing the <code>python</code> bit (even if no syntax highlight is applied)</li>
<li>Show a table of contents</li>
</ul>
<p>For URLizing the best solution I've found is <a href="https://github.com/r0wb0t/markdown-urlize">r0wb0t/markdown-urlize</a> - it's not available on PyPI so you have to download the <a href="https://github.com/r0wb0t/markdown-urlize/blob/master/mdx_urlize.py">mdx_urlize.py</a> file and import from it.</p>
<p>The other two features can be handled by the <a href="https://python-markdown.github.io/extensions/toc/" rel="nofollow">toc</a> and <a href="https://python-markdown.github.io/extensions/fenced_code_blocks/" rel="nofollow">fenced_code</a> extensions, both of which are included with Python Markdown and can be activated by passing their names as strings to the <code>extensions=</code> list.</p>
<p>Here's what I ended up doing (in a Django view):</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">mdx_urlize</span> <span class="pl-k">import</span> <span class="pl-v">UrlizeExtension</span>
<span class="pl-k">import</span> <span class="pl-s1">markdown</span>
<span class="pl-k">import</span> <span class="pl-s1">pathlib</span>

<span class="pl-k">def</span> <span class="pl-en">docs</span>(<span class="pl-s1">request</span>):
    <span class="pl-s1">content</span> <span class="pl-c1">=</span> (<span class="pl-s1">pathlib</span>.<span class="pl-v">Path</span>(<span class="pl-s1">__file__</span>).<span class="pl-s1">parent</span> <span class="pl-c1">/</span> <span class="pl-s">"docs.md"</span>).<span class="pl-en">read_text</span>()
    <span class="pl-s1">md</span> <span class="pl-c1">=</span> <span class="pl-s1">markdown</span>.<span class="pl-v">Markdown</span>(
        <span class="pl-s1">extensions</span><span class="pl-c1">=</span>[<span class="pl-s">"toc"</span>, <span class="pl-s">"fenced_code"</span>, <span class="pl-v">UrlizeExtension</span>()], <span class="pl-s1">output_format</span><span class="pl-c1">=</span><span class="pl-s">"html5"</span>
    )
    <span class="pl-s1">html</span> <span class="pl-c1">=</span> <span class="pl-s1">md</span>.<span class="pl-en">convert</span>(<span class="pl-s1">content</span>)
    <span class="pl-k">return</span> <span class="pl-en">render</span>(
        <span class="pl-s1">request</span>,
        <span class="pl-s">"docs.html"</span>,
        {
            <span class="pl-s">"content"</span>: <span class="pl-s1">html</span>,
            <span class="pl-s">"toc"</span>: <span class="pl-s1">md</span>.<span class="pl-s1">toc</span>,
        },
    )</pre></div>
<p>And in the <code>docs.html</code> template:</p>
<div class="highlight highlight-text-html-django"><pre><span class="pl-e">{%</span> <span class="pl-k">extends</span> <span class="pl-s">"base.html"</span> <span class="pl-e">%}</span>

<span class="pl-e">{%</span> <span class="pl-k">block</span> <span class="pl-s">title</span> <span class="pl-e">%}</span>Documentation<span class="pl-e">{%</span> <span class="pl-k">endblock</span> <span class="pl-e">%}</span>

<span class="pl-e">{%</span> <span class="pl-k">block</span> <span class="pl-s">content</span> <span class="pl-e">%}</span>
&lt;<span class="pl-ent">h1</span>&gt;Documentation&lt;/<span class="pl-ent">h1</span>&gt;
{{ toc|safe }}
{{ content|safe }}
<span class="pl-e">{%</span> <span class="pl-k">endblock</span> <span class="pl-e">%}</span></pre></div>


path: vscode_language-specific-indentation-settings.md
topic: vscode
title: Language-specific indentation settings in VS Code
url: https://github.com/simonw/til/blob/main/vscode/language-specific-indentation-settings.md
created: 2021-04-04T10:25:50-07:00
created_utc: 2021-04-04T17:25:50+00:00
updated: 2021-04-04T10:27:56-07:00
updated_utc: 2021-04-04T17:27:56+00:00
shot_hash: abc771108e9fae516b4b80024e0314e4
slug: language-specific-indentation-settings
body:
When I'm working with Python I like four space indents, but for JavaScript or HTML I like two space indents.

Today I figured out how to teach VS Code those defaults.

1. Hit `Shift+Command+P` to bring up the action menu
2. Search for the `Preferences: Configure Language Specific Settings...` item
3. Select the language, e.g. `HTML` or `Python`
4. Add `"editor.tabSize": 2` to the corresponding JSON object
5. Save that file

I ended up with the following in my JSON (plus a bunch of other stuff I've edited out):

```json
{
    "[html]": {
        "editor.tabSize": 2,
        "editor.defaultFormatter": "esbenp.prettier-vscode"
    },
    "[javascript]": {
        "editor.defaultFormatter": "esbenp.prettier-vscode",
        "editor.tabSize": 2
    },
    "[typescript]": {
        "editor.defaultFormatter": "esbenp.prettier-vscode",
        "editor.tabSize": 2
    },
    "[python]": {
        "editor.tabSize": 4,
        "editor.wordBasedSuggestions": false
    }
}
```
This file is stored on my machine at `~/Library/Application Support/Code/User/settings.json`.

html:
<p>When I'm working with Python I like four space indents, but for JavaScript or HTML I like two space indents.</p>
<p>Today I figured out how to teach VS Code those defaults.</p>
<ol>
<li>Hit <code>Shift+Command+P</code> to bring up the action menu</li>
<li>Search for the <code>Preferences: Configure Language Specific Settings...</code> item</li>
<li>Select the language, e.g. <code>HTML</code> or <code>Python</code>
</li>
<li>Add <code>"editor.tabSize": 2</code> to the corresponding JSON object</li>
<li>Save that file</li>
</ol>
<p>I ended up with the following in my JSON (plus a bunch of other stuff I've edited out):</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-s"><span class="pl-pds">"</span>[html]<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>editor.tabSize<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>,
        <span class="pl-s"><span class="pl-pds">"</span>editor.defaultFormatter<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>esbenp.prettier-vscode<span class="pl-pds">"</span></span>
    },
    <span class="pl-s"><span class="pl-pds">"</span>[javascript]<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>editor.defaultFormatter<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>esbenp.prettier-vscode<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>editor.tabSize<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
    },
    <span class="pl-s"><span class="pl-pds">"</span>[typescript]<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>editor.defaultFormatter<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>esbenp.prettier-vscode<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>editor.tabSize<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
    },
    <span class="pl-s"><span class="pl-pds">"</span>[python]<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>editor.tabSize<span class="pl-pds">"</span></span>: <span class="pl-c1">4</span>,
        <span class="pl-s"><span class="pl-pds">"</span>editor.wordBasedSuggestions<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>
    }
}</pre></div>
<p>This file is stored on my machine at <code>~/Library/Application Support/Code/User/settings.json</code>.</p>


path: django_efficient-bulk-deletions-in-django.md
topic: django
title: Efficient bulk deletions in Django
url: https://github.com/simonw/til/blob/main/django/efficient-bulk-deletions-in-django.md
created: 2021-04-09T10:58:37-07:00
created_utc: 2021-04-09T17:58:37+00:00
updated: 2022-03-20T21:56:38-07:00
updated_utc: 2022-03-21T04:56:38+00:00
shot_hash: 0b7d65f4eb063315a8e8369790c1f432
slug: efficient-bulk-deletions-in-django
body:
I needed to bulk-delete a large number of objects today. Django deletions are relatively inefficient by default, because Django implements its own version of cascading deletions and fires signals for each deleted object.

I knew that I wanted to avoid both of these and run a bulk `DELETE` SQL operation.

Django has an undocumented `queryset._raw_delete(db_connection)` method that can do this:

```python
reports_qs = Report.objects.filter(public_id__in=report_ids)
reports_qs._raw_delete(reports_qs.db)
```
But this failed for me, because my `Report` object has a many-to-many relationship with another table - and those records were not deleted.

I could have hand-crafted a PostgreSQL cascading delete here, but I instead decided to manually delete those many-to-many records first. Here's what that looked like:

```python
report_availability_tag_qs = (
    Report.availability_tags.through.objects.filter(
        report__public_id__in=report_ids
    )
)
report_availability_tag_qs._raw_delete(report_availability_tag_qs.db)
```
This didn't quite work either, because I have another model `Location` with foreign key references to those reports. So I added this:
```python
Location.objects.filter(latest_report__public_id__in=report_ids).update(
    latest_report=None
)
```
That combination worked! The Django debug toolbar confirmed that this executed one `UPDATE` followed by two efficient bulk `DELETE` operations.

html:
<p>I needed to bulk-delete a large number of objects today. Django deletions are relatively inefficient by default, because Django implements its own version of cascading deletions and fires signals for each deleted object.</p>
<p>I knew that I wanted to avoid both of these and run a bulk <code>DELETE</code> SQL operation.</p>
<p>Django has an undocumented <code>queryset._raw_delete(db_connection)</code> method that can do this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">reports_qs</span> <span class="pl-c1">=</span> <span class="pl-v">Report</span>.<span class="pl-s1">objects</span>.<span class="pl-en">filter</span>(<span class="pl-s1">public_id__in</span><span class="pl-c1">=</span><span class="pl-s1">report_ids</span>)
<span class="pl-s1">reports_qs</span>.<span class="pl-en">_raw_delete</span>(<span class="pl-s1">reports_qs</span>.<span class="pl-s1">db</span>)</pre></div>
<p>But this failed for me, because my <code>Report</code> object has a many-to-many relationship with another table - and those records were not deleted.</p>
<p>I could have hand-crafted a PostgreSQL cascading delete here, but I instead decided to manually delete those many-to-many records first. Here's what that looked like:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">report_availability_tag_qs</span> <span class="pl-c1">=</span> (
    <span class="pl-v">Report</span>.<span class="pl-s1">availability_tags</span>.<span class="pl-s1">through</span>.<span class="pl-s1">objects</span>.<span class="pl-en">filter</span>(
        <span class="pl-s1">report__public_id__in</span><span class="pl-c1">=</span><span class="pl-s1">report_ids</span>
    )
)
<span class="pl-s1">report_availability_tag_qs</span>.<span class="pl-en">_raw_delete</span>(<span class="pl-s1">report_availability_tag_qs</span>.<span class="pl-s1">db</span>)</pre></div>
<p>This didn't quite work either, because I have another model <code>Location</code> with foreign key references to those reports. So I added this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-v">Location</span>.<span class="pl-s1">objects</span>.<span class="pl-en">filter</span>(<span class="pl-s1">latest_report__public_id__in</span><span class="pl-c1">=</span><span class="pl-s1">report_ids</span>).<span class="pl-en">update</span>(
    <span class="pl-s1">latest_report</span><span class="pl-c1">=</span><span class="pl-c1">None</span>
)</pre></div>
<p>That combination worked! The Django debug toolbar confirmed that this executed one <code>UPDATE</code> followed by two efficient bulk <code>DELETE</code> operations.</p>


path: postgresql_unnest-csv.md
topic: postgresql
title: Using unnest() to use a comma-separated string as the input to an IN query
url: https://github.com/simonw/til/blob/main/postgresql/unnest-csv.md
created: 2021-04-10T16:31:32-07:00
created_utc: 2021-04-10T23:31:32+00:00
updated: 2021-04-13T05:31:43-07:00
updated_utc: 2021-04-13T12:31:43+00:00
shot_hash: 361385c28223e9da6149ba6a53e02645
slug: unnest-csv
body:
[django-sql-dashboard](https://github.com/simonw/django-sql-dashboard) lets you define a SQL query plus one or more text inputs that the user can provide in order to execute the query.

I wanted the user to provide a comma-separated list of IDs which I would then use as input to a `WHERE column IN ...` query.

I figured out how to do that using the `unnest()` function and `regexp_split_to_array`:

```sql
select * from report where id in (select unnest(regexp_split_to_array(%(ids)s, ',')))
```

The `ids` string passed to this query is split on commas and used for the IN clause.

Here's a simple demo of how `unnest()` works:

```sql
select unnest(regexp_split_to_array('1,2,3', ','))
```
Output:

|unnest|
|------|
|1     |
|2     |
|3     |

html:
<p><a href="https://github.com/simonw/django-sql-dashboard">django-sql-dashboard</a> lets you define a SQL query plus one or more text inputs that the user can provide in order to execute the query.</p>
<p>I wanted the user to provide a comma-separated list of IDs which I would then use as input to a <code>WHERE column IN ...</code> query.</p>
<p>I figured out how to do that using the <code>unnest()</code> function and <code>regexp_split_to_array</code>:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> <span class="pl-k">*</span> <span class="pl-k">from</span> report <span class="pl-k">where</span> id <span class="pl-k">in</span> (<span class="pl-k">select</span> unnest(regexp_split_to_array(%(ids)s, <span class="pl-s"><span class="pl-pds">'</span>,<span class="pl-pds">'</span></span>)))</pre></div>
<p>The <code>ids</code> string passed to this query is split on commas and used for the IN clause.</p>
<p>Here's a simple demo of how <code>unnest()</code> works:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> unnest(regexp_split_to_array(<span class="pl-s"><span class="pl-pds">'</span>1,2,3<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>,<span class="pl-pds">'</span></span>))</pre></div>
<p>Output:</p>
<table>
<thead>
<tr>
<th>unnest</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
</tr>
<tr>
<td>2</td>
</tr>
<tr>
<td>3</td>
</tr>
</tbody>
</table>


path: postgresql_json-extract-path.md
topic: postgresql
title: Using json_extract_path in PostgreSQL
url: https://github.com/simonw/til/blob/main/postgresql/json-extract-path.md
created: 2021-04-13T17:29:22-07:00
created_utc: 2021-04-14T00:29:22+00:00
updated: 2021-04-13T17:29:22-07:00
updated_utc: 2021-04-14T00:29:22+00:00
shot_hash: f745de4f4d904dba7830b6792a9ec055
slug: json-extract-path
body:
The `json_extract_path()` function in PostgreSQL can be used to extract specific items from JSON - but I couldn't find documentation for the path language it uses.

It turns out it's a variadic functions - it takes multiple arguments, so the path you want is split into separate arguments.

I had data that looks like this (from [django-reversion](https://github.com/etianen/django-reversion)) in a column called `serialized_data`:

```json
[
    {
        "model": "core.location",
        "pk": 119,
        "fields": {
            "name": "Vista Community Clinic- The Gary Center, S. Harbour Blvd",
            "full_address": "201 S. Harbor Boulevard, \nLa Habra, CA 90631"
        }
    }
]
```
I wanted just that `full_address` value. Here's how I got it:
```sql
select
  object_id,
  content_type_id,
  json_extract_path(
    serialized_data::json,
    '0',
    'fields',
    'full_address'
  ) as full_address
from
  reversion_version
```
That's a path of `0`, `fields`, `full_address` - note that arrays are accessed by passing a string integer.

The `::json` casting operater is required here because my JSON isn't stored in a PostgreSQL `jsonb` column, it's stored in a regular text column.

Without the `::json` I got the following error:

> function json_extract_path(text, unknown, unknown, unknown) does not exist LINE 7: json_extract_path( ^ HINT: No function matches the given name and argument types. You might need to add explicit type casts.

html:
<p>The <code>json_extract_path()</code> function in PostgreSQL can be used to extract specific items from JSON - but I couldn't find documentation for the path language it uses.</p>
<p>It turns out it's a variadic functions - it takes multiple arguments, so the path you want is split into separate arguments.</p>
<p>I had data that looks like this (from <a href="https://github.com/etianen/django-reversion">django-reversion</a>) in a column called <code>serialized_data</code>:</p>
<div class="highlight highlight-source-json"><pre>[
    {
        <span class="pl-s"><span class="pl-pds">"</span>model<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>core.location<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>pk<span class="pl-pds">"</span></span>: <span class="pl-c1">119</span>,
        <span class="pl-s"><span class="pl-pds">"</span>fields<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Vista Community Clinic- The Gary Center, S. Harbour Blvd<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>full_address<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>201 S. Harbor Boulevard, <span class="pl-cce">\n</span>La Habra, CA 90631<span class="pl-pds">"</span></span>
        }
    }
]</pre></div>
<p>I wanted just that <code>full_address</code> value. Here's how I got it:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  object_id,
  content_type_id,
  json_extract_path(
    serialized_data::json,
    <span class="pl-s"><span class="pl-pds">'</span>0<span class="pl-pds">'</span></span>,
    <span class="pl-s"><span class="pl-pds">'</span>fields<span class="pl-pds">'</span></span>,
    <span class="pl-s"><span class="pl-pds">'</span>full_address<span class="pl-pds">'</span></span>
  ) <span class="pl-k">as</span> full_address
<span class="pl-k">from</span>
  reversion_version</pre></div>
<p>That's a path of <code>0</code>, <code>fields</code>, <code>full_address</code> - note that arrays are accessed by passing a string integer.</p>
<p>The <code>::json</code> casting operater is required here because my JSON isn't stored in a PostgreSQL <code>jsonb</code> column, it's stored in a regular text column.</p>
<p>Without the <code>::json</code> I got the following error:</p>
<blockquote>
<p>function json_extract_path(text, unknown, unknown, unknown) does not exist LINE 7: json_extract_path( ^ HINT: No function matches the given name and argument types. You might need to add explicit type casts.</p>
</blockquote>


path: cloudrun_listing-cloudbuild-files.md
topic: cloudrun
title: Listing files uploaded to Cloud Build
url: https://github.com/simonw/til/blob/main/cloudrun/listing-cloudbuild-files.md
created: 2021-04-14T23:09:37-07:00
created_utc: 2021-04-15T06:09:37+00:00
updated: 2021-04-14T23:09:37-07:00
updated_utc: 2021-04-15T06:09:37+00:00
shot_hash: cbff23daed53586724ec425fc09017bd
slug: listing-cloudbuild-files
body:
Today while running `datasette publish cloudrun ...` I noticed the following:

```
Uploading tarball of [.] to [gs://datasette-222320_cloudbuild/source/1618465936.523167-939ed21aedff4cb8a2c914c099fb48cd.tgz]
```
`gs://` indicates a Google Cloud Storage bucket. Can I see what's in that `datasette-222320_cloudbuild` bucket?

Turns out I can:

```
~ % gsutil ls -l gs://datasette-222320_cloudbuild/source/ | head -n 10
     36929  2019-05-03T13:18:35Z  gs://datasette-222320_cloudbuild/source/1556889512.4-7ffeb30ed7bc4173a8101cc3e7d6e12e.tgz
     36929  2019-05-03T13:20:06Z  gs://datasette-222320_cloudbuild/source/1556889605.56-5a5251a73b9646cca36b9afef8e578fd.tgz
     36928  2019-05-03T13:20:23Z  gs://datasette-222320_cloudbuild/source/1556889623.22-5ccfa45f935e4810ac322c15593233dc.tgz
     36927  2019-05-03T13:21:33Z  gs://datasette-222320_cloudbuild/source/1556889692.37-44759f37332047d9849cfb3773ef5b28.tgz
     36962  2019-05-03T14:01:14Z  gs://datasette-222320_cloudbuild/source/1556892073.6-d99f13f412054e13b4fb36670f454e50.tgz
```
The `-l` option adds the size information.

Mine has 7438 objects in it! I panicked a bit when I saw this at the end:

```
~ % gsutil ls -l gs://datasette-222320_cloudbuild/source/ | tail -n 10
 152553673  2021-04-15T01:41:32Z  gs://datasette-222320_cloudbuild/source/1618450815.99-26109d7f15bc478d999423e993091fd0.tgz
   1283564  2021-04-15T02:23:47Z  gs://datasette-222320_cloudbuild/source/1618453427.2-0e6193003ae14bff8be813f734b038b2.tgz
   1284121  2021-04-15T03:11:09Z  gs://datasette-222320_cloudbuild/source/1618456268.44-11595af453a74c9fb122b818e56d152e.tgz
  18660297  2021-04-15T03:37:24Z  gs://datasette-222320_cloudbuild/source/1618457837.52-71dfc8e6527042c6ba7b25afe91d006c.tgz
   1283482  2021-04-15T04:10:28Z  gs://datasette-222320_cloudbuild/source/1618459828.02-db9803983d024e7da2593a8db4c87b65.tgz
   3654810  2021-04-15T04:39:26Z  gs://datasette-222320_cloudbuild/source/1618461564.31-a9cff151b6bd4baba4ce68972bef4549.tgz
   1283746  2021-04-15T05:11:01Z  gs://datasette-222320_cloudbuild/source/1618463460.43-b65504ef3f9243a4acd2e07f6c7e9f63.tgz
 152748131  2021-04-15T05:19:52Z  gs://datasette-222320_cloudbuild/source/1618463900.76-1b829400ac3644b69dc5554d56dc3eab.tgz
 845871549  2021-04-15T06:05:49Z  gs://datasette-222320_cloudbuild/source/1618465936.523167-939ed21aedff4cb8a2c914c099fb48cd.tgz
TOTAL: 7438 objects, 273755828462 bytes (254.95 GiB)
```
But... since storage costs $0.020 per GB per month that quarter of a TB is costing me only $5/month. Phew!

html:
<p>Today while running <code>datasette publish cloudrun ...</code> I noticed the following:</p>
<pre><code>Uploading tarball of [.] to [gs://datasette-222320_cloudbuild/source/1618465936.523167-939ed21aedff4cb8a2c914c099fb48cd.tgz]
</code></pre>
<p><code>gs://</code> indicates a Google Cloud Storage bucket. Can I see what's in that <code>datasette-222320_cloudbuild</code> bucket?</p>
<p>Turns out I can:</p>
<pre><code>~ % gsutil ls -l gs://datasette-222320_cloudbuild/source/ | head -n 10
     36929  2019-05-03T13:18:35Z  gs://datasette-222320_cloudbuild/source/1556889512.4-7ffeb30ed7bc4173a8101cc3e7d6e12e.tgz
     36929  2019-05-03T13:20:06Z  gs://datasette-222320_cloudbuild/source/1556889605.56-5a5251a73b9646cca36b9afef8e578fd.tgz
     36928  2019-05-03T13:20:23Z  gs://datasette-222320_cloudbuild/source/1556889623.22-5ccfa45f935e4810ac322c15593233dc.tgz
     36927  2019-05-03T13:21:33Z  gs://datasette-222320_cloudbuild/source/1556889692.37-44759f37332047d9849cfb3773ef5b28.tgz
     36962  2019-05-03T14:01:14Z  gs://datasette-222320_cloudbuild/source/1556892073.6-d99f13f412054e13b4fb36670f454e50.tgz
</code></pre>
<p>The <code>-l</code> option adds the size information.</p>
<p>Mine has 7438 objects in it! I panicked a bit when I saw this at the end:</p>
<pre><code>~ % gsutil ls -l gs://datasette-222320_cloudbuild/source/ | tail -n 10
 152553673  2021-04-15T01:41:32Z  gs://datasette-222320_cloudbuild/source/1618450815.99-26109d7f15bc478d999423e993091fd0.tgz
   1283564  2021-04-15T02:23:47Z  gs://datasette-222320_cloudbuild/source/1618453427.2-0e6193003ae14bff8be813f734b038b2.tgz
   1284121  2021-04-15T03:11:09Z  gs://datasette-222320_cloudbuild/source/1618456268.44-11595af453a74c9fb122b818e56d152e.tgz
  18660297  2021-04-15T03:37:24Z  gs://datasette-222320_cloudbuild/source/1618457837.52-71dfc8e6527042c6ba7b25afe91d006c.tgz
   1283482  2021-04-15T04:10:28Z  gs://datasette-222320_cloudbuild/source/1618459828.02-db9803983d024e7da2593a8db4c87b65.tgz
   3654810  2021-04-15T04:39:26Z  gs://datasette-222320_cloudbuild/source/1618461564.31-a9cff151b6bd4baba4ce68972bef4549.tgz
   1283746  2021-04-15T05:11:01Z  gs://datasette-222320_cloudbuild/source/1618463460.43-b65504ef3f9243a4acd2e07f6c7e9f63.tgz
 152748131  2021-04-15T05:19:52Z  gs://datasette-222320_cloudbuild/source/1618463900.76-1b829400ac3644b69dc5554d56dc3eab.tgz
 845871549  2021-04-15T06:05:49Z  gs://datasette-222320_cloudbuild/source/1618465936.523167-939ed21aedff4cb8a2c914c099fb48cd.tgz
TOTAL: 7438 objects, 273755828462 bytes (254.95 GiB)
</code></pre>
<p>But... since storage costs $0.020 per GB per month that quarter of a TB is costing me only $5/month. Phew!</p>


path: django_migration-postgresql-fuzzystrmatch.md
topic: django
title: Enabling the fuzzystrmatch extension in PostgreSQL with a Django migration
url: https://github.com/simonw/til/blob/main/django/migration-postgresql-fuzzystrmatch.md
created: 2021-04-18T12:32:58-07:00
created_utc: 2021-04-18T19:32:58+00:00
updated: 2021-04-18T12:32:58-07:00
updated_utc: 2021-04-18T19:32:58+00:00
shot_hash: a95d79750b014c729ca6aca5db4665ee
slug: migration-postgresql-fuzzystrmatch
body:
The PostgreSQL [fuzzystrmatch extension](https://www.postgresql.org/docs/13/fuzzystrmatch.html) enables several functions for fuzzy string matching: `soundex()`, `difference()`, `levenshtein()`, `levenshtein_less_equal()`, `metaphone()`, `dmetaphone()` and `dmetaphone_alt()`.

Enabling them for use with Django turns out to be really easy - it just takes a migration that looks something like this:

```python
from django.contrib.postgres.operations import CreateExtension
from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ("core", "0089_importrun_sourcelocation"),
    ]

    operations = [
        CreateExtension(name="fuzzystrmatch"),
    ]
```

html:
<p>The PostgreSQL <a href="https://www.postgresql.org/docs/13/fuzzystrmatch.html" rel="nofollow">fuzzystrmatch extension</a> enables several functions for fuzzy string matching: <code>soundex()</code>, <code>difference()</code>, <code>levenshtein()</code>, <code>levenshtein_less_equal()</code>, <code>metaphone()</code>, <code>dmetaphone()</code> and <code>dmetaphone_alt()</code>.</p>
<p>Enabling them for use with Django turns out to be really easy - it just takes a migration that looks something like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">postgres</span>.<span class="pl-s1">operations</span> <span class="pl-k">import</span> <span class="pl-v">CreateExtension</span>
<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">db</span> <span class="pl-k">import</span> <span class="pl-s1">migrations</span>


<span class="pl-k">class</span> <span class="pl-v">Migration</span>(<span class="pl-s1">migrations</span>.<span class="pl-v">Migration</span>):

    <span class="pl-s1">dependencies</span> <span class="pl-c1">=</span> [
        (<span class="pl-s">"core"</span>, <span class="pl-s">"0089_importrun_sourcelocation"</span>),
    ]

    <span class="pl-s1">operations</span> <span class="pl-c1">=</span> [
        <span class="pl-v">CreateExtension</span>(<span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"fuzzystrmatch"</span>),
    ]</pre></div>


path: django_django-admin-horizontal-scroll.md
topic: django
title: Usable horizontal scrollbars in the Django admin for mouse users
url: https://github.com/simonw/til/blob/main/django/django-admin-horizontal-scroll.md
created: 2021-04-20T12:06:55-07:00
created_utc: 2021-04-20T19:06:55+00:00
updated: 2021-04-21T09:48:50-07:00
updated_utc: 2021-04-21T16:48:50+00:00
shot_hash: a0304657cacaf66cbb241ccaf0671d50
slug: django-admin-horizontal-scroll
body:
I got a complaint from a Windows-with-mouse user of a Django admin project I'm working on: they couldn't see the right hand columns in a table without scrolling horizontally, but since the horizontal scrollbar was only available at the bottom of the page they had to scroll all the way to the bottom first in order to scroll sideways.

As a trackpad user I'm not affected by this, since I can two-finger scroll sideways anywhere on the table.

(I've had the same exact complaint about Datasette in the past, so I'm very interested in solutions).

Matthew Somerville [on Twitter](https://twitter.com/dracos/status/1384391599476641793) suggested setting the maxmimum height of the table to the height of the window, which would cause the horizontal scrollbar to always be available.

Here's the recipe I came up with for doing that for tables in the Django admin:

```html
<script>
function resizeTable() {
  /* So Windows mouse users can see the horizontal scrollbar
     https://github.com/CAVaccineInventory/vial/issues/363 */
  if (window.matchMedia('screen and (min-width: 800px)').matches) {
    let container = document.querySelector("#changelist-form .results");
    let paginator = document.querySelector("p.paginator");
    if (!container || !paginator) {
        return;
    }
    let height =
        window.innerHeight -
        container.getBoundingClientRect().top -
        paginator.getBoundingClientRect().height -
        10;
    container.style.overflowY = "auto";
    container.style.height = height + "px";
  }
}
window.addEventListener("load", resizeTable);
</script>
```
I added the `window.matchMedia()` check when I realized that this approach wasn't useful at mobile screen sizes.

Here `#changelist-form .results` is a `<div>` that wraps the main table on the page, and `p.paginator` is the pagination links shown directly below the table. I decided to set the vertically scrollable height to `window height - top-of-table - paginator height - 10px`.

I added this code to my project's custom `admin/base_site.html` template, which now looks something like this:

```html+django
{% extends "admin/base_site.html" %}

{% block footer %}
<div id="footer"></div>
<script>
/* Script goes here */
</script>
{% endblock %}
```
The end result looks like this:

<img width="1355" alt="Select_county_to_change___VIAL_admin" src="https://user-images.githubusercontent.com/9599/115450508-d4c6cd00-a1d0-11eb-8efd-5561a630337c.png">

html:
<p>I got a complaint from a Windows-with-mouse user of a Django admin project I'm working on: they couldn't see the right hand columns in a table without scrolling horizontally, but since the horizontal scrollbar was only available at the bottom of the page they had to scroll all the way to the bottom first in order to scroll sideways.</p>
<p>As a trackpad user I'm not affected by this, since I can two-finger scroll sideways anywhere on the table.</p>
<p>(I've had the same exact complaint about Datasette in the past, so I'm very interested in solutions).</p>
<p>Matthew Somerville <a href="https://twitter.com/dracos/status/1384391599476641793" rel="nofollow">on Twitter</a> suggested setting the maxmimum height of the table to the height of the window, which would cause the horizontal scrollbar to always be available.</p>
<p>Here's the recipe I came up with for doing that for tables in the Django admin:</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span>
<span class="pl-k">function</span> <span class="pl-en">resizeTable</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
  <span class="pl-c">/* So Windows mouse users can see the horizontal scrollbar</span>
<span class="pl-c">     https://github.com/CAVaccineInventory/vial/issues/363 */</span>
  <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-smi">window</span><span class="pl-kos">.</span><span class="pl-en">matchMedia</span><span class="pl-kos">(</span><span class="pl-s">'screen and (min-width: 800px)'</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-c1">matches</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-k">let</span> <span class="pl-s1">container</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">querySelector</span><span class="pl-kos">(</span><span class="pl-s">"#changelist-form .results"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-k">let</span> <span class="pl-s1">paginator</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">querySelector</span><span class="pl-kos">(</span><span class="pl-s">"p.paginator"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-c1">!</span><span class="pl-s1">container</span> <span class="pl-c1">||</span> <span class="pl-c1">!</span><span class="pl-s1">paginator</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-k">return</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span>
    <span class="pl-k">let</span> <span class="pl-s1">height</span> <span class="pl-c1">=</span>
        <span class="pl-smi">window</span><span class="pl-kos">.</span><span class="pl-c1">innerHeight</span> <span class="pl-c1">-</span>
        <span class="pl-s1">container</span><span class="pl-kos">.</span><span class="pl-en">getBoundingClientRect</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-c1">top</span> <span class="pl-c1">-</span>
        <span class="pl-s1">paginator</span><span class="pl-kos">.</span><span class="pl-en">getBoundingClientRect</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-c1">height</span> <span class="pl-c1">-</span>
        <span class="pl-c1">10</span><span class="pl-kos">;</span>
    <span class="pl-s1">container</span><span class="pl-kos">.</span><span class="pl-c1">style</span><span class="pl-kos">.</span><span class="pl-c1">overflowY</span> <span class="pl-c1">=</span> <span class="pl-s">"auto"</span><span class="pl-kos">;</span>
    <span class="pl-s1">container</span><span class="pl-kos">.</span><span class="pl-c1">style</span><span class="pl-kos">.</span><span class="pl-c1">height</span> <span class="pl-c1">=</span> <span class="pl-s1">height</span> <span class="pl-c1">+</span> <span class="pl-s">"px"</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span>
<span class="pl-kos">}</span>
<span class="pl-smi">window</span><span class="pl-kos">.</span><span class="pl-en">addEventListener</span><span class="pl-kos">(</span><span class="pl-s">"load"</span><span class="pl-kos">,</span> <span class="pl-s1">resizeTable</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">&lt;/</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span></pre></div>
<p>I added the <code>window.matchMedia()</code> check when I realized that this approach wasn't useful at mobile screen sizes.</p>
<p>Here <code>#changelist-form .results</code> is a <code>&lt;div&gt;</code> that wraps the main table on the page, and <code>p.paginator</code> is the pagination links shown directly below the table. I decided to set the vertically scrollable height to <code>window height - top-of-table - paginator height - 10px</code>.</p>
<p>I added this code to my project's custom <code>admin/base_site.html</code> template, which now looks something like this:</p>
<div class="highlight highlight-text-html-django"><pre><span class="pl-e">{%</span> <span class="pl-k">extends</span> <span class="pl-s">"admin/base_site.html"</span> <span class="pl-e">%}</span>

<span class="pl-e">{%</span> <span class="pl-k">block</span> <span class="pl-s">footer</span> <span class="pl-e">%}</span>
&lt;<span class="pl-ent">div</span> <span class="pl-e">id</span>=<span class="pl-s"><span class="pl-pds">"</span>footer<span class="pl-pds">"</span></span>&gt;&lt;/<span class="pl-ent">div</span>&gt;
&lt;<span class="pl-ent">script</span>&gt;<span class="pl-s1"></span>
<span class="pl-s1"><span class="pl-c"><span class="pl-c">/*</span> Script goes here <span class="pl-c">*/</span></span></span>
<span class="pl-s1"></span>&lt;/<span class="pl-ent">script</span>&gt;
<span class="pl-e">{%</span> <span class="pl-k">endblock</span> <span class="pl-e">%}</span></pre></div>
<p>The end result looks like this:</p>
<p><a href="https://user-images.githubusercontent.com/9599/115450508-d4c6cd00-a1d0-11eb-8efd-5561a630337c.png" target="_blank" rel="nofollow"><img width="1355" alt="Select_county_to_change___VIAL_admin" src="https://user-images.githubusercontent.com/9599/115450508-d4c6cd00-a1d0-11eb-8efd-5561a630337c.png" style="max-width:100%;"></a></p>


path: django_filter-by-comma-separated-values.md
topic: django
title: Filter by comma-separated values in the Django admin
url: https://github.com/simonw/til/blob/main/django/filter-by-comma-separated-values.md
created: 2021-04-21T09:31:55-07:00
created_utc: 2021-04-21T16:31:55+00:00
updated: 2021-04-28T16:23:10-07:00
updated_utc: 2021-04-28T23:23:10+00:00
shot_hash: b908bf07608a7730778650f861b6fb75
slug: filter-by-comma-separated-values
body:
I have a text column which contains comma-separated values - inherited from an older database schema.

I should refactor this into a many-to-many field (or maybe even a PostgreSQL array field), but I haven't done that yet. And I wanted to be able to filter by those values in the Django admin.

Since I'm using PostgreSQL, I decided to figure out how to do this using the PostgreSQL `regexp_split_to_array()` function.

There are two necessary SQL queries here: one to figure out all of the unique distinct values that are represented across all of those comma-separated lists, and one to filter for rows that include a specific value.

Here's what I came up with for the first:

```sql
select distinct unnest(
  regexp_split_to_array(my_column, ',\s*')
) from my_table
```
This uses `unnest()`, see [this TIL](https://til.simonwillison.net/postgresql/unnest-csv).

For filtering down to rows that contain a specific value in their comma-separated list, I figured out this:

```sql
select
  *
from
  my_table
where
  array_position(
    regexp_split_to_array(
      my_column, ',\s*'
    ),
    'MyValue'
  ) is not null
```
That second one, translated into the Django ORM, looks like this:
```python
from django.contrib.postgres.fields import ArrayField
from django.db.models import F, IntegerField, TextField, Value
from django.db.models.expressions import Func

queryset.annotate(
    value_array_position=Func(
        Func(
            F(my_column),
            Value(",\\s*"),
            function="regexp_split_to_array",
            output_field=ArrayField(TextField()),
        ),
        Value(my_value),
        function="array_position",
        output_field=IntegerField()
    )
).filter(value_array_position__isnull=False)
```
I didn't bother figuring out the ORM equivalent of that first `unnest()` SQL.

Here's the reusable admin filter factory I came up with using these:

```python
from django.contrib.admin import SimpleListFilter
from django.contrib.postgres.fields import ArrayField
from django.db import connection
from django.db.models import F, TextField, Value
from django.db.models.expressions import Func


def make_csv_filter(filter_title, filter_parameter_name, table, column):
    class CommaSeparatedValuesFilter(SimpleListFilter):
        title = filter_title
        parameter_name = filter_parameter_name

        def lookups(self, request, model_admin):
            sql = """
                select distinct unnest(
                    regexp_split_to_array({}, ',\\s*')
                ) from {}
            """.format(
                column, table
            )
            with connection.cursor() as cursor:
                cursor.execute(sql)
                values = [r[0] for r in cursor.fetchall() if r[0]]
            return zip(values, values)

        def queryset(self, request, queryset):
            value = self.value()
            if not value:
                return queryset
            else:
                return queryset.annotate(
                    value_array_position=Func(
                        Func(
                            F(column),
                            Value(",\\s*"),
                            function="regexp_split_to_array",
                            output_field=ArrayField(TextField()),
                        ),
                        Value(value),
                        function="array_position",
                        output_field=IntegerField()
                    )
                ).filter(value_array_position__isnull=False)

    return CommaSeparatedValuesFilter
```
Then you use it in a `ModelAdmin` subclass like this:
```python
@admin.register(Reporter)
class ReporterAdmin(admin.ModelAdmin):
    list_filter = (
        make_csv_filter(
            filter_title="Roles",
            filter_parameter_name="role",
            table="reporter",
            column="role_names",
        ),
    )
```

html:
<p>I have a text column which contains comma-separated values - inherited from an older database schema.</p>
<p>I should refactor this into a many-to-many field (or maybe even a PostgreSQL array field), but I haven't done that yet. And I wanted to be able to filter by those values in the Django admin.</p>
<p>Since I'm using PostgreSQL, I decided to figure out how to do this using the PostgreSQL <code>regexp_split_to_array()</code> function.</p>
<p>There are two necessary SQL queries here: one to figure out all of the unique distinct values that are represented across all of those comma-separated lists, and one to filter for rows that include a specific value.</p>
<p>Here's what I came up with for the first:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select distinct</span> unnest(
  regexp_split_to_array(my_column, <span class="pl-s"><span class="pl-pds">'</span>,<span class="pl-cce">\s</span>*<span class="pl-pds">'</span></span>)
) <span class="pl-k">from</span> my_table</pre></div>
<p>This uses <code>unnest()</code>, see <a href="https://til.simonwillison.net/postgresql/unnest-csv" rel="nofollow">this TIL</a>.</p>
<p>For filtering down to rows that contain a specific value in their comma-separated list, I figured out this:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  <span class="pl-k">*</span>
<span class="pl-k">from</span>
  my_table
<span class="pl-k">where</span>
  array_position(
    regexp_split_to_array(
      my_column, <span class="pl-s"><span class="pl-pds">'</span>,<span class="pl-cce">\s</span>*<span class="pl-pds">'</span></span>
    ),
    <span class="pl-s"><span class="pl-pds">'</span>MyValue<span class="pl-pds">'</span></span>
  ) <span class="pl-k">is not null</span></pre></div>
<p>That second one, translated into the Django ORM, looks like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">postgres</span>.<span class="pl-s1">fields</span> <span class="pl-k">import</span> <span class="pl-v">ArrayField</span>
<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">db</span>.<span class="pl-s1">models</span> <span class="pl-k">import</span> <span class="pl-v">F</span>, <span class="pl-v">IntegerField</span>, <span class="pl-v">TextField</span>, <span class="pl-v">Value</span>
<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">db</span>.<span class="pl-s1">models</span>.<span class="pl-s1">expressions</span> <span class="pl-k">import</span> <span class="pl-v">Func</span>

<span class="pl-s1">queryset</span>.<span class="pl-en">annotate</span>(
    <span class="pl-s1">value_array_position</span><span class="pl-c1">=</span><span class="pl-v">Func</span>(
        <span class="pl-v">Func</span>(
            <span class="pl-v">F</span>(<span class="pl-s1">my_column</span>),
            <span class="pl-v">Value</span>(<span class="pl-s">",<span class="pl-cce">\\</span>s*"</span>),
            <span class="pl-s1">function</span><span class="pl-c1">=</span><span class="pl-s">"regexp_split_to_array"</span>,
            <span class="pl-s1">output_field</span><span class="pl-c1">=</span><span class="pl-v">ArrayField</span>(<span class="pl-v">TextField</span>()),
        ),
        <span class="pl-v">Value</span>(<span class="pl-s1">my_value</span>),
        <span class="pl-s1">function</span><span class="pl-c1">=</span><span class="pl-s">"array_position"</span>,
        <span class="pl-s1">output_field</span><span class="pl-c1">=</span><span class="pl-v">IntegerField</span>()
    )
).<span class="pl-en">filter</span>(<span class="pl-s1">value_array_position__isnull</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)</pre></div>
<p>I didn't bother figuring out the ORM equivalent of that first <code>unnest()</code> SQL.</p>
<p>Here's the reusable admin filter factory I came up with using these:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">admin</span> <span class="pl-k">import</span> <span class="pl-v">SimpleListFilter</span>
<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">postgres</span>.<span class="pl-s1">fields</span> <span class="pl-k">import</span> <span class="pl-v">ArrayField</span>
<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">db</span> <span class="pl-k">import</span> <span class="pl-s1">connection</span>
<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">db</span>.<span class="pl-s1">models</span> <span class="pl-k">import</span> <span class="pl-v">F</span>, <span class="pl-v">TextField</span>, <span class="pl-v">Value</span>
<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">db</span>.<span class="pl-s1">models</span>.<span class="pl-s1">expressions</span> <span class="pl-k">import</span> <span class="pl-v">Func</span>


<span class="pl-k">def</span> <span class="pl-en">make_csv_filter</span>(<span class="pl-s1">filter_title</span>, <span class="pl-s1">filter_parameter_name</span>, <span class="pl-s1">table</span>, <span class="pl-s1">column</span>):
    <span class="pl-k">class</span> <span class="pl-v">CommaSeparatedValuesFilter</span>(<span class="pl-v">SimpleListFilter</span>):
        <span class="pl-s1">title</span> <span class="pl-c1">=</span> <span class="pl-s1">filter_title</span>
        <span class="pl-s1">parameter_name</span> <span class="pl-c1">=</span> <span class="pl-s1">filter_parameter_name</span>

        <span class="pl-k">def</span> <span class="pl-en">lookups</span>(<span class="pl-s1">self</span>, <span class="pl-s1">request</span>, <span class="pl-s1">model_admin</span>):
            <span class="pl-s1">sql</span> <span class="pl-c1">=</span> <span class="pl-s">"""</span>
<span class="pl-s">                select distinct unnest(</span>
<span class="pl-s">                    regexp_split_to_array({}, ',<span class="pl-cce">\\</span>s*')</span>
<span class="pl-s">                ) from {}</span>
<span class="pl-s">            """</span>.<span class="pl-en">format</span>(
                <span class="pl-s1">column</span>, <span class="pl-s1">table</span>
            )
            <span class="pl-k">with</span> <span class="pl-s1">connection</span>.<span class="pl-en">cursor</span>() <span class="pl-k">as</span> <span class="pl-s1">cursor</span>:
                <span class="pl-s1">cursor</span>.<span class="pl-en">execute</span>(<span class="pl-s1">sql</span>)
                <span class="pl-s1">values</span> <span class="pl-c1">=</span> [<span class="pl-s1">r</span>[<span class="pl-c1">0</span>] <span class="pl-k">for</span> <span class="pl-s1">r</span> <span class="pl-c1">in</span> <span class="pl-s1">cursor</span>.<span class="pl-en">fetchall</span>() <span class="pl-k">if</span> <span class="pl-s1">r</span>[<span class="pl-c1">0</span>]]
            <span class="pl-k">return</span> <span class="pl-en">zip</span>(<span class="pl-s1">values</span>, <span class="pl-s1">values</span>)

        <span class="pl-k">def</span> <span class="pl-en">queryset</span>(<span class="pl-s1">self</span>, <span class="pl-s1">request</span>, <span class="pl-s1">queryset</span>):
            <span class="pl-s1">value</span> <span class="pl-c1">=</span> <span class="pl-s1">self</span>.<span class="pl-en">value</span>()
            <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">value</span>:
                <span class="pl-k">return</span> <span class="pl-s1">queryset</span>
            <span class="pl-k">else</span>:
                <span class="pl-k">return</span> <span class="pl-s1">queryset</span>.<span class="pl-en">annotate</span>(
                    <span class="pl-s1">value_array_position</span><span class="pl-c1">=</span><span class="pl-v">Func</span>(
                        <span class="pl-v">Func</span>(
                            <span class="pl-v">F</span>(<span class="pl-s1">column</span>),
                            <span class="pl-v">Value</span>(<span class="pl-s">",<span class="pl-cce">\\</span>s*"</span>),
                            <span class="pl-s1">function</span><span class="pl-c1">=</span><span class="pl-s">"regexp_split_to_array"</span>,
                            <span class="pl-s1">output_field</span><span class="pl-c1">=</span><span class="pl-v">ArrayField</span>(<span class="pl-v">TextField</span>()),
                        ),
                        <span class="pl-v">Value</span>(<span class="pl-s1">value</span>),
                        <span class="pl-s1">function</span><span class="pl-c1">=</span><span class="pl-s">"array_position"</span>,
                        <span class="pl-s1">output_field</span><span class="pl-c1">=</span><span class="pl-v">IntegerField</span>()
                    )
                ).<span class="pl-en">filter</span>(<span class="pl-s1">value_array_position__isnull</span><span class="pl-c1">=</span><span class="pl-c1">False</span>)

    <span class="pl-k">return</span> <span class="pl-v">CommaSeparatedValuesFilter</span></pre></div>
<p>Then you use it in a <code>ModelAdmin</code> subclass like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">admin</span>.<span class="pl-en">register</span>(<span class="pl-v">Reporter</span>)</span>
<span class="pl-k">class</span> <span class="pl-v">ReporterAdmin</span>(<span class="pl-s1">admin</span>.<span class="pl-v">ModelAdmin</span>):
    <span class="pl-s1">list_filter</span> <span class="pl-c1">=</span> (
        <span class="pl-en">make_csv_filter</span>(
            <span class="pl-s1">filter_title</span><span class="pl-c1">=</span><span class="pl-s">"Roles"</span>,
            <span class="pl-s1">filter_parameter_name</span><span class="pl-c1">=</span><span class="pl-s">"role"</span>,
            <span class="pl-s1">table</span><span class="pl-c1">=</span><span class="pl-s">"reporter"</span>,
            <span class="pl-s1">column</span><span class="pl-c1">=</span><span class="pl-s">"role_names"</span>,
        ),
    )</pre></div>


path: postgresql_constructing-geojson-in-postgresql.md
topic: postgresql
title: Constructing GeoJSON in PostgreSQL
url: https://github.com/simonw/til/blob/main/postgresql/constructing-geojson-in-postgresql.md
created: 2021-04-24T11:55:27-07:00
created_utc: 2021-04-24T18:55:27+00:00
updated: 2021-05-09T19:28:26-07:00
updated_utc: 2021-05-10T02:28:26+00:00
shot_hash: ccd2b3315a1b219561b19fd4edcea187
slug: constructing-geojson-in-postgresql
body:
In order to efficiently generate a GeoJSON representation of a vast number of locations, I'm currently experimenting with generating the GeoJSON directly inside a PostgreSQL SQL query using `json_build_object()` and friends.

Here's my query so far, which illustrates various new patterns I've been learning - including a join against a CTE that extracts a many-to-many table into a format that can be represented in the final output as a JSON array.

```sql
with location_concordances as (
  select location.id, coalesce(
    jsonb_agg(ci.authority || ':' || ci.identifier) filter (
      where ci.authority IS NOT NULL
    ),
    '[]'
  ) as concordances
  from location
  left join concordance_location cl on location.id = cl.location_id
  left join concordance_identifier ci on ci.id = cl.concordanceidentifier_id
  group by location.id
)
SELECT
  json_build_object(
    'type', 'Feature',
    'properties', json_build_object(
      'id',
      location.public_id,
      'name',
      location.name,
      'state', state.abbreviation,
      'latitude', location.latitude,
      'longitude', location.longitude,
      'location_type', location_type.name,
      'import_ref', location.import_ref,
      'phone_number', location.phone_number,
      'full_address', location.full_address,
      'city', location.city,
      'county', county.name,
      'google_places_id', location.google_places_id,
      'vaccinefinder_location_id', location.vaccinefinder_location_id,
      'vaccinespotter_location_id', location.vaccinespotter_location_id,
      'zip_code', location.zip_code,
      'hours', location.hours,
      'website', location.website,
      'preferred_contact_method', location.preferred_contact_method,
      'provider', case 
        when provider.name is null then null
        else jsonb_build_object('name', provider.name, 'type', provider_type.name) 
      end,
      'concordances', location_concordances.concordances
    ),
    'geometry', json_build_object(
      'type', 'Point',
      'coordinates', json_build_array(location.longitude, location.latitude)
    )
  )
  from location
    join state on location.state_id = state.id
    join county on location.county_id = county.id
    join location_type on location.location_type_id = location_type.id
    join location_concordances on location.id = location_concordances.id
    left join provider on location.provider_id = provider.id
    left join provider_type on provider.provider_type_id = provider_type.id
limit 1
```
Example output from this query:
```json
{
    "type": "Feature",
    "properties": {
        "id": "rec00NpJzUnVDpLaQ",
        "name": "Kaiser Permanente Pharmacy #568",
        "state": "CA",
        "latitude": 34.081292,
        "longitude": -117.996576,
        "location_type": "Pharmacy",
        "import_ref": "vca-airtable:rec00NpJzUnVDpLaQ",
        "phone_number": "833-480-4700",
        "full_address": "12761 Schabarum Ave Plaza Level RM 1100, Irwindale, CA 91706",
        "city": null,
        "county": "Los Angeles",
        "google_places_id": "ChIJizKuijvXwoAR4VAXM2Ek4Nc",
        "vaccinefinder_location_id": null,
        "vaccinespotter_location_id": null,
        "zip_code": null,
        "hours": "Monday - Friday: 8:00 AM – 5:00 PM\nSaturday - Sunday: Closed",
        "website": null,
        "preferred_contact_method": null,
        "provider": {
            "name": "Kaiser Permanente",
            "type": "Health Plan"
        },
        "concordances": [
            "google_places:ChIJizKuijvXwoAR4VAXM2Ek4Nc"
        ]
    },
    "geometry": {
        "type": "Point",
        "coordinates": [
            -117.996576,
            34.081292
        ]
    }
}
```
## Using ST_AsGeoJSON

After writing this TIL I found a much quicker option: the `ST_AsGeoJSON()` function. This works if your table has a geometry column on it - in my case I [enhanced my location table](https://simonwillison.net/2021/May/3/adding-geodjango-to-an-existing-django-project/) to include a point column.

The following SQL generates a full GeoJSON feature row for each location in the database:

```sql
select ST_AsGeoJSON(location.*) from location
```
The `location.*` is required - if you try to use `ST_AsGeoJSON(*)` you get an error claiming that "No function matches the given name and argument types".

The output looks something like this:
```json
{
  "type": "Feature",
  "geometry": {
    "type": "Point",
    "coordinates": [
      -117.65499,
      34.10835
    ]
  },
  "properties": {
    "id": 4285,
    "name": "Vons Pharmacy #2681",
    "public_id": "recbECvQlveAyBorV",
    "import_json": {
      "Name": "Vons Pharmacy #2681",
      "Hours": "Monday - Friday: 8:00 AM \u2013 8:00 PM\nSaturday - Sunday: 9:00 AM \u2013 5:00 PM",
      "County": "San Bernardino County",
      "county_id": [
        "reclZ8DWOEuoluStG"
      ]
    }
  }
}
```
That `import_json` section is a PostgreSQL JSON column in the database - note that it gets output as nested data inside the `"properties"` object.

html:
<p>In order to efficiently generate a GeoJSON representation of a vast number of locations, I'm currently experimenting with generating the GeoJSON directly inside a PostgreSQL SQL query using <code>json_build_object()</code> and friends.</p>
<p>Here's my query so far, which illustrates various new patterns I've been learning - including a join against a CTE that extracts a many-to-many table into a format that can be represented in the final output as a JSON array.</p>
<div class="highlight highlight-source-sql"><pre>with location_concordances <span class="pl-k">as</span> (
  <span class="pl-k">select</span> <span class="pl-c1">location</span>.<span class="pl-c1">id</span>, coalesce(
    jsonb_agg(<span class="pl-c1">ci</span>.<span class="pl-c1">authority</span> <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>:<span class="pl-pds">'</span></span> <span class="pl-k">||</span> <span class="pl-c1">ci</span>.<span class="pl-c1">identifier</span>) filter (
      <span class="pl-k">where</span> <span class="pl-c1">ci</span>.<span class="pl-c1">authority</span> <span class="pl-k">IS NOT NULL</span>
    ),
    <span class="pl-s"><span class="pl-pds">'</span>[]<span class="pl-pds">'</span></span>
  ) <span class="pl-k">as</span> concordances
  <span class="pl-k">from</span> location
  <span class="pl-k">left join</span> concordance_location cl <span class="pl-k">on</span> <span class="pl-c1">location</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">cl</span>.<span class="pl-c1">location_id</span>
  <span class="pl-k">left join</span> concordance_identifier ci <span class="pl-k">on</span> <span class="pl-c1">ci</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">cl</span>.<span class="pl-c1">concordanceidentifier_id</span>
  <span class="pl-k">group by</span> <span class="pl-c1">location</span>.<span class="pl-c1">id</span>
)
<span class="pl-k">SELECT</span>
  json_build_object(
    <span class="pl-s"><span class="pl-pds">'</span>type<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>Feature<span class="pl-pds">'</span></span>,
    <span class="pl-s"><span class="pl-pds">'</span>properties<span class="pl-pds">'</span></span>, json_build_object(
      <span class="pl-s"><span class="pl-pds">'</span>id<span class="pl-pds">'</span></span>,
      <span class="pl-c1">location</span>.<span class="pl-c1">public_id</span>,
      <span class="pl-s"><span class="pl-pds">'</span>name<span class="pl-pds">'</span></span>,
      <span class="pl-c1">location</span>.<span class="pl-c1">name</span>,
      <span class="pl-s"><span class="pl-pds">'</span>state<span class="pl-pds">'</span></span>, <span class="pl-c1">state</span>.<span class="pl-c1">abbreviation</span>,
      <span class="pl-s"><span class="pl-pds">'</span>latitude<span class="pl-pds">'</span></span>, <span class="pl-c1">location</span>.<span class="pl-c1">latitude</span>,
      <span class="pl-s"><span class="pl-pds">'</span>longitude<span class="pl-pds">'</span></span>, <span class="pl-c1">location</span>.<span class="pl-c1">longitude</span>,
      <span class="pl-s"><span class="pl-pds">'</span>location_type<span class="pl-pds">'</span></span>, <span class="pl-c1">location_type</span>.<span class="pl-c1">name</span>,
      <span class="pl-s"><span class="pl-pds">'</span>import_ref<span class="pl-pds">'</span></span>, <span class="pl-c1">location</span>.<span class="pl-c1">import_ref</span>,
      <span class="pl-s"><span class="pl-pds">'</span>phone_number<span class="pl-pds">'</span></span>, <span class="pl-c1">location</span>.<span class="pl-c1">phone_number</span>,
      <span class="pl-s"><span class="pl-pds">'</span>full_address<span class="pl-pds">'</span></span>, <span class="pl-c1">location</span>.<span class="pl-c1">full_address</span>,
      <span class="pl-s"><span class="pl-pds">'</span>city<span class="pl-pds">'</span></span>, <span class="pl-c1">location</span>.<span class="pl-c1">city</span>,
      <span class="pl-s"><span class="pl-pds">'</span>county<span class="pl-pds">'</span></span>, <span class="pl-c1">county</span>.<span class="pl-c1">name</span>,
      <span class="pl-s"><span class="pl-pds">'</span>google_places_id<span class="pl-pds">'</span></span>, <span class="pl-c1">location</span>.<span class="pl-c1">google_places_id</span>,
      <span class="pl-s"><span class="pl-pds">'</span>vaccinefinder_location_id<span class="pl-pds">'</span></span>, <span class="pl-c1">location</span>.<span class="pl-c1">vaccinefinder_location_id</span>,
      <span class="pl-s"><span class="pl-pds">'</span>vaccinespotter_location_id<span class="pl-pds">'</span></span>, <span class="pl-c1">location</span>.<span class="pl-c1">vaccinespotter_location_id</span>,
      <span class="pl-s"><span class="pl-pds">'</span>zip_code<span class="pl-pds">'</span></span>, <span class="pl-c1">location</span>.<span class="pl-c1">zip_code</span>,
      <span class="pl-s"><span class="pl-pds">'</span>hours<span class="pl-pds">'</span></span>, <span class="pl-c1">location</span>.<span class="pl-c1">hours</span>,
      <span class="pl-s"><span class="pl-pds">'</span>website<span class="pl-pds">'</span></span>, <span class="pl-c1">location</span>.<span class="pl-c1">website</span>,
      <span class="pl-s"><span class="pl-pds">'</span>preferred_contact_method<span class="pl-pds">'</span></span>, <span class="pl-c1">location</span>.<span class="pl-c1">preferred_contact_method</span>,
      <span class="pl-s"><span class="pl-pds">'</span>provider<span class="pl-pds">'</span></span>, case 
        when <span class="pl-c1">provider</span>.<span class="pl-c1">name</span> is <span class="pl-k">null</span> then <span class="pl-k">null</span>
        else jsonb_build_object(<span class="pl-s"><span class="pl-pds">'</span>name<span class="pl-pds">'</span></span>, <span class="pl-c1">provider</span>.<span class="pl-c1">name</span>, <span class="pl-s"><span class="pl-pds">'</span>type<span class="pl-pds">'</span></span>, <span class="pl-c1">provider_type</span>.<span class="pl-c1">name</span>) 
      end,
      <span class="pl-s"><span class="pl-pds">'</span>concordances<span class="pl-pds">'</span></span>, <span class="pl-c1">location_concordances</span>.<span class="pl-c1">concordances</span>
    ),
    <span class="pl-s"><span class="pl-pds">'</span>geometry<span class="pl-pds">'</span></span>, json_build_object(
      <span class="pl-s"><span class="pl-pds">'</span>type<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>Point<span class="pl-pds">'</span></span>,
      <span class="pl-s"><span class="pl-pds">'</span>coordinates<span class="pl-pds">'</span></span>, json_build_array(<span class="pl-c1">location</span>.<span class="pl-c1">longitude</span>, <span class="pl-c1">location</span>.<span class="pl-c1">latitude</span>)
    )
  )
  <span class="pl-k">from</span> location
    <span class="pl-k">join</span> state <span class="pl-k">on</span> <span class="pl-c1">location</span>.<span class="pl-c1">state_id</span> <span class="pl-k">=</span> <span class="pl-c1">state</span>.<span class="pl-c1">id</span>
    <span class="pl-k">join</span> county <span class="pl-k">on</span> <span class="pl-c1">location</span>.<span class="pl-c1">county_id</span> <span class="pl-k">=</span> <span class="pl-c1">county</span>.<span class="pl-c1">id</span>
    <span class="pl-k">join</span> location_type <span class="pl-k">on</span> <span class="pl-c1">location</span>.<span class="pl-c1">location_type_id</span> <span class="pl-k">=</span> <span class="pl-c1">location_type</span>.<span class="pl-c1">id</span>
    <span class="pl-k">join</span> location_concordances <span class="pl-k">on</span> <span class="pl-c1">location</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">location_concordances</span>.<span class="pl-c1">id</span>
    <span class="pl-k">left join</span> provider <span class="pl-k">on</span> <span class="pl-c1">location</span>.<span class="pl-c1">provider_id</span> <span class="pl-k">=</span> <span class="pl-c1">provider</span>.<span class="pl-c1">id</span>
    <span class="pl-k">left join</span> provider_type <span class="pl-k">on</span> <span class="pl-c1">provider</span>.<span class="pl-c1">provider_type_id</span> <span class="pl-k">=</span> <span class="pl-c1">provider_type</span>.<span class="pl-c1">id</span>
<span class="pl-k">limit</span> <span class="pl-c1">1</span></pre></div>
<p>Example output from this query:</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Feature<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>properties<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>rec00NpJzUnVDpLaQ<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Kaiser Permanente Pharmacy #568<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>state<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>CA<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>latitude<span class="pl-pds">"</span></span>: <span class="pl-c1">34.081292</span>,
        <span class="pl-s"><span class="pl-pds">"</span>longitude<span class="pl-pds">"</span></span>: <span class="pl-c1">-117.996576</span>,
        <span class="pl-s"><span class="pl-pds">"</span>location_type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Pharmacy<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>import_ref<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>vca-airtable:rec00NpJzUnVDpLaQ<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>phone_number<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>833-480-4700<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>full_address<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>12761 Schabarum Ave Plaza Level RM 1100, Irwindale, CA 91706<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>city<span class="pl-pds">"</span></span>: <span class="pl-c1">null</span>,
        <span class="pl-s"><span class="pl-pds">"</span>county<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Los Angeles<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>google_places_id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>ChIJizKuijvXwoAR4VAXM2Ek4Nc<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>vaccinefinder_location_id<span class="pl-pds">"</span></span>: <span class="pl-c1">null</span>,
        <span class="pl-s"><span class="pl-pds">"</span>vaccinespotter_location_id<span class="pl-pds">"</span></span>: <span class="pl-c1">null</span>,
        <span class="pl-s"><span class="pl-pds">"</span>zip_code<span class="pl-pds">"</span></span>: <span class="pl-c1">null</span>,
        <span class="pl-s"><span class="pl-pds">"</span>hours<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Monday - Friday: 8:00 AM – 5:00 PM<span class="pl-cce">\n</span>Saturday - Sunday: Closed<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>website<span class="pl-pds">"</span></span>: <span class="pl-c1">null</span>,
        <span class="pl-s"><span class="pl-pds">"</span>preferred_contact_method<span class="pl-pds">"</span></span>: <span class="pl-c1">null</span>,
        <span class="pl-s"><span class="pl-pds">"</span>provider<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Kaiser Permanente<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Health Plan<span class="pl-pds">"</span></span>
        },
        <span class="pl-s"><span class="pl-pds">"</span>concordances<span class="pl-pds">"</span></span>: [
            <span class="pl-s"><span class="pl-pds">"</span>google_places:ChIJizKuijvXwoAR4VAXM2Ek4Nc<span class="pl-pds">"</span></span>
        ]
    },
    <span class="pl-s"><span class="pl-pds">"</span>geometry<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Point<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>coordinates<span class="pl-pds">"</span></span>: [
            <span class="pl-c1">-117.996576</span>,
            <span class="pl-c1">34.081292</span>
        ]
    }
}</pre></div>
<h2>
<a id="user-content-using-st_asgeojson" class="anchor" href="#using-st_asgeojson" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using ST_AsGeoJSON</h2>
<p>After writing this TIL I found a much quicker option: the <code>ST_AsGeoJSON()</code> function. This works if your table has a geometry column on it - in my case I <a href="https://simonwillison.net/2021/May/3/adding-geodjango-to-an-existing-django-project/" rel="nofollow">enhanced my location table</a> to include a point column.</p>
<p>The following SQL generates a full GeoJSON feature row for each location in the database:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> ST_AsGeoJSON(location.<span class="pl-k">*</span>) <span class="pl-k">from</span> location</pre></div>
<p>The <code>location.*</code> is required - if you try to use <code>ST_AsGeoJSON(*)</code> you get an error claiming that "No function matches the given name and argument types".</p>
<p>The output looks something like this:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Feature<span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>geometry<span class="pl-pds">"</span></span>: {
    <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Point<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>coordinates<span class="pl-pds">"</span></span>: [
      <span class="pl-c1">-117.65499</span>,
      <span class="pl-c1">34.10835</span>
    ]
  },
  <span class="pl-s"><span class="pl-pds">"</span>properties<span class="pl-pds">"</span></span>: {
    <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-c1">4285</span>,
    <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Vons Pharmacy #2681<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>public_id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>recbECvQlveAyBorV<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>import_json<span class="pl-pds">"</span></span>: {
      <span class="pl-s"><span class="pl-pds">"</span>Name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Vons Pharmacy #2681<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>Hours<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Monday - Friday: 8:00 AM <span class="pl-cce">\u2013</span> 8:00 PM<span class="pl-cce">\n</span>Saturday - Sunday: 9:00 AM <span class="pl-cce">\u2013</span> 5:00 PM<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>County<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>San Bernardino County<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>county_id<span class="pl-pds">"</span></span>: [
        <span class="pl-s"><span class="pl-pds">"</span>reclZ8DWOEuoluStG<span class="pl-pds">"</span></span>
      ]
    }
  }
}</pre></div>
<p>That <code>import_json</code> section is a PostgreSQL JSON column in the database - note that it gets output as nested data inside the <code>"properties"</code> object.</p>


path: django_export-csv-from-django-admin.md
topic: django
title: Django Admin action for exporting selected rows as CSV
url: https://github.com/simonw/til/blob/main/django/export-csv-from-django-admin.md
created: 2021-04-25T17:38:06-07:00
created_utc: 2021-04-26T00:38:06+00:00
updated: 2021-04-25T17:38:06-07:00
updated_utc: 2021-04-26T00:38:06+00:00
shot_hash: da3a8857be8af5bfa07a3e637e9929cc
slug: export-csv-from-django-admin
body:
I wanted to add an action option to the Django Admin for exporting the currently selected set of rows (or every row in the table) as a CSV file.

I ended up using a pattern inspired by [this Django Snippet](https://djangosnippets.org/snippets/10767/), but with an added touch for more efficient exports. In order to avoid using up too much memory for the export, I use keyset pagination to fetch 500 rows at a time.

The `keyset_pagination_iterator()` helper function accepts any queryset, orders it by the primary key and then repeatedly fetches 500 items. It then modifies the queryset to add a `WHERE id > $last_seen_id` clause. This is a relatively inexpensive way to paginate, so having an endpoint perform that query dozens or even hundreds of times should hopefully avoid adding too much load to the database.

The action itself uses a pattern that combines `StringIO` and `csv.writer()` to stream out the results as a CSV file.

Django's `StreamingHttpResponse` mechanism is really neat: it accepts a Python iterator or generator and returns a streaming response derived from that sequence.

The Django documentation says "Streaming responses will tie a worker process for the entire duration of the response. This may result in poor performance" - this particular project runs on Google Cloud Run so I'm less concerned about tying up a worker than I would be normally, plus the export option is only available to trusted staff users with access to the Django Admin interface.

To add the CSV export option to a `ModelAdmin` subclass, do the following:

```python
from .admin_actions import export_as_csv_action

@admin.register(County)
class CountyAdmin(admin.ModelAdmin):
    actions = [export_as_csv_action()]
```
Here's `admin_actions.py`:
```python
import csv
from io import StringIO

from django.http import StreamingHttpResponse


def keyset_pagination_iterator(input_queryset, batch_size=500):
    all_queryset = input_queryset.order_by("pk")
    last_pk = None
    while True:
        queryset = all_queryset
        if last_pk is not None:
            queryset = all_queryset.filter(pk__gt=last_pk)
        queryset = queryset[:batch_size]
        for row in queryset:
            last_pk = row.pk
            yield row
        if not queryset:
            break


def export_as_csv_action(description="Export selected rows to CSV"):
    def export_as_csv(modeladmin, request, queryset):
        def rows(queryset):

            csvfile = StringIO()
            csvwriter = csv.writer(csvfile)
            columns = [field.name for field in modeladmin.model._meta.fields]

            def read_and_flush():
                csvfile.seek(0)
                data = csvfile.read()
                csvfile.seek(0)
                csvfile.truncate()
                return data

            header = False

            if not header:
                header = True
                csvwriter.writerow(columns)
                yield read_and_flush()

            for row in keyset_pagination_iterator(queryset):
                csvwriter.writerow(getattr(row, column) for column in columns)
                yield read_and_flush()

        response = StreamingHttpResponse(rows(queryset), content_type="text/csv")
        response["Content-Disposition"] = (
            "attachment; filename=%s.csv" % modeladmin.model.__name__
        )

        return response

    export_as_csv.short_description = description
    return export_as_csv
```

html:
<p>I wanted to add an action option to the Django Admin for exporting the currently selected set of rows (or every row in the table) as a CSV file.</p>
<p>I ended up using a pattern inspired by <a href="https://djangosnippets.org/snippets/10767/" rel="nofollow">this Django Snippet</a>, but with an added touch for more efficient exports. In order to avoid using up too much memory for the export, I use keyset pagination to fetch 500 rows at a time.</p>
<p>The <code>keyset_pagination_iterator()</code> helper function accepts any queryset, orders it by the primary key and then repeatedly fetches 500 items. It then modifies the queryset to add a <code>WHERE id &gt; $last_seen_id</code> clause. This is a relatively inexpensive way to paginate, so having an endpoint perform that query dozens or even hundreds of times should hopefully avoid adding too much load to the database.</p>
<p>The action itself uses a pattern that combines <code>StringIO</code> and <code>csv.writer()</code> to stream out the results as a CSV file.</p>
<p>Django's <code>StreamingHttpResponse</code> mechanism is really neat: it accepts a Python iterator or generator and returns a streaming response derived from that sequence.</p>
<p>The Django documentation says "Streaming responses will tie a worker process for the entire duration of the response. This may result in poor performance" - this particular project runs on Google Cloud Run so I'm less concerned about tying up a worker than I would be normally, plus the export option is only available to trusted staff users with access to the Django Admin interface.</p>
<p>To add the CSV export option to a <code>ModelAdmin</code> subclass, do the following:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> .<span class="pl-s1">admin_actions</span> <span class="pl-k">import</span> <span class="pl-s1">export_as_csv_action</span>

<span class="pl-en">@<span class="pl-s1">admin</span>.<span class="pl-en">register</span>(<span class="pl-v">County</span>)</span>
<span class="pl-k">class</span> <span class="pl-v">CountyAdmin</span>(<span class="pl-s1">admin</span>.<span class="pl-v">ModelAdmin</span>):
    <span class="pl-s1">actions</span> <span class="pl-c1">=</span> [<span class="pl-en">export_as_csv_action</span>()]</pre></div>
<p>Here's <code>admin_actions.py</code>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">csv</span>
<span class="pl-k">from</span> <span class="pl-s1">io</span> <span class="pl-k">import</span> <span class="pl-v">StringIO</span>

<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">http</span> <span class="pl-k">import</span> <span class="pl-v">StreamingHttpResponse</span>


<span class="pl-k">def</span> <span class="pl-en">keyset_pagination_iterator</span>(<span class="pl-s1">input_queryset</span>, <span class="pl-s1">batch_size</span><span class="pl-c1">=</span><span class="pl-c1">500</span>):
    <span class="pl-s1">all_queryset</span> <span class="pl-c1">=</span> <span class="pl-s1">input_queryset</span>.<span class="pl-en">order_by</span>(<span class="pl-s">"pk"</span>)
    <span class="pl-s1">last_pk</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>
    <span class="pl-k">while</span> <span class="pl-c1">True</span>:
        <span class="pl-s1">queryset</span> <span class="pl-c1">=</span> <span class="pl-s1">all_queryset</span>
        <span class="pl-k">if</span> <span class="pl-s1">last_pk</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>:
            <span class="pl-s1">queryset</span> <span class="pl-c1">=</span> <span class="pl-s1">all_queryset</span>.<span class="pl-en">filter</span>(<span class="pl-s1">pk__gt</span><span class="pl-c1">=</span><span class="pl-s1">last_pk</span>)
        <span class="pl-s1">queryset</span> <span class="pl-c1">=</span> <span class="pl-s1">queryset</span>[:<span class="pl-s1">batch_size</span>]
        <span class="pl-k">for</span> <span class="pl-s1">row</span> <span class="pl-c1">in</span> <span class="pl-s1">queryset</span>:
            <span class="pl-s1">last_pk</span> <span class="pl-c1">=</span> <span class="pl-s1">row</span>.<span class="pl-s1">pk</span>
            <span class="pl-k">yield</span> <span class="pl-s1">row</span>
        <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">queryset</span>:
            <span class="pl-k">break</span>


<span class="pl-k">def</span> <span class="pl-en">export_as_csv_action</span>(<span class="pl-s1">description</span><span class="pl-c1">=</span><span class="pl-s">"Export selected rows to CSV"</span>):
    <span class="pl-k">def</span> <span class="pl-en">export_as_csv</span>(<span class="pl-s1">modeladmin</span>, <span class="pl-s1">request</span>, <span class="pl-s1">queryset</span>):
        <span class="pl-k">def</span> <span class="pl-en">rows</span>(<span class="pl-s1">queryset</span>):

            <span class="pl-s1">csvfile</span> <span class="pl-c1">=</span> <span class="pl-v">StringIO</span>()
            <span class="pl-s1">csvwriter</span> <span class="pl-c1">=</span> <span class="pl-s1">csv</span>.<span class="pl-en">writer</span>(<span class="pl-s1">csvfile</span>)
            <span class="pl-s1">columns</span> <span class="pl-c1">=</span> [<span class="pl-s1">field</span>.<span class="pl-s1">name</span> <span class="pl-k">for</span> <span class="pl-s1">field</span> <span class="pl-c1">in</span> <span class="pl-s1">modeladmin</span>.<span class="pl-s1">model</span>.<span class="pl-s1">_meta</span>.<span class="pl-s1">fields</span>]

            <span class="pl-k">def</span> <span class="pl-en">read_and_flush</span>():
                <span class="pl-s1">csvfile</span>.<span class="pl-en">seek</span>(<span class="pl-c1">0</span>)
                <span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">csvfile</span>.<span class="pl-en">read</span>()
                <span class="pl-s1">csvfile</span>.<span class="pl-en">seek</span>(<span class="pl-c1">0</span>)
                <span class="pl-s1">csvfile</span>.<span class="pl-en">truncate</span>()
                <span class="pl-k">return</span> <span class="pl-s1">data</span>

            <span class="pl-s1">header</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>

            <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">header</span>:
                <span class="pl-s1">header</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
                <span class="pl-s1">csvwriter</span>.<span class="pl-en">writerow</span>(<span class="pl-s1">columns</span>)
                <span class="pl-k">yield</span> <span class="pl-en">read_and_flush</span>()

            <span class="pl-k">for</span> <span class="pl-s1">row</span> <span class="pl-c1">in</span> <span class="pl-en">keyset_pagination_iterator</span>(<span class="pl-s1">queryset</span>):
                <span class="pl-s1">csvwriter</span>.<span class="pl-en">writerow</span>(<span class="pl-en">getattr</span>(<span class="pl-s1">row</span>, <span class="pl-s1">column</span>) <span class="pl-k">for</span> <span class="pl-s1">column</span> <span class="pl-c1">in</span> <span class="pl-s1">columns</span>)
                <span class="pl-k">yield</span> <span class="pl-en">read_and_flush</span>()

        <span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-v">StreamingHttpResponse</span>(<span class="pl-en">rows</span>(<span class="pl-s1">queryset</span>), <span class="pl-s1">content_type</span><span class="pl-c1">=</span><span class="pl-s">"text/csv"</span>)
        <span class="pl-s1">response</span>[<span class="pl-s">"Content-Disposition"</span>] <span class="pl-c1">=</span> (
            <span class="pl-s">"attachment; filename=%s.csv"</span> <span class="pl-c1">%</span> <span class="pl-s1">modeladmin</span>.<span class="pl-s1">model</span>.<span class="pl-s1">__name__</span>
        )

        <span class="pl-k">return</span> <span class="pl-s1">response</span>

    <span class="pl-s1">export_as_csv</span>.<span class="pl-s1">short_description</span> <span class="pl-c1">=</span> <span class="pl-s1">description</span>
    <span class="pl-k">return</span> <span class="pl-s1">export_as_csv</span></pre></div>


path: django_migrations-runsql-noop.md
topic: django
title: migrations.RunSQL.noop for reversible SQL migrations
url: https://github.com/simonw/til/blob/main/django/migrations-runsql-noop.md
created: 2021-05-02T10:48:46-07:00
created_utc: 2021-05-02T17:48:46+00:00
updated: 2021-05-02T10:48:46-07:00
updated_utc: 2021-05-02T17:48:46+00:00
shot_hash: 03f66f89626893e5a1a0374109ec84e8
slug: migrations-runsql-noop
body:
`migrations.RunSQL.noop` provides an easy way to create "reversible" Django SQL migrations, where the reverse operation does nothing (but keeps it possible to reverse back to a previous migration state without being blocked by an irreversible migration).

```python
from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ("app", "0114_last_migration"),
    ]

    operations = [
        migrations.RunSQL(
            sql="""
                update concordance_identifier
                set authority = replace(authority, ':', '_')
                where authority like '%:%'
            """,
            reverse_sql=migrations.RunSQL.noop,
        )
    ]
```

html:
<p><code>migrations.RunSQL.noop</code> provides an easy way to create "reversible" Django SQL migrations, where the reverse operation does nothing (but keeps it possible to reverse back to a previous migration state without being blocked by an irreversible migration).</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">db</span> <span class="pl-k">import</span> <span class="pl-s1">migrations</span>


<span class="pl-k">class</span> <span class="pl-v">Migration</span>(<span class="pl-s1">migrations</span>.<span class="pl-v">Migration</span>):

    <span class="pl-s1">dependencies</span> <span class="pl-c1">=</span> [
        (<span class="pl-s">"app"</span>, <span class="pl-s">"0114_last_migration"</span>),
    ]

    <span class="pl-s1">operations</span> <span class="pl-c1">=</span> [
        <span class="pl-s1">migrations</span>.<span class="pl-v">RunSQL</span>(
            <span class="pl-s1">sql</span><span class="pl-c1">=</span><span class="pl-s">"""</span>
<span class="pl-s">                update concordance_identifier</span>
<span class="pl-s">                set authority = replace(authority, ':', '_')</span>
<span class="pl-s">                where authority like '%:%'</span>
<span class="pl-s">            """</span>,
            <span class="pl-s1">reverse_sql</span><span class="pl-c1">=</span><span class="pl-s1">migrations</span>.<span class="pl-v">RunSQL</span>.<span class="pl-s1">noop</span>,
        )
    ]</pre></div>


path: datasette_datasette-on-replit.md
topic: datasette
title: Running Datasette on Replit
url: https://github.com/simonw/til/blob/main/datasette/datasette-on-replit.md
created: 2021-05-02T11:50:05-07:00
created_utc: 2021-05-02T18:50:05+00:00
updated: 2021-05-02T11:50:05-07:00
updated_utc: 2021-05-02T18:50:05+00:00
shot_hash: 96e900155fbcb773fcc32dfbbc2bf55c
slug: datasette-on-replit
body:
I figured out how to run Datasette on https://replit.com/

The trick is to start a new Python project and then drop the following into the `main.py` file:

```python
import uvicorn
from datasette.app import Datasette

ds = Datasette(memory=True, files=[])


if __name__ == "__main__":
    uvicorn.run(ds.app(), host="0.0.0.0", port=8000)
```
Replit is smart enough to automatically create a `pyproject.toml` file with `datasette` and `uvicorn` as dependencies. It will also notice that the application is running on port 8000 and set `https://name-of-prject.your-username.repl.co` to proxy to that port. Plus it will restart the server any time it recieves new traffic (and pause it in between groups of requests).

To serve a database file, download it using `wget` in the Replit console and add it to the `files=[]` argument. I ran this:

    wget https://datasette.io/content.db

Then changed that first line to:

```python
ds = Datasette(files=["content.db"])
```
And restarted the server.

html:
<p>I figured out how to run Datasette on <a href="https://replit.com/" rel="nofollow">https://replit.com/</a></p>
<p>The trick is to start a new Python project and then drop the following into the <code>main.py</code> file:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">uvicorn</span>
<span class="pl-k">from</span> <span class="pl-s1">datasette</span>.<span class="pl-s1">app</span> <span class="pl-k">import</span> <span class="pl-v">Datasette</span>

<span class="pl-s1">ds</span> <span class="pl-c1">=</span> <span class="pl-v">Datasette</span>(<span class="pl-s1">memory</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">files</span><span class="pl-c1">=</span>[])


<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">"__main__"</span>:
    <span class="pl-s1">uvicorn</span>.<span class="pl-en">run</span>(<span class="pl-s1">ds</span>.<span class="pl-en">app</span>(), <span class="pl-s1">host</span><span class="pl-c1">=</span><span class="pl-s">"0.0.0.0"</span>, <span class="pl-s1">port</span><span class="pl-c1">=</span><span class="pl-c1">8000</span>)</pre></div>
<p>Replit is smart enough to automatically create a <code>pyproject.toml</code> file with <code>datasette</code> and <code>uvicorn</code> as dependencies. It will also notice that the application is running on port 8000 and set <code>https://name-of-prject.your-username.repl.co</code> to proxy to that port. Plus it will restart the server any time it recieves new traffic (and pause it in between groups of requests).</p>
<p>To serve a database file, download it using <code>wget</code> in the Replit console and add it to the <code>files=[]</code> argument. I ran this:</p>
<pre><code>wget https://datasette.io/content.db
</code></pre>
<p>Then changed that first line to:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">ds</span> <span class="pl-c1">=</span> <span class="pl-v">Datasette</span>(<span class="pl-s1">files</span><span class="pl-c1">=</span>[<span class="pl-s">"content.db"</span>])</pre></div>
<p>And restarted the server.</p>


path: javascript_scroll-to-form-if-errors.md
topic: javascript
title: Scroll page to form if there are errors
url: https://github.com/simonw/til/blob/main/javascript/scroll-to-form-if-errors.md
created: 2021-05-08T10:13:03-07:00
created_utc: 2021-05-08T17:13:03+00:00
updated: 2021-05-08T10:13:03-07:00
updated_utc: 2021-05-08T17:13:03+00:00
shot_hash: d535a31814bff84323b55d4410cddbf3
slug: scroll-to-form-if-errors
body:
For a Django application I'm working on ([this issue](https://github.com/simonw/django-sql-dashboard/issues/44)) I have a form that can be quite a long way down the page.

If the form is displayed with errors, I want to scroll the user down to the form so they don't get confused.

Since Django forms display errors in an element with a `errorlist` class, this worked:

```javascript
window.addEventListener("load", () => {
  if (document.querySelector('.errorlist')) {
    document.querySelector('#my-form').scrollIntoView();
  }
});
```

[Element.scrollIntoView()](https://developer.mozilla.org/en-US/docs/Web/API/Element/scrollIntoView) on MDN, and [on Can I use](https://caniuse.com/scrollintoview).

html:
<p>For a Django application I'm working on (<a href="https://github.com/simonw/django-sql-dashboard/issues/44">this issue</a>) I have a form that can be quite a long way down the page.</p>
<p>If the form is displayed with errors, I want to scroll the user down to the form so they don't get confused.</p>
<p>Since Django forms display errors in an element with a <code>errorlist</code> class, this worked:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-smi">window</span><span class="pl-kos">.</span><span class="pl-en">addEventListener</span><span class="pl-kos">(</span><span class="pl-s">"load"</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
  <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">querySelector</span><span class="pl-kos">(</span><span class="pl-s">'.errorlist'</span><span class="pl-kos">)</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">querySelector</span><span class="pl-kos">(</span><span class="pl-s">'#my-form'</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">scrollIntoView</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<p><a href="https://developer.mozilla.org/en-US/docs/Web/API/Element/scrollIntoView" rel="nofollow">Element.scrollIntoView()</a> on MDN, and <a href="https://caniuse.com/scrollintoview" rel="nofollow">on Can I use</a>.</p>


path: wikipedia_page-stats-api.md
topic: wikipedia
title: The Wikipedia page stats API
url: https://github.com/simonw/til/blob/main/wikipedia/page-stats-api.md
created: 2021-05-13T12:26:08-07:00
created_utc: 2021-05-13T19:26:08+00:00
updated: 2021-05-13T12:26:08-07:00
updated_utc: 2021-05-13T19:26:08+00:00
shot_hash: 8168506a4ac3ddb1e6c1e8ab1998d48b
slug: page-stats-api
body:
Via https://alexgarcia.xyz/dataflow/examples/wiki-pageviews/ I found this API for retrieving daily pageview stats from Wikipedia for any article:

- https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/Python_(programming_language)/daily/20210101/20210501
- https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/Simon_Willison/daily/20210101/20210501

Example response:

```json
{
  "items": [
    {
      "project": "en.wikipedia",
      "article": "Python_(programming_language)",
      "granularity": "daily",
      "timestamp": "2021010100",
      "access": "all-access",
      "agent": "user",
      "views": 7238
    },
    {
      "project": "en.wikipedia",
      "article": "Python_(programming_language)",
      "granularity": "daily",
      "timestamp": "2021010200",
      "access": "all-access",
      "agent": "user",
      "views": 8449
    },
    {
      "project": "en.wikipedia",
      "article": "Python_(programming_language)",
      "granularity": "daily",
      "timestamp": "2021010300",
      "access": "all-access",
      "agent": "user",
      "views": 8669
    }
  ]
}
```

html:
<p>Via <a href="https://alexgarcia.xyz/dataflow/examples/wiki-pageviews/" rel="nofollow">https://alexgarcia.xyz/dataflow/examples/wiki-pageviews/</a> I found this API for retrieving daily pageview stats from Wikipedia for any article:</p>
<ul>
<li><a href="https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/Python_(programming_language)/daily/20210101/20210501" rel="nofollow">https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/Python_(programming_language)/daily/20210101/20210501</a></li>
<li><a href="https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/Simon_Willison/daily/20210101/20210501" rel="nofollow">https://wikimedia.org/api/rest_v1/metrics/pageviews/per-article/en.wikipedia.org/all-access/user/Simon_Willison/daily/20210101/20210501</a></li>
</ul>
<p>Example response:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-s"><span class="pl-pds">"</span>items<span class="pl-pds">"</span></span>: [
    {
      <span class="pl-s"><span class="pl-pds">"</span>project<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>en.wikipedia<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>article<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Python_(programming_language)<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>granularity<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>daily<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>timestamp<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021010100<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>access<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>all-access<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>agent<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>user<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>views<span class="pl-pds">"</span></span>: <span class="pl-c1">7238</span>
    },
    {
      <span class="pl-s"><span class="pl-pds">"</span>project<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>en.wikipedia<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>article<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Python_(programming_language)<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>granularity<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>daily<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>timestamp<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021010200<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>access<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>all-access<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>agent<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>user<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>views<span class="pl-pds">"</span></span>: <span class="pl-c1">8449</span>
    },
    {
      <span class="pl-s"><span class="pl-pds">"</span>project<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>en.wikipedia<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>article<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Python_(programming_language)<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>granularity<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>daily<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>timestamp<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2021010300<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>access<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>all-access<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>agent<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>user<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>views<span class="pl-pds">"</span></span>: <span class="pl-c1">8669</span>
    }
  ]
}</pre></div>


path: vega_bar-chart-ordering.md
topic: vega
title: Vega-Lite bar charts in the same order as the data
url: https://github.com/simonw/til/blob/main/vega/bar-chart-ordering.md
created: 2021-05-15T17:25:32-07:00
created_utc: 2021-05-16T00:25:32+00:00
updated: 2021-05-15T17:25:32-07:00
updated_utc: 2021-05-16T00:25:32+00:00
shot_hash: bdf5d642743e1177b23c5b648dd7b26c
slug: bar-chart-ordering
body:
I've been puzzling over this one for a couple of years now, and I finally figured out the solution.

The Vega-Lite charting library can generate bar charts using a JSON definition that looks like this:

```json
{
  "$schema": "https://vega.github.io/schema/vega-lite/v5.json",
  "description": "A simple bar chart with embedded data.",
  "data": {
    "values": [
      {"bar_label": "startups", "bar_quantity": 157},
      {"bar_label": "webdevelopment", "bar_quantity": 166},
      {"bar_label": "quora", "bar_quantity": 1003},
      {"bar_label": "conferences", "bar_quantity": 152},
      {"bar_label": "datasette", "bar_quantity": 111}
    ]
  },
  "mark": "bar",
  "encoding": {
    "x": {
      "field": "bar_label",
      "title": "Label",
      "type": "nominal",
      "axis": {"labelAngle": 90}
    },
    "y": {"field": "bar_quantity", "title": "Quantity", "type": "quantitative"}
  }
}
```
There's just one catch: Vega-Lite defaults to sorting the bars alphabetically by their nominal label.

<img width="840" alt="Vega_Editor" src="https://user-images.githubusercontent.com/9599/118381682-46acef00-b5a2-11eb-994c-65b8aed11be4.png">

But sometimes you might want to control that sort order - to have the order that the bars are displayed in directly reflect the order of the data that you passed in.

To do this, pass `"sort": null` as an option to the `x` encoding:

<img width="830" alt="Vega_Editor" src="https://user-images.githubusercontent.com/9599/118381690-5debdc80-b5a2-11eb-93db-bbd847afae81.png">


Vega-Lite [Sorting documentation](https://vega.github.io/vega-lite/docs/sort.html).

html:
<p>I've been puzzling over this one for a couple of years now, and I finally figured out the solution.</p>
<p>The Vega-Lite charting library can generate bar charts using a JSON definition that looks like this:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-s"><span class="pl-pds">"</span>$schema<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://vega.github.io/schema/vega-lite/v5.json<span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>description<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>A simple bar chart with embedded data.<span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>data<span class="pl-pds">"</span></span>: {
    <span class="pl-s"><span class="pl-pds">"</span>values<span class="pl-pds">"</span></span>: [
      {<span class="pl-s"><span class="pl-pds">"</span>bar_label<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>startups<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>bar_quantity<span class="pl-pds">"</span></span>: <span class="pl-c1">157</span>},
      {<span class="pl-s"><span class="pl-pds">"</span>bar_label<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>webdevelopment<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>bar_quantity<span class="pl-pds">"</span></span>: <span class="pl-c1">166</span>},
      {<span class="pl-s"><span class="pl-pds">"</span>bar_label<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>quora<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>bar_quantity<span class="pl-pds">"</span></span>: <span class="pl-c1">1003</span>},
      {<span class="pl-s"><span class="pl-pds">"</span>bar_label<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>conferences<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>bar_quantity<span class="pl-pds">"</span></span>: <span class="pl-c1">152</span>},
      {<span class="pl-s"><span class="pl-pds">"</span>bar_label<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>datasette<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>bar_quantity<span class="pl-pds">"</span></span>: <span class="pl-c1">111</span>}
    ]
  },
  <span class="pl-s"><span class="pl-pds">"</span>mark<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>bar<span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>encoding<span class="pl-pds">"</span></span>: {
    <span class="pl-s"><span class="pl-pds">"</span>x<span class="pl-pds">"</span></span>: {
      <span class="pl-s"><span class="pl-pds">"</span>field<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>bar_label<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>title<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Label<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>nominal<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>axis<span class="pl-pds">"</span></span>: {<span class="pl-s"><span class="pl-pds">"</span>labelAngle<span class="pl-pds">"</span></span>: <span class="pl-c1">90</span>}
    },
    <span class="pl-s"><span class="pl-pds">"</span>y<span class="pl-pds">"</span></span>: {<span class="pl-s"><span class="pl-pds">"</span>field<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>bar_quantity<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>title<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Quantity<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>type<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>quantitative<span class="pl-pds">"</span></span>}
  }
}</pre></div>
<p>There's just one catch: Vega-Lite defaults to sorting the bars alphabetically by their nominal label.</p>
<p><a href="https://user-images.githubusercontent.com/9599/118381682-46acef00-b5a2-11eb-994c-65b8aed11be4.png" target="_blank" rel="nofollow"><img width="840" alt="Vega_Editor" src="https://user-images.githubusercontent.com/9599/118381682-46acef00-b5a2-11eb-994c-65b8aed11be4.png" style="max-width:100%;"></a></p>
<p>But sometimes you might want to control that sort order - to have the order that the bars are displayed in directly reflect the order of the data that you passed in.</p>
<p>To do this, pass <code>"sort": null</code> as an option to the <code>x</code> encoding:</p>
<p><a href="https://user-images.githubusercontent.com/9599/118381690-5debdc80-b5a2-11eb-93db-bbd847afae81.png" target="_blank" rel="nofollow"><img width="830" alt="Vega_Editor" src="https://user-images.githubusercontent.com/9599/118381690-5debdc80-b5a2-11eb-93db-bbd847afae81.png" style="max-width:100%;"></a></p>
<p>Vega-Lite <a href="https://vega.github.io/vega-lite/docs/sort.html" rel="nofollow">Sorting documentation</a>.</p>


path: django_enabling-gin-index.md
topic: django
title: Enabling a gin index for faster LIKE queries
url: https://github.com/simonw/til/blob/main/django/enabling-gin-index.md
created: 2021-05-16T17:59:05-07:00
created_utc: 2021-05-17T00:59:05+00:00
updated: 2021-05-16T17:59:05-07:00
updated_utc: 2021-05-17T00:59:05+00:00
shot_hash: ab4106d1cd70b2fabb2cb63117d18edd
slug: enabling-gin-index
body:
I tried using a gin index to speed up `LIKE '%term%'` queries against a column.

[PostgreSQL: More performance for LIKE and ILIKE statements](https://www.cybertec-postgresql.com/en/postgresql-more-performance-for-like-and-ilike-statements/) provided useful background. The raw-SQL way to do this is to install the extension like so:

```sql
CREATE EXTENSION pg_trgm;
```
And then create an index like this:
```sql
CREATE INDEX idx_gin ON mytable USING gin (mycolumn gin_trgm_ops);
```
This translates to two migrations in Django. The first, to enable the extension, looks like this:
```python
from django.contrib.postgres.operations import TrigramExtension
from django.db import migrations


class Migration(migrations.Migration):

    dependencies = [
        ("blog", "0014_entry_custom_template"),
    ]

    operations = [TrigramExtension()]
```
Then to configure the index for a model you can add this to the model's `Meta` class:
```python
class Entry(models.Model):
    title = models.CharField(max_length=255)
    body = models.TextField()

    class Meta:
        indexes = [
            GinIndex(
                name="idx_blog_entry_body_gin",
                fields=["body"],
                opclasses=["gin_trgm_ops"],
            ),
        ]
```
The `opclasses=["gin_trgm_ops"]` line is necessary to have the same efect as the `CREATE INDEX` statement shown above. The `name=` option is required if you specify `opclasses`.

Run `./manage.py makemigrations` and Django will automatically create the correct migration to add the new index.

I ended up not shipping this for my blog because with less than 10,000 rows in the table it made no difference at all to my query performance.

html:
<p>I tried using a gin index to speed up <code>LIKE '%term%'</code> queries against a column.</p>
<p><a href="https://www.cybertec-postgresql.com/en/postgresql-more-performance-for-like-and-ilike-statements/" rel="nofollow">PostgreSQL: More performance for LIKE and ILIKE statements</a> provided useful background. The raw-SQL way to do this is to install the extension like so:</p>
<div class="highlight highlight-source-sql"><pre>CREATE EXTENSION pg_trgm;</pre></div>
<p>And then create an index like this:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">CREATE</span> <span class="pl-k">INDEX</span> <span class="pl-en">idx_gin</span> <span class="pl-k">ON</span> mytable USING gin (mycolumn gin_trgm_ops);</pre></div>
<p>This translates to two migrations in Django. The first, to enable the extension, looks like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">contrib</span>.<span class="pl-s1">postgres</span>.<span class="pl-s1">operations</span> <span class="pl-k">import</span> <span class="pl-v">TrigramExtension</span>
<span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">db</span> <span class="pl-k">import</span> <span class="pl-s1">migrations</span>


<span class="pl-k">class</span> <span class="pl-v">Migration</span>(<span class="pl-s1">migrations</span>.<span class="pl-v">Migration</span>):

    <span class="pl-s1">dependencies</span> <span class="pl-c1">=</span> [
        (<span class="pl-s">"blog"</span>, <span class="pl-s">"0014_entry_custom_template"</span>),
    ]

    <span class="pl-s1">operations</span> <span class="pl-c1">=</span> [<span class="pl-v">TrigramExtension</span>()]</pre></div>
<p>Then to configure the index for a model you can add this to the model's <code>Meta</code> class:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">class</span> <span class="pl-v">Entry</span>(<span class="pl-s1">models</span>.<span class="pl-v">Model</span>):
    <span class="pl-s1">title</span> <span class="pl-c1">=</span> <span class="pl-s1">models</span>.<span class="pl-v">CharField</span>(<span class="pl-s1">max_length</span><span class="pl-c1">=</span><span class="pl-c1">255</span>)
    <span class="pl-s1">body</span> <span class="pl-c1">=</span> <span class="pl-s1">models</span>.<span class="pl-v">TextField</span>()

    <span class="pl-k">class</span> <span class="pl-v">Meta</span>:
        <span class="pl-s1">indexes</span> <span class="pl-c1">=</span> [
            <span class="pl-v">GinIndex</span>(
                <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"idx_blog_entry_body_gin"</span>,
                <span class="pl-s1">fields</span><span class="pl-c1">=</span>[<span class="pl-s">"body"</span>],
                <span class="pl-s1">opclasses</span><span class="pl-c1">=</span>[<span class="pl-s">"gin_trgm_ops"</span>],
            ),
        ]</pre></div>
<p>The <code>opclasses=["gin_trgm_ops"]</code> line is necessary to have the same efect as the <code>CREATE INDEX</code> statement shown above. The <code>name=</code> option is required if you specify <code>opclasses</code>.</p>
<p>Run <code>./manage.py makemigrations</code> and Django will automatically create the correct migration to add the new index.</p>
<p>I ended up not shipping this for my blog because with less than 10,000 rows in the table it made no difference at all to my query performance.</p>


path: spatialite_knn.md
topic: spatialite
title: KNN queries with SpatiaLite
url: https://github.com/simonw/til/blob/main/spatialite/knn.md
created: 2021-05-16T18:12:57-07:00
created_utc: 2021-05-17T01:12:57+00:00
updated: 2021-05-20T14:22:10-07:00
updated_utc: 2021-05-20T21:22:10+00:00
shot_hash: 63dddb9364f6ce9011060c5f46e98a22
slug: knn
body:
The latest version of SpatiaLite adds KNN support, which makes it easy to efficiently answer the question "what are the X closest records to this point".

The USGS earthquakes GeoJSON is a great dataset for experimenting with these features.

Documentation for that is here: https://earthquake.usgs.gov/earthquakes/feed/v1.0/geojson.php

`https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson` currently contains 10,642 features.

To turn that into a SpatiaLite database using the latest version of [geojson-to-sqlite](https://github.com/simonw/geojson-to-sqlite):
```bash
curl 'https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson' | \
  geojson-to-sqlite usgs.db quakes - --spatial-index
```
This will create a `usgs.db` SpatiaLite database with a spatial index.

This should have a `knn` virtual table already. If one does not exist you can create one like this:

    sqlite-utils --load-extension=spatialite usgs.db 'CREATE VIRTUAL TABLE knn USING VirtualKNN();'

Open it in Datasette like this:

```bash
datasette --load-extension=spatialite usgs.db
```

(Running `datasette install datasette-cluster-map` first will let you see them on a map.)

Use this SQL query to run KNN searches returning the ten closest earthquakes to a point:
```sql
SELECT
  knn.distance,
  quakes.title,
  quakes.mag,
  quakes.time,
  quakes.url,
  y(quakes.geometry) as latitude,
  x(quakes.geometry) as longitude
FROM
  knn
  join quakes on knn.fid = quakes.rowid
WHERE
  f_table_name = 'quakes'
  AND ref_geometry = MakePoint(cast(:longitude as real), cast(:latitude as real))
  AND max_items = 10
```

html:
<p>The latest version of SpatiaLite adds KNN support, which makes it easy to efficiently answer the question "what are the X closest records to this point".</p>
<p>The USGS earthquakes GeoJSON is a great dataset for experimenting with these features.</p>
<p>Documentation for that is here: <a href="https://earthquake.usgs.gov/earthquakes/feed/v1.0/geojson.php" rel="nofollow">https://earthquake.usgs.gov/earthquakes/feed/v1.0/geojson.php</a></p>
<p><code>https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson</code> currently contains 10,642 features.</p>
<p>To turn that into a SpatiaLite database using the latest version of <a href="https://github.com/simonw/geojson-to-sqlite">geojson-to-sqlite</a>:</p>
<div class="highlight highlight-source-shell"><pre>curl <span class="pl-s"><span class="pl-pds">'</span>https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.geojson<span class="pl-pds">'</span></span> <span class="pl-k">|</span> \
  geojson-to-sqlite usgs.db quakes - --spatial-index</pre></div>
<p>This will create a <code>usgs.db</code> SpatiaLite database with a spatial index.</p>
<p>This should have a <code>knn</code> virtual table already. If one does not exist you can create one like this:</p>
<pre><code>sqlite-utils --load-extension=spatialite usgs.db 'CREATE VIRTUAL TABLE knn USING VirtualKNN();'
</code></pre>
<p>Open it in Datasette like this:</p>
<div class="highlight highlight-source-shell"><pre>datasette --load-extension=spatialite usgs.db</pre></div>
<p>(Running <code>datasette install datasette-cluster-map</code> first will let you see them on a map.)</p>
<p>Use this SQL query to run KNN searches returning the ten closest earthquakes to a point:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">SELECT</span>
  <span class="pl-c1">knn</span>.<span class="pl-c1">distance</span>,
  <span class="pl-c1">quakes</span>.<span class="pl-c1">title</span>,
  <span class="pl-c1">quakes</span>.<span class="pl-c1">mag</span>,
  <span class="pl-c1">quakes</span>.<span class="pl-c1">time</span>,
  <span class="pl-c1">quakes</span>.<span class="pl-c1">url</span>,
  y(<span class="pl-c1">quakes</span>.<span class="pl-c1">geometry</span>) <span class="pl-k">as</span> latitude,
  x(<span class="pl-c1">quakes</span>.<span class="pl-c1">geometry</span>) <span class="pl-k">as</span> longitude
<span class="pl-k">FROM</span>
  knn
  <span class="pl-k">join</span> quakes <span class="pl-k">on</span> <span class="pl-c1">knn</span>.<span class="pl-c1">fid</span> <span class="pl-k">=</span> <span class="pl-c1">quakes</span>.<span class="pl-c1">rowid</span>
<span class="pl-k">WHERE</span>
  f_table_name <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">'</span>quakes<span class="pl-pds">'</span></span>
  <span class="pl-k">AND</span> ref_geometry <span class="pl-k">=</span> MakePoint(cast(:longitude <span class="pl-k">as</span> <span class="pl-k">real</span>), cast(:latitude <span class="pl-k">as</span> <span class="pl-k">real</span>))
  <span class="pl-k">AND</span> max_items <span class="pl-k">=</span> <span class="pl-c1">10</span></pre></div>


path: django_migration-using-cte.md
topic: django
title: Django data migration using a PostgreSQL CTE
url: https://github.com/simonw/til/blob/main/django/migration-using-cte.md
created: 2021-05-17T17:04:29-07:00
created_utc: 2021-05-18T00:04:29+00:00
updated: 2021-05-17T17:04:29-07:00
updated_utc: 2021-05-18T00:04:29+00:00
shot_hash: 4f4a982442ef5d9b9bb40127c0d7949e
slug: migration-using-cte
body:
I figured out how to use a PostgreSQL CTE as part of an update statement in a Django data migration. The trick here is mainly understanding how to combine CTEs with a PostgreSQL update - here's the pattern for that:

```sql
with something as (
  select id, created_at from ...
)
update mytable
  set
    created_at = something.created_at
  from
    something
  where
    mytable.id = something.id
```
Here's the full migration I wrote:
```python
from django.db import migrations

SQL = """
with created_at_by_reversion as (
  select
    location.id as id, min(date_created) as created_at
  from location
    join reversion_version on (location.id = reversion_version.object_id::integer and reversion_version.content_type_id = 18)
    join reversion_revision on reversion_revision.id = reversion_version.revision_id
  group by location.id
),
created_at_by_source_location as (
  select
    location.id as id, min(source_location.created_at) as created_at
  from source_location
    join location on source_location.matched_location_id = location.id
  group by location.id
),
new_created_at_for_locations as (
  select
    location.id,
    created_at_by_reversion.created_at as created_at_by_reversion,
    created_at_by_source_location.created_at as created_at_by_source_location,
    coalesce(created_at_by_reversion.created_at, created_at_by_source_location.created_at) as new_created_at
  from location
    left join created_at_by_source_location on created_at_by_source_location.id = location.id
    left join created_at_by_reversion on created_at_by_reversion.id = location.id
)
update location
  set
    created_at = new_created_at_for_locations.new_created_at
  from
    new_created_at_for_locations
  where
    location.id = new_created_at_for_locations.id
"""


class Migration(migrations.Migration):

    dependencies = [
        ("core", "0132_location_created_at_created_by"),
    ]

    operations = [
        migrations.RunSQL(
            sql=SQL,
            reverse_sql=migrations.RunSQL.noop,
        ),
    ]
```

html:
<p>I figured out how to use a PostgreSQL CTE as part of an update statement in a Django data migration. The trick here is mainly understanding how to combine CTEs with a PostgreSQL update - here's the pattern for that:</p>
<div class="highlight highlight-source-sql"><pre>with something <span class="pl-k">as</span> (
  <span class="pl-k">select</span> id, created_at <span class="pl-k">from</span> ...
)
<span class="pl-k">update</span> mytable
  <span class="pl-k">set</span>
    created_at <span class="pl-k">=</span> <span class="pl-c1">something</span>.<span class="pl-c1">created_at</span>
  <span class="pl-k">from</span>
    something
  <span class="pl-k">where</span>
    <span class="pl-c1">mytable</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">something</span>.<span class="pl-c1">id</span></pre></div>
<p>Here's the full migration I wrote:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">django</span>.<span class="pl-s1">db</span> <span class="pl-k">import</span> <span class="pl-s1">migrations</span>

<span class="pl-v">SQL</span> <span class="pl-c1">=</span> <span class="pl-s">"""</span>
<span class="pl-s">with created_at_by_reversion as (</span>
<span class="pl-s">  select</span>
<span class="pl-s">    location.id as id, min(date_created) as created_at</span>
<span class="pl-s">  from location</span>
<span class="pl-s">    join reversion_version on (location.id = reversion_version.object_id::integer and reversion_version.content_type_id = 18)</span>
<span class="pl-s">    join reversion_revision on reversion_revision.id = reversion_version.revision_id</span>
<span class="pl-s">  group by location.id</span>
<span class="pl-s">),</span>
<span class="pl-s">created_at_by_source_location as (</span>
<span class="pl-s">  select</span>
<span class="pl-s">    location.id as id, min(source_location.created_at) as created_at</span>
<span class="pl-s">  from source_location</span>
<span class="pl-s">    join location on source_location.matched_location_id = location.id</span>
<span class="pl-s">  group by location.id</span>
<span class="pl-s">),</span>
<span class="pl-s">new_created_at_for_locations as (</span>
<span class="pl-s">  select</span>
<span class="pl-s">    location.id,</span>
<span class="pl-s">    created_at_by_reversion.created_at as created_at_by_reversion,</span>
<span class="pl-s">    created_at_by_source_location.created_at as created_at_by_source_location,</span>
<span class="pl-s">    coalesce(created_at_by_reversion.created_at, created_at_by_source_location.created_at) as new_created_at</span>
<span class="pl-s">  from location</span>
<span class="pl-s">    left join created_at_by_source_location on created_at_by_source_location.id = location.id</span>
<span class="pl-s">    left join created_at_by_reversion on created_at_by_reversion.id = location.id</span>
<span class="pl-s">)</span>
<span class="pl-s">update location</span>
<span class="pl-s">  set</span>
<span class="pl-s">    created_at = new_created_at_for_locations.new_created_at</span>
<span class="pl-s">  from</span>
<span class="pl-s">    new_created_at_for_locations</span>
<span class="pl-s">  where</span>
<span class="pl-s">    location.id = new_created_at_for_locations.id</span>
<span class="pl-s">"""</span>


<span class="pl-k">class</span> <span class="pl-v">Migration</span>(<span class="pl-s1">migrations</span>.<span class="pl-v">Migration</span>):

    <span class="pl-s1">dependencies</span> <span class="pl-c1">=</span> [
        (<span class="pl-s">"core"</span>, <span class="pl-s">"0132_location_created_at_created_by"</span>),
    ]

    <span class="pl-s1">operations</span> <span class="pl-c1">=</span> [
        <span class="pl-s1">migrations</span>.<span class="pl-v">RunSQL</span>(
            <span class="pl-s1">sql</span><span class="pl-c1">=</span><span class="pl-v">SQL</span>,
            <span class="pl-s1">reverse_sql</span><span class="pl-c1">=</span><span class="pl-s1">migrations</span>.<span class="pl-v">RunSQL</span>.<span class="pl-s1">noop</span>,
        ),
    ]</pre></div>


path: cloudrun_multiple-gcloud-accounts.md
topic: cloudrun
title: Switching between gcloud accounts
url: https://github.com/simonw/til/blob/main/cloudrun/multiple-gcloud-accounts.md
created: 2021-05-18T14:35:37-07:00
created_utc: 2021-05-18T21:35:37+00:00
updated: 2021-09-23T11:42:26-07:00
updated_utc: 2021-09-23T18:42:26+00:00
shot_hash: be7b6208a2e3f8d86967eb00019c5db0
slug: multiple-gcloud-accounts
body:
I have two different Google Cloud accounts active at the moment. Here's how to list them with `gcloud auth list`:

```
% gcloud auth list
    Credentialed Accounts
ACTIVE  ACCOUNT
        simon@example.com
*       me@gmail.com

To set the active account, run:
    $ gcloud config set account `ACCOUNT`
```
And to switch between them with `gcloud config set account`:

```
% gcloud config set account me@gmail.com
Updated property [core/account].
```

html:
<p>I have two different Google Cloud accounts active at the moment. Here's how to list them with <code>gcloud auth list</code>:</p>
<pre><code>% gcloud auth list
    Credentialed Accounts
ACTIVE  ACCOUNT
        simon@example.com
*       me@gmail.com

To set the active account, run:
    $ gcloud config set account `ACCOUNT`
</code></pre>
<p>And to switch between them with <code>gcloud config set account</code>:</p>
<pre><code>% gcloud config set account me@gmail.com
Updated property [core/account].
</code></pre>


path: sql_finding-dupes-by-name-and-distance.md
topic: sql
title: Finding duplicate records by matching name and nearby distance
url: https://github.com/simonw/til/blob/main/sql/finding-dupes-by-name-and-distance.md
created: 2021-05-19T20:52:04-07:00
created_utc: 2021-05-20T03:52:04+00:00
updated: 2021-05-19T20:52:04-07:00
updated_utc: 2021-05-20T03:52:04+00:00
shot_hash: 83d5e34daae013bcb6d251bd816ea6b5
slug: finding-dupes-by-name-and-distance
body:
I wanted to find potentially duplicate records in my data, based on having the exact same name and being geographically located within 500 meters of each other.

This worked:
```sql
with potential_duplicates as (
  select
    a.id as one,
    b.id as two,
    ST_Distance(a.point, b.point) as distance_m
  from location a, location b 
    where a.name = b.name
    and a.id > b.id
    and ST_Distance(a.point, b.point) < 500
)
select * from potential_duplicates
```
I'm using a CTE here because it makes it easy to further customize the output with an additional query.

A few tricks in here:

- Alias the `location` twice as `a` and `b` in order to join against itself to find duplicates
- The `ST_Distance(a.point, b.point) < 500` clause returns locations within 500m of each other
- The `a.id > b.id` clause solves a problem I had with the first version of this query where each pairing was returned twice, with `one` and `two` swapped. By requiring `a` to have a higher `id` than `b` I avoid this problem entirely - and also prevent rows from matching themselves (where `a.id = b.id`).

html:
<p>I wanted to find potentially duplicate records in my data, based on having the exact same name and being geographically located within 500 meters of each other.</p>
<p>This worked:</p>
<div class="highlight highlight-source-sql"><pre>with potential_duplicates <span class="pl-k">as</span> (
  <span class="pl-k">select</span>
    <span class="pl-c1">a</span>.<span class="pl-c1">id</span> <span class="pl-k">as</span> one,
    <span class="pl-c1">b</span>.<span class="pl-c1">id</span> <span class="pl-k">as</span> two,
    ST_Distance(<span class="pl-c1">a</span>.<span class="pl-c1">point</span>, <span class="pl-c1">b</span>.<span class="pl-c1">point</span>) <span class="pl-k">as</span> distance_m
  <span class="pl-k">from</span> location a, location b 
    <span class="pl-k">where</span> <span class="pl-c1">a</span>.<span class="pl-c1">name</span> <span class="pl-k">=</span> <span class="pl-c1">b</span>.<span class="pl-c1">name</span>
    <span class="pl-k">and</span> <span class="pl-c1">a</span>.<span class="pl-c1">id</span> <span class="pl-k">&gt;</span> <span class="pl-c1">b</span>.<span class="pl-c1">id</span>
    <span class="pl-k">and</span> ST_Distance(<span class="pl-c1">a</span>.<span class="pl-c1">point</span>, <span class="pl-c1">b</span>.<span class="pl-c1">point</span>) <span class="pl-k">&lt;</span> <span class="pl-c1">500</span>
)
<span class="pl-k">select</span> <span class="pl-k">*</span> <span class="pl-k">from</span> potential_duplicates</pre></div>
<p>I'm using a CTE here because it makes it easy to further customize the output with an additional query.</p>
<p>A few tricks in here:</p>
<ul>
<li>Alias the <code>location</code> twice as <code>a</code> and <code>b</code> in order to join against itself to find duplicates</li>
<li>The <code>ST_Distance(a.point, b.point) &lt; 500</code> clause returns locations within 500m of each other</li>
<li>The <code>a.id &gt; b.id</code> clause solves a problem I had with the first version of this query where each pairing was returned twice, with <code>one</code> and <code>two</code> swapped. By requiring <code>a</code> to have a higher <code>id</code> than <code>b</code> I avoid this problem entirely - and also prevent rows from matching themselves (where <code>a.id = b.id</code>).</li>
</ul>


