# til

path: linux_allow-sudo-without-password-specific-command.md
topic: linux
title: Enabling a user to execute a specific command as root without a password
url: https://github.com/simonw/til/blob/main/linux/allow-sudo-without-password-specific-command.md
created: 2022-02-13T11:53:13-08:00
created_utc: 2022-02-13T19:53:13+00:00
updated: 2022-02-13T11:53:13-08:00
updated_utc: 2022-02-13T19:53:13+00:00
shot_hash: af91c24e82e83e745aac4bb3e1e3162f
slug: allow-sudo-without-password-specific-command
body:
I wanted a script running as a non-root user to be able to restart a systemd service on my Ubuntu machine without needing a password.

I figured out how to do that by adding the following line to the `sudoers` file, which can be edited as root using the `visudo` command:
```
# dogsheep user can restart datasette service
dogsheep  ALL = (root) NOPASSWD: /usr/bin/systemctl restart datasette
```
Having added this line, my `dogsheep` user account could now run the following:

```
$ sudo /usr/bin/systemctl restart datasette
```
But if it tries to run the command with any other arguments it gets prompted for a password:
```
$ sudo /usr/bin/systemctl restart datasette2
[sudo] password for dogsheep: 
```

html:
<p>I wanted a script running as a non-root user to be able to restart a systemd service on my Ubuntu machine without needing a password.</p>
<p>I figured out how to do that by adding the following line to the <code>sudoers</code> file, which can be edited as root using the <code>visudo</code> command:</p>
<pre><code># dogsheep user can restart datasette service
dogsheep  ALL = (root) NOPASSWD: /usr/bin/systemctl restart datasette
</code></pre>
<p>Having added this line, my <code>dogsheep</code> user account could now run the following:</p>
<pre><code>$ sudo /usr/bin/systemctl restart datasette
</code></pre>
<p>But if it tries to run the command with any other arguments it gets prompted for a password:</p>
<pre><code>$ sudo /usr/bin/systemctl restart datasette2
[sudo] password for dogsheep: 
</code></pre>


path: bash_nullglob-in-bash.md
topic: bash
title: nullglob in bash
url: https://github.com/simonw/til/blob/main/bash/nullglob-in-bash.md
created: 2022-02-14T21:16:05-08:00
created_utc: 2022-02-15T05:16:05+00:00
updated: 2022-02-14T21:16:05-08:00
updated_utc: 2022-02-15T05:16:05+00:00
shot_hash: 617dfd393565706d61b6bf41b1401c65
slug: nullglob-in-bash
body:
I ran into a tricky problem while working [on this issue](https://github.com/simonw/datasette-publish-fly/issues/17): the following line was behaving in an unexpected way for me:

    datasette content.db *.db --create

What I expect this to do is to create a `content.db` database if one does not exist, and then start Datasette with both that database and any other databases that exist in the directory.

The surprising behaviour occurred when the directory started off empty. Running the above in `bash` caused a file called `*.db` to be created in the directory.

It turns out if `bash` can't find any files matching a wildcard it passes that wildcard as a literal value to the underlying command!

`sh` does the same thing. `zsh` returns an error:

```
% datasette content.db *.db --create
zsh: no matches found: *.db
```
The solution, for `bash`, is to set the `nullglob` shell option. That can be done like this:

    shopt -s nullglob

This lasts for the rest of the interactive session, and causes `bash` to behave the way I expected it to, completely ignoring the `*.db` wildcard if it has no matches.

## Using this in a Dockerfile

I originally ran into this because I had a `Dockerfile` with a last line that looked like this:

`CMD datasette serve --host 0.0.0.0 --cors --inspect-file inspect-data.json --metadata metadata.json /data/tiddlywiki.db --create --port $PORT /data/*.db`

The goal here was to serve any existing databases in the `/data/` mounted volume, and to explicitly create that `tiddlywiki.db` database if it did not exist.

But it created a `*.db` database file if the folder was empty, due to the issue described above.

I ended up using this recipe to work around the problem:

`CMD ["/bin/bash", "-c", "shopt -s nullglob && datasette serve --host 0.0.0.0 --cors --inspect-file inspect-data.json /data/tiddlywiki.db --create --port $PORT /data/*.db"]`

This uses `CMD` to execute `/bin/bash` and pass it a one-liner that sets `nullglob` and then calls Datasette. This worked as intended.

html:
<p>I ran into a tricky problem while working <a href="https://github.com/simonw/datasette-publish-fly/issues/17">on this issue</a>: the following line was behaving in an unexpected way for me:</p>
<pre><code>datasette content.db *.db --create
</code></pre>
<p>What I expect this to do is to create a <code>content.db</code> database if one does not exist, and then start Datasette with both that database and any other databases that exist in the directory.</p>
<p>The surprising behaviour occurred when the directory started off empty. Running the above in <code>bash</code> caused a file called <code>*.db</code> to be created in the directory.</p>
<p>It turns out if <code>bash</code> can't find any files matching a wildcard it passes that wildcard as a literal value to the underlying command!</p>
<p><code>sh</code> does the same thing. <code>zsh</code> returns an error:</p>
<pre><code>% datasette content.db *.db --create
zsh: no matches found: *.db
</code></pre>
<p>The solution, for <code>bash</code>, is to set the <code>nullglob</code> shell option. That can be done like this:</p>
<pre><code>shopt -s nullglob
</code></pre>
<p>This lasts for the rest of the interactive session, and causes <code>bash</code> to behave the way I expected it to, completely ignoring the <code>*.db</code> wildcard if it has no matches.</p>
<h2>
<a id="user-content-using-this-in-a-dockerfile" class="anchor" href="#using-this-in-a-dockerfile" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using this in a Dockerfile</h2>
<p>I originally ran into this because I had a <code>Dockerfile</code> with a last line that looked like this:</p>
<p><code>CMD datasette serve --host 0.0.0.0 --cors --inspect-file inspect-data.json --metadata metadata.json /data/tiddlywiki.db --create --port $PORT /data/*.db</code></p>
<p>The goal here was to serve any existing databases in the <code>/data/</code> mounted volume, and to explicitly create that <code>tiddlywiki.db</code> database if it did not exist.</p>
<p>But it created a <code>*.db</code> database file if the folder was empty, due to the issue described above.</p>
<p>I ended up using this recipe to work around the problem:</p>
<p><code>CMD ["/bin/bash", "-c", "shopt -s nullglob &amp;&amp; datasette serve --host 0.0.0.0 --cors --inspect-file inspect-data.json /data/tiddlywiki.db --create --port $PORT /data/*.db"]</code></p>
<p>This uses <code>CMD</code> to execute <code>/bin/bash</code> and pass it a one-liner that sets <code>nullglob</code> and then calls Datasette. This worked as intended.</p>


path: sqlite_substr-instr.md
topic: sqlite
title: Combining substr and instr to extract text
url: https://github.com/simonw/til/blob/main/sqlite/substr-instr.md
created: 2022-02-15T18:05:02-08:00
created_utc: 2022-02-16T02:05:02+00:00
updated: 2022-02-15T18:05:02-08:00
updated_utc: 2022-02-16T02:05:02+00:00
shot_hash: e404c1bef14a89754ef78bb43f9ce90f
slug: substr-instr
body:
Derek Willis has [a Datasette instance full of political campaign emails](https://political-emails.herokuapp.com/) running on Heroku.

Matt Hodges [pointed out](https://twitter.com/hodgesmr/status/1493359553488076802) that [a lot of these emails](https://political-emails.herokuapp.com/emails?sql=select%0D%0A++body%0D%0Afrom%0D%0A++emails%0D%0Awhere%0D%0A++body+LIKE+%27%25refcode%25%27) include `refcode=` codes, which are used by ActBlue campaigns to track clicks.

They look like this:

- `...c-email?refcode=220210_FR_midmonth1kavin_plain] Jessica Mason...`
- `...hmp-footer?refcode=2021_footer&amount=25&a...`

I thought it would be fun to extract just the codes.

The [datasette-rure](https://datasette.io/plugins/datasette-rure) plugin adds regular expression support which can be used for this kind of thing, but in the absence of a plugin like that the only way to do it is with the SQLite [instr()](https://www.sqlite.org/lang_corefunc.html#instr) and [substr()](https://www.sqlite.org/lang_corefunc.html#substr) functions.

Here's the query I figured out:

```sql
with snippets as (
  select
    substr(body, instr(body, 'refcode=') + 8, 128) as snippet
  from
    emails
  where
    body LIKE '%refcode%'
),
refcodes as (
  select
    snippet,
    substr(
      snippet,
      0,
      min(
        case
          when instr(snippet, '&') > 0 then instr(snippet, '&')
          else 128
        end,
        case
          when instr(snippet, ']') > 0 then instr(snippet, ']')
          else 128
        end,
        case
          when instr(snippet, ' ') > 0 then instr(snippet, ' ')
          else 128
        end,
        case
          when instr(snippet, '.') > 0 then instr(snippet, '.')
          else 128
        end
      )
    ) as refcode
  from
    snippets
)
select
  refcode,
  count(*) as n
from
  refcodes
group by
  refcode
order by
  n desc
```

I started by pulling out just the 128 characters following each `refcode=` - I picked 128 characters at random just to make the data easier to look at:
```sql
    substr(body, instr(body, 'refcode=') + 8, 128) as snippet
```
`instr(body, 'refcode=') + 8` gives the character after the `=` sign, because `refcode=` is 8 characters long.

Next I needed to find the first character following the refcode that was either a `&`, a `]`, a `  space or a `.`. That's what this bit does:
```sql
    substr(
      snippet,
      0,
      min(
        case
          when instr(snippet, '&') > 0 then instr(snippet, '&')
          else 128
        end,
        case
          when instr(snippet, ']') > 0 then instr(snippet, ']')
          else 128
        end,
        case
          when instr(snippet, ' ') > 0 then instr(snippet, ' ')
          else 128
        end,
        case
          when instr(snippet, '.') > 0 then instr(snippet, '.')
          else 128
        end
      )
    ) as refcode
```
I'm trying to find the first instance of any of those characters - so I use `instr` to find them, but ignore any results where that returns `0` for "character not found" - in those cases I use the number `128` picked earlier. I can then grab the minimum of those scores.

Then finally I do a group-by/count to find the most common refcodes:

```sql
select
  refcode,
  count(*) as n
from
  refcodes
group by
  refcode
order by
  n desc
```

Top results were:

| refcode                                            |   n |
|----------------------------------------------------|-----|
| email_footer                                       | 527 |
| em_pt                                              | 352 |
| em_fr_2020                                         | 285 |
| pt                                                 | 254 |
| emfooter                                           | 242 |
| em_fr_2021                                         | 242 |
| footer-bio                                         | 198 |
| footer_button                                      | 192 |
| em_footer                                          | 173 |
| email-footer                                       | 168 |
|                                                    | 168 |
| footer                                             | 164 |
| em2021                                             | 135 |
| em_jc_fr_footer_link                               | 107 |
| em_fr_2019                                         | 107 |
| em-footer                                          | 104 |
| em_fr_2018                                         |  84 |
| ABD_EM_FR_2021                                     |  67 |

html:
<p>Derek Willis has <a href="https://political-emails.herokuapp.com/" rel="nofollow">a Datasette instance full of political campaign emails</a> running on Heroku.</p>
<p>Matt Hodges <a href="https://twitter.com/hodgesmr/status/1493359553488076802" rel="nofollow">pointed out</a> that <a href="https://political-emails.herokuapp.com/emails?sql=select%0D%0A++body%0D%0Afrom%0D%0A++emails%0D%0Awhere%0D%0A++body+LIKE+%27%25refcode%25%27" rel="nofollow">a lot of these emails</a> include <code>refcode=</code> codes, which are used by ActBlue campaigns to track clicks.</p>
<p>They look like this:</p>
<ul>
<li><code>...c-email?refcode=220210_FR_midmonth1kavin_plain] Jessica Mason...</code></li>
<li><code>...hmp-footer?refcode=2021_footer&amp;amount=25&amp;a...</code></li>
</ul>
<p>I thought it would be fun to extract just the codes.</p>
<p>The <a href="https://datasette.io/plugins/datasette-rure" rel="nofollow">datasette-rure</a> plugin adds regular expression support which can be used for this kind of thing, but in the absence of a plugin like that the only way to do it is with the SQLite <a href="https://www.sqlite.org/lang_corefunc.html#instr" rel="nofollow">instr()</a> and <a href="https://www.sqlite.org/lang_corefunc.html#substr" rel="nofollow">substr()</a> functions.</p>
<p>Here's the query I figured out:</p>
<div class="highlight highlight-source-sql"><pre>with snippets <span class="pl-k">as</span> (
  <span class="pl-k">select</span>
    substr(body, instr(body, <span class="pl-s"><span class="pl-pds">'</span>refcode=<span class="pl-pds">'</span></span>) <span class="pl-k">+</span> <span class="pl-c1">8</span>, <span class="pl-c1">128</span>) <span class="pl-k">as</span> snippet
  <span class="pl-k">from</span>
    emails
  <span class="pl-k">where</span>
    body <span class="pl-k">LIKE</span> <span class="pl-s"><span class="pl-pds">'</span>%refcode%<span class="pl-pds">'</span></span>
),
refcodes <span class="pl-k">as</span> (
  <span class="pl-k">select</span>
    snippet,
    substr(
      snippet,
      <span class="pl-c1">0</span>,
      <span class="pl-c1">min</span>(
        case
          when instr(snippet, <span class="pl-s"><span class="pl-pds">'</span>&amp;<span class="pl-pds">'</span></span>) <span class="pl-k">&gt;</span> <span class="pl-c1">0</span> then instr(snippet, <span class="pl-s"><span class="pl-pds">'</span>&amp;<span class="pl-pds">'</span></span>)
          else <span class="pl-c1">128</span>
        end,
        case
          when instr(snippet, <span class="pl-s"><span class="pl-pds">'</span>]<span class="pl-pds">'</span></span>) <span class="pl-k">&gt;</span> <span class="pl-c1">0</span> then instr(snippet, <span class="pl-s"><span class="pl-pds">'</span>]<span class="pl-pds">'</span></span>)
          else <span class="pl-c1">128</span>
        end,
        case
          when instr(snippet, <span class="pl-s"><span class="pl-pds">'</span> <span class="pl-pds">'</span></span>) <span class="pl-k">&gt;</span> <span class="pl-c1">0</span> then instr(snippet, <span class="pl-s"><span class="pl-pds">'</span> <span class="pl-pds">'</span></span>)
          else <span class="pl-c1">128</span>
        end,
        case
          when instr(snippet, <span class="pl-s"><span class="pl-pds">'</span>.<span class="pl-pds">'</span></span>) <span class="pl-k">&gt;</span> <span class="pl-c1">0</span> then instr(snippet, <span class="pl-s"><span class="pl-pds">'</span>.<span class="pl-pds">'</span></span>)
          else <span class="pl-c1">128</span>
        end
      )
    ) <span class="pl-k">as</span> refcode
  <span class="pl-k">from</span>
    snippets
)
<span class="pl-k">select</span>
  refcode,
  <span class="pl-c1">count</span>(<span class="pl-k">*</span>) <span class="pl-k">as</span> n
<span class="pl-k">from</span>
  refcodes
<span class="pl-k">group by</span>
  refcode
<span class="pl-k">order by</span>
  n <span class="pl-k">desc</span></pre></div>
<p>I started by pulling out just the 128 characters following each <code>refcode=</code> - I picked 128 characters at random just to make the data easier to look at:</p>
<div class="highlight highlight-source-sql"><pre>    substr(body, instr(body, <span class="pl-s"><span class="pl-pds">'</span>refcode=<span class="pl-pds">'</span></span>) <span class="pl-k">+</span> <span class="pl-c1">8</span>, <span class="pl-c1">128</span>) <span class="pl-k">as</span> snippet</pre></div>
<p><code>instr(body, 'refcode=') + 8</code> gives the character after the <code>=</code> sign, because <code>refcode=</code> is 8 characters long.</p>
<p>Next I needed to find the first character following the refcode that was either a <code>&amp;</code>, a <code>]</code>, a <code> space or a</code>.`. That's what this bit does:</p>
<div class="highlight highlight-source-sql"><pre>    substr(
      snippet,
      <span class="pl-c1">0</span>,
      <span class="pl-c1">min</span>(
        case
          when instr(snippet, <span class="pl-s"><span class="pl-pds">'</span>&amp;<span class="pl-pds">'</span></span>) <span class="pl-k">&gt;</span> <span class="pl-c1">0</span> then instr(snippet, <span class="pl-s"><span class="pl-pds">'</span>&amp;<span class="pl-pds">'</span></span>)
          else <span class="pl-c1">128</span>
        end,
        case
          when instr(snippet, <span class="pl-s"><span class="pl-pds">'</span>]<span class="pl-pds">'</span></span>) <span class="pl-k">&gt;</span> <span class="pl-c1">0</span> then instr(snippet, <span class="pl-s"><span class="pl-pds">'</span>]<span class="pl-pds">'</span></span>)
          else <span class="pl-c1">128</span>
        end,
        case
          when instr(snippet, <span class="pl-s"><span class="pl-pds">'</span> <span class="pl-pds">'</span></span>) <span class="pl-k">&gt;</span> <span class="pl-c1">0</span> then instr(snippet, <span class="pl-s"><span class="pl-pds">'</span> <span class="pl-pds">'</span></span>)
          else <span class="pl-c1">128</span>
        end,
        case
          when instr(snippet, <span class="pl-s"><span class="pl-pds">'</span>.<span class="pl-pds">'</span></span>) <span class="pl-k">&gt;</span> <span class="pl-c1">0</span> then instr(snippet, <span class="pl-s"><span class="pl-pds">'</span>.<span class="pl-pds">'</span></span>)
          else <span class="pl-c1">128</span>
        end
      )
    ) <span class="pl-k">as</span> refcode</pre></div>
<p>I'm trying to find the first instance of any of those characters - so I use <code>instr</code> to find them, but ignore any results where that returns <code>0</code> for "character not found" - in those cases I use the number <code>128</code> picked earlier. I can then grab the minimum of those scores.</p>
<p>Then finally I do a group-by/count to find the most common refcodes:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  refcode,
  <span class="pl-c1">count</span>(<span class="pl-k">*</span>) <span class="pl-k">as</span> n
<span class="pl-k">from</span>
  refcodes
<span class="pl-k">group by</span>
  refcode
<span class="pl-k">order by</span>
  n <span class="pl-k">desc</span></pre></div>
<p>Top results were:</p>
<table>
<thead>
<tr>
<th>refcode</th>
<th>n</th>
</tr>
</thead>
<tbody>
<tr>
<td>email_footer</td>
<td>527</td>
</tr>
<tr>
<td>em_pt</td>
<td>352</td>
</tr>
<tr>
<td>em_fr_2020</td>
<td>285</td>
</tr>
<tr>
<td>pt</td>
<td>254</td>
</tr>
<tr>
<td>emfooter</td>
<td>242</td>
</tr>
<tr>
<td>em_fr_2021</td>
<td>242</td>
</tr>
<tr>
<td>footer-bio</td>
<td>198</td>
</tr>
<tr>
<td>footer_button</td>
<td>192</td>
</tr>
<tr>
<td>em_footer</td>
<td>173</td>
</tr>
<tr>
<td>email-footer</td>
<td>168</td>
</tr>
<tr>
<td></td>
<td>168</td>
</tr>
<tr>
<td>footer</td>
<td>164</td>
</tr>
<tr>
<td>em2021</td>
<td>135</td>
</tr>
<tr>
<td>em_jc_fr_footer_link</td>
<td>107</td>
</tr>
<tr>
<td>em_fr_2019</td>
<td>107</td>
</tr>
<tr>
<td>em-footer</td>
<td>104</td>
</tr>
<tr>
<td>em_fr_2018</td>
<td>84</td>
</tr>
<tr>
<td>ABD_EM_FR_2021</td>
<td>67</td>
</tr>
</tbody>
</table>


path: googlecloud_google-oauth-cli-application.md
topic: googlecloud
title: Google OAuth for a CLI application
url: https://github.com/simonw/til/blob/main/googlecloud/google-oauth-cli-application.md
created: 2022-02-16T20:30:06-08:00
created_utc: 2022-02-17T04:30:06+00:00
updated: 2022-02-21T09:13:07-08:00
updated_utc: 2022-02-21T17:13:07+00:00
shot_hash: 530025116ece9a5f92bb3fcbfbf22f07
slug: google-oauth-cli-application
body:
> :warning: The OOB flow described in this document has been [scheduled for deprecation](https://developers.googleblog.com/2022/02/making-oauth-flows-safer.html?m=1#disallowed-oo) by Google. See [issue #39](https://github.com/simonw/google-drive-to-sqlite/issues/39) for notes on how to replace it.

I had to figure out how to do the OAuth flow while building my [google-drive-to-sqlite](https://github.com/simonw/google-drive-to-sqlite) CLI application - [issue here](https://github.com/simonw/google-drive-to-sqlite/issues/2).

For CLI apps, the flow is that you show the user a URL, they click on it, sign into their Google account, grant your application permission and the Google website then shows them a code. They copy and paste that code back into your application and you can exchange it for an `access_token` and a `refresh_token`. The `access_token` will work for an hour, but you can store the `refresh_token` and use it to obtain a new `access_token` any time you like.

## Setting up the OAuth app

In the Google Cloud console you need to navigate to "APIs and Services", "Credentials", "Create Credentials" and select "OAuth client ID". You need to create a client ID for application type "Desktop app".

![Screenshot of the Google Cloud API key interface](https://raw.githubusercontent.com/simonw/til/main/googlecloud/google-oauth-cli-application-oauth-client-id.png)

Completing this form will give you a `google_client_id` and a `google_client_secret`. Even though it's called a secret it's safe to distribute this in your application and include it in your code in a public GitHub repository. The [Google documentation](https://developers.google.com/identity/protocols/oauth2#installed) says:

> The process results in a client ID and, in some cases, a client secret, which you embed in the source code of your application. (In this context, the client secret is obviously not treated as a secret.)

## Providing an authentication link

The link the user clicks on should look like this (broken into multiple lines for readability):

    https://accounts.google.com/o/oauth2/v2/auth
      ?access_type=offline
      &client_id=YOUR_CLIENT_ID_HERE
      &redirect_uri=urn:ietf:wg:oauth:2.0:oob
      &response_type=code
      &scope=https://www.googleapis.com/auth/drive.readonly

In this case I am using a `scope` of `https://www.googleapis.com/auth/drive.readonly` because I want to access files in the user's Google Drive - you'll need to hunt around in the documentation to figure out which scope you need.

Here's a live demo link for my `google-drive-to-sqlite` application:

https://accounts.google.com/o/oauth2/v2/auth?access_type=offline&client_id=148933860554-98i3hter1bsn24sa6fcq1tcrhcrujrnl.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&response_type=code&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.readonly

Once the user completes that flow, they will be given a code to copy and paste. When they give you that code you can exchange it for tokens using this API (illustrated with Python):

```python
copied_code = '4/1A...'

import httpx
response = httpx.post("https://www.googleapis.com/oauth2/v4/token", data={
    "code": copied_code,
    "client_id": google_client_id,
    "client_secret": google_client_secret,
    "redirect_uri": "urn:ietf:wg:oauth:2.0:oob",
    "grant_type": "authorization_code",
})
tokens = response.json()
```
The `tokens` variable is now a Python dictionary with `access_token` and `refresh_token` keys.

## Making authenticated calls

You can use the `access_key` straight away for calls like this one:

```python
response = httpx.get("https://www.googleapis.com/drive/v3/files", headers={
    "Authorization": "Bearer {}".format(access_key)
})
print(response.json())
```

## Exchanging a refresh_token for an access_token

Here's my code for exchanging the `refresh_token` for a new `access_token`:

```python
data = httpx.post(
    "https://www.googleapis.com/oauth2/v4/token",
    data={
        "grant_type": "refresh_token",
        "refresh_token": refresh_token,
        "client_id": GOOGLE_CLIENT_ID,
        "client_secret": GOOGLE_CLIENT_SECRET,
    },
).json()
if "error" in data:
    raise click.ClickException(str(data))
return data["access_token"]
```
## The rest of my code

Most of my code that handles this can be found in the [cli.py module](https://github.com/simonw/google-drive-to-sqlite/blob/9377505af30366d57adafd1aa7846a2a6f4e990a/google_drive_to_sqlite/cli.py) in `google-drive-to-sqlite`.

html:
<blockquote>
<p><g-emoji class="g-emoji" alias="warning" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/26a0.png">⚠️</g-emoji> The OOB flow described in this document has been <a href="https://developers.googleblog.com/2022/02/making-oauth-flows-safer.html?m=1#disallowed-oo" rel="nofollow">scheduled for deprecation</a> by Google. See <a href="https://github.com/simonw/google-drive-to-sqlite/issues/39">issue #39</a> for notes on how to replace it.</p>
</blockquote>
<p>I had to figure out how to do the OAuth flow while building my <a href="https://github.com/simonw/google-drive-to-sqlite">google-drive-to-sqlite</a> CLI application - <a href="https://github.com/simonw/google-drive-to-sqlite/issues/2">issue here</a>.</p>
<p>For CLI apps, the flow is that you show the user a URL, they click on it, sign into their Google account, grant your application permission and the Google website then shows them a code. They copy and paste that code back into your application and you can exchange it for an <code>access_token</code> and a <code>refresh_token</code>. The <code>access_token</code> will work for an hour, but you can store the <code>refresh_token</code> and use it to obtain a new <code>access_token</code> any time you like.</p>
<h2>
<a id="user-content-setting-up-the-oauth-app" class="anchor" href="#setting-up-the-oauth-app" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Setting up the OAuth app</h2>
<p>In the Google Cloud console you need to navigate to "APIs and Services", "Credentials", "Create Credentials" and select "OAuth client ID". You need to create a client ID for application type "Desktop app".</p>
<p><a href="https://raw.githubusercontent.com/simonw/til/main/googlecloud/google-oauth-cli-application-oauth-client-id.png" target="_blank" rel="nofollow"><img src="https://raw.githubusercontent.com/simonw/til/main/googlecloud/google-oauth-cli-application-oauth-client-id.png" alt="Screenshot of the Google Cloud API key interface" style="max-width:100%;"></a></p>
<p>Completing this form will give you a <code>google_client_id</code> and a <code>google_client_secret</code>. Even though it's called a secret it's safe to distribute this in your application and include it in your code in a public GitHub repository. The <a href="https://developers.google.com/identity/protocols/oauth2#installed" rel="nofollow">Google documentation</a> says:</p>
<blockquote>
<p>The process results in a client ID and, in some cases, a client secret, which you embed in the source code of your application. (In this context, the client secret is obviously not treated as a secret.)</p>
</blockquote>
<h2>
<a id="user-content-providing-an-authentication-link" class="anchor" href="#providing-an-authentication-link" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Providing an authentication link</h2>
<p>The link the user clicks on should look like this (broken into multiple lines for readability):</p>
<pre><code>https://accounts.google.com/o/oauth2/v2/auth
  ?access_type=offline
  &amp;client_id=YOUR_CLIENT_ID_HERE
  &amp;redirect_uri=urn:ietf:wg:oauth:2.0:oob
  &amp;response_type=code
  &amp;scope=https://www.googleapis.com/auth/drive.readonly
</code></pre>
<p>In this case I am using a <code>scope</code> of <code>https://www.googleapis.com/auth/drive.readonly</code> because I want to access files in the user's Google Drive - you'll need to hunt around in the documentation to figure out which scope you need.</p>
<p>Here's a live demo link for my <code>google-drive-to-sqlite</code> application:</p>
<p><a href="https://accounts.google.com/o/oauth2/v2/auth?access_type=offline&amp;client_id=148933860554-98i3hter1bsn24sa6fcq1tcrhcrujrnl.apps.googleusercontent.com&amp;redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&amp;response_type=code&amp;scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.readonly" rel="nofollow">https://accounts.google.com/o/oauth2/v2/auth?access_type=offline&amp;client_id=148933860554-98i3hter1bsn24sa6fcq1tcrhcrujrnl.apps.googleusercontent.com&amp;redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&amp;response_type=code&amp;scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.readonly</a></p>
<p>Once the user completes that flow, they will be given a code to copy and paste. When they give you that code you can exchange it for tokens using this API (illustrated with Python):</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">copied_code</span> <span class="pl-c1">=</span> <span class="pl-s">'4/1A...'</span>

<span class="pl-k">import</span> <span class="pl-s1">httpx</span>
<span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">httpx</span>.<span class="pl-en">post</span>(<span class="pl-s">"https://www.googleapis.com/oauth2/v4/token"</span>, <span class="pl-s1">data</span><span class="pl-c1">=</span>{
    <span class="pl-s">"code"</span>: <span class="pl-s1">copied_code</span>,
    <span class="pl-s">"client_id"</span>: <span class="pl-s1">google_client_id</span>,
    <span class="pl-s">"client_secret"</span>: <span class="pl-s1">google_client_secret</span>,
    <span class="pl-s">"redirect_uri"</span>: <span class="pl-s">"urn:ietf:wg:oauth:2.0:oob"</span>,
    <span class="pl-s">"grant_type"</span>: <span class="pl-s">"authorization_code"</span>,
})
<span class="pl-s1">tokens</span> <span class="pl-c1">=</span> <span class="pl-s1">response</span>.<span class="pl-en">json</span>()</pre></div>
<p>The <code>tokens</code> variable is now a Python dictionary with <code>access_token</code> and <code>refresh_token</code> keys.</p>
<h2>
<a id="user-content-making-authenticated-calls" class="anchor" href="#making-authenticated-calls" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Making authenticated calls</h2>
<p>You can use the <code>access_key</code> straight away for calls like this one:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">httpx</span>.<span class="pl-en">get</span>(<span class="pl-s">"https://www.googleapis.com/drive/v3/files"</span>, <span class="pl-s1">headers</span><span class="pl-c1">=</span>{
    <span class="pl-s">"Authorization"</span>: <span class="pl-s">"Bearer {}"</span>.<span class="pl-en">format</span>(<span class="pl-s1">access_key</span>)
})
<span class="pl-en">print</span>(<span class="pl-s1">response</span>.<span class="pl-en">json</span>())</pre></div>
<h2>
<a id="user-content-exchanging-a-refresh_token-for-an-access_token" class="anchor" href="#exchanging-a-refresh_token-for-an-access_token" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Exchanging a refresh_token for an access_token</h2>
<p>Here's my code for exchanging the <code>refresh_token</code> for a new <code>access_token</code>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">httpx</span>.<span class="pl-en">post</span>(
    <span class="pl-s">"https://www.googleapis.com/oauth2/v4/token"</span>,
    <span class="pl-s1">data</span><span class="pl-c1">=</span>{
        <span class="pl-s">"grant_type"</span>: <span class="pl-s">"refresh_token"</span>,
        <span class="pl-s">"refresh_token"</span>: <span class="pl-s1">refresh_token</span>,
        <span class="pl-s">"client_id"</span>: <span class="pl-v">GOOGLE_CLIENT_ID</span>,
        <span class="pl-s">"client_secret"</span>: <span class="pl-v">GOOGLE_CLIENT_SECRET</span>,
    },
).<span class="pl-en">json</span>()
<span class="pl-k">if</span> <span class="pl-s">"error"</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>:
    <span class="pl-k">raise</span> <span class="pl-s1">click</span>.<span class="pl-v">ClickException</span>(<span class="pl-en">str</span>(<span class="pl-s1">data</span>))
<span class="pl-k">return</span> <span class="pl-s1">data</span>[<span class="pl-s">"access_token"</span>]</pre></div>
<h2>
<a id="user-content-the-rest-of-my-code" class="anchor" href="#the-rest-of-my-code" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>The rest of my code</h2>
<p>Most of my code that handles this can be found in the <a href="https://github.com/simonw/google-drive-to-sqlite/blob/9377505af30366d57adafd1aa7846a2a6f4e990a/google_drive_to_sqlite/cli.py">cli.py module</a> in <code>google-drive-to-sqlite</code>.</p>


path: googlecloud_recursive-fetch-google-drive.md
topic: googlecloud
title: Recursively fetching metadata for all files in a Google Drive folder
url: https://github.com/simonw/til/blob/main/googlecloud/recursive-fetch-google-drive.md
created: 2022-02-16T20:56:05-08:00
created_utc: 2022-02-17T04:56:05+00:00
updated: 2022-02-17T09:49:18-08:00
updated_utc: 2022-02-17T17:49:18+00:00
shot_hash: f82768dbf3ef4bae4dab5b40d36f75aa
slug: recursive-fetch-google-drive
body:
For [google-drive-to-sqlite](https://github.com/simonw/google-drive-to-sqlite) I wanted a mechanism to recursively return metadata on every file in a specified Goole Drive folder.

## Fetching files in a single folder

To fetch files in a single folder with the ID `FOLDER_ID` you need to use this API:

    https://www.googleapis.com/drive/v3/files?q=...

The `q=` parameter needs to be fed the following [special search query](https://developers.google.com/drive/api/v3/search-files):

    "FOLDER_ID" in parents

So the URL for a folder with ID `1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i` ends up looking like this:

    https://www.googleapis.com/drive/v3/files?q=%221E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i%22%20in%20parents

You need to accompany that with a `Authorization: Bearer YOUR_ACCESS_TOKEN` header - see [Google OAuth for CLI applications](https://til.simonwillison.net/googlecloud/google-oauth-cli-application) for more on that.

## Fetching specific fields

The metadata representation you get back by default is pretty thin.

You can add `&fields=*` to the URL to get back the full JSON representation - but this can actually be a little bit too much data, mainly because it includes a list of every user who has permissions relating to every file.

You can specify individual fields, but it's not instantly obvious how to do so. The syntax turns out to look like this:

    nextPageToken, files(id, kind, name)

Each token corresonds to a key, The `files(id, kind, name)` bit selects the fields you want from items in the `files` array.

The full list of file keys I've been using are:

```json
[
    "kind",
    "id",
    "name",
    "mimeType",
    "starred",
    "trashed",
    "explicitlyTrashed",
    "parents",
    "spaces",
    "version",
    "webViewLink",
    "iconLink",
    "hasThumbnail",
    "thumbnailVersion",
    "viewedByMe",
    "createdTime",
    "modifiedTime",
    "modifiedByMe",
    "owners",
    "lastModifyingUser",
    "shared",
    "ownedByMe",
    "viewersCanCopyContent",
    "copyRequiresWriterPermission",
    "writersCanShare",
    "folderColorRgb",
    "quotaBytesUsed",
    "isAppAuthorized",
    "linkShareMetadata",
]
```

## Recursively fetching from sub-folders

This will get you the files and folders within a folder... but you have to do extra work if you want to recursively fetch files from every sub-folder.

A sub-folder in the response from that API looks like this:

```json
{
    "kind": "drive#file",
    "id": "1ixwEcEUmZ5-RZY2AQrwG-s1rvOF9LtqX",
    "name": "My sub-folder",
    "mimeType": "application/vnd.google-apps.folder"
}
```
So fetching everything involves fetching the contents of a folder, then filtering out anything with a `mimeType` of `application/vnd.google-apps.folder`, extracting the `id` and recursively fetching the contents of that folder too.

You also need to handle pagination, in case a folder contains more than 100 items. That's done using the `nextPageToken` key in the response JSON, which you can then pass as `?pageToken=` to the API to retrieve the next page.

Note that while the Google Drive API implies that a file can live in more than one folder - `parents` is an array of IDs - Google Drive [simplified their model in September 2020](https://cloud.google.com/blog/products/g-suite/simplifying-google-drives-folder-structure-and-sharing-models) such that a file can only be in a single folder.

## Code

Here's [the code I used](https://github.com/simonw/google-drive-to-sqlite/blob/0.1a0/google_drive_to_sqlite/utils.py) to implement recursive fetching of folder contents:

```python
def paginate_files(access_token, *, q=None, fields=None):
    pageToken = None
    files_url = "https://www.googleapis.com/drive/v3/files"
    params = {}
    if fields is not None:
        params["fields"] = "nextPageToken, files({})".format(",".join(fields))
    if q is not None:
        params["q"] = q
    while True:
        if pageToken is not None:
            params["pageToken"] = pageToken
        else:
            params.pop("pageToken", None)
        data = httpx.get(
            files_url,
            params=params,
            headers={"Authorization": "Bearer {}".format(access_token)},
            timeout=30.0,
        ).json()
        if "error" in data:
            raise Exception(data)
        yield from data["files"]
        pageToken = data.get("nextPageToken", None)
        if pageToken is None:
            break


def files_in_folder_recursive(access_token, folder_id, fields):
    for file in paginate_files(
        access_token, q='"{}" in parents'.format(folder_id), fields=fields
    ):
        yield file
        if file["mimeType"] == "application/vnd.google-apps.folder":
            yield from files_in_folder_recursive(access_token, file["id"], fields)
```

html:
<p>For <a href="https://github.com/simonw/google-drive-to-sqlite">google-drive-to-sqlite</a> I wanted a mechanism to recursively return metadata on every file in a specified Goole Drive folder.</p>
<h2>
<a id="user-content-fetching-files-in-a-single-folder" class="anchor" href="#fetching-files-in-a-single-folder" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Fetching files in a single folder</h2>
<p>To fetch files in a single folder with the ID <code>FOLDER_ID</code> you need to use this API:</p>
<pre><code>https://www.googleapis.com/drive/v3/files?q=...
</code></pre>
<p>The <code>q=</code> parameter needs to be fed the following <a href="https://developers.google.com/drive/api/v3/search-files" rel="nofollow">special search query</a>:</p>
<pre><code>"FOLDER_ID" in parents
</code></pre>
<p>So the URL for a folder with ID <code>1E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i</code> ends up looking like this:</p>
<pre><code>https://www.googleapis.com/drive/v3/files?q=%221E6Zg2X2bjjtPzVfX8YqdXZDCoB3AVA7i%22%20in%20parents
</code></pre>
<p>You need to accompany that with a <code>Authorization: Bearer YOUR_ACCESS_TOKEN</code> header - see <a href="https://til.simonwillison.net/googlecloud/google-oauth-cli-application" rel="nofollow">Google OAuth for CLI applications</a> for more on that.</p>
<h2>
<a id="user-content-fetching-specific-fields" class="anchor" href="#fetching-specific-fields" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Fetching specific fields</h2>
<p>The metadata representation you get back by default is pretty thin.</p>
<p>You can add <code>&amp;fields=*</code> to the URL to get back the full JSON representation - but this can actually be a little bit too much data, mainly because it includes a list of every user who has permissions relating to every file.</p>
<p>You can specify individual fields, but it's not instantly obvious how to do so. The syntax turns out to look like this:</p>
<pre><code>nextPageToken, files(id, kind, name)
</code></pre>
<p>Each token corresonds to a key, The <code>files(id, kind, name)</code> bit selects the fields you want from items in the <code>files</code> array.</p>
<p>The full list of file keys I've been using are:</p>
<div class="highlight highlight-source-json"><pre>[
    <span class="pl-s"><span class="pl-pds">"</span>kind<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>mimeType<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>starred<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>trashed<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>explicitlyTrashed<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>parents<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>spaces<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>version<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>webViewLink<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>iconLink<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>hasThumbnail<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>thumbnailVersion<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>viewedByMe<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>createdTime<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>modifiedTime<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>modifiedByMe<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>owners<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>lastModifyingUser<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>shared<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>ownedByMe<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>viewersCanCopyContent<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>copyRequiresWriterPermission<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>writersCanShare<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>folderColorRgb<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>quotaBytesUsed<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>isAppAuthorized<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>linkShareMetadata<span class="pl-pds">"</span></span>,
]</pre></div>
<h2>
<a id="user-content-recursively-fetching-from-sub-folders" class="anchor" href="#recursively-fetching-from-sub-folders" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Recursively fetching from sub-folders</h2>
<p>This will get you the files and folders within a folder... but you have to do extra work if you want to recursively fetch files from every sub-folder.</p>
<p>A sub-folder in the response from that API looks like this:</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-ent">"kind"</span>: <span class="pl-s"><span class="pl-pds">"</span>drive#file<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"id"</span>: <span class="pl-s"><span class="pl-pds">"</span>1ixwEcEUmZ5-RZY2AQrwG-s1rvOF9LtqX<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>My sub-folder<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"mimeType"</span>: <span class="pl-s"><span class="pl-pds">"</span>application/vnd.google-apps.folder<span class="pl-pds">"</span></span>
}</pre></div>
<p>So fetching everything involves fetching the contents of a folder, then filtering out anything with a <code>mimeType</code> of <code>application/vnd.google-apps.folder</code>, extracting the <code>id</code> and recursively fetching the contents of that folder too.</p>
<p>You also need to handle pagination, in case a folder contains more than 100 items. That's done using the <code>nextPageToken</code> key in the response JSON, which you can then pass as <code>?pageToken=</code> to the API to retrieve the next page.</p>
<p>Note that while the Google Drive API implies that a file can live in more than one folder - <code>parents</code> is an array of IDs - Google Drive <a href="https://cloud.google.com/blog/products/g-suite/simplifying-google-drives-folder-structure-and-sharing-models" rel="nofollow">simplified their model in September 2020</a> such that a file can only be in a single folder.</p>
<h2>
<a id="user-content-code" class="anchor" href="#code" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Code</h2>
<p>Here's <a href="https://github.com/simonw/google-drive-to-sqlite/blob/0.1a0/google_drive_to_sqlite/utils.py">the code I used</a> to implement recursive fetching of folder contents:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">paginate_files</span>(<span class="pl-s1">access_token</span>, *, <span class="pl-s1">q</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">fields</span><span class="pl-c1">=</span><span class="pl-c1">None</span>):
    <span class="pl-s1">pageToken</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>
    <span class="pl-s1">files_url</span> <span class="pl-c1">=</span> <span class="pl-s">"https://www.googleapis.com/drive/v3/files"</span>
    <span class="pl-s1">params</span> <span class="pl-c1">=</span> {}
    <span class="pl-k">if</span> <span class="pl-s1">fields</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>:
        <span class="pl-s1">params</span>[<span class="pl-s">"fields"</span>] <span class="pl-c1">=</span> <span class="pl-s">"nextPageToken, files({})"</span>.<span class="pl-en">format</span>(<span class="pl-s">","</span>.<span class="pl-en">join</span>(<span class="pl-s1">fields</span>))
    <span class="pl-k">if</span> <span class="pl-s1">q</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>:
        <span class="pl-s1">params</span>[<span class="pl-s">"q"</span>] <span class="pl-c1">=</span> <span class="pl-s1">q</span>
    <span class="pl-k">while</span> <span class="pl-c1">True</span>:
        <span class="pl-k">if</span> <span class="pl-s1">pageToken</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>:
            <span class="pl-s1">params</span>[<span class="pl-s">"pageToken"</span>] <span class="pl-c1">=</span> <span class="pl-s1">pageToken</span>
        <span class="pl-k">else</span>:
            <span class="pl-s1">params</span>.<span class="pl-en">pop</span>(<span class="pl-s">"pageToken"</span>, <span class="pl-c1">None</span>)
        <span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">httpx</span>.<span class="pl-en">get</span>(
            <span class="pl-s1">files_url</span>,
            <span class="pl-s1">params</span><span class="pl-c1">=</span><span class="pl-s1">params</span>,
            <span class="pl-s1">headers</span><span class="pl-c1">=</span>{<span class="pl-s">"Authorization"</span>: <span class="pl-s">"Bearer {}"</span>.<span class="pl-en">format</span>(<span class="pl-s1">access_token</span>)},
            <span class="pl-s1">timeout</span><span class="pl-c1">=</span><span class="pl-c1">30.0</span>,
        ).<span class="pl-en">json</span>()
        <span class="pl-k">if</span> <span class="pl-s">"error"</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>:
            <span class="pl-k">raise</span> <span class="pl-v">Exception</span>(<span class="pl-s1">data</span>)
        <span class="pl-k">yield</span> <span class="pl-k">from</span> <span class="pl-s1">data</span>[<span class="pl-s">"files"</span>]
        <span class="pl-s1">pageToken</span> <span class="pl-c1">=</span> <span class="pl-s1">data</span>.<span class="pl-en">get</span>(<span class="pl-s">"nextPageToken"</span>, <span class="pl-c1">None</span>)
        <span class="pl-k">if</span> <span class="pl-s1">pageToken</span> <span class="pl-c1">is</span> <span class="pl-c1">None</span>:
            <span class="pl-k">break</span>


<span class="pl-k">def</span> <span class="pl-en">files_in_folder_recursive</span>(<span class="pl-s1">access_token</span>, <span class="pl-s1">folder_id</span>, <span class="pl-s1">fields</span>):
    <span class="pl-k">for</span> <span class="pl-s1">file</span> <span class="pl-c1">in</span> <span class="pl-en">paginate_files</span>(
        <span class="pl-s1">access_token</span>, <span class="pl-s1">q</span><span class="pl-c1">=</span><span class="pl-s">'"{}" in parents'</span>.<span class="pl-en">format</span>(<span class="pl-s1">folder_id</span>), <span class="pl-s1">fields</span><span class="pl-c1">=</span><span class="pl-s1">fields</span>
    ):
        <span class="pl-k">yield</span> <span class="pl-s1">file</span>
        <span class="pl-k">if</span> <span class="pl-s1">file</span>[<span class="pl-s">"mimeType"</span>] <span class="pl-c1">==</span> <span class="pl-s">"application/vnd.google-apps.folder"</span>:
            <span class="pl-k">yield</span> <span class="pl-k">from</span> <span class="pl-en">files_in_folder_recursive</span>(<span class="pl-s1">access_token</span>, <span class="pl-s1">file</span>[<span class="pl-s">"id"</span>], <span class="pl-s1">fields</span>)</pre></div>


path: graphql_graphql-with-curl.md
topic: graphql
title: Using curl to run GraphQL queries from the command line
url: https://github.com/simonw/til/blob/main/graphql/graphql-with-curl.md
created: 2022-02-21T16:11:12-08:00
created_utc: 2022-02-22T00:11:12+00:00
updated: 2022-02-21T16:59:31-08:00
updated_utc: 2022-02-22T00:59:31+00:00
shot_hash: 20657a418ecc01f8655596cffbf740f5
slug: graphql-with-curl
body:
I wanted to run a query against the GitHub GraphQL API using `curl` on the command line, while keeping the query itself as readable as possible. Here's the recipe I came up with (tested in both `bash` and `zsh`), with TOKEN replaced by my GitHub API personal access token:
```
curl -s https://api.github.com/graphql -X POST \
-H "Authorization: Bearer TOKEN" \
-H "Content-Type: application/json" \
-d "$(jq -c -n --arg query '
{
  search(type: REPOSITORY, query: "user:simonw topic:git-scraping", first: 100) {
    repositoryCount
    nodes {
      __typename
      ... on Repository {
        nameWithOwner
        description
        defaultBranchRef {
          name
          target {
            ... on Commit {
              committedDate
              url
              message
            }
          }
        }
      }
    }
  }
}' '{"query":$query}')"
```
As you can see, the GraphQL query itself is embedded in plain text inside a complex set of escaping tricks.

## Building a JSON document with jq

I needed to encode the query as part of a JSON document that looks like this:

```json
{"query":"\n{\n  search(type: REPOSITORY, query: \"user:simonw topic:git-scraping\", first: 100) {\n    repositoryCount\n    nodes {\n      __typename\n      ... on Repository {\n        nameWithOwner\n        description\n        defaultBranchRef {\n          name\n          target {\n            ... on Commit {\n              committedDate\n              url\n              message\n            }\n          }\n        }\n      }\n    }\n  }\n}"}
```
The `jq` recipe handles the construction of that document for me:
```
jq -c -n --arg query '
{
  search(type: REPOSITORY, query: "user:simonw topic:git-scraping", first: 100) {
    repositoryCount
    nodes {
      __typename
      ... on Repository {
        nameWithOwner
        description
        defaultBranchRef {
          name
          target {
            ... on Commit {
              committedDate
              url
              message
            }
          }
        }
      }
    }
  }
}' '{"query":$query}'
```
`jq -c` means "compact syntax" - so the JSON itself is produced as a single line.

The `-n` option stands for `--null-input` - which is described as:

> Don´t read any input at all! Instead, the filter is run once using null as the input. This is useful when using `jq` as a simple calculator or to construct JSON data from scratch.

Then `--arg query '...'` sets a variable within jQuery to the string representing my GraphQL query.

Finally I evaluate the jQuery expression `'{"query":$query}'` which constructs the final document with my GraphQL query as the value for the `"query"` key.

## Passing that to curl with "$()"

Having constructed the JSON document, I needed to pass it to the `curl -d` option to submit it to the server.

The recipe for doing that is:

```
-d "$(jq -c -n --arg query ...)"
```

I tried doing this with `-d $(jq ...)` first, and it didn't work - because whitespace inside the substition was treated as separate tokens passed to `curl`.

Adding the wrapping double quotes caused the substition result to be treated as a single value.

I was worried that double quotes within the string itself would break out of the pattern, but [this Stackoverflow answer](https://unix.stackexchange.com/questions/289574/nested-double-quotes-in-assignment-with-command-substitut) reassured me otherwise:

> Once one is inside `$(...)`, quoting starts all over from scratch.
>
> In other words, `"..."` and `$(...)` can nest within each other. Command substitution, `$(...)`, can contain one or more complete double-quoted strings.

The resulting (but truncated) JSON from the GraphQL query looks like this:

```json
{
  "data": {
    "search": {
      "repositoryCount": 22,
      "nodes": [
        {
          "__typename": "Repository",
          "nameWithOwner": "simonw/csv-diff",
          "description": "Python CLI tool and library for diffing CSV and JSON files",
          "defaultBranchRef": {
            "name": "main",
            "target": {
              "committedDate": "2021-02-23T02:53:11Z",
              "url": "https://github.com/simonw/csv-diff/commit/33e0a5918283c02a339a1fb507fc7a9cda89a198",
              "message": "Handle missing JSON keys, refs #13"
            }
          }
        },
```

## Combining it with sqlite-utils insert

My end goal was to create a SQLite database with a record for each of my GitHub repositories that were tagged `git-scraping` that included the date of their most recent commit. Here's how I did that:

```
curl https://api.github.com/graphql -X POST \
-H "Authorization: Bearer ..." \
-H "Content-Type: application/json" \
-d "$(jq -c -n --arg query '
{
  search(type: REPOSITORY, query: "user:simonw topic:git-scraping", first: 100) {
    repositoryCount
    nodes {
      __typename
      ... on Repository {
        nameWithOwner
        description
        defaultBranchRef {
          name
          target {
            ... on Commit {
              committedDate
              url
              message
            }
          }
        }
      }
    }
  }
}' '{"query":$query}')" \
  | jq .data.search.nodes | sqlite-utils insert /tmp/github.db repos - --flatten
```
The line doing the work at the end is:

    | jq .data.search.nodes | sqlite-utils insert /tmp/github.db repos - --flatten

This uses `jq` to pull out the `{"data": {"search": {"nodes": [...]` array from the returned JSON, then pipes that into [sqlite-utils insert](https://sqlite-utils.datasette.io/en/stable/cli.html#cli-inserting-data).

This line does the rest:

    sqlite-utils insert /tmp/github.db repos - --flatten

That reads from standard input (`-`) and creates a `repos` table in the new `github.db` SQLite file.

The `--flatten` option at the end ensures that nested fields such as `{"defaultBranchRef": {"target": {"committedDate": ...` are flattened to columns with names like `defaultBranchRef_target_committedDate`.

The final table schema looks like this:

```
% sqlite-utils schema /tmp/github.db 
CREATE TABLE [repos] (
   [__typename] TEXT,
   [nameWithOwner] TEXT,
   [description] TEXT,
   [defaultBranchRef_name] TEXT,
   [defaultBranchRef_target_committedDate] TEXT,
   [defaultBranchRef_target_url] TEXT,
   [defaultBranchRef_target_message] TEXT
);
```

html:
<p>I wanted to run a query against the GitHub GraphQL API using <code>curl</code> on the command line, while keeping the query itself as readable as possible. Here's the recipe I came up with (tested in both <code>bash</code> and <code>zsh</code>), with TOKEN replaced by my GitHub API personal access token:</p>
<pre><code>curl -s https://api.github.com/graphql -X POST \
-H "Authorization: Bearer TOKEN" \
-H "Content-Type: application/json" \
-d "$(jq -c -n --arg query '
{
  search(type: REPOSITORY, query: "user:simonw topic:git-scraping", first: 100) {
    repositoryCount
    nodes {
      __typename
      ... on Repository {
        nameWithOwner
        description
        defaultBranchRef {
          name
          target {
            ... on Commit {
              committedDate
              url
              message
            }
          }
        }
      }
    }
  }
}' '{"query":$query}')"
</code></pre>
<p>As you can see, the GraphQL query itself is embedded in plain text inside a complex set of escaping tricks.</p>
<h2>
<a id="user-content-building-a-json-document-with-jq" class="anchor" href="#building-a-json-document-with-jq" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Building a JSON document with jq</h2>
<p>I needed to encode the query as part of a JSON document that looks like this:</p>
<div class="highlight highlight-source-json"><pre>{<span class="pl-ent">"query"</span>:<span class="pl-s"><span class="pl-pds">"</span><span class="pl-cce">\n</span>{<span class="pl-cce">\n</span>  search(type: REPOSITORY, query: <span class="pl-cce">\"</span>user:simonw topic:git-scraping<span class="pl-cce">\"</span>, first: 100) {<span class="pl-cce">\n</span>    repositoryCount<span class="pl-cce">\n</span>    nodes {<span class="pl-cce">\n</span>      __typename<span class="pl-cce">\n</span>      ... on Repository {<span class="pl-cce">\n</span>        nameWithOwner<span class="pl-cce">\n</span>        description<span class="pl-cce">\n</span>        defaultBranchRef {<span class="pl-cce">\n</span>          name<span class="pl-cce">\n</span>          target {<span class="pl-cce">\n</span>            ... on Commit {<span class="pl-cce">\n</span>              committedDate<span class="pl-cce">\n</span>              url<span class="pl-cce">\n</span>              message<span class="pl-cce">\n</span>            }<span class="pl-cce">\n</span>          }<span class="pl-cce">\n</span>        }<span class="pl-cce">\n</span>      }<span class="pl-cce">\n</span>    }<span class="pl-cce">\n</span>  }<span class="pl-cce">\n</span>}<span class="pl-pds">"</span></span>}</pre></div>
<p>The <code>jq</code> recipe handles the construction of that document for me:</p>
<pre><code>jq -c -n --arg query '
{
  search(type: REPOSITORY, query: "user:simonw topic:git-scraping", first: 100) {
    repositoryCount
    nodes {
      __typename
      ... on Repository {
        nameWithOwner
        description
        defaultBranchRef {
          name
          target {
            ... on Commit {
              committedDate
              url
              message
            }
          }
        }
      }
    }
  }
}' '{"query":$query}'
</code></pre>
<p><code>jq -c</code> means "compact syntax" - so the JSON itself is produced as a single line.</p>
<p>The <code>-n</code> option stands for <code>--null-input</code> - which is described as:</p>
<blockquote>
<p>Don´t read any input at all! Instead, the filter is run once using null as the input. This is useful when using <code>jq</code> as a simple calculator or to construct JSON data from scratch.</p>
</blockquote>
<p>Then <code>--arg query '...'</code> sets a variable within jQuery to the string representing my GraphQL query.</p>
<p>Finally I evaluate the jQuery expression <code>'{"query":$query}'</code> which constructs the final document with my GraphQL query as the value for the <code>"query"</code> key.</p>
<h2>
<a id="user-content-passing-that-to-curl-with-" class="anchor" href="#passing-that-to-curl-with-" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Passing that to curl with "$()"</h2>
<p>Having constructed the JSON document, I needed to pass it to the <code>curl -d</code> option to submit it to the server.</p>
<p>The recipe for doing that is:</p>
<pre><code>-d "$(jq -c -n --arg query ...)"
</code></pre>
<p>I tried doing this with <code>-d $(jq ...)</code> first, and it didn't work - because whitespace inside the substition was treated as separate tokens passed to <code>curl</code>.</p>
<p>Adding the wrapping double quotes caused the substition result to be treated as a single value.</p>
<p>I was worried that double quotes within the string itself would break out of the pattern, but <a href="https://unix.stackexchange.com/questions/289574/nested-double-quotes-in-assignment-with-command-substitut" rel="nofollow">this Stackoverflow answer</a> reassured me otherwise:</p>
<blockquote>
<p>Once one is inside <code>$(...)</code>, quoting starts all over from scratch.</p>
<p>In other words, <code>"..."</code> and <code>$(...)</code> can nest within each other. Command substitution, <code>$(...)</code>, can contain one or more complete double-quoted strings.</p>
</blockquote>
<p>The resulting (but truncated) JSON from the GraphQL query looks like this:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-ent">"data"</span>: {
    <span class="pl-ent">"search"</span>: {
      <span class="pl-ent">"repositoryCount"</span>: <span class="pl-c1">22</span>,
      <span class="pl-ent">"nodes"</span>: [
        {
          <span class="pl-ent">"__typename"</span>: <span class="pl-s"><span class="pl-pds">"</span>Repository<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"nameWithOwner"</span>: <span class="pl-s"><span class="pl-pds">"</span>simonw/csv-diff<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"description"</span>: <span class="pl-s"><span class="pl-pds">"</span>Python CLI tool and library for diffing CSV and JSON files<span class="pl-pds">"</span></span>,
          <span class="pl-ent">"defaultBranchRef"</span>: {
            <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>main<span class="pl-pds">"</span></span>,
            <span class="pl-ent">"target"</span>: {
              <span class="pl-ent">"committedDate"</span>: <span class="pl-s"><span class="pl-pds">"</span>2021-02-23T02:53:11Z<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"url"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://github.com/simonw/csv-diff/commit/33e0a5918283c02a339a1fb507fc7a9cda89a198<span class="pl-pds">"</span></span>,
              <span class="pl-ent">"message"</span>: <span class="pl-s"><span class="pl-pds">"</span>Handle missing JSON keys, refs #13<span class="pl-pds">"</span></span>
            }
          }
        },</pre></div>
<h2>
<a id="user-content-combining-it-with-sqlite-utils-insert" class="anchor" href="#combining-it-with-sqlite-utils-insert" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Combining it with sqlite-utils insert</h2>
<p>My end goal was to create a SQLite database with a record for each of my GitHub repositories that were tagged <code>git-scraping</code> that included the date of their most recent commit. Here's how I did that:</p>
<pre><code>curl https://api.github.com/graphql -X POST \
-H "Authorization: Bearer ..." \
-H "Content-Type: application/json" \
-d "$(jq -c -n --arg query '
{
  search(type: REPOSITORY, query: "user:simonw topic:git-scraping", first: 100) {
    repositoryCount
    nodes {
      __typename
      ... on Repository {
        nameWithOwner
        description
        defaultBranchRef {
          name
          target {
            ... on Commit {
              committedDate
              url
              message
            }
          }
        }
      }
    }
  }
}' '{"query":$query}')" \
  | jq .data.search.nodes | sqlite-utils insert /tmp/github.db repos - --flatten
</code></pre>
<p>The line doing the work at the end is:</p>
<pre><code>| jq .data.search.nodes | sqlite-utils insert /tmp/github.db repos - --flatten
</code></pre>
<p>This uses <code>jq</code> to pull out the <code>{"data": {"search": {"nodes": [...]</code> array from the returned JSON, then pipes that into <a href="https://sqlite-utils.datasette.io/en/stable/cli.html#cli-inserting-data" rel="nofollow">sqlite-utils insert</a>.</p>
<p>This line does the rest:</p>
<pre><code>sqlite-utils insert /tmp/github.db repos - --flatten
</code></pre>
<p>That reads from standard input (<code>-</code>) and creates a <code>repos</code> table in the new <code>github.db</code> SQLite file.</p>
<p>The <code>--flatten</code> option at the end ensures that nested fields such as <code>{"defaultBranchRef": {"target": {"committedDate": ...</code> are flattened to columns with names like <code>defaultBranchRef_target_committedDate</code>.</p>
<p>The final table schema looks like this:</p>
<pre><code>% sqlite-utils schema /tmp/github.db 
CREATE TABLE [repos] (
   [__typename] TEXT,
   [nameWithOwner] TEXT,
   [description] TEXT,
   [defaultBranchRef_name] TEXT,
   [defaultBranchRef_target_committedDate] TEXT,
   [defaultBranchRef_target_url] TEXT,
   [defaultBranchRef_target_message] TEXT
);
</code></pre>


path: datasette_crawling-datasette-with-datasette.md
topic: datasette
title: Crawling Datasette with Datasette
url: https://github.com/simonw/til/blob/main/datasette/crawling-datasette-with-datasette.md
created: 2022-02-27T22:37:16-08:00
created_utc: 2022-02-28T06:37:16+00:00
updated: 2022-02-27T22:37:16-08:00
updated_utc: 2022-02-28T06:37:16+00:00
shot_hash: 20fa7576084d0bd54463685c205d2be5
slug: crawling-datasette-with-datasette
body:
I wanted to add the new tutorials on https://datasette.io/tutorials to the search index that is used by the https://datasette.io/-/beta search engine.

To do this, I needed the content of those tutorials in a SQLite database table. But the tutorials are implemented as static pages in [templates/pages/tutorials](https://github.com/simonw/datasette.io/tree/9dffe361b0210b9d8b1f2fb820a3f2193f0f2fc7/templates/pages/tutorials) - so I needed to crawl that content and insert it into a table.

I ended up using a combination of the `datasette.client` mechanism ([documented here](https://docs.datasette.io/en/stable/internals.html#internals-datasette-client)), [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) and [sqlite-utils](https://sqlite-utils.readthedocs.io/) - all wrapped up in [a Python script](https://github.com/simonw/datasette.io/blob/9dffe361b0210b9d8b1f2fb820a3f2193f0f2fc7/index_tutorials.py) that's now called as part of [the GitHub Actions build process](https://github.com/simonw/datasette.io/blob/9dffe361b0210b9d8b1f2fb820a3f2193f0f2fc7/scripts/build.sh#L35) for the site.

I'm also using [configuration directory mode](https://docs.datasette.io/en/stable/settings.html#config-dir).

Here's the annotated script:

```python
import asyncio
from bs4 import BeautifulSoup as Soup
from datasette.app import Datasette
import pathlib
import sqlite_utils

# This is an async def function because it needs to call await ds.client
async def main():
    db = sqlite_utils.Database("content.db")
    # We need to simulate the full https://datasette.io/ site - including all
    # of its custom templates and plugins. On the command-line we would do this
    # by running "datasette ." - using configuration directory mode. This is
    # the equivalent of that when constructing the Datasette object directly:
    ds = Datasette(config_dir=pathlib.Path("."))
    # Equivalent of fetching the HTML from https://datasette.io/tutorials
    index_response = await ds.client.get("/tutorials")
    index_soup = Soup(index_response.text, "html5lib")
    # We want to crawl the links inside <div class="content"><ul>...<a href="">
    tutorial_links = index_soup.select(".content ul a")
    for link in tutorial_links:
        # For each one fetch the HTML, e.g. from /tutorials/learn-sql
        tutorial_response = await ds.client.get(link["href"])
        # The script should fail loudly if it encounters a broken link
        assert tutorial_response.status_code == 200
        # Now we can parse the page and extract the <h1> and <div class="content">
        soup = Soup(tutorial_response.text, "html5lib")
        # Beautiful Soup makes extracting text easy:
        title = soup.select("h1")[0].text
        body = soup.select(".content")[0].text
        # Insert this into the "tutorials" table, creating it if it does not exist
        db["tutorials"].insert(
            {
                "path": link["href"],
                "title": title,
                "body": body.strip(),
            },
            # Treat path, e.g. /tutorials/learn-sql, as the primary key
            pk="path",
            # This will over-write any existing records with the same path
            replace=True,
        )


if __name__ == "__main__":
    # This idiom executes the async function in an event loop:
    asyncio.run(main())
```
It's then added to the search index by this [Dogsheep Beta](https://datasette.io/tools/dogsheep-beta) search configuration [fragment](https://github.com/simonw/datasette.io/blob/9dffe361b0210b9d8b1f2fb820a3f2193f0f2fc7/templates/dogsheep-beta.yml#L209-L229):
```yaml
content.db:
    tutorials:
        sql: |-
            select
              path as key,
              title,
              body as search_1,
              1 as is_public
            from
              tutorials
        display_sql: |-
            select
              highlight(
                body, :q
              ) as snippet
            from
              tutorials
            where
              tutorials.path = :key
        display: |-
            <h3>Tutorial: <a href="{{ key }}">{{ title }}</a></h3>
            <p>{{ display.snippet|safe }}</p>
```
See [Building a search engine for datasette.io](https://simonwillison.net/2020/Dec/19/dogsheep-beta/) for more details on exactly how this works.

html:
<p>I wanted to add the new tutorials on <a href="https://datasette.io/tutorials" rel="nofollow">https://datasette.io/tutorials</a> to the search index that is used by the <a href="https://datasette.io/-/beta" rel="nofollow">https://datasette.io/-/beta</a> search engine.</p>
<p>To do this, I needed the content of those tutorials in a SQLite database table. But the tutorials are implemented as static pages in <a href="https://github.com/simonw/datasette.io/tree/9dffe361b0210b9d8b1f2fb820a3f2193f0f2fc7/templates/pages/tutorials">templates/pages/tutorials</a> - so I needed to crawl that content and insert it into a table.</p>
<p>I ended up using a combination of the <code>datasette.client</code> mechanism (<a href="https://docs.datasette.io/en/stable/internals.html#internals-datasette-client" rel="nofollow">documented here</a>), <a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/" rel="nofollow">Beautiful Soup</a> and <a href="https://sqlite-utils.readthedocs.io/" rel="nofollow">sqlite-utils</a> - all wrapped up in <a href="https://github.com/simonw/datasette.io/blob/9dffe361b0210b9d8b1f2fb820a3f2193f0f2fc7/index_tutorials.py">a Python script</a> that's now called as part of <a href="https://github.com/simonw/datasette.io/blob/9dffe361b0210b9d8b1f2fb820a3f2193f0f2fc7/scripts/build.sh#L35">the GitHub Actions build process</a> for the site.</p>
<p>I'm also using <a href="https://docs.datasette.io/en/stable/settings.html#config-dir" rel="nofollow">configuration directory mode</a>.</p>
<p>Here's the annotated script:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">asyncio</span>
<span class="pl-k">from</span> <span class="pl-s1">bs4</span> <span class="pl-k">import</span> <span class="pl-v">BeautifulSoup</span> <span class="pl-k">as</span> <span class="pl-v">Soup</span>
<span class="pl-k">from</span> <span class="pl-s1">datasette</span>.<span class="pl-s1">app</span> <span class="pl-k">import</span> <span class="pl-v">Datasette</span>
<span class="pl-k">import</span> <span class="pl-s1">pathlib</span>
<span class="pl-k">import</span> <span class="pl-s1">sqlite_utils</span>

<span class="pl-c"># This is an async def function because it needs to call await ds.client</span>
<span class="pl-k">async</span> <span class="pl-k">def</span> <span class="pl-en">main</span>():
    <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite_utils</span>.<span class="pl-v">Database</span>(<span class="pl-s">"content.db"</span>)
    <span class="pl-c"># We need to simulate the full https://datasette.io/ site - including all</span>
    <span class="pl-c"># of its custom templates and plugins. On the command-line we would do this</span>
    <span class="pl-c"># by running "datasette ." - using configuration directory mode. This is</span>
    <span class="pl-c"># the equivalent of that when constructing the Datasette object directly:</span>
    <span class="pl-s1">ds</span> <span class="pl-c1">=</span> <span class="pl-v">Datasette</span>(<span class="pl-s1">config_dir</span><span class="pl-c1">=</span><span class="pl-s1">pathlib</span>.<span class="pl-v">Path</span>(<span class="pl-s">"."</span>))
    <span class="pl-c"># Equivalent of fetching the HTML from https://datasette.io/tutorials</span>
    <span class="pl-s1">index_response</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">ds</span>.<span class="pl-s1">client</span>.<span class="pl-en">get</span>(<span class="pl-s">"/tutorials"</span>)
    <span class="pl-s1">index_soup</span> <span class="pl-c1">=</span> <span class="pl-v">Soup</span>(<span class="pl-s1">index_response</span>.<span class="pl-s1">text</span>, <span class="pl-s">"html5lib"</span>)
    <span class="pl-c"># We want to crawl the links inside &lt;div class="content"&gt;&lt;ul&gt;...&lt;a href=""&gt;</span>
    <span class="pl-s1">tutorial_links</span> <span class="pl-c1">=</span> <span class="pl-s1">index_soup</span>.<span class="pl-en">select</span>(<span class="pl-s">".content ul a"</span>)
    <span class="pl-k">for</span> <span class="pl-s1">link</span> <span class="pl-c1">in</span> <span class="pl-s1">tutorial_links</span>:
        <span class="pl-c"># For each one fetch the HTML, e.g. from /tutorials/learn-sql</span>
        <span class="pl-s1">tutorial_response</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">ds</span>.<span class="pl-s1">client</span>.<span class="pl-en">get</span>(<span class="pl-s1">link</span>[<span class="pl-s">"href"</span>])
        <span class="pl-c"># The script should fail loudly if it encounters a broken link</span>
        <span class="pl-k">assert</span> <span class="pl-s1">tutorial_response</span>.<span class="pl-s1">status_code</span> <span class="pl-c1">==</span> <span class="pl-c1">200</span>
        <span class="pl-c"># Now we can parse the page and extract the &lt;h1&gt; and &lt;div class="content"&gt;</span>
        <span class="pl-s1">soup</span> <span class="pl-c1">=</span> <span class="pl-v">Soup</span>(<span class="pl-s1">tutorial_response</span>.<span class="pl-s1">text</span>, <span class="pl-s">"html5lib"</span>)
        <span class="pl-c"># Beautiful Soup makes extracting text easy:</span>
        <span class="pl-s1">title</span> <span class="pl-c1">=</span> <span class="pl-s1">soup</span>.<span class="pl-en">select</span>(<span class="pl-s">"h1"</span>)[<span class="pl-c1">0</span>].<span class="pl-s1">text</span>
        <span class="pl-s1">body</span> <span class="pl-c1">=</span> <span class="pl-s1">soup</span>.<span class="pl-en">select</span>(<span class="pl-s">".content"</span>)[<span class="pl-c1">0</span>].<span class="pl-s1">text</span>
        <span class="pl-c"># Insert this into the "tutorials" table, creating it if it does not exist</span>
        <span class="pl-s1">db</span>[<span class="pl-s">"tutorials"</span>].<span class="pl-en">insert</span>(
            {
                <span class="pl-s">"path"</span>: <span class="pl-s1">link</span>[<span class="pl-s">"href"</span>],
                <span class="pl-s">"title"</span>: <span class="pl-s1">title</span>,
                <span class="pl-s">"body"</span>: <span class="pl-s1">body</span>.<span class="pl-en">strip</span>(),
            },
            <span class="pl-c"># Treat path, e.g. /tutorials/learn-sql, as the primary key</span>
            <span class="pl-s1">pk</span><span class="pl-c1">=</span><span class="pl-s">"path"</span>,
            <span class="pl-c"># This will over-write any existing records with the same path</span>
            <span class="pl-s1">replace</span><span class="pl-c1">=</span><span class="pl-c1">True</span>,
        )


<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">"__main__"</span>:
    <span class="pl-c"># This idiom executes the async function in an event loop:</span>
    <span class="pl-s1">asyncio</span>.<span class="pl-en">run</span>(<span class="pl-en">main</span>())</pre></div>
<p>It's then added to the search index by this <a href="https://datasette.io/tools/dogsheep-beta" rel="nofollow">Dogsheep Beta</a> search configuration <a href="https://github.com/simonw/datasette.io/blob/9dffe361b0210b9d8b1f2fb820a3f2193f0f2fc7/templates/dogsheep-beta.yml#L209-L229">fragment</a>:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">content.db</span>:
    <span class="pl-ent">tutorials</span>:
        <span class="pl-ent">sql</span>: <span class="pl-s">|-</span>
<span class="pl-s">            select</span>
<span class="pl-s">              path as key,</span>
<span class="pl-s">              title,</span>
<span class="pl-s">              body as search_1,</span>
<span class="pl-s">              1 as is_public</span>
<span class="pl-s">            from</span>
<span class="pl-s">              tutorials</span>
<span class="pl-s"></span>        <span class="pl-ent">display_sql</span>: <span class="pl-s">|-</span>
<span class="pl-s">            select</span>
<span class="pl-s">              highlight(</span>
<span class="pl-s">                body, :q</span>
<span class="pl-s">              ) as snippet</span>
<span class="pl-s">            from</span>
<span class="pl-s">              tutorials</span>
<span class="pl-s">            where</span>
<span class="pl-s">              tutorials.path = :key</span>
<span class="pl-s"></span>        <span class="pl-ent">display</span>: <span class="pl-s">|-</span>
<span class="pl-s">            &lt;h3&gt;Tutorial: &lt;a href="{{ key }}"&gt;{{ title }}&lt;/a&gt;&lt;/h3&gt;</span>
<span class="pl-s">            &lt;p&gt;{{ display.snippet|safe }}&lt;/p&gt;</span></pre></div>
<p>See <a href="https://simonwillison.net/2020/Dec/19/dogsheep-beta/" rel="nofollow">Building a search engine for datasette.io</a> for more details on exactly how this works.</p>


path: homebrew_latest-sqlite.md
topic: homebrew
title: Running the latest SQLite in Datasette using Homebrew
url: https://github.com/simonw/til/blob/main/homebrew/latest-sqlite.md
created: 2022-02-28T12:09:11-08:00
created_utc: 2022-02-28T20:09:11+00:00
updated: 2022-02-28T12:41:50-08:00
updated_utc: 2022-02-28T20:41:50+00:00
shot_hash: d4b773aa2f8f7ed09c3a49c71fb61224
slug: latest-sqlite
body:
I made a pleasant discovery today: Homebrew are very quick to update to the latest SQLite release (here's [their formula](https://github.com/Homebrew/homebrew-core/blob/master/Formula/sqlite.rb)), and since [Datasette](https://datasette.io/) when installed via Homebrew uses that version, this means you can use `brew update sqlite` to ensure you are running the most recent SQLite version within Datasette.

If you've installed Datasette using Homebrew:

    brew install datasette

You can see the version of SQLite it uses either by running `datasette` and navigating to http://127.0.0.1:8001/-/versions - or you can see it from the command-line using:

    % datasette --get /-/versions.json | jq .sqlite.version
    "3.37.2"

To upgrade SQLite, run the following:

    brew upgrade sqlite

After doing that I ran the above command again and confirmed I had been upgraded to SQLite 3.38.0:

    % datasette --get /-/versions.json | jq .sqlite.version
    "3.38.0"

html:
<p>I made a pleasant discovery today: Homebrew are very quick to update to the latest SQLite release (here's <a href="https://github.com/Homebrew/homebrew-core/blob/master/Formula/sqlite.rb">their formula</a>), and since <a href="https://datasette.io/" rel="nofollow">Datasette</a> when installed via Homebrew uses that version, this means you can use <code>brew update sqlite</code> to ensure you are running the most recent SQLite version within Datasette.</p>
<p>If you've installed Datasette using Homebrew:</p>
<pre><code>brew install datasette
</code></pre>
<p>You can see the version of SQLite it uses either by running <code>datasette</code> and navigating to <a href="http://127.0.0.1:8001/-/versions" rel="nofollow">http://127.0.0.1:8001/-/versions</a> - or you can see it from the command-line using:</p>
<pre><code>% datasette --get /-/versions.json | jq .sqlite.version
"3.37.2"
</code></pre>
<p>To upgrade SQLite, run the following:</p>
<pre><code>brew upgrade sqlite
</code></pre>
<p>After doing that I ran the above command again and confirmed I had been upgraded to SQLite 3.38.0:</p>
<pre><code>% datasette --get /-/versions.json | jq .sqlite.version
"3.38.0"
</code></pre>


path: macos_python-installer-macos.md
topic: macos
title: Installing Python on macOS with the official Python installer
url: https://github.com/simonw/til/blob/main/macos/python-installer-macos.md
created: 2022-02-28T12:33:46-08:00
created_utc: 2022-02-28T20:33:46+00:00
updated: 2022-02-28T12:42:11-08:00
updated_utc: 2022-02-28T20:42:11+00:00
shot_hash: 1af76f01103484fccc017ceb8b3090d9
slug: python-installer-macos
body:
I usually use Homebrew on macOS, but I decided to try using the official Python installer based on [this Twitter conversation](https://twitter.com/rtpg_/status/1498115527465914371).

My big fear was that it would add Yet Another Python to my system in a way that made the [XKCD 1987](https://xkcd.com/1987/) situation even worse!

I downloaded the installer using the prompt on the Python.org homepage (in the "Downloads" menu):

<img width="967" alt="image" src="https://user-images.githubusercontent.com/9599/156053747-0cad53c1-6646-4398-a36d-582179bf4a93.png">

After running the installer and accepting the default options, a Python 3.10 folder was added to my `/Applications` folder:

<img width="540" alt="image" src="https://user-images.githubusercontent.com/9599/156053911-a9f9d28a-ed1c-43c9-a5ce-098444b30fd3.png">

More importantly though, running `python3` in my terminal now runs the version of Python that was installed by the installer - which lives in `/Library/Frameworks/Python.framework/`:
```
~ % which python3
/Library/Frameworks/Python.framework/Versions/3.10/bin/python3
~ % python3
Python 3.10.2 (v3.10.2:a58ebcc701, Jan 13 2022, 14:50:16) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
```
I was confused as to how it had added itself to my path - a tip [from Ned Deily](https://twitter.com/baybryj/status/1498388515742224387) (a Python release manager) helped me figure that out:

```
~ % grep -d skip '3.10' ~/.*
/Users/simon/.zprofile:# Setting PATH for Python 3.10
/Users/simon/.zprofile:PATH="/Library/Frameworks/Python.framework/Versions/3.10/bin:${PATH}"
```
So the installer added a line to my `.zprofile` file adding the `bin` directory for that release to my path.

Running `python3.10` is the most specific way to execute that version of Python. I can install new command-line tools into that `bin` directory like so:

    % python3.10 -m pip install google-drive-to-sqlite
    Collecting google-drive-to-sqlite
      Downloading google_drive_to_sqlite-0.4-py3-none-any.whl (20 kB)
    ...
    % google-drive-to-sqlite --version
    google-drive-to-sqlite, version 0.4
    % which google-drive-to-sqlite
    /Library/Frameworks/Python.framework/Versions/3.10/bin/google-drive-to-sqlite

html:
<p>I usually use Homebrew on macOS, but I decided to try using the official Python installer based on <a href="https://twitter.com/rtpg_/status/1498115527465914371" rel="nofollow">this Twitter conversation</a>.</p>
<p>My big fear was that it would add Yet Another Python to my system in a way that made the <a href="https://xkcd.com/1987/" rel="nofollow">XKCD 1987</a> situation even worse!</p>
<p>I downloaded the installer using the prompt on the Python.org homepage (in the "Downloads" menu):</p>
<p><a href="https://user-images.githubusercontent.com/9599/156053747-0cad53c1-6646-4398-a36d-582179bf4a93.png" target="_blank" rel="nofollow"><img width="967" alt="image" src="https://user-images.githubusercontent.com/9599/156053747-0cad53c1-6646-4398-a36d-582179bf4a93.png" style="max-width:100%;"></a></p>
<p>After running the installer and accepting the default options, a Python 3.10 folder was added to my <code>/Applications</code> folder:</p>
<p><a href="https://user-images.githubusercontent.com/9599/156053911-a9f9d28a-ed1c-43c9-a5ce-098444b30fd3.png" target="_blank" rel="nofollow"><img width="540" alt="image" src="https://user-images.githubusercontent.com/9599/156053911-a9f9d28a-ed1c-43c9-a5ce-098444b30fd3.png" style="max-width:100%;"></a></p>
<p>More importantly though, running <code>python3</code> in my terminal now runs the version of Python that was installed by the installer - which lives in <code>/Library/Frameworks/Python.framework/</code>:</p>
<pre><code>~ % which python3
/Library/Frameworks/Python.framework/Versions/3.10/bin/python3
~ % python3
Python 3.10.2 (v3.10.2:a58ebcc701, Jan 13 2022, 14:50:16) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
</code></pre>
<p>I was confused as to how it had added itself to my path - a tip <a href="https://twitter.com/baybryj/status/1498388515742224387" rel="nofollow">from Ned Deily</a> (a Python release manager) helped me figure that out:</p>
<pre><code>~ % grep -d skip '3.10' ~/.*
/Users/simon/.zprofile:# Setting PATH for Python 3.10
/Users/simon/.zprofile:PATH="/Library/Frameworks/Python.framework/Versions/3.10/bin:${PATH}"
</code></pre>
<p>So the installer added a line to my <code>.zprofile</code> file adding the <code>bin</code> directory for that release to my path.</p>
<p>Running <code>python3.10</code> is the most specific way to execute that version of Python. I can install new command-line tools into that <code>bin</code> directory like so:</p>
<pre><code>% python3.10 -m pip install google-drive-to-sqlite
Collecting google-drive-to-sqlite
  Downloading google_drive_to_sqlite-0.4-py3-none-any.whl (20 kB)
...
% google-drive-to-sqlite --version
google-drive-to-sqlite, version 0.4
% which google-drive-to-sqlite
/Library/Frameworks/Python.framework/Versions/3.10/bin/google-drive-to-sqlite
</code></pre>


path: gis_natural-earth-in-spatialite-and-datasette.md
topic: gis
title: Natural Earth in SpatiaLite and Datasette
url: https://github.com/simonw/til/blob/main/gis/natural-earth-in-spatialite-and-datasette.md
created: 2022-03-04T11:11:55-08:00
created_utc: 2022-03-04T19:11:55+00:00
updated: 2022-03-04T16:10:50-08:00
updated_utc: 2022-03-05T00:10:50+00:00
shot_hash: 5d119fc759211694df235623daa93943
slug: natural-earth-in-spatialite-and-datasette
body:
Natural Earth ([website](https://www.naturalearthdata.com/), [Wikipedia](https://en.wikipedia.org/wiki/Natural_Earth)) is a a public domain map dataset.

It's distributed in a bunch of different formats - one of them is a SQLite database file.

http://naciscdn.org/naturalearth/packages/natural_earth_vector.sqlite.zip - this is a 423MB file which decompresses to provide a 791MB `packages/natural_earth_vector.sqlite` file.

I opened it in Datasette like this:

    datasette --load-extension spatialite \
      ~/Downloads/natural_earth_vector.sqlite/packages/natural_earth_vector.sqlite

I had previously installed Datasette and SpatiaLite using Homebrew:

    brew install datasette spatialite-tools

## Database format

The database contains 181 tables, for different layers at different scales. Those tables are listed below.

Each table has a bunch of columns and a `GEOMETRY` column. That geometry column contains data stored in WKB - Well-Known Binary format.

If you have SpatiaLite you can convert that column to GeoJSON like so:

    AsGeoJSON(GeomFromWKB(GEOMETRY))

For example, here are the largest "urban areas" at 50m scale:

```sql
select
  AsGeoJSON(GeomFromWKB(GEOMETRY))
from
  ne_50m_urban_areas
order by area_sqkm desc
```
Every country at 50m scale (a good balance between detail and overall size):
```sql
select
  AsGeoJSON(GeomFromWKB(GEOMETRY)), *
from
  ne_50m_admin_0_countries
```

This query draws a coloured map of countries using the `datasette-geojson-map` and `sqlite-colorbrewer` plugins:

```sql
select
  ogc_fid,
  GeomFromWKB(GEOMETRY) as geometry,
  colorbrewer('Paired', 9, MAPCOLOR9 - 1) as fill
from
  ne_10m_admin_0_countries
```

<img width="1098" alt="Screenshot of a map showing different countries in random colours" src="https://user-images.githubusercontent.com/9599/156858327-08f99300-29fd-4ca8-a268-f8c2ec659349.png">

The `ne_10m_admin_1_states_provinces` table is useful: it has subdivisions for a bunch of different countries. Here's the UK divided into counties:

```sql
select
  ogc_fid,
  GeomFromWKB(GEOMETRY) as geometry,
  featurecla,
  scalerank,
  adm1_code,
  diss_me,
  iso_3166_2,
  wikipedia,
  iso_a2,
  adm0_sr,
  name,
  name_alt,
  type,
  type_en,
  area_sqkm,
  latitude,
  longitude,
  admin
from
  ne_10m_admin_1_states_provinces
where
  admin = 'United Kingdom'
```
I tried this with `select *, GeomFromWKB(GEOMETRY) as geometry` first but it didn't work with `datasette-geojson-map` because the `*` picked up the original `GEOMETRY` column in the wrong format.

The scales are:

- Large scale data, 1:10m - most detailed
- Medium scale data, 1:50m
- Small scale data, 1:110m - least detailed

## Exploring with Datasette plugins

With the `datasette-leaflet-geojson` plugin installed, any column that returns GeoJSON (from `AsGeoJSON(GeomFromWKB(GEOMETRY))`) will render as a little map, no matter what the column name.

If you install `datasette-geojson-map` you can seee a single map with all of the shapes on - you need to create a `geometry` column containing a SpatiaLite geometry, which you can do like this:
```sql
select
  ogc_fid, GeomFromWKB(GEOMETRY) as geometry, *
from
  ne_50m_coastline
```
## Full list of tables

- `ne_10m_admin_0_antarctic_claim_limit_lines`
- `ne_10m_admin_0_antarctic_claims`
- `ne_10m_admin_0_boundary_lines_disputed_areas`
- `ne_10m_admin_0_boundary_lines_land`
- `ne_10m_admin_0_boundary_lines_map_units`
- `ne_10m_admin_0_boundary_lines_maritime_indicator`
- `ne_10m_admin_0_boundary_lines_maritime_indicator_chn`
- `ne_10m_admin_0_countries`
- `ne_10m_admin_0_countries_arg`
- `ne_10m_admin_0_countries_bdg`
- `ne_10m_admin_0_countries_bra`
- `ne_10m_admin_0_countries_chn`
- `ne_10m_admin_0_countries_deu`
- `ne_10m_admin_0_countries_egy`
- `ne_10m_admin_0_countries_esp`
- `ne_10m_admin_0_countries_fra`
- `ne_10m_admin_0_countries_gbr`
- `ne_10m_admin_0_countries_grc`
- `ne_10m_admin_0_countries_idn`
- `ne_10m_admin_0_countries_ind`
- `ne_10m_admin_0_countries_isr`
- `ne_10m_admin_0_countries_ita`
- `ne_10m_admin_0_countries_jpn`
- `ne_10m_admin_0_countries_kor`
- `ne_10m_admin_0_countries_lakes`
- `ne_10m_admin_0_countries_mar`
- `ne_10m_admin_0_countries_nep`
- `ne_10m_admin_0_countries_nld`
- `ne_10m_admin_0_countries_pak`
- `ne_10m_admin_0_countries_pol`
- `ne_10m_admin_0_countries_prt`
- `ne_10m_admin_0_countries_pse`
- `ne_10m_admin_0_countries_rus`
- `ne_10m_admin_0_countries_sau`
- `ne_10m_admin_0_countries_swe`
- `ne_10m_admin_0_countries_tur`
- `ne_10m_admin_0_countries_twn`
- `ne_10m_admin_0_countries_ukr`
- `ne_10m_admin_0_countries_usa`
- `ne_10m_admin_0_countries_vnm`
- `ne_10m_admin_0_disputed_areas`
- `ne_10m_admin_0_disputed_areas_scale_rank_minor_islands`
- `ne_10m_admin_0_label_points`
- `ne_10m_admin_0_map_subunits`
- `ne_10m_admin_0_map_units`
- `ne_10m_admin_0_names`
- `ne_10m_admin_0_pacific_groupings`
- `ne_10m_admin_0_scale_rank`
- `ne_10m_admin_0_scale_rank_minor_islands`
- `ne_10m_admin_0_seams`
- `ne_10m_admin_0_sovereignty`
- `ne_10m_admin_1_label_points`
- `ne_10m_admin_1_label_points_details`
- `ne_10m_admin_1_seams`
- `ne_10m_admin_1_sel`
- `ne_10m_admin_1_states_provinces`
- `ne_10m_admin_1_states_provinces_lakes`
- `ne_10m_admin_1_states_provinces_lines`
- `ne_10m_admin_1_states_provinces_scale_rank`
- `ne_10m_admin_1_states_provinces_scale_rank_minor_islands`
- `ne_10m_admin_2_counties`
- `ne_10m_admin_2_counties_lakes`
- `ne_10m_admin_2_counties_lines`
- `ne_10m_admin_2_counties_scale_rank`
- `ne_10m_admin_2_counties_scale_rank_minor_islands`
- `ne_10m_admin_2_counties_to_match`
- `ne_10m_admin_2_label_points`
- `ne_10m_admin_2_label_points_details`
- `ne_10m_airports`
- `ne_10m_antarctic_ice_shelves_lines`
- `ne_10m_antarctic_ice_shelves_polys`
- `ne_10m_coastline`
- `ne_10m_geographic_lines`
- `ne_10m_geography_marine_polys`
- `ne_10m_geography_regions_elevation_points`
- `ne_10m_geography_regions_points`
- `ne_10m_geography_regions_polys`
- `ne_10m_glaciated_areas`
- `ne_10m_lakes`
- `ne_10m_lakes_australia`
- `ne_10m_lakes_europe`
- `ne_10m_lakes_historic`
- `ne_10m_lakes_north_america`
- `ne_10m_lakes_pluvial`
- `ne_10m_land`
- `ne_10m_land_ocean_label_points`
- `ne_10m_land_ocean_seams`
- `ne_10m_land_scale_rank`
- `ne_10m_minor_islands`
- `ne_10m_minor_islands_coastline`
- `ne_10m_minor_islands_label_points`
- `ne_10m_ocean`
- `ne_10m_ocean_scale_rank`
- `ne_10m_parks_and_protected_lands_area`
- `ne_10m_parks_and_protected_lands_line`
- `ne_10m_parks_and_protected_lands_point`
- `ne_10m_parks_and_protected_lands_scale_rank`
- `ne_10m_playas`
- `ne_10m_populated_places`
- `ne_10m_populated_places_simple`
- `ne_10m_ports`
- `ne_10m_railroads`
- `ne_10m_railroads_north_america`
- `ne_10m_reefs`
- `ne_10m_rivers_australia`
- `ne_10m_rivers_europe`
- `ne_10m_rivers_lake_centerlines`
- `ne_10m_rivers_lake_centerlines_scale_rank`
- `ne_10m_rivers_north_america`
- `ne_10m_roads`
- `ne_10m_roads_north_america`
- `ne_10m_time_zones`
- `ne_10m_urban_areas`
- `ne_10m_urban_areas_landscan`
- `ne_110m_admin_0_boundary_lines_land`
- `ne_110m_admin_0_countries`
- `ne_110m_admin_0_countries_lakes`
- `ne_110m_admin_0_map_units`
- `ne_110m_admin_0_pacific_groupings`
- `ne_110m_admin_0_scale_rank`
- `ne_110m_admin_0_sovereignty`
- `ne_110m_admin_0_tiny_countries`
- `ne_110m_admin_1_states_provinces`
- `ne_110m_admin_1_states_provinces_lakes`
- `ne_110m_admin_1_states_provinces_lines`
- `ne_110m_admin_1_states_provinces_scale_rank`
- `ne_110m_coastline`
- `ne_110m_geographic_lines`
- `ne_110m_geography_marine_polys`
- `ne_110m_geography_regions_elevation_points`
- `ne_110m_geography_regions_points`
- `ne_110m_geography_regions_polys`
- `ne_110m_glaciated_areas`
- `ne_110m_lakes`
- `ne_110m_land`
- `ne_110m_ocean`
- `ne_110m_populated_places`
- `ne_110m_populated_places_simple`
- `ne_110m_rivers_lake_centerlines`
- `ne_50m_admin_0_boundary_lines_disputed_areas`
- `ne_50m_admin_0_boundary_lines_land`
- `ne_50m_admin_0_boundary_lines_maritime_indicator`
- `ne_50m_admin_0_boundary_lines_maritime_indicator_chn`
- `ne_50m_admin_0_boundary_map_units`
- `ne_50m_admin_0_breakaway_disputed_areas`
- `ne_50m_admin_0_breakaway_disputed_areas_scale_rank`
- `ne_50m_admin_0_countries`
- `ne_50m_admin_0_countries_lakes`
- `ne_50m_admin_0_map_subunits`
- `ne_50m_admin_0_map_units`
- `ne_50m_admin_0_pacific_groupings`
- `ne_50m_admin_0_scale_rank`
- `ne_50m_admin_0_sovereignty`
- `ne_50m_admin_0_tiny_countries`
- `ne_50m_admin_0_tiny_countries_scale_rank`
- `ne_50m_admin_1_seams`
- `ne_50m_admin_1_states_provinces`
- `ne_50m_admin_1_states_provinces_lakes`
- `ne_50m_admin_1_states_provinces_lines`
- `ne_50m_admin_1_states_provinces_scale_rank`
- `ne_50m_airports`
- `ne_50m_antarctic_ice_shelves_lines`
- `ne_50m_antarctic_ice_shelves_polys`
- `ne_50m_coastline`
- `ne_50m_geographic_lines`
- `ne_50m_geography_marine_polys`
- `ne_50m_geography_regions_elevation_points`
- `ne_50m_geography_regions_points`
- `ne_50m_geography_regions_polys`
- `ne_50m_glaciated_areas`
- `ne_50m_lakes`
- `ne_50m_lakes_historic`
- `ne_50m_land`
- `ne_50m_ocean`
- `ne_50m_playas`
- `ne_50m_populated_places`
- `ne_50m_populated_places_simple`
- `ne_50m_ports`
- `ne_50m_rivers_lake_centerlines`
- `ne_50m_rivers_lake_centerlines_scale_rank`
- `ne_50m_urban_areas`

html:
<p>Natural Earth (<a href="https://www.naturalearthdata.com/" rel="nofollow">website</a>, <a href="https://en.wikipedia.org/wiki/Natural_Earth" rel="nofollow">Wikipedia</a>) is a a public domain map dataset.</p>
<p>It's distributed in a bunch of different formats - one of them is a SQLite database file.</p>
<p><a href="http://naciscdn.org/naturalearth/packages/natural_earth_vector.sqlite.zip" rel="nofollow">http://naciscdn.org/naturalearth/packages/natural_earth_vector.sqlite.zip</a> - this is a 423MB file which decompresses to provide a 791MB <code>packages/natural_earth_vector.sqlite</code> file.</p>
<p>I opened it in Datasette like this:</p>
<pre><code>datasette --load-extension spatialite \
  ~/Downloads/natural_earth_vector.sqlite/packages/natural_earth_vector.sqlite
</code></pre>
<p>I had previously installed Datasette and SpatiaLite using Homebrew:</p>
<pre><code>brew install datasette spatialite-tools
</code></pre>
<h2>
<a id="user-content-database-format" class="anchor" href="#database-format" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Database format</h2>
<p>The database contains 181 tables, for different layers at different scales. Those tables are listed below.</p>
<p>Each table has a bunch of columns and a <code>GEOMETRY</code> column. That geometry column contains data stored in WKB - Well-Known Binary format.</p>
<p>If you have SpatiaLite you can convert that column to GeoJSON like so:</p>
<pre><code>AsGeoJSON(GeomFromWKB(GEOMETRY))
</code></pre>
<p>For example, here are the largest "urban areas" at 50m scale:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  AsGeoJSON(GeomFromWKB(GEOMETRY))
<span class="pl-k">from</span>
  ne_50m_urban_areas
<span class="pl-k">order by</span> area_sqkm <span class="pl-k">desc</span></pre></div>
<p>Every country at 50m scale (a good balance between detail and overall size):</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  AsGeoJSON(GeomFromWKB(GEOMETRY)), <span class="pl-k">*</span>
<span class="pl-k">from</span>
  ne_50m_admin_0_countries</pre></div>
<p>This query draws a coloured map of countries using the <code>datasette-geojson-map</code> and <code>sqlite-colorbrewer</code> plugins:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  ogc_fid,
  GeomFromWKB(GEOMETRY) <span class="pl-k">as</span> geometry,
  colorbrewer(<span class="pl-s"><span class="pl-pds">'</span>Paired<span class="pl-pds">'</span></span>, <span class="pl-c1">9</span>, MAPCOLOR9 <span class="pl-k">-</span> <span class="pl-c1">1</span>) <span class="pl-k">as</span> fill
<span class="pl-k">from</span>
  ne_10m_admin_0_countries</pre></div>
<p><a href="https://user-images.githubusercontent.com/9599/156858327-08f99300-29fd-4ca8-a268-f8c2ec659349.png" target="_blank" rel="nofollow"><img width="1098" alt="Screenshot of a map showing different countries in random colours" src="https://user-images.githubusercontent.com/9599/156858327-08f99300-29fd-4ca8-a268-f8c2ec659349.png" style="max-width:100%;"></a></p>
<p>The <code>ne_10m_admin_1_states_provinces</code> table is useful: it has subdivisions for a bunch of different countries. Here's the UK divided into counties:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  ogc_fid,
  GeomFromWKB(GEOMETRY) <span class="pl-k">as</span> geometry,
  featurecla,
  scalerank,
  adm1_code,
  diss_me,
  iso_3166_2,
  wikipedia,
  iso_a2,
  adm0_sr,
  name,
  name_alt,
  type,
  type_en,
  area_sqkm,
  latitude,
  longitude,
  admin
<span class="pl-k">from</span>
  ne_10m_admin_1_states_provinces
<span class="pl-k">where</span>
  admin <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">'</span>United Kingdom<span class="pl-pds">'</span></span></pre></div>
<p>I tried this with <code>select *, GeomFromWKB(GEOMETRY) as geometry</code> first but it didn't work with <code>datasette-geojson-map</code> because the <code>*</code> picked up the original <code>GEOMETRY</code> column in the wrong format.</p>
<p>The scales are:</p>
<ul>
<li>Large scale data, 1:10m - most detailed</li>
<li>Medium scale data, 1:50m</li>
<li>Small scale data, 1:110m - least detailed</li>
</ul>
<h2>
<a id="user-content-exploring-with-datasette-plugins" class="anchor" href="#exploring-with-datasette-plugins" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Exploring with Datasette plugins</h2>
<p>With the <code>datasette-leaflet-geojson</code> plugin installed, any column that returns GeoJSON (from <code>AsGeoJSON(GeomFromWKB(GEOMETRY))</code>) will render as a little map, no matter what the column name.</p>
<p>If you install <code>datasette-geojson-map</code> you can seee a single map with all of the shapes on - you need to create a <code>geometry</code> column containing a SpatiaLite geometry, which you can do like this:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  ogc_fid, GeomFromWKB(GEOMETRY) <span class="pl-k">as</span> geometry, <span class="pl-k">*</span>
<span class="pl-k">from</span>
  ne_50m_coastline</pre></div>
<h2>
<a id="user-content-full-list-of-tables" class="anchor" href="#full-list-of-tables" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Full list of tables</h2>
<ul>
<li><code>ne_10m_admin_0_antarctic_claim_limit_lines</code></li>
<li><code>ne_10m_admin_0_antarctic_claims</code></li>
<li><code>ne_10m_admin_0_boundary_lines_disputed_areas</code></li>
<li><code>ne_10m_admin_0_boundary_lines_land</code></li>
<li><code>ne_10m_admin_0_boundary_lines_map_units</code></li>
<li><code>ne_10m_admin_0_boundary_lines_maritime_indicator</code></li>
<li><code>ne_10m_admin_0_boundary_lines_maritime_indicator_chn</code></li>
<li><code>ne_10m_admin_0_countries</code></li>
<li><code>ne_10m_admin_0_countries_arg</code></li>
<li><code>ne_10m_admin_0_countries_bdg</code></li>
<li><code>ne_10m_admin_0_countries_bra</code></li>
<li><code>ne_10m_admin_0_countries_chn</code></li>
<li><code>ne_10m_admin_0_countries_deu</code></li>
<li><code>ne_10m_admin_0_countries_egy</code></li>
<li><code>ne_10m_admin_0_countries_esp</code></li>
<li><code>ne_10m_admin_0_countries_fra</code></li>
<li><code>ne_10m_admin_0_countries_gbr</code></li>
<li><code>ne_10m_admin_0_countries_grc</code></li>
<li><code>ne_10m_admin_0_countries_idn</code></li>
<li><code>ne_10m_admin_0_countries_ind</code></li>
<li><code>ne_10m_admin_0_countries_isr</code></li>
<li><code>ne_10m_admin_0_countries_ita</code></li>
<li><code>ne_10m_admin_0_countries_jpn</code></li>
<li><code>ne_10m_admin_0_countries_kor</code></li>
<li><code>ne_10m_admin_0_countries_lakes</code></li>
<li><code>ne_10m_admin_0_countries_mar</code></li>
<li><code>ne_10m_admin_0_countries_nep</code></li>
<li><code>ne_10m_admin_0_countries_nld</code></li>
<li><code>ne_10m_admin_0_countries_pak</code></li>
<li><code>ne_10m_admin_0_countries_pol</code></li>
<li><code>ne_10m_admin_0_countries_prt</code></li>
<li><code>ne_10m_admin_0_countries_pse</code></li>
<li><code>ne_10m_admin_0_countries_rus</code></li>
<li><code>ne_10m_admin_0_countries_sau</code></li>
<li><code>ne_10m_admin_0_countries_swe</code></li>
<li><code>ne_10m_admin_0_countries_tur</code></li>
<li><code>ne_10m_admin_0_countries_twn</code></li>
<li><code>ne_10m_admin_0_countries_ukr</code></li>
<li><code>ne_10m_admin_0_countries_usa</code></li>
<li><code>ne_10m_admin_0_countries_vnm</code></li>
<li><code>ne_10m_admin_0_disputed_areas</code></li>
<li><code>ne_10m_admin_0_disputed_areas_scale_rank_minor_islands</code></li>
<li><code>ne_10m_admin_0_label_points</code></li>
<li><code>ne_10m_admin_0_map_subunits</code></li>
<li><code>ne_10m_admin_0_map_units</code></li>
<li><code>ne_10m_admin_0_names</code></li>
<li><code>ne_10m_admin_0_pacific_groupings</code></li>
<li><code>ne_10m_admin_0_scale_rank</code></li>
<li><code>ne_10m_admin_0_scale_rank_minor_islands</code></li>
<li><code>ne_10m_admin_0_seams</code></li>
<li><code>ne_10m_admin_0_sovereignty</code></li>
<li><code>ne_10m_admin_1_label_points</code></li>
<li><code>ne_10m_admin_1_label_points_details</code></li>
<li><code>ne_10m_admin_1_seams</code></li>
<li><code>ne_10m_admin_1_sel</code></li>
<li><code>ne_10m_admin_1_states_provinces</code></li>
<li><code>ne_10m_admin_1_states_provinces_lakes</code></li>
<li><code>ne_10m_admin_1_states_provinces_lines</code></li>
<li><code>ne_10m_admin_1_states_provinces_scale_rank</code></li>
<li><code>ne_10m_admin_1_states_provinces_scale_rank_minor_islands</code></li>
<li><code>ne_10m_admin_2_counties</code></li>
<li><code>ne_10m_admin_2_counties_lakes</code></li>
<li><code>ne_10m_admin_2_counties_lines</code></li>
<li><code>ne_10m_admin_2_counties_scale_rank</code></li>
<li><code>ne_10m_admin_2_counties_scale_rank_minor_islands</code></li>
<li><code>ne_10m_admin_2_counties_to_match</code></li>
<li><code>ne_10m_admin_2_label_points</code></li>
<li><code>ne_10m_admin_2_label_points_details</code></li>
<li><code>ne_10m_airports</code></li>
<li><code>ne_10m_antarctic_ice_shelves_lines</code></li>
<li><code>ne_10m_antarctic_ice_shelves_polys</code></li>
<li><code>ne_10m_coastline</code></li>
<li><code>ne_10m_geographic_lines</code></li>
<li><code>ne_10m_geography_marine_polys</code></li>
<li><code>ne_10m_geography_regions_elevation_points</code></li>
<li><code>ne_10m_geography_regions_points</code></li>
<li><code>ne_10m_geography_regions_polys</code></li>
<li><code>ne_10m_glaciated_areas</code></li>
<li><code>ne_10m_lakes</code></li>
<li><code>ne_10m_lakes_australia</code></li>
<li><code>ne_10m_lakes_europe</code></li>
<li><code>ne_10m_lakes_historic</code></li>
<li><code>ne_10m_lakes_north_america</code></li>
<li><code>ne_10m_lakes_pluvial</code></li>
<li><code>ne_10m_land</code></li>
<li><code>ne_10m_land_ocean_label_points</code></li>
<li><code>ne_10m_land_ocean_seams</code></li>
<li><code>ne_10m_land_scale_rank</code></li>
<li><code>ne_10m_minor_islands</code></li>
<li><code>ne_10m_minor_islands_coastline</code></li>
<li><code>ne_10m_minor_islands_label_points</code></li>
<li><code>ne_10m_ocean</code></li>
<li><code>ne_10m_ocean_scale_rank</code></li>
<li><code>ne_10m_parks_and_protected_lands_area</code></li>
<li><code>ne_10m_parks_and_protected_lands_line</code></li>
<li><code>ne_10m_parks_and_protected_lands_point</code></li>
<li><code>ne_10m_parks_and_protected_lands_scale_rank</code></li>
<li><code>ne_10m_playas</code></li>
<li><code>ne_10m_populated_places</code></li>
<li><code>ne_10m_populated_places_simple</code></li>
<li><code>ne_10m_ports</code></li>
<li><code>ne_10m_railroads</code></li>
<li><code>ne_10m_railroads_north_america</code></li>
<li><code>ne_10m_reefs</code></li>
<li><code>ne_10m_rivers_australia</code></li>
<li><code>ne_10m_rivers_europe</code></li>
<li><code>ne_10m_rivers_lake_centerlines</code></li>
<li><code>ne_10m_rivers_lake_centerlines_scale_rank</code></li>
<li><code>ne_10m_rivers_north_america</code></li>
<li><code>ne_10m_roads</code></li>
<li><code>ne_10m_roads_north_america</code></li>
<li><code>ne_10m_time_zones</code></li>
<li><code>ne_10m_urban_areas</code></li>
<li><code>ne_10m_urban_areas_landscan</code></li>
<li><code>ne_110m_admin_0_boundary_lines_land</code></li>
<li><code>ne_110m_admin_0_countries</code></li>
<li><code>ne_110m_admin_0_countries_lakes</code></li>
<li><code>ne_110m_admin_0_map_units</code></li>
<li><code>ne_110m_admin_0_pacific_groupings</code></li>
<li><code>ne_110m_admin_0_scale_rank</code></li>
<li><code>ne_110m_admin_0_sovereignty</code></li>
<li><code>ne_110m_admin_0_tiny_countries</code></li>
<li><code>ne_110m_admin_1_states_provinces</code></li>
<li><code>ne_110m_admin_1_states_provinces_lakes</code></li>
<li><code>ne_110m_admin_1_states_provinces_lines</code></li>
<li><code>ne_110m_admin_1_states_provinces_scale_rank</code></li>
<li><code>ne_110m_coastline</code></li>
<li><code>ne_110m_geographic_lines</code></li>
<li><code>ne_110m_geography_marine_polys</code></li>
<li><code>ne_110m_geography_regions_elevation_points</code></li>
<li><code>ne_110m_geography_regions_points</code></li>
<li><code>ne_110m_geography_regions_polys</code></li>
<li><code>ne_110m_glaciated_areas</code></li>
<li><code>ne_110m_lakes</code></li>
<li><code>ne_110m_land</code></li>
<li><code>ne_110m_ocean</code></li>
<li><code>ne_110m_populated_places</code></li>
<li><code>ne_110m_populated_places_simple</code></li>
<li><code>ne_110m_rivers_lake_centerlines</code></li>
<li><code>ne_50m_admin_0_boundary_lines_disputed_areas</code></li>
<li><code>ne_50m_admin_0_boundary_lines_land</code></li>
<li><code>ne_50m_admin_0_boundary_lines_maritime_indicator</code></li>
<li><code>ne_50m_admin_0_boundary_lines_maritime_indicator_chn</code></li>
<li><code>ne_50m_admin_0_boundary_map_units</code></li>
<li><code>ne_50m_admin_0_breakaway_disputed_areas</code></li>
<li><code>ne_50m_admin_0_breakaway_disputed_areas_scale_rank</code></li>
<li><code>ne_50m_admin_0_countries</code></li>
<li><code>ne_50m_admin_0_countries_lakes</code></li>
<li><code>ne_50m_admin_0_map_subunits</code></li>
<li><code>ne_50m_admin_0_map_units</code></li>
<li><code>ne_50m_admin_0_pacific_groupings</code></li>
<li><code>ne_50m_admin_0_scale_rank</code></li>
<li><code>ne_50m_admin_0_sovereignty</code></li>
<li><code>ne_50m_admin_0_tiny_countries</code></li>
<li><code>ne_50m_admin_0_tiny_countries_scale_rank</code></li>
<li><code>ne_50m_admin_1_seams</code></li>
<li><code>ne_50m_admin_1_states_provinces</code></li>
<li><code>ne_50m_admin_1_states_provinces_lakes</code></li>
<li><code>ne_50m_admin_1_states_provinces_lines</code></li>
<li><code>ne_50m_admin_1_states_provinces_scale_rank</code></li>
<li><code>ne_50m_airports</code></li>
<li><code>ne_50m_antarctic_ice_shelves_lines</code></li>
<li><code>ne_50m_antarctic_ice_shelves_polys</code></li>
<li><code>ne_50m_coastline</code></li>
<li><code>ne_50m_geographic_lines</code></li>
<li><code>ne_50m_geography_marine_polys</code></li>
<li><code>ne_50m_geography_regions_elevation_points</code></li>
<li><code>ne_50m_geography_regions_points</code></li>
<li><code>ne_50m_geography_regions_polys</code></li>
<li><code>ne_50m_glaciated_areas</code></li>
<li><code>ne_50m_lakes</code></li>
<li><code>ne_50m_lakes_historic</code></li>
<li><code>ne_50m_land</code></li>
<li><code>ne_50m_ocean</code></li>
<li><code>ne_50m_playas</code></li>
<li><code>ne_50m_populated_places</code></li>
<li><code>ne_50m_populated_places_simple</code></li>
<li><code>ne_50m_ports</code></li>
<li><code>ne_50m_rivers_lake_centerlines</code></li>
<li><code>ne_50m_rivers_lake_centerlines_scale_rank</code></li>
<li><code>ne_50m_urban_areas</code></li>
</ul>


path: pytest_coverage-with-context.md
topic: pytest
title: pytest coverage with context
url: https://github.com/simonw/til/blob/main/pytest/coverage-with-context.md
created: 2022-03-04T16:53:50-08:00
created_utc: 2022-03-05T00:53:50+00:00
updated: 2022-03-04T16:53:50-08:00
updated_utc: 2022-03-05T00:53:50+00:00
shot_hash: c21dbc2b5d071379a39aa24acc3cd6c5
slug: coverage-with-context
body:
[This tweet](https://twitter.com/mariatta/status/1499863816489734146) from \@Mariatta tipped me off to the ability to measure "contexts" when [running coverage](https://coverage.readthedocs.io/en/6.3.2/contexts.html#context-reporting) - as a way to tell which tests exercise which specific lines of code.

My [sqlite-utils](https://github.com/simonw/sqlite-utils) project uses `pytest` for the test suite. I decided to figure out how to get this working with [pytest-cov](https://pypi.org/project/pytest-cov/).

After some experimentation, this is the recipe that worked for me:

```
# In the virtual environment, make sure pytest-cov is installed:
% pip install pytest-cov
# First, run pytest to calculate coverage of the `sqlite_utils` package, with context
% pytest --cov=sqlite_utils --cov-context=test
# The .coverage file is actually a SQLite database:
% ls -lah .coverage
-rw-r--r--@ 1 simon  staff   716K Mar  4 16:39 .coverage
# This command generates the HTML coverage report in `htmlcov/`
% coverage html --show-contexts
# Open the report in a browser`
% open htmlcov/index.html
```

Here's what one of the pages looks like, displaying the context for some lines of code:

![The code has an expandable section which reveals which tests executed each individual line.](https://user-images.githubusercontent.com/9599/156860441-66e35994-653a-4ab7-b690-4d901fc57750.png)

## The .coverage schema

Since `.coverage` is a SQLite database, here's the schema - generated using `sqlite-utils schema .coverage`:
```sql
CREATE TABLE coverage_schema (
    -- One row, to record the version of the schema in this db.
    version integer
);
CREATE TABLE meta (
    -- Key-value pairs, to record metadata about the data
    key text,
    value text,
    unique (key)
    -- Keys:
    --  'has_arcs' boolean      -- Is this data recording branches?
    --  'sys_argv' text         -- The coverage command line that recorded the data.
    --  'version' text          -- The version of coverage.py that made the file.
    --  'when' text             -- Datetime when the file was created.
);
CREATE TABLE file (
    -- A row per file measured.
    id integer primary key,
    path text,
    unique (path)
);
CREATE TABLE context (
    -- A row per context measured.
    id integer primary key,
    context text,
    unique (context)
);
CREATE TABLE line_bits (
    -- If recording lines, a row per context per file executed.
    -- All of the line numbers for that file/context are in one numbits.
    file_id integer,            -- foreign key to `file`.
    context_id integer,         -- foreign key to `context`.
    numbits blob,               -- see the numbits functions in coverage.numbits
    foreign key (file_id) references file (id),
    foreign key (context_id) references context (id),
    unique (file_id, context_id)
);
CREATE TABLE arc (
    -- If recording branches, a row per context per from/to line transition executed.
    file_id integer,            -- foreign key to `file`.
    context_id integer,         -- foreign key to `context`.
    fromno integer,             -- line number jumped from.
    tono integer,               -- line number jumped to.
    foreign key (file_id) references file (id),
    foreign key (context_id) references context (id),
    unique (file_id, context_id, fromno, tono)
);
CREATE TABLE tracer (
    -- A row per file indicating the tracer used for that file.
    file_id integer primary key,
    tracer text,
    foreign key (file_id) references file (id)
);
```

html:
<p><a href="https://twitter.com/mariatta/status/1499863816489734146" rel="nofollow">This tweet</a> from @Mariatta tipped me off to the ability to measure "contexts" when <a href="https://coverage.readthedocs.io/en/6.3.2/contexts.html#context-reporting" rel="nofollow">running coverage</a> - as a way to tell which tests exercise which specific lines of code.</p>
<p>My <a href="https://github.com/simonw/sqlite-utils">sqlite-utils</a> project uses <code>pytest</code> for the test suite. I decided to figure out how to get this working with <a href="https://pypi.org/project/pytest-cov/" rel="nofollow">pytest-cov</a>.</p>
<p>After some experimentation, this is the recipe that worked for me:</p>
<pre><code># In the virtual environment, make sure pytest-cov is installed:
% pip install pytest-cov
# First, run pytest to calculate coverage of the `sqlite_utils` package, with context
% pytest --cov=sqlite_utils --cov-context=test
# The .coverage file is actually a SQLite database:
% ls -lah .coverage
-rw-r--r--@ 1 simon  staff   716K Mar  4 16:39 .coverage
# This command generates the HTML coverage report in `htmlcov/`
% coverage html --show-contexts
# Open the report in a browser`
% open htmlcov/index.html
</code></pre>
<p>Here's what one of the pages looks like, displaying the context for some lines of code:</p>
<p><a href="https://user-images.githubusercontent.com/9599/156860441-66e35994-653a-4ab7-b690-4d901fc57750.png" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/156860441-66e35994-653a-4ab7-b690-4d901fc57750.png" alt="The code has an expandable section which reveals which tests executed each individual line." style="max-width:100%;"></a></p>
<h2>
<a id="user-content-the-coverage-schema" class="anchor" href="#the-coverage-schema" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>The .coverage schema</h2>
<p>Since <code>.coverage</code> is a SQLite database, here's the schema - generated using <code>sqlite-utils schema .coverage</code>:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> <span class="pl-en">coverage_schema</span> (
    <span class="pl-c"><span class="pl-c">--</span> One row, to record the version of the schema in this db.</span>
    version <span class="pl-k">integer</span>
);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> <span class="pl-en">meta</span> (
    <span class="pl-c"><span class="pl-c">--</span> Key-value pairs, to record metadata about the data</span>
    key <span class="pl-k">text</span>,
    value <span class="pl-k">text</span>,
    unique (key)
    <span class="pl-c"><span class="pl-c">--</span> Keys:</span>
    <span class="pl-c"><span class="pl-c">--</span>  'has_arcs' boolean      -- Is this data recording branches?</span>
    <span class="pl-c"><span class="pl-c">--</span>  'sys_argv' text         -- The coverage command line that recorded the data.</span>
    <span class="pl-c"><span class="pl-c">--</span>  'version' text          -- The version of coverage.py that made the file.</span>
    <span class="pl-c"><span class="pl-c">--</span>  'when' text             -- Datetime when the file was created.</span>
);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> <span class="pl-en">file</span> (
    <span class="pl-c"><span class="pl-c">--</span> A row per file measured.</span>
    id <span class="pl-k">integer</span> <span class="pl-k">primary key</span>,
    <span class="pl-k">path</span> <span class="pl-k">text</span>,
    unique (<span class="pl-k">path</span>)
);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> <span class="pl-en">context</span> (
    <span class="pl-c"><span class="pl-c">--</span> A row per context measured.</span>
    id <span class="pl-k">integer</span> <span class="pl-k">primary key</span>,
    context <span class="pl-k">text</span>,
    unique (context)
);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> <span class="pl-en">line_bits</span> (
    <span class="pl-c"><span class="pl-c">--</span> If recording lines, a row per context per file executed.</span>
    <span class="pl-c"><span class="pl-c">--</span> All of the line numbers for that file/context are in one numbits.</span>
    file_id <span class="pl-k">integer</span>,            <span class="pl-c"><span class="pl-c">--</span> foreign key to `file`.</span>
    context_id <span class="pl-k">integer</span>,         <span class="pl-c"><span class="pl-c">--</span> foreign key to `context`.</span>
    numbits blob,               <span class="pl-c"><span class="pl-c">--</span> see the numbits functions in coverage.numbits</span>
    <span class="pl-k">foreign key</span> (file_id) <span class="pl-k">references</span> file (id),
    <span class="pl-k">foreign key</span> (context_id) <span class="pl-k">references</span> context (id),
    unique (file_id, context_id)
);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> <span class="pl-en">arc</span> (
    <span class="pl-c"><span class="pl-c">--</span> If recording branches, a row per context per from/to line transition executed.</span>
    file_id <span class="pl-k">integer</span>,            <span class="pl-c"><span class="pl-c">--</span> foreign key to `file`.</span>
    context_id <span class="pl-k">integer</span>,         <span class="pl-c"><span class="pl-c">--</span> foreign key to `context`.</span>
    fromno <span class="pl-k">integer</span>,             <span class="pl-c"><span class="pl-c">--</span> line number jumped from.</span>
    tono <span class="pl-k">integer</span>,               <span class="pl-c"><span class="pl-c">--</span> line number jumped to.</span>
    <span class="pl-k">foreign key</span> (file_id) <span class="pl-k">references</span> file (id),
    <span class="pl-k">foreign key</span> (context_id) <span class="pl-k">references</span> context (id),
    unique (file_id, context_id, fromno, tono)
);
<span class="pl-k">CREATE</span> <span class="pl-k">TABLE</span> <span class="pl-en">tracer</span> (
    <span class="pl-c"><span class="pl-c">--</span> A row per file indicating the tracer used for that file.</span>
    file_id <span class="pl-k">integer</span> <span class="pl-k">primary key</span>,
    tracer <span class="pl-k">text</span>,
    <span class="pl-k">foreign key</span> (file_id) <span class="pl-k">references</span> file (id)
);</pre></div>


path: google-sheets_concatenate.md
topic: google-sheets
title: Concatenating strings and newlines in Google Sheets
url: https://github.com/simonw/til/blob/main/google-sheets/concatenate.md
created: 2022-03-15T10:11:27-07:00
created_utc: 2022-03-15T17:11:27+00:00
updated: 2022-03-15T12:20:56-07:00
updated_utc: 2022-03-15T19:20:56+00:00
shot_hash: 5b7c6c0a7e1729ff9953a80a1779049e
slug: concatenate
body:
I was asked if there was a way to run [shot-scraper](https://github.com/simonw/shot-scraper) against a list of URLs in a Google Sheet.

I came up with [this example sheet](https://docs.google.com/spreadsheets/d/1VCrG1VzGtdv0DxCdCFf7G8yIurd63O6rvp1FkLyp4kI/edit#gid=0) which uses a formula to code-generate the YAML configuration needed by `shot-scraper multi`:

<img width="542" alt="image" src="https://user-images.githubusercontent.com/9599/158430766-cc4b7e3d-2ff8-4ab1-a83c-a80ba65244a9.png">

The formula is:

    =if(isblank(A4), "", "- url: " & A4)

The `&` character is used for string concatenation.

The `if(condition, if-true, if-false)` function is used to return an empty string if the cell is blank, or a concatenated string otherwise.

## Adding newlines to the output text

I decided to try generating this output instead:

```yaml
- url: https://simonwillison.net/
  width: 800
- url: https://datasette.io/
  width: 800
- url: https://www.example.com/
  width: 800
```

My first attempt was to include the newline in the formula - you can do this by hitting Ctrl+Enter while editing the cell:

```
=if(isblank(A2), "", "- url: " & A2 & "
  width: 800")
```
This looks like it does the right thing:

<img width="502" alt="image" src="https://user-images.githubusercontent.com/9599/158431961-c0063dc3-ccf9-4173-9a08-126514416042.png">

But... when you copy and paste out the result, you get additional unwanted double quotes!

```
"- url: https://simonwillison.net/
  width: 800"
"- url: https://datasette.io/
  width: 800"
"- url: https://www.example.com/
  width: 800"
```
Thanks to [this answer](https://webapps.stackexchange.com/a/104802) on StackOverflow I found a workaround.

Use `char(13)` where you want a newline. This is actually the character code for `\r` rather than `\n` - the result looks like this:

<img width="502" alt="image" src="https://user-images.githubusercontent.com/9599/158432445-72ded18f-d406-46f9-8918-e7cada57b9d2.png">

But... when you copy and paste out the column into a VS Code file you get this:
```
- url: https://simonwillison.net/
  width: 800
- url: https://datasette.io/
  width: 800
- url: https://www.example.com/
  width: 800
```
Hitting save in VS Code (for me on my Mac) resulted in a file with `\n` lines in it.
```
% python -c "print(repr(open('/tmp/saved.txt', 'rb').read()))"
b'- url: https://simonwillison.net/\n  width: 800\n- url: https://datasette.io/\n  width: 800\n- url: https://www.example.com/\n  width: 800'
```
You can confirm that the original clipboard text contained `\r` and not `\n` by doing this:

```
% pbpaste | python -c 'print(repr(__import__("sys").stdin.buffer.read()))'
b'- url: https://simonwillison.net/\r  width: 800\n- url: https://datasette.io/\r  width: 800\n- url: https://www.example.com/\r  width: 800'
```

html:
<p>I was asked if there was a way to run <a href="https://github.com/simonw/shot-scraper">shot-scraper</a> against a list of URLs in a Google Sheet.</p>
<p>I came up with <a href="https://docs.google.com/spreadsheets/d/1VCrG1VzGtdv0DxCdCFf7G8yIurd63O6rvp1FkLyp4kI/edit#gid=0" rel="nofollow">this example sheet</a> which uses a formula to code-generate the YAML configuration needed by <code>shot-scraper multi</code>:</p>
<p><a href="https://user-images.githubusercontent.com/9599/158430766-cc4b7e3d-2ff8-4ab1-a83c-a80ba65244a9.png" target="_blank" rel="nofollow"><img width="542" alt="image" src="https://user-images.githubusercontent.com/9599/158430766-cc4b7e3d-2ff8-4ab1-a83c-a80ba65244a9.png" style="max-width:100%;"></a></p>
<p>The formula is:</p>
<pre><code>=if(isblank(A4), "", "- url: " &amp; A4)
</code></pre>
<p>The <code>&amp;</code> character is used for string concatenation.</p>
<p>The <code>if(condition, if-true, if-false)</code> function is used to return an empty string if the cell is blank, or a concatenated string otherwise.</p>
<h2>
<a id="user-content-adding-newlines-to-the-output-text" class="anchor" href="#adding-newlines-to-the-output-text" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Adding newlines to the output text</h2>
<p>I decided to try generating this output instead:</p>
<div class="highlight highlight-source-yaml"><pre>- <span class="pl-ent">url</span>: <span class="pl-s">https://simonwillison.net/</span>
  <span class="pl-ent">width</span>: <span class="pl-c1">800</span>
- <span class="pl-ent">url</span>: <span class="pl-s">https://datasette.io/</span>
  <span class="pl-ent">width</span>: <span class="pl-c1">800</span>
- <span class="pl-ent">url</span>: <span class="pl-s">https://www.example.com/</span>
  <span class="pl-ent">width</span>: <span class="pl-c1">800</span></pre></div>
<p>My first attempt was to include the newline in the formula - you can do this by hitting Ctrl+Enter while editing the cell:</p>
<pre><code>=if(isblank(A2), "", "- url: " &amp; A2 &amp; "
  width: 800")
</code></pre>
<p>This looks like it does the right thing:</p>
<p><a href="https://user-images.githubusercontent.com/9599/158431961-c0063dc3-ccf9-4173-9a08-126514416042.png" target="_blank" rel="nofollow"><img width="502" alt="image" src="https://user-images.githubusercontent.com/9599/158431961-c0063dc3-ccf9-4173-9a08-126514416042.png" style="max-width:100%;"></a></p>
<p>But... when you copy and paste out the result, you get additional unwanted double quotes!</p>
<pre><code>"- url: https://simonwillison.net/
  width: 800"
"- url: https://datasette.io/
  width: 800"
"- url: https://www.example.com/
  width: 800"
</code></pre>
<p>Thanks to <a href="https://webapps.stackexchange.com/a/104802" rel="nofollow">this answer</a> on StackOverflow I found a workaround.</p>
<p>Use <code>char(13)</code> where you want a newline. This is actually the character code for <code>\r</code> rather than <code>\n</code> - the result looks like this:</p>
<p><a href="https://user-images.githubusercontent.com/9599/158432445-72ded18f-d406-46f9-8918-e7cada57b9d2.png" target="_blank" rel="nofollow"><img width="502" alt="image" src="https://user-images.githubusercontent.com/9599/158432445-72ded18f-d406-46f9-8918-e7cada57b9d2.png" style="max-width:100%;"></a></p>
<p>But... when you copy and paste out the column into a VS Code file you get this:</p>
<pre><code>- url: https://simonwillison.net/
  width: 800
- url: https://datasette.io/
  width: 800
- url: https://www.example.com/
  width: 800
</code></pre>
<p>Hitting save in VS Code (for me on my Mac) resulted in a file with <code>\n</code> lines in it.</p>
<pre><code>% python -c "print(repr(open('/tmp/saved.txt', 'rb').read()))"
b'- url: https://simonwillison.net/\n  width: 800\n- url: https://datasette.io/\n  width: 800\n- url: https://www.example.com/\n  width: 800'
</code></pre>
<p>You can confirm that the original clipboard text contained <code>\r</code> and not <code>\n</code> by doing this:</p>
<pre><code>% pbpaste | python -c 'print(repr(__import__("sys").stdin.buffer.read()))'
b'- url: https://simonwillison.net/\r  width: 800\n- url: https://datasette.io/\r  width: 800\n- url: https://www.example.com/\r  width: 800'
</code></pre>


path: pytest_async-fixtures.md
topic: pytest
title: Async fixtures with pytest-asyncio
url: https://github.com/simonw/til/blob/main/pytest/async-fixtures.md
created: 2022-03-19T15:48:59-07:00
created_utc: 2022-03-19T22:48:59+00:00
updated: 2022-03-19T15:48:59-07:00
updated_utc: 2022-03-19T22:48:59+00:00
shot_hash: 7eb855c34dc73456ac1fd9335dfea05e
slug: async-fixtures
body:
I wanted to use a fixture with `pytest-asyncio` that was itsef as `async def` function, so that it could execute `await` statements.

Since I'm using a `pytest.ini` file containing `asyncio_mode = strict` I had to use the `@pytest_asyncio.fixture` fixture to get this to work. Without that fixture I got this error:

```
    assert _has_explicit_asyncio_mark(fixturedef.func)
E   AssertionError: assert False
E    +  where False = _has_explicit_asyncio_mark(<function ds_with_route at 0x11332d2d0>)
E    +    where <function ds_with_route at 0x11332d2d0> = <FixtureDef argname='ds_with_route' scope='function' baseid='tests/test_routes.py'>.func
```

Swapping `@pytest.fixture` for `@pytest_asyncio.fixture` fixed this problem:

```python
import pytest_asyncio

@pytest_asyncio.fixture
async def ds_with_route():
    ds = Datasette()
    db = Database(ds, is_memory=True, memory_name="route-name-db")
    ds.add_database(db, name="name", route="route-name")
    await db.execute_write_script(
        """
        create table if not exists t (id integer primary key);
        insert or replace into t (id) values (1);
    """
    )
    return ds
```

html:
<p>I wanted to use a fixture with <code>pytest-asyncio</code> that was itsef as <code>async def</code> function, so that it could execute <code>await</code> statements.</p>
<p>Since I'm using a <code>pytest.ini</code> file containing <code>asyncio_mode = strict</code> I had to use the <code>@pytest_asyncio.fixture</code> fixture to get this to work. Without that fixture I got this error:</p>
<pre><code>    assert _has_explicit_asyncio_mark(fixturedef.func)
E   AssertionError: assert False
E    +  where False = _has_explicit_asyncio_mark(&lt;function ds_with_route at 0x11332d2d0&gt;)
E    +    where &lt;function ds_with_route at 0x11332d2d0&gt; = &lt;FixtureDef argname='ds_with_route' scope='function' baseid='tests/test_routes.py'&gt;.func
</code></pre>
<p>Swapping <code>@pytest.fixture</code> for <code>@pytest_asyncio.fixture</code> fixed this problem:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">pytest_asyncio</span>

<span class="pl-en">@<span class="pl-s1">pytest_asyncio</span>.<span class="pl-s1">fixture</span></span>
<span class="pl-k">async</span> <span class="pl-k">def</span> <span class="pl-en">ds_with_route</span>():
    <span class="pl-s1">ds</span> <span class="pl-c1">=</span> <span class="pl-v">Datasette</span>()
    <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-v">Database</span>(<span class="pl-s1">ds</span>, <span class="pl-s1">is_memory</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">memory_name</span><span class="pl-c1">=</span><span class="pl-s">"route-name-db"</span>)
    <span class="pl-s1">ds</span>.<span class="pl-en">add_database</span>(<span class="pl-s1">db</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"name"</span>, <span class="pl-s1">route</span><span class="pl-c1">=</span><span class="pl-s">"route-name"</span>)
    <span class="pl-k">await</span> <span class="pl-s1">db</span>.<span class="pl-en">execute_write_script</span>(
        <span class="pl-s">"""</span>
<span class="pl-s">        create table if not exists t (id integer primary key);</span>
<span class="pl-s">        insert or replace into t (id) values (1);</span>
<span class="pl-s">    """</span>
    )
    <span class="pl-k">return</span> <span class="pl-s1">ds</span></pre></div>


path: sqlite_simple-recursive-cte.md
topic: sqlite
title: The simplest recursive CTE
url: https://github.com/simonw/til/blob/main/sqlite/simple-recursive-cte.md
created: 2022-03-20T21:32:23-07:00
created_utc: 2022-03-21T04:32:23+00:00
updated: 2022-03-25T12:23:53-07:00
updated_utc: 2022-03-25T19:23:53+00:00
shot_hash: bb0418730118bc597a8eb6c49a1f144d
slug: simple-recursive-cte
body:
I found this really simple recursive CTE useful for ensuring I understood how to write recursive CTEs.
```sql
with recursive counter(x) as (
  select 0
    union
  select x + 1 from counter
)
select * from counter limit 5;
```
This query [returns five rows](https://latest.datasette.io/_memory?sql=with+recursive+counter%28x%29+as+%28%0D%0A++select+0%0D%0A++++union%0D%0A++select+x+%2B+1+from+counter%0D%0A%29%0D%0Aselect+*+from+counter+limit+10%3B) from a single column `x` - from 0 to 4.

|   x |
|-----|
|   0 |
|   1 |
|   2 |
|   3 |
|   4 |

If you write `with recursive counter as ...`, omitting the `(x)`, you get the following error:

> `no such column: x`

You can fix that by assigning `x` as the alias in the first part of that union:
```sql
with recursive counter as (
  select 0 as x
    union
  select x + 1 from counter
)
select * from counter limit 5;
```
So that `counter(x)` formulation is really just a way to define the column names up front.

This query returns two columns, `x` and `y`:

```sql
with recursive counter(x, y) as (
  select 0 as x, 1 as y
    union
  select x + 1, y + 2 from counter
)
select * from counter limit 5;
```
|   x |   y |
|-----|-----|
|   0 |   1 |
|   1 |   3 |
|   2 |   5 |
|   3 |   7 |
|   4 |   9 |

html:
<p>I found this really simple recursive CTE useful for ensuring I understood how to write recursive CTEs.</p>
<div class="highlight highlight-source-sql"><pre>with recursive counter(x) <span class="pl-k">as</span> (
  <span class="pl-k">select</span> <span class="pl-c1">0</span>
    <span class="pl-k">union</span>
  <span class="pl-k">select</span> x <span class="pl-k">+</span> <span class="pl-c1">1</span> <span class="pl-k">from</span> counter
)
<span class="pl-k">select</span> <span class="pl-k">*</span> <span class="pl-k">from</span> counter <span class="pl-k">limit</span> <span class="pl-c1">5</span>;</pre></div>
<p>This query <a href="https://latest.datasette.io/_memory?sql=with+recursive+counter%28x%29+as+%28%0D%0A++select+0%0D%0A++++union%0D%0A++select+x+%2B+1+from+counter%0D%0A%29%0D%0Aselect+*+from+counter+limit+10%3B" rel="nofollow">returns five rows</a> from a single column <code>x</code> - from 0 to 4.</p>
<table>
<thead>
<tr>
<th>x</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
</tr>
<tr>
<td>1</td>
</tr>
<tr>
<td>2</td>
</tr>
<tr>
<td>3</td>
</tr>
<tr>
<td>4</td>
</tr>
</tbody>
</table>
<p>If you write <code>with recursive counter as ...</code>, omitting the <code>(x)</code>, you get the following error:</p>
<blockquote>
<p><code>no such column: x</code></p>
</blockquote>
<p>You can fix that by assigning <code>x</code> as the alias in the first part of that union:</p>
<div class="highlight highlight-source-sql"><pre>with recursive counter <span class="pl-k">as</span> (
  <span class="pl-k">select</span> <span class="pl-c1">0</span> <span class="pl-k">as</span> x
    <span class="pl-k">union</span>
  <span class="pl-k">select</span> x <span class="pl-k">+</span> <span class="pl-c1">1</span> <span class="pl-k">from</span> counter
)
<span class="pl-k">select</span> <span class="pl-k">*</span> <span class="pl-k">from</span> counter <span class="pl-k">limit</span> <span class="pl-c1">5</span>;</pre></div>
<p>So that <code>counter(x)</code> formulation is really just a way to define the column names up front.</p>
<p>This query returns two columns, <code>x</code> and <code>y</code>:</p>
<div class="highlight highlight-source-sql"><pre>with recursive counter(x, y) <span class="pl-k">as</span> (
  <span class="pl-k">select</span> <span class="pl-c1">0</span> <span class="pl-k">as</span> x, <span class="pl-c1">1</span> <span class="pl-k">as</span> y
    <span class="pl-k">union</span>
  <span class="pl-k">select</span> x <span class="pl-k">+</span> <span class="pl-c1">1</span>, y <span class="pl-k">+</span> <span class="pl-c1">2</span> <span class="pl-k">from</span> counter
)
<span class="pl-k">select</span> <span class="pl-k">*</span> <span class="pl-k">from</span> counter <span class="pl-k">limit</span> <span class="pl-c1">5</span>;</pre></div>
<table>
<thead>
<tr>
<th>x</th>
<th>y</th>
</tr>
</thead>
<tbody>
<tr>
<td>0</td>
<td>1</td>
</tr>
<tr>
<td>1</td>
<td>3</td>
</tr>
<tr>
<td>2</td>
<td>5</td>
</tr>
<tr>
<td>3</td>
<td>7</td>
</tr>
<tr>
<td>4</td>
<td>9</td>
</tr>
</tbody>
</table>


path: sqlite_counting-vm-ops.md
topic: sqlite
title: Counting SQLite virtual machine operations
url: https://github.com/simonw/til/blob/main/sqlite/counting-vm-ops.md
created: 2022-03-20T21:45:22-07:00
created_utc: 2022-03-21T04:45:22+00:00
updated: 2022-03-20T21:45:22-07:00
updated_utc: 2022-03-21T04:45:22+00:00
shot_hash: 2b53d64907cf27a549b1be9ab869b1cd
slug: counting-vm-ops
body:
When SQLite executes a query, it does so by executing a sequence of virtual machine operations.

There are mechanisms for cancelling a query after a specific number of these operations, to protect against long-running queries.

To work with those mechanisms, it's useful to get a feel for how many operations different queries execute.

Thanks to [tips on the SQLite forum](https://sqlite.org/forum/forumpost/dfb178d58c185ee4) I now know how to count these operations using the `sqlite3` command line tool, using `.stats vmstats` to turn on display of that number after each query:

```
% sqlite3 fixtures.db 
SQLite version 3.36.0 2021-06-18 18:58:49
Enter ".help" for usage hints.
sqlite> .stats vmstep
sqlite> select * from facetable;
1|2019-01-14 08:00:00|1|1|CA|1|Mission|["tag1", "tag2"]|[{"foo": "bar"}]|one
2|2019-01-14 08:00:00|1|1|CA|1|Dogpatch|["tag1", "tag3"]|[]|two
3|2019-01-14 08:00:00|1|1|CA|1|SOMA|[]|[]|
4|2019-01-14 08:00:00|1|1|CA|1|Tenderloin|[]|[]|
5|2019-01-15 08:00:00|1|1|CA|1|Bernal Heights|[]|[]|
6|2019-01-15 08:00:00|1|1|CA|1|Hayes Valley|[]|[]|
7|2019-01-15 08:00:00|1|1|CA|2|Hollywood|[]|[]|
8|2019-01-15 08:00:00|1|1|CA|2|Downtown|[]|[]|
9|2019-01-16 08:00:00|1|1|CA|2|Los Feliz|[]|[]|
10|2019-01-16 08:00:00|1|1|CA|2|Koreatown|[]|[]|
11|2019-01-16 08:00:00|1|1|MI|3|Downtown|[]|[]|
12|2019-01-17 08:00:00|1|1|MI|3|Greektown|[]|[]|
13|2019-01-17 08:00:00|1|1|MI|3|Corktown|[]|[]|
14|2019-01-17 08:00:00|1|1|MI|3|Mexicantown|[]|[]|
15|2019-01-17 08:00:00|2|0|MC|4|Arcadia Planitia|[]|[]|
VM-steps: 187
sqlite> select * from facetable limit 3;
1|2019-01-14 08:00:00|1|1|CA|1|Mission|["tag1", "tag2"]|[{"foo": "bar"}]|one
2|2019-01-14 08:00:00|1|1|CA|1|Dogpatch|["tag1", "tag3"]|[]|two
3|2019-01-14 08:00:00|1|1|CA|1|SOMA|[]|[]|
VM-steps: 46
```
Using `.stats on` shows a more detailed group of statistics about the query:
```
sqlite> .stats on
sqlite> select * from facetable limit 3;
1|2019-01-14 08:00:00|1|1|CA|1|Mission|["tag1", "tag2"]|[{"foo": "bar"}]|one
2|2019-01-14 08:00:00|1|1|CA|1|Dogpatch|["tag1", "tag3"]|[]|two
3|2019-01-14 08:00:00|1|1|CA|1|SOMA|[]|[]|
Memory Used:                         195520 (max 195776) bytes
Number of Outstanding Allocations:   525 (max 526)
Number of Pcache Overflow Bytes:     5696 (max 5696) bytes
Largest Allocation:                  122400 bytes
Largest Pcache Allocation:           4104 bytes
Lookaside Slots Used:                94 (max 122)
Successful lookaside attempts:       431
Lookaside failures due to size:      4
Lookaside failures due to OOM:       0
Pager Heap Usage:                    23008 bytes
Page cache hits:                     6
Page cache misses:                   4
Page cache writes:                   0
Page cache spills:                   0
Schema Heap Usage:                   25216 bytes
Statement Heap/Lookaside Usage:      15504 bytes
Fullscan Steps:                      2
Sort Operations:                     0
Autoindex Inserts:                   0
Virtual Machine Steps:               46
Reprepare operations:                0
Number of times run:                 1
Memory used by prepared stmt:        15504
```
There is also a `sqlite_stmt` virtual table ([documented here](https://sqlite.org/stmt.html)) which can be enabled using a loadable module, but I have not yet managed to get this to compile and load into the Python `sqlite3` environment.

## Cancelling queries after a specified number of opcodes

You can set an upper limit on the number of opcodes in the `sqlite3` command line using `.progress 100 --limit 1`, as [described here](https://sqlite.org/forum/forumpost/4a669c7b50f9a64c):

```
sqlite> .progress 100 --limit 1
sqlite> select * from facetable limit 3;
1|2019-01-14 08:00:00|1|1|CA|1|Mission|["tag1", "tag2"]|[{"foo": "bar"}]|one
2|2019-01-14 08:00:00|1|1|CA|1|Dogpatch|["tag1", "tag3"]|[]|two
3|2019-01-14 08:00:00|1|1|CA|1|SOMA|[]|[]|
sqlite> select * from facetable;
1|2019-01-14 08:00:00|1|1|CA|1|Mission|["tag1", "tag2"]|[{"foo": "bar"}]|one
2|2019-01-14 08:00:00|1|1|CA|1|Dogpatch|["tag1", "tag3"]|[]|two
3|2019-01-14 08:00:00|1|1|CA|1|SOMA|[]|[]|
4|2019-01-14 08:00:00|1|1|CA|1|Tenderloin|[]|[]|
5|2019-01-15 08:00:00|1|1|CA|1|Bernal Heights|[]|[]|
6|2019-01-15 08:00:00|1|1|CA|1|Hayes Valley|[]|[]|
7|2019-01-15 08:00:00|1|1|CA|2|Hollywood|[]|[]|
Progress limit reached (1)
Error: interrupted
```
In Python code this can be achieved like so:

```python
conn.set_progress_handler(lambda: 1, 100)
```
This sets a progress handler which will be called after every 100 opcodes - but by returning `1` the handler causes the query to be cancelled:
```pycon
>>> import sqlite3
>>> conn = sqlite3.connect("fixtures.db")
>>> conn.set_progress_handler(lambda: 1, 200)
>>> conn.execute("select * from facetable limit 3").rows()
# Outputs rows
>>> conn.execute("select * from facetable").fetchall()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
sqlite3.OperationalError: interrupted
```
This didn't exactly work how I expected - I had to change the number from 100 to 200 and I'm not sure why. But it illustrates the principle.

html:
<p>When SQLite executes a query, it does so by executing a sequence of virtual machine operations.</p>
<p>There are mechanisms for cancelling a query after a specific number of these operations, to protect against long-running queries.</p>
<p>To work with those mechanisms, it's useful to get a feel for how many operations different queries execute.</p>
<p>Thanks to <a href="https://sqlite.org/forum/forumpost/dfb178d58c185ee4" rel="nofollow">tips on the SQLite forum</a> I now know how to count these operations using the <code>sqlite3</code> command line tool, using <code>.stats vmstats</code> to turn on display of that number after each query:</p>
<pre><code>% sqlite3 fixtures.db 
SQLite version 3.36.0 2021-06-18 18:58:49
Enter ".help" for usage hints.
sqlite&gt; .stats vmstep
sqlite&gt; select * from facetable;
1|2019-01-14 08:00:00|1|1|CA|1|Mission|["tag1", "tag2"]|[{"foo": "bar"}]|one
2|2019-01-14 08:00:00|1|1|CA|1|Dogpatch|["tag1", "tag3"]|[]|two
3|2019-01-14 08:00:00|1|1|CA|1|SOMA|[]|[]|
4|2019-01-14 08:00:00|1|1|CA|1|Tenderloin|[]|[]|
5|2019-01-15 08:00:00|1|1|CA|1|Bernal Heights|[]|[]|
6|2019-01-15 08:00:00|1|1|CA|1|Hayes Valley|[]|[]|
7|2019-01-15 08:00:00|1|1|CA|2|Hollywood|[]|[]|
8|2019-01-15 08:00:00|1|1|CA|2|Downtown|[]|[]|
9|2019-01-16 08:00:00|1|1|CA|2|Los Feliz|[]|[]|
10|2019-01-16 08:00:00|1|1|CA|2|Koreatown|[]|[]|
11|2019-01-16 08:00:00|1|1|MI|3|Downtown|[]|[]|
12|2019-01-17 08:00:00|1|1|MI|3|Greektown|[]|[]|
13|2019-01-17 08:00:00|1|1|MI|3|Corktown|[]|[]|
14|2019-01-17 08:00:00|1|1|MI|3|Mexicantown|[]|[]|
15|2019-01-17 08:00:00|2|0|MC|4|Arcadia Planitia|[]|[]|
VM-steps: 187
sqlite&gt; select * from facetable limit 3;
1|2019-01-14 08:00:00|1|1|CA|1|Mission|["tag1", "tag2"]|[{"foo": "bar"}]|one
2|2019-01-14 08:00:00|1|1|CA|1|Dogpatch|["tag1", "tag3"]|[]|two
3|2019-01-14 08:00:00|1|1|CA|1|SOMA|[]|[]|
VM-steps: 46
</code></pre>
<p>Using <code>.stats on</code> shows a more detailed group of statistics about the query:</p>
<pre><code>sqlite&gt; .stats on
sqlite&gt; select * from facetable limit 3;
1|2019-01-14 08:00:00|1|1|CA|1|Mission|["tag1", "tag2"]|[{"foo": "bar"}]|one
2|2019-01-14 08:00:00|1|1|CA|1|Dogpatch|["tag1", "tag3"]|[]|two
3|2019-01-14 08:00:00|1|1|CA|1|SOMA|[]|[]|
Memory Used:                         195520 (max 195776) bytes
Number of Outstanding Allocations:   525 (max 526)
Number of Pcache Overflow Bytes:     5696 (max 5696) bytes
Largest Allocation:                  122400 bytes
Largest Pcache Allocation:           4104 bytes
Lookaside Slots Used:                94 (max 122)
Successful lookaside attempts:       431
Lookaside failures due to size:      4
Lookaside failures due to OOM:       0
Pager Heap Usage:                    23008 bytes
Page cache hits:                     6
Page cache misses:                   4
Page cache writes:                   0
Page cache spills:                   0
Schema Heap Usage:                   25216 bytes
Statement Heap/Lookaside Usage:      15504 bytes
Fullscan Steps:                      2
Sort Operations:                     0
Autoindex Inserts:                   0
Virtual Machine Steps:               46
Reprepare operations:                0
Number of times run:                 1
Memory used by prepared stmt:        15504
</code></pre>
<p>There is also a <code>sqlite_stmt</code> virtual table (<a href="https://sqlite.org/stmt.html" rel="nofollow">documented here</a>) which can be enabled using a loadable module, but I have not yet managed to get this to compile and load into the Python <code>sqlite3</code> environment.</p>
<h2>
<a id="user-content-cancelling-queries-after-a-specified-number-of-opcodes" class="anchor" href="#cancelling-queries-after-a-specified-number-of-opcodes" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Cancelling queries after a specified number of opcodes</h2>
<p>You can set an upper limit on the number of opcodes in the <code>sqlite3</code> command line using <code>.progress 100 --limit 1</code>, as <a href="https://sqlite.org/forum/forumpost/4a669c7b50f9a64c" rel="nofollow">described here</a>:</p>
<pre><code>sqlite&gt; .progress 100 --limit 1
sqlite&gt; select * from facetable limit 3;
1|2019-01-14 08:00:00|1|1|CA|1|Mission|["tag1", "tag2"]|[{"foo": "bar"}]|one
2|2019-01-14 08:00:00|1|1|CA|1|Dogpatch|["tag1", "tag3"]|[]|two
3|2019-01-14 08:00:00|1|1|CA|1|SOMA|[]|[]|
sqlite&gt; select * from facetable;
1|2019-01-14 08:00:00|1|1|CA|1|Mission|["tag1", "tag2"]|[{"foo": "bar"}]|one
2|2019-01-14 08:00:00|1|1|CA|1|Dogpatch|["tag1", "tag3"]|[]|two
3|2019-01-14 08:00:00|1|1|CA|1|SOMA|[]|[]|
4|2019-01-14 08:00:00|1|1|CA|1|Tenderloin|[]|[]|
5|2019-01-15 08:00:00|1|1|CA|1|Bernal Heights|[]|[]|
6|2019-01-15 08:00:00|1|1|CA|1|Hayes Valley|[]|[]|
7|2019-01-15 08:00:00|1|1|CA|2|Hollywood|[]|[]|
Progress limit reached (1)
Error: interrupted
</code></pre>
<p>In Python code this can be achieved like so:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">conn</span>.<span class="pl-en">set_progress_handler</span>(<span class="pl-k">lambda</span>: <span class="pl-c1">1</span>, <span class="pl-c1">100</span>)</pre></div>
<p>This sets a progress handler which will be called after every 100 opcodes - but by returning <code>1</code> the handler causes the query to be cancelled:</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; <span class="pl-k">import</span> sqlite3
&gt;&gt;&gt; conn <span class="pl-k">=</span> sqlite3.connect(<span class="pl-s"><span class="pl-pds">"</span>fixtures.db<span class="pl-pds">"</span></span>)
&gt;&gt;&gt; conn.set_progress_handler(<span class="pl-k">lambda</span>: <span class="pl-c1">1</span>, <span class="pl-c1">200</span>)
&gt;&gt;&gt; conn.execute(<span class="pl-s"><span class="pl-pds">"</span>select * from facetable limit 3<span class="pl-pds">"</span></span>).rows()
# Outputs rows
&gt;&gt;&gt; conn.execute(<span class="pl-s"><span class="pl-pds">"</span>select * from facetable<span class="pl-pds">"</span></span>).fetchall()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
sqlite3.OperationalError: interrupted</pre></div>
<p>This didn't exactly work how I expected - I had to change the number from 100 to 200 and I'm not sure why. But it illustrates the principle.</p>


path: github-actions_npm-cache-with-npx-no-package.md
topic: github-actions
title: Using the GitHub Actions cache with npx and no package.json
url: https://github.com/simonw/til/blob/main/github-actions/npm-cache-with-npx-no-package.md
created: 2022-03-22T14:57:52-07:00
created_utc: 2022-03-22T21:57:52+00:00
updated: 2022-03-22T20:54:04-07:00
updated_utc: 2022-03-23T03:54:04+00:00
shot_hash: 3edb0616e57f1ee464602030bbf43ced
slug: npm-cache-with-npx-no-package
body:
Some of my repositories have GitHub Actions workflows that execute commands using `npx`, for example my [graphql-scraper](https://github.com/simonw/graphql-scraper) repo using `npx` to [install and run](https://github.com/simonw/help-scraper/blob/54ef9b1fa579dc2d3f68055ffdac9996fd6dae9c/.github/workflows/scrape.yml#L76-L80) the `get-graphql-schema` tool:

    npx get-graphql-schema https://api.fly.io/graphql > flyctl/fly.graphql

This was downloading the package from npm every time the action ran. This feels rude! I wanted to use the [actions/cache](https://github.com/actions/cache) mechanism to cache the download in between runs.

One problem: the documentation for that action assumes that projects are using a `package.json` file which can be used as part of the cache key. But my projects don't have one of those files, they just want to use `npx` utilities as part of their workflows.

Here's the pattern I came up with to configure the cache:

```yaml
    - uses: actions/setup-node@v2
      with:
        node-version: '14'
        cache: 'npm'
        cache-dependency-path: '**/.github/workflows/*.yml'
```
This is using [actions/setup-node](https://github.com/actions/setup-node) which itself can setup the cache. I'm telling it that I want to cache `npm` assets - which usually requires a `package.json` file. But it turns out you can set a custom `cache-dependency-path` - and in this case I'm bundling together ALL of my GitHub Actions wokflow YAML files as that cache key.

I could be more specific and indicate the exact filename of the workflow in question, but I wanted something I could copy-and-paste between projects easily.

With this in place, `npx ...` lines now get to reuse the previously downloaded version of the tool. The entire cache is invalidated any time any content of any of the workflow YAML files changes, which I think is fine.

html:
<p>Some of my repositories have GitHub Actions workflows that execute commands using <code>npx</code>, for example my <a href="https://github.com/simonw/graphql-scraper">graphql-scraper</a> repo using <code>npx</code> to <a href="https://github.com/simonw/help-scraper/blob/54ef9b1fa579dc2d3f68055ffdac9996fd6dae9c/.github/workflows/scrape.yml#L76-L80">install and run</a> the <code>get-graphql-schema</code> tool:</p>
<pre><code>npx get-graphql-schema https://api.fly.io/graphql &gt; flyctl/fly.graphql
</code></pre>
<p>This was downloading the package from npm every time the action ran. This feels rude! I wanted to use the <a href="https://github.com/actions/cache">actions/cache</a> mechanism to cache the download in between runs.</p>
<p>One problem: the documentation for that action assumes that projects are using a <code>package.json</code> file which can be used as part of the cache key. But my projects don't have one of those files, they just want to use <code>npx</code> utilities as part of their workflows.</p>
<p>Here's the pattern I came up with to configure the cache:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/setup-node@v2</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">node-version</span>: <span class="pl-s"><span class="pl-pds">'</span>14<span class="pl-pds">'</span></span>
        <span class="pl-ent">cache</span>: <span class="pl-s"><span class="pl-pds">'</span>npm<span class="pl-pds">'</span></span>
        <span class="pl-ent">cache-dependency-path</span>: <span class="pl-s"><span class="pl-pds">'</span>**/.github/workflows/*.yml<span class="pl-pds">'</span></span></pre></div>
<p>This is using <a href="https://github.com/actions/setup-node">actions/setup-node</a> which itself can setup the cache. I'm telling it that I want to cache <code>npm</code> assets - which usually requires a <code>package.json</code> file. But it turns out you can set a custom <code>cache-dependency-path</code> - and in this case I'm bundling together ALL of my GitHub Actions wokflow YAML files as that cache key.</p>
<p>I could be more specific and indicate the exact filename of the workflow in question, but I wanted something I could copy-and-paste between projects easily.</p>
<p>With this in place, <code>npx ...</code> lines now get to reuse the previously downloaded version of the tool. The entire cache is invalidated any time any content of any of the workflow YAML files changes, which I think is fine.</p>


path: git_rewrite-repo-specific-files.md
topic: git
title: Rewriting a repo to contain the history of just specific files
url: https://github.com/simonw/til/blob/main/git/rewrite-repo-specific-files.md
created: 2022-03-22T16:11:45-07:00
created_utc: 2022-03-22T23:11:45+00:00
updated: 2022-03-22T16:21:51-07:00
updated_utc: 2022-03-22T23:21:51+00:00
shot_hash: 291f781770fee270d63e579044dee9c0
slug: rewrite-repo-specific-files
body:
I wanted to start [a new git repository](https://github.com/simonw/graphql-scraper/tree/828a1efc4307cca6cd378c394c2d33eac2eceb52) containing just the history of two specific files from my [help-scraper repository](https://github.com/simonw/help-scraper).

I started out planning to use `git filter-branch` for this, but got put off when [this StackOverflow thread](https://stackoverflow.com/questions/2982055/detach-many-subdirectories-into-a-new-separate-git-repository) started talking about the need to understand the differences between macOS `sed` and regular GNU `sed`.

That thread also pointed me to [git-filter-repo](https://github.com/newren/git-filter-repo), a really neat Python script that makes this *so much easier*.

## Installing git-filter-repo

`git-filter-repo` is written in Python but has zero dependencies on anything else - all you need to do is place the script somewhere on your path.

I ran `echo $PATH` to check which directories were on my path - one of them is `.local/bin` - so I decided to put it there:

    cd ~/.local/bin
    wget https://raw.githubusercontent.com/newren/git-filter-repo/main/git-filter-repo
    chmod 755 git-filter-repo

It didn't work until I ran `chmod 755` on it.

Now I can run this:

    % git filter-repo
    No arguments specified.

Confirming the new command is installed!

## Rewriting my repository

The `--path` option can be used to preserve just the history of specified paths. I ran this:

    cd /tmp
    git clone https://github.com/simonw/help-scraper
    cd help-scraper
    git filter-repo --path flyctl/fly.graphql --path github/github.graphql

The command output was:

    Parsed 132 commits
    New history written in 0.33 seconds; now repacking/cleaning...
    Repacking your repo and cleaning out old unneeded objects
    HEAD is now at 828a1efc GitHub: Tue Mar 22 15:09:04 UTC 2022
    Enumerating objects: 144, done.
    Counting objects: 100% (144/144), done.
    Delta compression using up to 12 threads
    Compressing objects: 100% (69/69), done.
    Writing objects: 100% (144/144), done.
    Total 144 (delta 34), reused 55 (delta 34), pack-reused 0
    Completely finished after 1.33 seconds.

And sure enough, my repo now only contains those two files:
```
% find . | grep -v .git
.
./flyctl
./flyctl/fly.graphql
```
And the commit history for those files:
```
% git log --stat --pretty=oneline | head -n 10
828a1efc4307cca6cd378c394c2d33eac2eceb52 GitHub: Tue Mar 22 15:09:04 UTC 2022
 github/github.graphql | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
44fc9ebb49a255ad1c90f7761ab2ab1267135ff9 GitHub: Fri Mar 18 15:08:38 UTC 2022
 github/github.graphql | 1 +
 1 file changed, 1 insertion(+)
dae52d5b94761145333867bc6641e8210409c3b7 GitHub: Thu Mar 17 16:58:45 UTC 2022
 github/github.graphql | 3 +++
 1 file changed, 3 insertions(+)
33e0691bc579bb1e26ea6d05556cf047f5c120da GitHub: Wed Mar 16 15:05:35 UTC 2022
help-scraper % git log --stat --pretty=oneline | head -n 20
828a1efc4307cca6cd378c394c2d33eac2eceb52 GitHub: Tue Mar 22 15:09:04 UTC 2022
 github/github.graphql | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
44fc9ebb49a255ad1c90f7761ab2ab1267135ff9 GitHub: Fri Mar 18 15:08:38 UTC 2022
 github/github.graphql | 1 +
 1 file changed, 1 insertion(+)
dae52d5b94761145333867bc6641e8210409c3b7 GitHub: Thu Mar 17 16:58:45 UTC 2022
 github/github.graphql | 3 +++
 1 file changed, 3 insertions(+)
33e0691bc579bb1e26ea6d05556cf047f5c120da GitHub: Wed Mar 16 15:05:35 UTC 2022
 github/github.graphql | 54 ++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 51 insertions(+), 3 deletions(-)
3dae87045436c3a8c6c84409f9e7ec5ae3bfae74 GitHub: Tue Mar 15 15:08:59 UTC 2022
 github/github.graphql | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
74a3f34c254f789948ee0257981f5d30a094d19f Fly: Tue Mar 15 06:23:03 UTC 2022
 flyctl/fly.graphql | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
63c92c60285b08a898b874b9296b28935f0e7ea8 GitHub: Mon Mar 14 15:09:42 UTC 2022
 github/github.graphql | 5 ++++-
```

html:
<p>I wanted to start <a href="https://github.com/simonw/graphql-scraper/tree/828a1efc4307cca6cd378c394c2d33eac2eceb52">a new git repository</a> containing just the history of two specific files from my <a href="https://github.com/simonw/help-scraper">help-scraper repository</a>.</p>
<p>I started out planning to use <code>git filter-branch</code> for this, but got put off when <a href="https://stackoverflow.com/questions/2982055/detach-many-subdirectories-into-a-new-separate-git-repository" rel="nofollow">this StackOverflow thread</a> started talking about the need to understand the differences between macOS <code>sed</code> and regular GNU <code>sed</code>.</p>
<p>That thread also pointed me to <a href="https://github.com/newren/git-filter-repo">git-filter-repo</a>, a really neat Python script that makes this <em>so much easier</em>.</p>
<h2>
<a id="user-content-installing-git-filter-repo" class="anchor" href="#installing-git-filter-repo" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing git-filter-repo</h2>
<p><code>git-filter-repo</code> is written in Python but has zero dependencies on anything else - all you need to do is place the script somewhere on your path.</p>
<p>I ran <code>echo $PATH</code> to check which directories were on my path - one of them is <code>.local/bin</code> - so I decided to put it there:</p>
<pre><code>cd ~/.local/bin
wget https://raw.githubusercontent.com/newren/git-filter-repo/main/git-filter-repo
chmod 755 git-filter-repo
</code></pre>
<p>It didn't work until I ran <code>chmod 755</code> on it.</p>
<p>Now I can run this:</p>
<pre><code>% git filter-repo
No arguments specified.
</code></pre>
<p>Confirming the new command is installed!</p>
<h2>
<a id="user-content-rewriting-my-repository" class="anchor" href="#rewriting-my-repository" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Rewriting my repository</h2>
<p>The <code>--path</code> option can be used to preserve just the history of specified paths. I ran this:</p>
<pre><code>cd /tmp
git clone https://github.com/simonw/help-scraper
cd help-scraper
git filter-repo --path flyctl/fly.graphql --path github/github.graphql
</code></pre>
<p>The command output was:</p>
<pre><code>Parsed 132 commits
New history written in 0.33 seconds; now repacking/cleaning...
Repacking your repo and cleaning out old unneeded objects
HEAD is now at 828a1efc GitHub: Tue Mar 22 15:09:04 UTC 2022
Enumerating objects: 144, done.
Counting objects: 100% (144/144), done.
Delta compression using up to 12 threads
Compressing objects: 100% (69/69), done.
Writing objects: 100% (144/144), done.
Total 144 (delta 34), reused 55 (delta 34), pack-reused 0
Completely finished after 1.33 seconds.
</code></pre>
<p>And sure enough, my repo now only contains those two files:</p>
<pre><code>% find . | grep -v .git
.
./flyctl
./flyctl/fly.graphql
</code></pre>
<p>And the commit history for those files:</p>
<pre><code>% git log --stat --pretty=oneline | head -n 10
828a1efc4307cca6cd378c394c2d33eac2eceb52 GitHub: Tue Mar 22 15:09:04 UTC 2022
 github/github.graphql | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
44fc9ebb49a255ad1c90f7761ab2ab1267135ff9 GitHub: Fri Mar 18 15:08:38 UTC 2022
 github/github.graphql | 1 +
 1 file changed, 1 insertion(+)
dae52d5b94761145333867bc6641e8210409c3b7 GitHub: Thu Mar 17 16:58:45 UTC 2022
 github/github.graphql | 3 +++
 1 file changed, 3 insertions(+)
33e0691bc579bb1e26ea6d05556cf047f5c120da GitHub: Wed Mar 16 15:05:35 UTC 2022
help-scraper % git log --stat --pretty=oneline | head -n 20
828a1efc4307cca6cd378c394c2d33eac2eceb52 GitHub: Tue Mar 22 15:09:04 UTC 2022
 github/github.graphql | 27 +++++++++++++++++++++++++++
 1 file changed, 27 insertions(+)
44fc9ebb49a255ad1c90f7761ab2ab1267135ff9 GitHub: Fri Mar 18 15:08:38 UTC 2022
 github/github.graphql | 1 +
 1 file changed, 1 insertion(+)
dae52d5b94761145333867bc6641e8210409c3b7 GitHub: Thu Mar 17 16:58:45 UTC 2022
 github/github.graphql | 3 +++
 1 file changed, 3 insertions(+)
33e0691bc579bb1e26ea6d05556cf047f5c120da GitHub: Wed Mar 16 15:05:35 UTC 2022
 github/github.graphql | 54 ++++++++++++++++++++++++++++++++++++++++++++++++---
 1 file changed, 51 insertions(+), 3 deletions(-)
3dae87045436c3a8c6c84409f9e7ec5ae3bfae74 GitHub: Tue Mar 15 15:08:59 UTC 2022
 github/github.graphql | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)
74a3f34c254f789948ee0257981f5d30a094d19f Fly: Tue Mar 15 06:23:03 UTC 2022
 flyctl/fly.graphql | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)
63c92c60285b08a898b874b9296b28935f0e7ea8 GitHub: Mon Mar 14 15:09:42 UTC 2022
 github/github.graphql | 5 ++++-
</code></pre>


path: twitter_export-edit-twitter-spaces.md
topic: twitter
title: Exporting and editing a Twitter Spaces recording
url: https://github.com/simonw/til/blob/main/twitter/export-edit-twitter-spaces.md
created: 2022-03-23T14:55:06-07:00
created_utc: 2022-03-23T21:55:06+00:00
updated: 2022-03-25T11:58:55-07:00
updated_utc: 2022-03-25T18:58:55+00:00
shot_hash: 544f6f279b51a411afaed57c107f73d8
slug: export-edit-twitter-spaces
body:
I hosted [a Twitter Spaces conversation](https://twitter.com/simonw/status/1504604448202518529) the other day. The recording is only available for 30 days afterwards, and I wanted to publish it elsewhere.

## Exporting the recording

Right now [the only way](https://media.twitter.com/en/articles/products/2021/twitter-spaces/recorded-spaces) to export a recording is to request an export of your *entire* Twitter account, and then wait around 24 hours for the export to be generated.

I hoped I could instead hack around in my browser network panel to extract the audio, but it's served up as hundreds of smaller files so this wasn't feasible.

So I hit export, waited 24 hours and downloaded my 900MB archive zip file.

Within the zip file was a `spaces_media/` folder containing a 49.8MB `space-1ypKdEXvkMLGW_0.ts` audio recording.

## Export using youtube-dl

 Jan-Erik Rediger [sent me this tip](https://twitter.com/badboy_/status/1507315981970030593) on Twitter:
 
 > So regarding "Exporting the recording": the thing is served as a m3u playlist. If you find that URL in your browser's network tab, you can throw it against youtube-dl and it will give you the full recording as a single file (mp4 for reasons, but it's really just aac in there)

This works! I ran the Firefox developer tools console against the Twitter spaces page, filtered for "m3u" and hit "Play" on the recording to kick off the audio load and capture the URL to that playlist file:

<img width="602" alt="image" src="https://user-images.githubusercontent.com/9599/160168737-7f7e779b-4d8d-491b-9c86-be99d6c2f980.png">

Then I pasted the resulting URL to `youtube-dl`:

```
% youtube-dl "https://prod-fastly-us-west-1.video.pscp.tv/Transcoding/v1/hls/GPI6dSzgZcfqRfMLplfNp_0xu1QXQ8iDEEA0KymUd5WuqOZCZ9LGGKY6vBQdumX7YV1TT2fGtMdXdl2qqtVvPA/non_transcode/us-west-1/periscope-replay-direct-prod-us-west-1-public/audio-space/playlist_16798763063413909336.m3u8?type=replay"
[generic] playlist_16798763063413909336: Requesting header
[generic] playlist_16798763063413909336: Downloading m3u8 information
[download] Destination: playlist_16798763063413909336-playlist_16798763063413909336.mp4
ffmpeg version 4.2.1 Copyright (c) 2000-2019 the FFmpeg developers
...
size=   44469kB time=01:03:45.66 bitrate=  95.2kbits/s speed=65.5x    
video:0kB audio:44982kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[ffmpeg] Downloaded 45536276 bytes
[download] 100% of 43.43MiB in 01:00
% ls -lh
total 90120
-rw-r--r--  1 simon  wheel    43M Mar 25 10:08 playlist_16798763063413909336-playlist_16798763063413909336.mp4
```

## Converting that to mp3

I used `ffmpeg` (installed via `brew install ffmpeg`) to convert that file to an MP3:
```
spaces_media % ffmpeg -i space-1ypKdEXvkMLGW_0.ts space.mp3 
ffmpeg version 4.2.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.2.0 (Alpine 9.2.0)
  configuration: --prefix=/usr --enable-avresample --enable-avfilter --enable-gnutls --enable-gpl --enable-libass --enable-libmp3lame --enable-libvorbis --enable-libvpx --enable-libxvid --enable-libx264 --enable-libx265 --enable-libtheora --enable-libv4l2 --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-libxcb --disable-stripping --disable-static --disable-librtmp --enable-vaapi --enable-vdpau --enable-libopus --disable-debug
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[aac @ 0x7f59653ad180] Format aac detected only with low score of 25, misdetection possible!
[aac @ 0x7f59653ad180] Estimating duration from bitrate, this may be inaccurate
Input #0, aac, from 'space-1ypKdEXvkMLGW_0.ts':
  Metadata:
    TIT3            : 3856965984.391
    id3v2_priv.com.apple.streaming.transportStreamTimestamp: \x00\x00\x00\x01\xc9\xdd\xe3\x82
    JSONMetadata    : {"HydraVersion":4,"ntp":3856965984.391}
    HydraParticipants: []
    HydraAudioLevel : [0]
  Duration: 01:09:09.41, bitrate: 96 kb/s
    Stream #0:0: Audio: aac (LC), 48000 Hz, stereo, fltp, 96 kb/s
Stream mapping:
  Stream #0:0 -> #0:0 (aac (native) -> mp3 (libmp3lame))
Press [q] to stop, [?] for help
Output #0, mp3, to 'space.mp3':
  Metadata:
    TIT3            : 3856965984.391
    id3v2_priv.com.apple.streaming.transportStreamTimestamp: \x00\x00\x00\x01\xc9\xdd\xe3\x82
    JSONMetadata    : {"HydraVersion":4,"ntp":3856965984.391}
    HydraParticipants: []
    HydraAudioLevel : [0]
    TSSE            : Lavf58.29.100
    Stream #0:0: Audio: mp3 (libmp3lame), 48000 Hz, stereo, fltp
    Metadata:
      encoder         : Lavc58.54.100 libmp3lame
size=   59777kB time=01:03:45.69 bitrate= 128.0kbits/s speed=  56x    
video:0kB audio:59777kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000740%
```

## Editing the file

I wanted to make a *very* basic edit to the file - just to cut off the first six minutes.

I used QuickTime Player on my Mac to do this.

I opened the file in that application, then moved the play head to the point that I wanted to start the track and used `Edit -> Split Clip` to split the clip - then deleted the section before the split.

Then I used `File -> Export As -> Audio Only...` to export the audio. This gave me a `.m4a` file.

I uploaded that file directly [to SoundCloud](https://soundcloud.com/simon-willison/sqlite-happy-hour-22nd-march-2022).

html:
<p>I hosted <a href="https://twitter.com/simonw/status/1504604448202518529" rel="nofollow">a Twitter Spaces conversation</a> the other day. The recording is only available for 30 days afterwards, and I wanted to publish it elsewhere.</p>
<h2>
<a id="user-content-exporting-the-recording" class="anchor" href="#exporting-the-recording" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Exporting the recording</h2>
<p>Right now <a href="https://media.twitter.com/en/articles/products/2021/twitter-spaces/recorded-spaces" rel="nofollow">the only way</a> to export a recording is to request an export of your <em>entire</em> Twitter account, and then wait around 24 hours for the export to be generated.</p>
<p>I hoped I could instead hack around in my browser network panel to extract the audio, but it's served up as hundreds of smaller files so this wasn't feasible.</p>
<p>So I hit export, waited 24 hours and downloaded my 900MB archive zip file.</p>
<p>Within the zip file was a <code>spaces_media/</code> folder containing a 49.8MB <code>space-1ypKdEXvkMLGW_0.ts</code> audio recording.</p>
<h2>
<a id="user-content-export-using-youtube-dl" class="anchor" href="#export-using-youtube-dl" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Export using youtube-dl</h2>
<p>Jan-Erik Rediger <a href="https://twitter.com/badboy_/status/1507315981970030593" rel="nofollow">sent me this tip</a> on Twitter:</p>
<blockquote>
<p>So regarding "Exporting the recording": the thing is served as a m3u playlist. If you find that URL in your browser's network tab, you can throw it against youtube-dl and it will give you the full recording as a single file (mp4 for reasons, but it's really just aac in there)</p>
</blockquote>
<p>This works! I ran the Firefox developer tools console against the Twitter spaces page, filtered for "m3u" and hit "Play" on the recording to kick off the audio load and capture the URL to that playlist file:</p>
<p><a href="https://user-images.githubusercontent.com/9599/160168737-7f7e779b-4d8d-491b-9c86-be99d6c2f980.png" target="_blank" rel="nofollow"><img width="602" alt="image" src="https://user-images.githubusercontent.com/9599/160168737-7f7e779b-4d8d-491b-9c86-be99d6c2f980.png" style="max-width:100%;"></a></p>
<p>Then I pasted the resulting URL to <code>youtube-dl</code>:</p>
<pre><code>% youtube-dl "https://prod-fastly-us-west-1.video.pscp.tv/Transcoding/v1/hls/GPI6dSzgZcfqRfMLplfNp_0xu1QXQ8iDEEA0KymUd5WuqOZCZ9LGGKY6vBQdumX7YV1TT2fGtMdXdl2qqtVvPA/non_transcode/us-west-1/periscope-replay-direct-prod-us-west-1-public/audio-space/playlist_16798763063413909336.m3u8?type=replay"
[generic] playlist_16798763063413909336: Requesting header
[generic] playlist_16798763063413909336: Downloading m3u8 information
[download] Destination: playlist_16798763063413909336-playlist_16798763063413909336.mp4
ffmpeg version 4.2.1 Copyright (c) 2000-2019 the FFmpeg developers
...
size=   44469kB time=01:03:45.66 bitrate=  95.2kbits/s speed=65.5x    
video:0kB audio:44982kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
[ffmpeg] Downloaded 45536276 bytes
[download] 100% of 43.43MiB in 01:00
% ls -lh
total 90120
-rw-r--r--  1 simon  wheel    43M Mar 25 10:08 playlist_16798763063413909336-playlist_16798763063413909336.mp4
</code></pre>
<h2>
<a id="user-content-converting-that-to-mp3" class="anchor" href="#converting-that-to-mp3" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Converting that to mp3</h2>
<p>I used <code>ffmpeg</code> (installed via <code>brew install ffmpeg</code>) to convert that file to an MP3:</p>
<pre><code>spaces_media % ffmpeg -i space-1ypKdEXvkMLGW_0.ts space.mp3 
ffmpeg version 4.2.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with gcc 9.2.0 (Alpine 9.2.0)
  configuration: --prefix=/usr --enable-avresample --enable-avfilter --enable-gnutls --enable-gpl --enable-libass --enable-libmp3lame --enable-libvorbis --enable-libvpx --enable-libxvid --enable-libx264 --enable-libx265 --enable-libtheora --enable-libv4l2 --enable-postproc --enable-pic --enable-pthreads --enable-shared --enable-libxcb --disable-stripping --disable-static --disable-librtmp --enable-vaapi --enable-vdpau --enable-libopus --disable-debug
  libavutil      56. 31.100 / 56. 31.100
  libavcodec     58. 54.100 / 58. 54.100
  libavformat    58. 29.100 / 58. 29.100
  libavdevice    58.  8.100 / 58.  8.100
  libavfilter     7. 57.100 /  7. 57.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  5.100 /  5.  5.100
  libswresample   3.  5.100 /  3.  5.100
  libpostproc    55.  5.100 / 55.  5.100
[aac @ 0x7f59653ad180] Format aac detected only with low score of 25, misdetection possible!
[aac @ 0x7f59653ad180] Estimating duration from bitrate, this may be inaccurate
Input #0, aac, from 'space-1ypKdEXvkMLGW_0.ts':
  Metadata:
    TIT3            : 3856965984.391
    id3v2_priv.com.apple.streaming.transportStreamTimestamp: \x00\x00\x00\x01\xc9\xdd\xe3\x82
    JSONMetadata    : {"HydraVersion":4,"ntp":3856965984.391}
    HydraParticipants: []
    HydraAudioLevel : [0]
  Duration: 01:09:09.41, bitrate: 96 kb/s
    Stream #0:0: Audio: aac (LC), 48000 Hz, stereo, fltp, 96 kb/s
Stream mapping:
  Stream #0:0 -&gt; #0:0 (aac (native) -&gt; mp3 (libmp3lame))
Press [q] to stop, [?] for help
Output #0, mp3, to 'space.mp3':
  Metadata:
    TIT3            : 3856965984.391
    id3v2_priv.com.apple.streaming.transportStreamTimestamp: \x00\x00\x00\x01\xc9\xdd\xe3\x82
    JSONMetadata    : {"HydraVersion":4,"ntp":3856965984.391}
    HydraParticipants: []
    HydraAudioLevel : [0]
    TSSE            : Lavf58.29.100
    Stream #0:0: Audio: mp3 (libmp3lame), 48000 Hz, stereo, fltp
    Metadata:
      encoder         : Lavc58.54.100 libmp3lame
size=   59777kB time=01:03:45.69 bitrate= 128.0kbits/s speed=  56x    
video:0kB audio:59777kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.000740%
</code></pre>
<h2>
<a id="user-content-editing-the-file" class="anchor" href="#editing-the-file" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Editing the file</h2>
<p>I wanted to make a <em>very</em> basic edit to the file - just to cut off the first six minutes.</p>
<p>I used QuickTime Player on my Mac to do this.</p>
<p>I opened the file in that application, then moved the play head to the point that I wanted to start the track and used <code>Edit -&gt; Split Clip</code> to split the clip - then deleted the section before the split.</p>
<p>Then I used <code>File -&gt; Export As -&gt; Audio Only...</code> to export the audio. This gave me a <code>.m4a</code> file.</p>
<p>I uploaded that file directly <a href="https://soundcloud.com/simon-willison/sqlite-happy-hour-22nd-march-2022" rel="nofollow">to SoundCloud</a>.</p>


path: shot-scraper_readability.md
topic: shot-scraper
title: Extracting web page content using Readability.js and shot-scraper
url: https://github.com/simonw/til/blob/main/shot-scraper/readability.md
created: 2022-03-24T12:44:17-07:00
created_utc: 2022-03-24T19:44:17+00:00
updated: 2022-03-25T15:56:19-07:00
updated_utc: 2022-03-25T22:56:19+00:00
shot_hash: 242ccd23ab4d8ba19c9af2d470a647db
slug: readability
body:
[Readability.js](https://github.com/mozilla/readability) is "A standalone version of the readability library used for Firefox Reader View".

My [shot-scraper](https://datasette.io/tools/shot-scraper) tool has a `shot-scraper javascript` command which can load a web page in a headless browser (via [Playwright](https://playwright.dev/)), execute JavaScript against that page and return the result to the console as JSON - see [Scraping web pages from the command line with shot-scraper](https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/).

I figured out how to use the two of these together to extract the core content from a web page using the command line.

Here is the recipe:

    shot-scraper javascript https://simonwillison.net/2022/Mar/24/datasette-061/ "
    async () => {
      const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
      return (new readability.Readability(document)).parse();
    }"

It's using Skypack to [load the module](https://www.skypack.dev/view/@mozilla/readability). Playwright (the tech that powers `shot-scraper`) knows how to execute `async ()` functions and return their results.

This outputs the JSON structure created by Readability directly to the console.
```
% shot-scraper javascript https://simonwillison.net/2022/Mar/24/datasette-061/ "
    async () => {
      const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
      return (new readability.Readability(document)).parse();
    }"
{
    "title": "Datasette 0.61: The annotated release notes",
    "byline": null,
    "dir": null,
    "lang": "en-gb",
    "content": "<div id=\"readability-page-1\" cla...
```
Piping to `jq keys` shows the keys in the returned object:
```
% shot-scraper javascript https://simonwillison.net/2022/Mar/24/datasette-061/ "
    async () => {
      const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
      return (new readability.Readability(document)).parse();
    }" | jq keys
[
  "byline",
  "content",
  "dir",
  "excerpt",
  "lang",
  "length",
  "siteName",
  "textContent",
  "title"
]
```

To get just the text content, use `jq .textContent -r` (the `-r` returns the raw string, without the surrounding double quotes):
```
% shot-scraper javascript https://simonwillison.net/2022/Mar/24/datasette-061/ "
    async () => {
      const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
      return (new readability.Readability(document)).parse();
    }" | jq .textContent -r
I released Datasette 0.61 this morning—closely followed by 0.61.1 to fix a minor bug.
Here are the annotated release notes.

In preparation for Datasette 1.0, this release includes ...
```
I can even pipe it directly into `sqlite-utils` to insert it into a SQLite database table:

```
% shot-scraper javascript https://simonwillison.net/2021/Oct/19/datasette-059/ "
async () => {    
  const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
  return (new readability.Readability(document)).parse();
}" | sqlite-utils insert articles.db articles -
```

## Saving the script to a file

If you want to run this a lot, you can avoid copying-and-pasting it by saving it to a file called `readability.js`:

```javascript
async () => {    
  const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
  return (new readability.Readability(document)).parse();
}
```

And then running:

    shot-scraper javascript https://til.simonwillison.net/shot-scraper/readability -i readability.js

Or even set that up as an alias, by adding the following to `~/.zshrc`:

    alias readability="shot-scraper javascript -i ~/readability.js"

Then restart a terminal window and run:

    readability https://til.simonwillison.net/shot-scraper/readability

## A note on safety

The [Readability.js README says](https://github.com/mozilla/readability/blob/0.4.1/README.md#security):

> If you're going to use Readability with untrusted input (whether in HTML or DOM form), we **strongly** recommend you use a sanitizer library like [DOMPurify](https://github.com/cure53/DOMPurify) to avoid script injection when you use the output of Readability.

It looks like DOMPurify could be executed against the output of Readability using the same import mechanism shown above.

html:
<p><a href="https://github.com/mozilla/readability">Readability.js</a> is "A standalone version of the readability library used for Firefox Reader View".</p>
<p>My <a href="https://datasette.io/tools/shot-scraper" rel="nofollow">shot-scraper</a> tool has a <code>shot-scraper javascript</code> command which can load a web page in a headless browser (via <a href="https://playwright.dev/" rel="nofollow">Playwright</a>), execute JavaScript against that page and return the result to the console as JSON - see <a href="https://simonwillison.net/2022/Mar/14/scraping-web-pages-shot-scraper/" rel="nofollow">Scraping web pages from the command line with shot-scraper</a>.</p>
<p>I figured out how to use the two of these together to extract the core content from a web page using the command line.</p>
<p>Here is the recipe:</p>
<pre><code>shot-scraper javascript https://simonwillison.net/2022/Mar/24/datasette-061/ "
async () =&gt; {
  const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
  return (new readability.Readability(document)).parse();
}"
</code></pre>
<p>It's using Skypack to <a href="https://www.skypack.dev/view/@mozilla/readability" rel="nofollow">load the module</a>. Playwright (the tech that powers <code>shot-scraper</code>) knows how to execute <code>async ()</code> functions and return their results.</p>
<p>This outputs the JSON structure created by Readability directly to the console.</p>
<pre><code>% shot-scraper javascript https://simonwillison.net/2022/Mar/24/datasette-061/ "
    async () =&gt; {
      const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
      return (new readability.Readability(document)).parse();
    }"
{
    "title": "Datasette 0.61: The annotated release notes",
    "byline": null,
    "dir": null,
    "lang": "en-gb",
    "content": "&lt;div id=\"readability-page-1\" cla...
</code></pre>
<p>Piping to <code>jq keys</code> shows the keys in the returned object:</p>
<pre><code>% shot-scraper javascript https://simonwillison.net/2022/Mar/24/datasette-061/ "
    async () =&gt; {
      const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
      return (new readability.Readability(document)).parse();
    }" | jq keys
[
  "byline",
  "content",
  "dir",
  "excerpt",
  "lang",
  "length",
  "siteName",
  "textContent",
  "title"
]
</code></pre>
<p>To get just the text content, use <code>jq .textContent -r</code> (the <code>-r</code> returns the raw string, without the surrounding double quotes):</p>
<pre><code>% shot-scraper javascript https://simonwillison.net/2022/Mar/24/datasette-061/ "
    async () =&gt; {
      const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
      return (new readability.Readability(document)).parse();
    }" | jq .textContent -r
I released Datasette 0.61 this morning—closely followed by 0.61.1 to fix a minor bug.
Here are the annotated release notes.

In preparation for Datasette 1.0, this release includes ...
</code></pre>
<p>I can even pipe it directly into <code>sqlite-utils</code> to insert it into a SQLite database table:</p>
<pre><code>% shot-scraper javascript https://simonwillison.net/2021/Oct/19/datasette-059/ "
async () =&gt; {    
  const readability = await import('https://cdn.skypack.dev/@mozilla/readability');
  return (new readability.Readability(document)).parse();
}" | sqlite-utils insert articles.db articles -
</code></pre>
<h2>
<a id="user-content-saving-the-script-to-a-file" class="anchor" href="#saving-the-script-to-a-file" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Saving the script to a file</h2>
<p>If you want to run this a lot, you can avoid copying-and-pasting it by saving it to a file called <code>readability.js</code>:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">async</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>    
  <span class="pl-k">const</span> <span class="pl-s1">readability</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-k">import</span><span class="pl-kos">(</span><span class="pl-s">'https://cdn.skypack.dev/@mozilla/readability'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">return</span> <span class="pl-kos">(</span><span class="pl-k">new</span> <span class="pl-s1">readability</span><span class="pl-kos">.</span><span class="pl-c1">Readability</span><span class="pl-kos">(</span><span class="pl-smi">document</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">parse</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span></pre></div>
<p>And then running:</p>
<pre><code>shot-scraper javascript https://til.simonwillison.net/shot-scraper/readability -i readability.js
</code></pre>
<p>Or even set that up as an alias, by adding the following to <code>~/.zshrc</code>:</p>
<pre><code>alias readability="shot-scraper javascript -i ~/readability.js"
</code></pre>
<p>Then restart a terminal window and run:</p>
<pre><code>readability https://til.simonwillison.net/shot-scraper/readability
</code></pre>
<h2>
<a id="user-content-a-note-on-safety" class="anchor" href="#a-note-on-safety" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>A note on safety</h2>
<p>The <a href="https://github.com/mozilla/readability/blob/0.4.1/README.md#security">Readability.js README says</a>:</p>
<blockquote>
<p>If you're going to use Readability with untrusted input (whether in HTML or DOM form), we <strong>strongly</strong> recommend you use a sanitizer library like <a href="https://github.com/cure53/DOMPurify">DOMPurify</a> to avoid script injection when you use the output of Readability.</p>
</blockquote>
<p>It looks like DOMPurify could be executed against the output of Readability using the same import mechanism shown above.</p>


path: auth0_oauth-with-auth0.md
topic: auth0
title: Simplest possible OAuth authentication with Auth0
url: https://github.com/simonw/til/blob/main/auth0/oauth-with-auth0.md
created: 2022-03-26T14:57:42-07:00
created_utc: 2022-03-26T21:57:42+00:00
updated: 2022-03-26T19:27:47-07:00
updated_utc: 2022-03-27T02:27:47+00:00
shot_hash: 1b78ef3b2c3b7a00a06af069ac7e9070
slug: oauth-with-auth0
body:
[Auth0](https://auth0.com/) provides an authentication API which you can use to avoid having to deal with user accounts in your own web application.

We used them last year for [VaccinateCA VIAL](https://github.com/CAVaccineInventory/vial), using the [Python Social Auth](https://github.com/python-social-auth/social-app-django) library recommended by the [Auth0 Django tutorial](https://auth0.com/docs/quickstart/webapp/django/01-login).

That was quite a lot of code, so today I decided to figure out how to implement Auth0 authentication from first principles.

Auth0 uses standard OAuth 2. Their documentation [leans very heavily](https://auth0.com/docs/quickstart/webapp) towards client libraries, but if you dig around enough you can find the [Authentication API](https://auth0.com/docs/api/authentication) documentation with the information you need.

I found that pretty late, and figured out most of this by following [their Flask tutorial](https://auth0.com/docs/quickstart/webapp/python) and then [reverse engineering](https://github.com/natbat/pillarpointstewards/issues/6) what the prototype was actually doing.

## Initial setup

To start, you need to create a new Auth0 application and note down three values. Mine looked something like this:
```python
AUTH0_DOMAIN = "pillarpointstewards.us.auth0.com"
AUTH0_CLIENT_ID = "DLXBMPbtamC2STUyV7R6OFJFDsSTHqEA"
AUTH0_CLIENT_SECRET = "..." # Get it from that page
```
You also need to decide on the "callback URL" that authenticated users will be redirected to, then add that to the "Allowed Callback URLs" setting in Auth0. You can set this as a comma-separated list.

My callback URL started out as `http://localhost:8000/callback`.

## Redirecting to Auth0

The first step is to redirect the user to Auth0 to sign in. The redirect URL looks something like this:
```
https://pillarpointstewards.us.auth0.com/authorize?
  response_type=code
  &client_id=DLXBMPbtamC2STUyV7R6OFJFDsSTHqEA
  &redirect_uri=http%3A%2F%2Flocalhost%3A8000%2Fcallback
  &scope=openid+profile+email
  &state=FtYFQBczDZemVurdBan5PjRiePPGhU
```
You can also hit https://pillarpointstewards.us.auth0.com/.well-known/openid-configuration to get back JSON describing all of the end points, but I prefer to hard-code them in rather than take on the performance overhead of that additional HTTP request.

The `state=` field there is a random string that you generate. You should store this in a cookie so you can compare it later on to [protect against CSRF attacks](https://auth0.com/docs/secure/attack-protection/state-parameters).

## User redirects back to your callback URL

The user signs in on Auth0 (which they may do via Google SSO, or by creating or using an Auth0 account). Auth0 then redirects them back to your callback URL, like this:

    https://your-site/callback?code=CODE_HERE&state=STATE_YOU_PROVIDED

Check that state against the cookie you set earlier (optional but a good idea).

Now you need to exchange the `code=` for an access token.

You do that with an authenticated server-side HTTP POST to the following URL:

`https://pillarpointstewards.us.auth0.com/oauth/token`

With these POST parameters:

```
grant_type=authorization_code
&redirect_uri=http%3A%2F%2Flocalhost%3A3000%2Fcallback
&code=804-RPsfJb9zIiLNtgP5LVKUnYor8_lN7vltl7DkRpxF-
```
The redirect URI is the same as before. The code is the code that was passed to you in the URL to the `/callback` page.

This call needs to be authenticated using HTTP basic authentication with the username set to your client ID and the password set to your client secret.

In HTTP, that looks like a `Authorization: Basic BASE64` header, where `BASE64` is the base-64 encoded string of `username:password` (or `client_id:client_secret`.

In Python using [HTTPX](https://www.python-httpx.org) that looks like this:

```python
response = httpx.post(
    "https://{}/oauth/token".format(config["domain"]),
    data={
        "grant_type": "authorization_code",
        "redirect_uri": redirect_uri,
        "code": code,
    },
    auth=(config["client_id"], config["client_secret"]),
)
access_token = response.json()["access_token"]
```
The response from that is a JSON object with a `"access_token"` key containing an access token.

## Fetching the user information

The `access_token` can then be used to make an authenticated API call to `https://pillarpointstewards.us.auth0.com/userinfo` to get back the user's profile:

```python
profile_response = httpx.get(
    "https://{}/userinfo".format(config["domain"]),
    headers={"Authorization": "Bearer {}".format(access_token)},
)
profile = profile_response.json()
```
And that's it!

## datasette-auth0

I implemented this in a new authentication plugin for Datasette called [datasette-auth0](https://datasette.io/plugins/datasette-auth0).

The bulk of the implementation is in this file: <https://github.com/simonw/datasette-auth0/blob/0.1a0/datasette_auth0/__init__.py>

html:
<p><a href="https://auth0.com/" rel="nofollow">Auth0</a> provides an authentication API which you can use to avoid having to deal with user accounts in your own web application.</p>
<p>We used them last year for <a href="https://github.com/CAVaccineInventory/vial">VaccinateCA VIAL</a>, using the <a href="https://github.com/python-social-auth/social-app-django">Python Social Auth</a> library recommended by the <a href="https://auth0.com/docs/quickstart/webapp/django/01-login" rel="nofollow">Auth0 Django tutorial</a>.</p>
<p>That was quite a lot of code, so today I decided to figure out how to implement Auth0 authentication from first principles.</p>
<p>Auth0 uses standard OAuth 2. Their documentation <a href="https://auth0.com/docs/quickstart/webapp" rel="nofollow">leans very heavily</a> towards client libraries, but if you dig around enough you can find the <a href="https://auth0.com/docs/api/authentication" rel="nofollow">Authentication API</a> documentation with the information you need.</p>
<p>I found that pretty late, and figured out most of this by following <a href="https://auth0.com/docs/quickstart/webapp/python" rel="nofollow">their Flask tutorial</a> and then <a href="https://github.com/natbat/pillarpointstewards/issues/6">reverse engineering</a> what the prototype was actually doing.</p>
<h2>
<a id="user-content-initial-setup" class="anchor" href="#initial-setup" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Initial setup</h2>
<p>To start, you need to create a new Auth0 application and note down three values. Mine looked something like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-v">AUTH0_DOMAIN</span> <span class="pl-c1">=</span> <span class="pl-s">"pillarpointstewards.us.auth0.com"</span>
<span class="pl-v">AUTH0_CLIENT_ID</span> <span class="pl-c1">=</span> <span class="pl-s">"DLXBMPbtamC2STUyV7R6OFJFDsSTHqEA"</span>
<span class="pl-v">AUTH0_CLIENT_SECRET</span> <span class="pl-c1">=</span> <span class="pl-s">"..."</span> <span class="pl-c"># Get it from that page</span></pre></div>
<p>You also need to decide on the "callback URL" that authenticated users will be redirected to, then add that to the "Allowed Callback URLs" setting in Auth0. You can set this as a comma-separated list.</p>
<p>My callback URL started out as <code>http://localhost:8000/callback</code>.</p>
<h2>
<a id="user-content-redirecting-to-auth0" class="anchor" href="#redirecting-to-auth0" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Redirecting to Auth0</h2>
<p>The first step is to redirect the user to Auth0 to sign in. The redirect URL looks something like this:</p>
<pre><code>https://pillarpointstewards.us.auth0.com/authorize?
  response_type=code
  &amp;client_id=DLXBMPbtamC2STUyV7R6OFJFDsSTHqEA
  &amp;redirect_uri=http%3A%2F%2Flocalhost%3A8000%2Fcallback
  &amp;scope=openid+profile+email
  &amp;state=FtYFQBczDZemVurdBan5PjRiePPGhU
</code></pre>
<p>You can also hit <a href="https://pillarpointstewards.us.auth0.com/.well-known/openid-configuration" rel="nofollow">https://pillarpointstewards.us.auth0.com/.well-known/openid-configuration</a> to get back JSON describing all of the end points, but I prefer to hard-code them in rather than take on the performance overhead of that additional HTTP request.</p>
<p>The <code>state=</code> field there is a random string that you generate. You should store this in a cookie so you can compare it later on to <a href="https://auth0.com/docs/secure/attack-protection/state-parameters" rel="nofollow">protect against CSRF attacks</a>.</p>
<h2>
<a id="user-content-user-redirects-back-to-your-callback-url" class="anchor" href="#user-redirects-back-to-your-callback-url" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>User redirects back to your callback URL</h2>
<p>The user signs in on Auth0 (which they may do via Google SSO, or by creating or using an Auth0 account). Auth0 then redirects them back to your callback URL, like this:</p>
<pre><code>https://your-site/callback?code=CODE_HERE&amp;state=STATE_YOU_PROVIDED
</code></pre>
<p>Check that state against the cookie you set earlier (optional but a good idea).</p>
<p>Now you need to exchange the <code>code=</code> for an access token.</p>
<p>You do that with an authenticated server-side HTTP POST to the following URL:</p>
<p><code>https://pillarpointstewards.us.auth0.com/oauth/token</code></p>
<p>With these POST parameters:</p>
<pre><code>grant_type=authorization_code
&amp;redirect_uri=http%3A%2F%2Flocalhost%3A3000%2Fcallback
&amp;code=804-RPsfJb9zIiLNtgP5LVKUnYor8_lN7vltl7DkRpxF-
</code></pre>
<p>The redirect URI is the same as before. The code is the code that was passed to you in the URL to the <code>/callback</code> page.</p>
<p>This call needs to be authenticated using HTTP basic authentication with the username set to your client ID and the password set to your client secret.</p>
<p>In HTTP, that looks like a <code>Authorization: Basic BASE64</code> header, where <code>BASE64</code> is the base-64 encoded string of <code>username:password</code> (or <code>client_id:client_secret</code>.</p>
<p>In Python using <a href="https://www.python-httpx.org" rel="nofollow">HTTPX</a> that looks like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">httpx</span>.<span class="pl-en">post</span>(
    <span class="pl-s">"https://{}/oauth/token"</span>.<span class="pl-en">format</span>(<span class="pl-s1">config</span>[<span class="pl-s">"domain"</span>]),
    <span class="pl-s1">data</span><span class="pl-c1">=</span>{
        <span class="pl-s">"grant_type"</span>: <span class="pl-s">"authorization_code"</span>,
        <span class="pl-s">"redirect_uri"</span>: <span class="pl-s1">redirect_uri</span>,
        <span class="pl-s">"code"</span>: <span class="pl-s1">code</span>,
    },
    <span class="pl-s1">auth</span><span class="pl-c1">=</span>(<span class="pl-s1">config</span>[<span class="pl-s">"client_id"</span>], <span class="pl-s1">config</span>[<span class="pl-s">"client_secret"</span>]),
)
<span class="pl-s1">access_token</span> <span class="pl-c1">=</span> <span class="pl-s1">response</span>.<span class="pl-en">json</span>()[<span class="pl-s">"access_token"</span>]</pre></div>
<p>The response from that is a JSON object with a <code>"access_token"</code> key containing an access token.</p>
<h2>
<a id="user-content-fetching-the-user-information" class="anchor" href="#fetching-the-user-information" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Fetching the user information</h2>
<p>The <code>access_token</code> can then be used to make an authenticated API call to <code>https://pillarpointstewards.us.auth0.com/userinfo</code> to get back the user's profile:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">profile_response</span> <span class="pl-c1">=</span> <span class="pl-s1">httpx</span>.<span class="pl-en">get</span>(
    <span class="pl-s">"https://{}/userinfo"</span>.<span class="pl-en">format</span>(<span class="pl-s1">config</span>[<span class="pl-s">"domain"</span>]),
    <span class="pl-s1">headers</span><span class="pl-c1">=</span>{<span class="pl-s">"Authorization"</span>: <span class="pl-s">"Bearer {}"</span>.<span class="pl-en">format</span>(<span class="pl-s1">access_token</span>)},
)
<span class="pl-s1">profile</span> <span class="pl-c1">=</span> <span class="pl-s1">profile_response</span>.<span class="pl-en">json</span>()</pre></div>
<p>And that's it!</p>
<h2>
<a id="user-content-datasette-auth0" class="anchor" href="#datasette-auth0" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>datasette-auth0</h2>
<p>I implemented this in a new authentication plugin for Datasette called <a href="https://datasette.io/plugins/datasette-auth0" rel="nofollow">datasette-auth0</a>.</p>
<p>The bulk of the implementation is in this file: <a href="https://github.com/simonw/datasette-auth0/blob/0.1a0/datasette_auth0/__init__.py">https://github.com/simonw/datasette-auth0/blob/0.1a0/datasette_auth0/__init__.py</a></p>


path: github-actions_deploy-live-demo-when-tests-pass.md
topic: github-actions
title: Deploying a live Datasette demo when the tests pass
url: https://github.com/simonw/til/blob/main/github-actions/deploy-live-demo-when-tests-pass.md
created: 2022-03-27T20:16:50-07:00
created_utc: 2022-03-28T03:16:50+00:00
updated: 2022-03-27T20:16:50-07:00
updated_utc: 2022-03-28T03:16:50+00:00
shot_hash: 7512db1a0c8703bd517605a7eda793a8
slug: deploy-live-demo-when-tests-pass
body:
I've implemented this pattern a bunch of times now - here's the version I've settled on for my [datasette-auth0 plugin](https://github.com/simonw/datasette-auth0) repository.

For publishing to Cloud Run, it needs two GitHub Actions secrets to be configured: `GCP_SA_EMAIL` and `GCP_SA_KEY`.

See below for publishing to Vercel.

In `.github/workflows/test.yml`:

```yaml
name: Test

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.7", "3.8", "3.9", "3.10"]
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python-version }}
    - uses: actions/cache@v2
      name: Configure pip caching
      with:
        path: ~/.cache/pip
        key: ${{ runner.os }}-pip-${{ hashFiles('**/setup.py') }}
        restore-keys: |
          ${{ runner.os }}-pip-
    - name: Install dependencies
      run: |
        pip install -e '.[test]'
    - name: Run tests
      run: |
        pytest
  deploy_demo:
    runs-on: ubuntu-latest
    needs: [test]
    if: github.ref == 'refs/heads/main'
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python 3.10
      uses: actions/setup-python@v2
      with:
        python-version: "3.10"
        cache: pip
        cache-dependency-path: "**/setup.py"
    - name: Install datasette
      run: pip install datasette
    - name: Set up Cloud Run
      uses: google-github-actions/setup-gcloud@v0
      with:
        version: '275.0.0'
        service_account_email: ${{ secrets.GCP_SA_EMAIL }}
        service_account_key: ${{ secrets.GCP_SA_KEY }}
    - name: Deploy demo to Cloud Run
      env:
        CLIENT_SECRET: ${{ secrets.AUTH0_CLIENT_SECRET }}
      run: |-
        gcloud config set run/region us-central1
        gcloud config set project datasette-222320
        wget https://latest.datasette.io/fixtures.db
        datasette publish cloudrun fixtures.db \
        --install https://github.com/simonw/datasette-auth0/archive/$GITHUB_SHA.zip \
        --plugin-secret datasette-auth0 domain "datasette.us.auth0.com" \
        --plugin-secret datasette-auth0 client_id "n9eaHS0ckIsujoyZNZ1wVgcPevjAcAXn" \
        --plugin-secret datasette-auth0 client_secret "$CLIENT_SECRET" \
        --about "datasette-auth0" \
        --about_url "https://datasette.io/plugins/datasette-auth0" \
        --service datasette-auth0-demo
```
The first job called `test` runs the Python tests in the repo. The second `deploy_demo` block is where things get interesting.

```yaml
  deploy_demo:
    runs-on: ubuntu-latest
    needs: [test]
    if: github.ref == 'refs/heads/main'
```
The `needs: [test]` bit ensures this only runs if the tests pass first.

`if: github.ref == 'refs/heads/main'` causes the deploy to only run on pushes to the `main` branch.

The most interesting bit of the deploy command is this bit:
```
datasette publish cloudrun fixtures.db \
--install https://github.com/simonw/datasette-auth0/archive/$GITHUB_SHA.zip \
...
```
`$GITHUB_SHA` is the commit hash that triggered the wokrflow. The `--install` line there constructs a URL to the zip archive of that version from the GitHub repository - so that exact version will be treated as a plugin and installed as part of deploying the Datasette demo instance.

## Deploying to Vercel

[This example](https://github.com/simonw/datasette-hashed-urls/blob/659614c23cbc544915079c44b09b09b090400ff8/.github/workflows/test.yml) deploys to Vercel instead. The key difference is this:

```yaml
    - name: Install datasette
      run: pip install datasette datasette-publish-vercel
    - name: Deploy demo to Vercel
      env:
        VERCEL_TOKEN: ${{ secrets.VERCEL_TOKEN }}
      run: |-
        wget https://latest.datasette.io/fixtures.db
        datasette publish vercel fixtures.db \
          --project datasette-hashed-urls \
          --install https://github.com/simonw/datasette-hashed-urls/archive/$GITHUB_SHA.zip \
          --token $VERCEL_TOKEN \
          --scope datasette
```
The `--token $VERCEL_TOKEN` passes a token created in the Vercel dashboard. I needed `--scope datasette` here because I was deploying to a Vercel team of that name - if deploying to your personal account you can leave this off.

html:
<p>I've implemented this pattern a bunch of times now - here's the version I've settled on for my <a href="https://github.com/simonw/datasette-auth0">datasette-auth0 plugin</a> repository.</p>
<p>For publishing to Cloud Run, it needs two GitHub Actions secrets to be configured: <code>GCP_SA_EMAIL</code> and <code>GCP_SA_KEY</code>.</p>
<p>See below for publishing to Vercel.</p>
<p>In <code>.github/workflows/test.yml</code>:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Test</span>

<span class="pl-ent">on</span>: <span class="pl-s">[push]</span>

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">test</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">strategy</span>:
      <span class="pl-ent">matrix</span>:
        <span class="pl-ent">python-version</span>: <span class="pl-s">["3.7", "3.8", "3.9", "3.10"]</span>
    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v2</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Set up Python ${{ matrix.python-version }}</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">actions/setup-python@v2</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">python-version</span>: <span class="pl-s">${{ matrix.python-version }}</span>
    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/cache@v2</span>
      <span class="pl-ent">name</span>: <span class="pl-s">Configure pip caching</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">path</span>: <span class="pl-s">~/.cache/pip</span>
        <span class="pl-ent">key</span>: <span class="pl-s">${{ runner.os }}-pip-${{ hashFiles('**/setup.py') }}</span>
        <span class="pl-ent">restore-keys</span>: <span class="pl-s">|</span>
<span class="pl-s">          ${{ runner.os }}-pip-</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Install dependencies</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        pip install -e '.[test]'</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Run tests</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        pytest</span>
<span class="pl-s"></span>  <span class="pl-ent">deploy_demo</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">needs</span>: <span class="pl-s">[test]</span>
    <span class="pl-ent">if</span>: <span class="pl-s">github.ref == 'refs/heads/main'</span>
    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v2</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Set up Python 3.10</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">actions/setup-python@v2</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">python-version</span>: <span class="pl-s"><span class="pl-pds">"</span>3.10<span class="pl-pds">"</span></span>
        <span class="pl-ent">cache</span>: <span class="pl-s">pip</span>
        <span class="pl-ent">cache-dependency-path</span>: <span class="pl-s"><span class="pl-pds">"</span>**/setup.py<span class="pl-pds">"</span></span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Install datasette</span>
      <span class="pl-ent">run</span>: <span class="pl-s">pip install datasette</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Set up Cloud Run</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">google-github-actions/setup-gcloud@v0</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">version</span>: <span class="pl-s"><span class="pl-pds">'</span>275.0.0<span class="pl-pds">'</span></span>
        <span class="pl-ent">service_account_email</span>: <span class="pl-s">${{ secrets.GCP_SA_EMAIL }}</span>
        <span class="pl-ent">service_account_key</span>: <span class="pl-s">${{ secrets.GCP_SA_KEY }}</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Deploy demo to Cloud Run</span>
      <span class="pl-ent">env</span>:
        <span class="pl-ent">CLIENT_SECRET</span>: <span class="pl-s">${{ secrets.AUTH0_CLIENT_SECRET }}</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        gcloud config set run/region us-central1</span>
<span class="pl-s">        gcloud config set project datasette-222320</span>
<span class="pl-s">        wget https://latest.datasette.io/fixtures.db</span>
<span class="pl-s">        datasette publish cloudrun fixtures.db \</span>
<span class="pl-s">        --install https://github.com/simonw/datasette-auth0/archive/$GITHUB_SHA.zip \</span>
<span class="pl-s">        --plugin-secret datasette-auth0 domain "datasette.us.auth0.com" \</span>
<span class="pl-s">        --plugin-secret datasette-auth0 client_id "n9eaHS0ckIsujoyZNZ1wVgcPevjAcAXn" \</span>
<span class="pl-s">        --plugin-secret datasette-auth0 client_secret "$CLIENT_SECRET" \</span>
<span class="pl-s">        --about "datasette-auth0" \</span>
<span class="pl-s">        --about_url "https://datasette.io/plugins/datasette-auth0" \</span>
<span class="pl-s">        --service datasette-auth0-demo</span></pre></div>
<p>The first job called <code>test</code> runs the Python tests in the repo. The second <code>deploy_demo</code> block is where things get interesting.</p>
<div class="highlight highlight-source-yaml"><pre>  <span class="pl-ent">deploy_demo</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">needs</span>: <span class="pl-s">[test]</span>
    <span class="pl-ent">if</span>: <span class="pl-s">github.ref == 'refs/heads/main'</span></pre></div>
<p>The <code>needs: [test]</code> bit ensures this only runs if the tests pass first.</p>
<p><code>if: github.ref == 'refs/heads/main'</code> causes the deploy to only run on pushes to the <code>main</code> branch.</p>
<p>The most interesting bit of the deploy command is this bit:</p>
<pre><code>datasette publish cloudrun fixtures.db \
--install https://github.com/simonw/datasette-auth0/archive/$GITHUB_SHA.zip \
...
</code></pre>
<p><code>$GITHUB_SHA</code> is the commit hash that triggered the wokrflow. The <code>--install</code> line there constructs a URL to the zip archive of that version from the GitHub repository - so that exact version will be treated as a plugin and installed as part of deploying the Datasette demo instance.</p>
<h2>
<a id="user-content-deploying-to-vercel" class="anchor" href="#deploying-to-vercel" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Deploying to Vercel</h2>
<p><a href="https://github.com/simonw/datasette-hashed-urls/blob/659614c23cbc544915079c44b09b09b090400ff8/.github/workflows/test.yml">This example</a> deploys to Vercel instead. The key difference is this:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Install datasette</span>
      <span class="pl-ent">run</span>: <span class="pl-s">pip install datasette datasette-publish-vercel</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Deploy demo to Vercel</span>
      <span class="pl-ent">env</span>:
        <span class="pl-ent">VERCEL_TOKEN</span>: <span class="pl-s">${{ secrets.VERCEL_TOKEN }}</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        wget https://latest.datasette.io/fixtures.db</span>
<span class="pl-s">        datasette publish vercel fixtures.db \</span>
<span class="pl-s">          --project datasette-hashed-urls \</span>
<span class="pl-s">          --install https://github.com/simonw/datasette-hashed-urls/archive/$GITHUB_SHA.zip \</span>
<span class="pl-s">          --token $VERCEL_TOKEN \</span>
<span class="pl-s">          --scope datasette</span></pre></div>
<p>The <code>--token $VERCEL_TOKEN</code> passes a token created in the Vercel dashboard. I needed <code>--scope datasette</code> here because I was deploying to a Vercel team of that name - if deploying to your personal account you can leave this off.</p>


path: webassembly_compile-to-wasm-llvm-macos.md
topic: webassembly
title: Compiling to WASM with llvm on macOS
url: https://github.com/simonw/til/blob/main/webassembly/compile-to-wasm-llvm-macos.md
created: 2022-03-28T11:41:17-07:00
created_utc: 2022-03-28T18:41:17+00:00
updated: 2022-03-28T12:06:11-07:00
updated_utc: 2022-03-28T19:06:11+00:00
shot_hash: 77faa37512337f48092a39a61e569410
slug: compile-to-wasm-llvm-macos
body:
[howto-wasm-minimal](https://github.com/ern0/howto-wasm-minimal) by Zalka Ernő ([my fork here](https://github.com/simonw/howto-wasm-minimal)) is a neat demo of a minimal WASM module. It uses C++ to define functions for simple image manipulation including blurring an image, compiles it to WASM using llvm/clang++, then uses JavaScript to run those functions against an image loaded into a `<canvas>` element.

I decided to try compiling it myself:

```
cd /tmp
git clone https://github.com/ern0/howto-wasm-minimal
cd howto-wasm-minimal
./build.sh
```
This gave me the following error:
```
error: unable to create target: 'No available targets are compatible with triple "wasm32"'
```
Searching for the error lead me to [this comment](https://github.com/WebAssembly/wasi-sdk/issues/172#issuecomment-772399153):

> You need to install `llvm` from `Homebrew`. Xcode's `clang` doesn't have support for WebAssembly.

So I installed `llvm` from Homebrew:

```
% brew install llvm
...
To use the bundled libc++ please add the following LDFLAGS:
  LDFLAGS="-L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib"

llvm is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have llvm first in your PATH, run:
  echo 'export PATH="/usr/local/opt/llvm/bin:$PATH"' >> ~/.zshrc

For compilers to find llvm you may need to set:
  export LDFLAGS="-L/usr/local/opt/llvm/lib"
  export CPPFLAGS="-I/usr/local/opt/llvm/include"
...
```
I've quoted the relevant output. I'm not ready to permanently replace the system ``llvm`` with the one from Homebrew, so I ran the build script like this instead:
```
% PATH="/usr/local/opt/llvm/bin:$PATH" ./build.sh 
0000000 00 61 73 6d 01 00 00 00 01 14 04 60 00 00 60 01
```
That created a `inc.wasm` file in my current folder.

I tried opening `index.html` in Firefox and got the following error:

> Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at file:///private/tmp/howto-wasm-minimal/inc.wasm. (Reason: CORS request not http).

So I ran a localhost web server instead using this Python one-liner:
```
% python3 -m http.server 8004
Serving HTTP on :: port 8004 (http://[::]:8004/) ...
```
This worked! http://localhost:8004/ displayed the working demo:

<img width="655" alt="A blurry picture of a car" src="https://user-images.githubusercontent.com/9599/160464900-c4c964ac-b825-4a49-b381-e17da3543187.png">

For reference, here's that `build.sh` script in full:

```bash
#!/bin/bash
set -e

clang++ \
	--target=wasm32 \
	-nostdlib \
	-O3 \
	-Wl,--no-entry \
	-Wl,--export-all \
	-Wl,--lto-O3 \
	-Wl,--allow-undefined \
	-Wl,--import-memory \
	-o inc.wasm \
	inc.cpp

hexdump inc.wasm | head -n 1
#wasm-objdump -x inc.wasm
```

## Bonus: Web Assembly Binary Toolkit

The last commented-out line of that script caught my attention:

    wasm-objdump -x inc.wasm

I searched, and `wasm-objdump` is part of the tools provided by the [WebAssembly Binary Toolkit ](https://github.com/WebAssembly/wabt).

You can install that with Homebrew by running `brew install wabt`.

Then this worked:

```
% wasm-objdump -x inc.wasm

inc.wasm:	file format wasm 0x1

Section Details:

Type[4]:
 - type[0] () -> nil
 - type[1] (i32) -> i32
 - type[2] (i32, i32) -> nil
 - type[3] (i32, i32, i32) -> nil
Import[1]:
 - memory[0] pages: initial=2 <- env.memory
Function[7]:
 - func[0] sig=0 <__wasm_call_ctors>
 - func[1] sig=1 <inc>
 - func[2] sig=0 <incmem>
 - func[3] sig=2 <gray>
 - func[4] sig=2 <swaprg>
 - func[5] sig=3 <swap_red>
 - func[6] sig=2 <blur>
Global[7]:
 - global[0] i32 mutable=1 <__stack_pointer> - init i32=66560
 - global[1] i32 mutable=0 <__dso_handle> - init i32=1024
 - global[2] i32 mutable=0 <__data_end> - init i32=1024
 - global[3] i32 mutable=0 <__global_base> - init i32=1024
 - global[4] i32 mutable=0 <__heap_base> - init i32=66560
 - global[5] i32 mutable=0 <__memory_base> - init i32=0
 - global[6] i32 mutable=0 <__table_base> - init i32=1
Export[13]:
 - func[0] <__wasm_call_ctors> -> "__wasm_call_ctors"
 - func[1] <inc> -> "inc"
 - func[2] <incmem> -> "incmem"
 - func[3] <gray> -> "gray"
 - func[4] <swaprg> -> "swaprg"
 - func[5] <swap_red> -> "swap_red"
 - func[6] <blur> -> "blur"
 - global[1] -> "__dso_handle"
 - global[2] -> "__data_end"
 - global[3] -> "__global_base"
 - global[4] -> "__heap_base"
 - global[5] -> "__memory_base"
 - global[6] -> "__table_base"
Code[7]:
 - func[0] size=2 <__wasm_call_ctors>
 - func[1] size=7 <inc>
 - func[2] size=69 <incmem>
 - func[3] size=101 <gray>
 - func[4] size=215 <swaprg>
 - func[5] size=1332 <swap_red>
 - func[6] size=615 <blur>
Custom:
 - name: "name"
 - func[0] <__wasm_call_ctors>
 - func[1] <inc>
 - func[2] <incmem>
 - func[3] <gray>
 - func[4] <swaprg>
 - func[5] <swap_red>
 - func[6] <blur>
 - global[0] <__stack_pointer>
Custom:
 - name: "producers"
```
Also fun: `wasm-decompile` to "decompile a wasm binary into readable C-like syntax":
```
% wasm-decompile inc.wasm
import memory env_memory;

global stack_pointer:int = 66560;
export global dso_handle:int = 1024;
export global data_end:int = 1024;
export global global_base:int = 1024;
export global heap_base:int = 66560;
...
```

And `wasm-opcodecnt` to "count opcode usage for instructions":
```
% wasm-opcodecnt inc.wasm 
Total opcodes: 1188

Opcode counts:
local.get: 263
i32.const: 240
i32.add: 172
local.set: 88
...
```
More in the README: https://github.com/WebAssembly/wabt/blob/main/README.md

html:
<p><a href="https://github.com/ern0/howto-wasm-minimal">howto-wasm-minimal</a> by Zalka Ernő (<a href="https://github.com/simonw/howto-wasm-minimal">my fork here</a>) is a neat demo of a minimal WASM module. It uses C++ to define functions for simple image manipulation including blurring an image, compiles it to WASM using llvm/clang++, then uses JavaScript to run those functions against an image loaded into a <code>&lt;canvas&gt;</code> element.</p>
<p>I decided to try compiling it myself:</p>
<pre><code>cd /tmp
git clone https://github.com/ern0/howto-wasm-minimal
cd howto-wasm-minimal
./build.sh
</code></pre>
<p>This gave me the following error:</p>
<pre><code>error: unable to create target: 'No available targets are compatible with triple "wasm32"'
</code></pre>
<p>Searching for the error lead me to <a href="https://github.com/WebAssembly/wasi-sdk/issues/172#issuecomment-772399153">this comment</a>:</p>
<blockquote>
<p>You need to install <code>llvm</code> from <code>Homebrew</code>. Xcode's <code>clang</code> doesn't have support for WebAssembly.</p>
</blockquote>
<p>So I installed <code>llvm</code> from Homebrew:</p>
<pre><code>% brew install llvm
...
To use the bundled libc++ please add the following LDFLAGS:
  LDFLAGS="-L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib"

llvm is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

If you need to have llvm first in your PATH, run:
  echo 'export PATH="/usr/local/opt/llvm/bin:$PATH"' &gt;&gt; ~/.zshrc

For compilers to find llvm you may need to set:
  export LDFLAGS="-L/usr/local/opt/llvm/lib"
  export CPPFLAGS="-I/usr/local/opt/llvm/include"
...
</code></pre>
<p>I've quoted the relevant output. I'm not ready to permanently replace the system <code>llvm</code> with the one from Homebrew, so I ran the build script like this instead:</p>
<pre><code>% PATH="/usr/local/opt/llvm/bin:$PATH" ./build.sh 
0000000 00 61 73 6d 01 00 00 00 01 14 04 60 00 00 60 01
</code></pre>
<p>That created a <code>inc.wasm</code> file in my current folder.</p>
<p>I tried opening <code>index.html</code> in Firefox and got the following error:</p>
<blockquote>
<p>Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at file:///private/tmp/howto-wasm-minimal/inc.wasm. (Reason: CORS request not http).</p>
</blockquote>
<p>So I ran a localhost web server instead using this Python one-liner:</p>
<pre><code>% python3 -m http.server 8004
Serving HTTP on :: port 8004 (http://[::]:8004/) ...
</code></pre>
<p>This worked! <a href="http://localhost:8004/" rel="nofollow">http://localhost:8004/</a> displayed the working demo:</p>
<p><a href="https://user-images.githubusercontent.com/9599/160464900-c4c964ac-b825-4a49-b381-e17da3543187.png" target="_blank" rel="nofollow"><img width="655" alt="A blurry picture of a car" src="https://user-images.githubusercontent.com/9599/160464900-c4c964ac-b825-4a49-b381-e17da3543187.png" style="max-width:100%;"></a></p>
<p>For reference, here's that <code>build.sh</code> script in full:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c"><span class="pl-c">#!</span>/bin/bash</span>
<span class="pl-c1">set</span> -e

clang++ \
	--target=wasm32 \
	-nostdlib \
	-O3 \
	-Wl,--no-entry \
	-Wl,--export-all \
	-Wl,--lto-O3 \
	-Wl,--allow-undefined \
	-Wl,--import-memory \
	-o inc.wasm \
	inc.cpp

hexdump inc.wasm <span class="pl-k">|</span> head -n 1
<span class="pl-c"><span class="pl-c">#</span>wasm-objdump -x inc.wasm</span></pre></div>
<h2>
<a id="user-content-bonus-web-assembly-binary-toolkit" class="anchor" href="#bonus-web-assembly-binary-toolkit" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Bonus: Web Assembly Binary Toolkit</h2>
<p>The last commented-out line of that script caught my attention:</p>
<pre><code>wasm-objdump -x inc.wasm
</code></pre>
<p>I searched, and <code>wasm-objdump</code> is part of the tools provided by the <a href="https://github.com/WebAssembly/wabt">WebAssembly Binary Toolkit </a>.</p>
<p>You can install that with Homebrew by running <code>brew install wabt</code>.</p>
<p>Then this worked:</p>
<pre><code>% wasm-objdump -x inc.wasm

inc.wasm:	file format wasm 0x1

Section Details:

Type[4]:
 - type[0] () -&gt; nil
 - type[1] (i32) -&gt; i32
 - type[2] (i32, i32) -&gt; nil
 - type[3] (i32, i32, i32) -&gt; nil
Import[1]:
 - memory[0] pages: initial=2 &lt;- env.memory
Function[7]:
 - func[0] sig=0 &lt;__wasm_call_ctors&gt;
 - func[1] sig=1 &lt;inc&gt;
 - func[2] sig=0 &lt;incmem&gt;
 - func[3] sig=2 &lt;gray&gt;
 - func[4] sig=2 &lt;swaprg&gt;
 - func[5] sig=3 &lt;swap_red&gt;
 - func[6] sig=2 &lt;blur&gt;
Global[7]:
 - global[0] i32 mutable=1 &lt;__stack_pointer&gt; - init i32=66560
 - global[1] i32 mutable=0 &lt;__dso_handle&gt; - init i32=1024
 - global[2] i32 mutable=0 &lt;__data_end&gt; - init i32=1024
 - global[3] i32 mutable=0 &lt;__global_base&gt; - init i32=1024
 - global[4] i32 mutable=0 &lt;__heap_base&gt; - init i32=66560
 - global[5] i32 mutable=0 &lt;__memory_base&gt; - init i32=0
 - global[6] i32 mutable=0 &lt;__table_base&gt; - init i32=1
Export[13]:
 - func[0] &lt;__wasm_call_ctors&gt; -&gt; "__wasm_call_ctors"
 - func[1] &lt;inc&gt; -&gt; "inc"
 - func[2] &lt;incmem&gt; -&gt; "incmem"
 - func[3] &lt;gray&gt; -&gt; "gray"
 - func[4] &lt;swaprg&gt; -&gt; "swaprg"
 - func[5] &lt;swap_red&gt; -&gt; "swap_red"
 - func[6] &lt;blur&gt; -&gt; "blur"
 - global[1] -&gt; "__dso_handle"
 - global[2] -&gt; "__data_end"
 - global[3] -&gt; "__global_base"
 - global[4] -&gt; "__heap_base"
 - global[5] -&gt; "__memory_base"
 - global[6] -&gt; "__table_base"
Code[7]:
 - func[0] size=2 &lt;__wasm_call_ctors&gt;
 - func[1] size=7 &lt;inc&gt;
 - func[2] size=69 &lt;incmem&gt;
 - func[3] size=101 &lt;gray&gt;
 - func[4] size=215 &lt;swaprg&gt;
 - func[5] size=1332 &lt;swap_red&gt;
 - func[6] size=615 &lt;blur&gt;
Custom:
 - name: "name"
 - func[0] &lt;__wasm_call_ctors&gt;
 - func[1] &lt;inc&gt;
 - func[2] &lt;incmem&gt;
 - func[3] &lt;gray&gt;
 - func[4] &lt;swaprg&gt;
 - func[5] &lt;swap_red&gt;
 - func[6] &lt;blur&gt;
 - global[0] &lt;__stack_pointer&gt;
Custom:
 - name: "producers"
</code></pre>
<p>Also fun: <code>wasm-decompile</code> to "decompile a wasm binary into readable C-like syntax":</p>
<pre><code>% wasm-decompile inc.wasm
import memory env_memory;

global stack_pointer:int = 66560;
export global dso_handle:int = 1024;
export global data_end:int = 1024;
export global global_base:int = 1024;
export global heap_base:int = 66560;
...
</code></pre>
<p>And <code>wasm-opcodecnt</code> to "count opcode usage for instructions":</p>
<pre><code>% wasm-opcodecnt inc.wasm 
Total opcodes: 1188

Opcode counts:
local.get: 263
i32.const: 240
i32.add: 172
local.set: 88
...
</code></pre>
<p>More in the README: <a href="https://github.com/WebAssembly/wabt/blob/main/README.md">https://github.com/WebAssembly/wabt/blob/main/README.md</a></p>


path: python_calendar-weeks.md
topic: python
title: Generating a calendar week grid with the Python Calendar module
url: https://github.com/simonw/til/blob/main/python/calendar-weeks.md
created: 2022-03-31T09:59:28-07:00
created_utc: 2022-03-31T16:59:28+00:00
updated: 2022-03-31T09:59:28-07:00
updated_utc: 2022-03-31T16:59:28+00:00
shot_hash: 3a75d4a93a238bc0d0455dfbc63c8d92
slug: calendar-weeks
body:
I needed to generate a grid calendar that looks like this (design [by Natalie Downe](https://github.com/natbat/pillarpointstewards/issues/23)):

<img alt="A calendar grid showing April with rows for each week, starting 28th March and finishing 1st May" src="https://user-images.githubusercontent.com/9599/161109554-b10c3aaa-57b3-4de5-8cae-84c88536ab24.png" width="600">

The Python standard library [calendar](https://docs.python.org/3/library/calendar.html) module has a utility function that really helps here:

```python
import calendar

cal = calendar.Calendar()
list(cal.monthdatescalendar(2022, 4))
```
Outputs:
```python
[[datetime.date(2022, 3, 28),
  datetime.date(2022, 3, 29),
  datetime.date(2022, 3, 30),
  datetime.date(2022, 3, 31),
  datetime.date(2022, 4, 1),
  datetime.date(2022, 4, 2),
  datetime.date(2022, 4, 3)],
 [datetime.date(2022, 4, 4),
  datetime.date(2022, 4, 5),
  datetime.date(2022, 4, 6),
  datetime.date(2022, 4, 7),
  datetime.date(2022, 4, 8),
  datetime.date(2022, 4, 9),
  datetime.date(2022, 4, 10)],
 [datetime.date(2022, 4, 11),
  datetime.date(2022, 4, 12),
  datetime.date(2022, 4, 13),
  datetime.date(2022, 4, 14),
  datetime.date(2022, 4, 15),
  datetime.date(2022, 4, 16),
  datetime.date(2022, 4, 17)],
 [datetime.date(2022, 4, 18),
  datetime.date(2022, 4, 19),
  datetime.date(2022, 4, 20),
  datetime.date(2022, 4, 21),
  datetime.date(2022, 4, 22),
  datetime.date(2022, 4, 23),
  datetime.date(2022, 4, 24)],
 [datetime.date(2022, 4, 25),
  datetime.date(2022, 4, 26),
  datetime.date(2022, 4, 27),
  datetime.date(2022, 4, 28),
  datetime.date(2022, 4, 29),
  datetime.date(2022, 4, 30),
  datetime.date(2022, 5, 1)]]
```

html:
<p>I needed to generate a grid calendar that looks like this (design <a href="https://github.com/natbat/pillarpointstewards/issues/23">by Natalie Downe</a>):</p>
<p><a href="https://user-images.githubusercontent.com/9599/161109554-b10c3aaa-57b3-4de5-8cae-84c88536ab24.png" target="_blank" rel="nofollow"><img alt="A calendar grid showing April with rows for each week, starting 28th March and finishing 1st May" src="https://user-images.githubusercontent.com/9599/161109554-b10c3aaa-57b3-4de5-8cae-84c88536ab24.png" width="600" style="max-width:100%;"></a></p>
<p>The Python standard library <a href="https://docs.python.org/3/library/calendar.html" rel="nofollow">calendar</a> module has a utility function that really helps here:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">calendar</span>

<span class="pl-s1">cal</span> <span class="pl-c1">=</span> <span class="pl-s1">calendar</span>.<span class="pl-v">Calendar</span>()
<span class="pl-en">list</span>(<span class="pl-s1">cal</span>.<span class="pl-en">monthdatescalendar</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>))</pre></div>
<p>Outputs:</p>
<div class="highlight highlight-source-python"><pre>[[<span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">3</span>, <span class="pl-c1">28</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">3</span>, <span class="pl-c1">29</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">3</span>, <span class="pl-c1">30</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">3</span>, <span class="pl-c1">31</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">1</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">2</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">3</span>)],
 [<span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">4</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">5</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">6</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">7</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">8</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">9</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">10</span>)],
 [<span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">11</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">12</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">13</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">14</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">15</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">16</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">17</span>)],
 [<span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">18</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">19</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">20</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">21</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">22</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">23</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">24</span>)],
 [<span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">25</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">26</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">27</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">28</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">29</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">4</span>, <span class="pl-c1">30</span>),
  <span class="pl-s1">datetime</span>.<span class="pl-en">date</span>(<span class="pl-c1">2022</span>, <span class="pl-c1">5</span>, <span class="pl-c1">1</span>)]]</pre></div>


path: docker_docker-for-mac-container-to-postgresql-on-host.md
topic: docker
title: Allowing a container in Docker Desktop for Mac to talk to a PostgreSQL server on the host machine
url: https://github.com/simonw/til/blob/main/docker/docker-for-mac-container-to-postgresql-on-host.md
created: 2022-03-31T22:48:17-07:00
created_utc: 2022-04-01T05:48:17+00:00
updated: 2022-03-31T22:48:17-07:00
updated_utc: 2022-04-01T05:48:17+00:00
shot_hash: 9ea725043c83df0d505051bd25506766
slug: docker-for-mac-container-to-postgresql-on-host
body:
I like using [Postgres.app](https://postgresapp.com/) to run PostgreSQL on my macOS laptop. I use it for a bunch of different projects.

When I deploy applications to Fly.io I build them as Docker containers and inject the Fly PostgreSQL database details as a `DATABASE_URL` environment variable.

In order to test those containers on my laptop, I needed to figure out a way to set a `DATABASE_URL` that would point to the PostgreSQL I have running on my own laptop - so that I didn't need to spin up another PostgreSQL Docker container just for testing purposes.

## host.docker.internal

The first thing to know is that Docker for Desktop sets `host.docker.internal` as a magic hostname inside the container that refers back to the IP address of the host machine.

So ideally something like this should work:

    docker run --env DATABASE_URL="postgres://docker:docker-password@host.docker.internal:5432/pillarpointstewards" \
       -p 8080:8000 pillarpointstewards

I'm using `-p 8080:8000` here to set port 8080 on my laptop to forward to the Django application server running on port 8000 inside the container.

## Creating the account and granting permissions

To create that PostgreSQL account with username `docker` and password `docker-password` (but pick a better password than that) I used Postico to open a connection to my `postgres` database and ran the following:

    create role docker login password 'docker-password';

Then I connected to my application database (in this case `pillarpointstewards`) and ran the following to grant permissions to that user:
```sql
GRANT ALL ON ALL TABLES IN SCHEMA "public" TO docker;
```
Having done this, the container run with the above `DATABASE_URL` environment variable was able to both connect to the server and run Django migrations too.

html:
<p>I like using <a href="https://postgresapp.com/" rel="nofollow">Postgres.app</a> to run PostgreSQL on my macOS laptop. I use it for a bunch of different projects.</p>
<p>When I deploy applications to Fly.io I build them as Docker containers and inject the Fly PostgreSQL database details as a <code>DATABASE_URL</code> environment variable.</p>
<p>In order to test those containers on my laptop, I needed to figure out a way to set a <code>DATABASE_URL</code> that would point to the PostgreSQL I have running on my own laptop - so that I didn't need to spin up another PostgreSQL Docker container just for testing purposes.</p>
<h2>
<a id="user-content-hostdockerinternal" class="anchor" href="#hostdockerinternal" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>host.docker.internal</h2>
<p>The first thing to know is that Docker for Desktop sets <code>host.docker.internal</code> as a magic hostname inside the container that refers back to the IP address of the host machine.</p>
<p>So ideally something like this should work:</p>
<pre><code>docker run --env DATABASE_URL="postgres://docker:docker-password@host.docker.internal:5432/pillarpointstewards" \
   -p 8080:8000 pillarpointstewards
</code></pre>
<p>I'm using <code>-p 8080:8000</code> here to set port 8080 on my laptop to forward to the Django application server running on port 8000 inside the container.</p>
<h2>
<a id="user-content-creating-the-account-and-granting-permissions" class="anchor" href="#creating-the-account-and-granting-permissions" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating the account and granting permissions</h2>
<p>To create that PostgreSQL account with username <code>docker</code> and password <code>docker-password</code> (but pick a better password than that) I used Postico to open a connection to my <code>postgres</code> database and ran the following:</p>
<pre><code>create role docker login password 'docker-password';
</code></pre>
<p>Then I connected to my application database (in this case <code>pillarpointstewards</code>) and ran the following to grant permissions to that user:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">GRANT</span> ALL <span class="pl-k">ON</span> ALL TABLES <span class="pl-k">IN</span> SCHEMA <span class="pl-s"><span class="pl-pds">"</span>public<span class="pl-pds">"</span></span> TO docker;</pre></div>
<p>Having done this, the container run with the above <code>DATABASE_URL</code> environment variable was able to both connect to the server and run Django migrations too.</p>


path: pytest_treat-warnings-as-errors.md
topic: pytest
title: Treating warnings as errors in pytest
url: https://github.com/simonw/til/blob/main/pytest/treat-warnings-as-errors.md
created: 2022-04-01T10:35:54-07:00
created_utc: 2022-04-01T17:35:54+00:00
updated: 2022-04-03T19:31:51-07:00
updated_utc: 2022-04-04T02:31:51+00:00
shot_hash: 0192bfff96a1b8cba4512c381a3336a5
slug: treat-warnings-as-errors
body:
I was seeing this warning in a Django project when I thought I was correctly using timezone-aware dates everywhere:

> RuntimeWarning: DateTimeField Shift.shift_start received a naive datetime (2022-04-01 00:00:00) while time zone support is active

Running `pytest -Werror` turns those warnings into errors that fail the tests.

Which means you can investigate them in the Python debugger by running:

    pytest -Werror --pdb -x

The `--pdb` starts the debugger at the warning (now error) and the `-x` stops the tests after the first failure.

## In pytest.ini

You can also set this in `pytest.ini` - useful if you want ALL warnings to be failures in both development and CI.

Add the following to the `pytest.ini` file:

```ini
[pytest]
# ...
filterwarnings =
    error
```

## Ignoring specific warnings

If you do this you may find there are warnings you cannot fix (because they are in dependency libraries). You can ignore those like this:

```ini
[pytest]
# ...
filterwarnings =
    error
    ignore::arrow.factory.ArrowParseWarning
```

You need to figure out the full path to the warning. I used `--pdb` to figure this out.

html:
<p>I was seeing this warning in a Django project when I thought I was correctly using timezone-aware dates everywhere:</p>
<blockquote>
<p>RuntimeWarning: DateTimeField Shift.shift_start received a naive datetime (2022-04-01 00:00:00) while time zone support is active</p>
</blockquote>
<p>Running <code>pytest -Werror</code> turns those warnings into errors that fail the tests.</p>
<p>Which means you can investigate them in the Python debugger by running:</p>
<pre><code>pytest -Werror --pdb -x
</code></pre>
<p>The <code>--pdb</code> starts the debugger at the warning (now error) and the <code>-x</code> stops the tests after the first failure.</p>
<h2>
<a id="user-content-in-pytestini" class="anchor" href="#in-pytestini" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>In pytest.ini</h2>
<p>You can also set this in <code>pytest.ini</code> - useful if you want ALL warnings to be failures in both development and CI.</p>
<p>Add the following to the <code>pytest.ini</code> file:</p>
<div class="highlight highlight-source-ini"><pre><span class="pl-en">[pytest]</span>
<span class="pl-c"><span class="pl-c">#</span> ...</span>
<span class="pl-k">filterwarnings</span> =
    error</pre></div>
<h2>
<a id="user-content-ignoring-specific-warnings" class="anchor" href="#ignoring-specific-warnings" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Ignoring specific warnings</h2>
<p>If you do this you may find there are warnings you cannot fix (because they are in dependency libraries). You can ignore those like this:</p>
<div class="highlight highlight-source-ini"><pre><span class="pl-en">[pytest]</span>
<span class="pl-c"><span class="pl-c">#</span> ...</span>
<span class="pl-k">filterwarnings</span> =
    error
    ignore::arrow.factory.ArrowParseWarning</pre></div>
<p>You need to figure out the full path to the warning. I used <code>--pdb</code> to figure this out.</p>


path: auth0_auth0-logout.md
topic: auth0
title: Logging users out of Auth0
url: https://github.com/simonw/til/blob/main/auth0/auth0-logout.md
created: 2022-04-03T18:18:12-07:00
created_utc: 2022-04-04T01:18:12+00:00
updated: 2022-04-03T18:18:12-07:00
updated_utc: 2022-04-04T01:18:12+00:00
shot_hash: ca445cd68963cdace05d56bb313c46e9
slug: auth0-logout
body:
If you [implement Auth0](https://til.simonwillison.net/auth0/oauth-with-auth0) for login, you may be tempted to skip implementing logout. I started out just with a `/logout/` page that cleared my own site's cookies, ignoring the Auth0 side of it.

Since users were still signed in to Auth0 (still had cookies there), this meant that if they clicked "login" again after clicking "logout" they would be logged straight in without needing to authenticate at all.

There are two problems with this approach:

1. It defies user expectations. If someone logged out they want to be logged out. Users don't understand the difference between being logged out in your own site and logged out for Auth0.
2. Sometimes people have a legitimate reason for wanting to properly log out - if they are on a shared computer and they need to be able to sign out and then sign back in as a different account.

For example, a couple who share the same computer and want to sign into their own separate accounts. I ran into this use-case pretty quickly!

## Logging users out of Auth0

The good news is this is easy to implement via a redirect. Clear your own site's cookies and then send them to:

    https://YOURDOMAIN.us.auth0.com/v2/logout?client_id=YOUR_CLIENT_ID&returnTo=URL

That `returnTo` URL is where Auth0 will return them to. I used my site's homepage.

It needs to be listed under "Allowed Logout URLs" in the Auth0 settings.

Relevant Auth0 documentation:

- [Logout](https://auth0.com/docs/authenticate/login/logout) Auth0 high level documentation
- [Log Users Out of Auth0](https://auth0.com/docs/authenticate/login/logout/log-users-out-of-auth0) describes how you can log them out of Auth0 (what I wanted) or you can additionally log them out of Google SSO (not what I wanted)
- [GET /v2/logout](https://auth0.com/docs/api/authentication#logout) API documentation

I implemented this for [pillarpointstewards/issues/54](https://github.com/natbat/pillarpointstewards/issues/54), in [this commit](https://github.com/natbat/pillarpointstewards/commit/2a79046b52da79216e6548d2ef386decf6b65656).

html:
<p>If you <a href="https://til.simonwillison.net/auth0/oauth-with-auth0" rel="nofollow">implement Auth0</a> for login, you may be tempted to skip implementing logout. I started out just with a <code>/logout/</code> page that cleared my own site's cookies, ignoring the Auth0 side of it.</p>
<p>Since users were still signed in to Auth0 (still had cookies there), this meant that if they clicked "login" again after clicking "logout" they would be logged straight in without needing to authenticate at all.</p>
<p>There are two problems with this approach:</p>
<ol>
<li>It defies user expectations. If someone logged out they want to be logged out. Users don't understand the difference between being logged out in your own site and logged out for Auth0.</li>
<li>Sometimes people have a legitimate reason for wanting to properly log out - if they are on a shared computer and they need to be able to sign out and then sign back in as a different account.</li>
</ol>
<p>For example, a couple who share the same computer and want to sign into their own separate accounts. I ran into this use-case pretty quickly!</p>
<h2>
<a id="user-content-logging-users-out-of-auth0" class="anchor" href="#logging-users-out-of-auth0" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Logging users out of Auth0</h2>
<p>The good news is this is easy to implement via a redirect. Clear your own site's cookies and then send them to:</p>
<pre><code>https://YOURDOMAIN.us.auth0.com/v2/logout?client_id=YOUR_CLIENT_ID&amp;returnTo=URL
</code></pre>
<p>That <code>returnTo</code> URL is where Auth0 will return them to. I used my site's homepage.</p>
<p>It needs to be listed under "Allowed Logout URLs" in the Auth0 settings.</p>
<p>Relevant Auth0 documentation:</p>
<ul>
<li>
<a href="https://auth0.com/docs/authenticate/login/logout" rel="nofollow">Logout</a> Auth0 high level documentation</li>
<li>
<a href="https://auth0.com/docs/authenticate/login/logout/log-users-out-of-auth0" rel="nofollow">Log Users Out of Auth0</a> describes how you can log them out of Auth0 (what I wanted) or you can additionally log them out of Google SSO (not what I wanted)</li>
<li>
<a href="https://auth0.com/docs/api/authentication#logout" rel="nofollow">GET /v2/logout</a> API documentation</li>
</ul>
<p>I implemented this for <a href="https://github.com/natbat/pillarpointstewards/issues/54">pillarpointstewards/issues/54</a>, in <a href="https://github.com/natbat/pillarpointstewards/commit/2a79046b52da79216e6548d2ef386decf6b65656">this commit</a>.</p>


path: bash_use-awk-to-add-a-prefix.md
topic: bash
title: Using awk to add a prefix
url: https://github.com/simonw/til/blob/main/bash/use-awk-to-add-a-prefix.md
created: 2022-04-08T09:25:04-07:00
created_utc: 2022-04-08T16:25:04+00:00
updated: 2022-04-08T09:25:04-07:00
updated_utc: 2022-04-08T16:25:04+00:00
shot_hash: 801ca3f33198f55a114494b5608cb6c1
slug: use-awk-to-add-a-prefix
body:
I wanted to dynamically run the following command against all files in a directory:

```bash
pypi-to-sqlite content.db -f /tmp/pypi-datasette-packages/packages/airtable-export.json \
-f /tmp/pypi-datasette-packages/packages/csv-diff.json \
--prefix pypi_
```

I can't use `/tmp/pypi-datasette-packages/packages/*.json` here because I need each file to be processed using the `-f` option.

I found a solution using `awk`. The `awk` program `'{print "-f "$0}'` adds a prefix to the input, for example:
```
% echo "blah" | awk '{print "-f "$0}'      
-f blah
```
I wanted that trailing backslash too, so I used this:

```awk
{print "-f "$0 " \\"}
```
Piping to `awk` works, so I combined that with `ls ../*.json` like so:

```
% ls /tmp/pypi-datasette-packages/packages/*.json | awk '{print "-f "$0 " \\"}' 
-f /tmp/pypi-datasette-packages/packages/airtable-export.json \
-f /tmp/pypi-datasette-packages/packages/csv-diff.json \
-f /tmp/pypi-datasette-packages/packages/csvs-to-sqlite.json \
```
Then I used `eval` to execute the command. The full recipe looks like this:
```bash
args=$(ls /tmp/pypi-datasette-packages/packages/*.json | awk '{print "-f "$0 " \\"}')
eval "pypi-to-sqlite content.db $args
--prefix pypi_"
```
Full details in [datasette.io issue 98](https://github.com/simonw/datasette.io/issues/98).

html:
<p>I wanted to dynamically run the following command against all files in a directory:</p>
<div class="highlight highlight-source-shell"><pre>pypi-to-sqlite content.db -f /tmp/pypi-datasette-packages/packages/airtable-export.json \
-f /tmp/pypi-datasette-packages/packages/csv-diff.json \
--prefix pypi_</pre></div>
<p>I can't use <code>/tmp/pypi-datasette-packages/packages/*.json</code> here because I need each file to be processed using the <code>-f</code> option.</p>
<p>I found a solution using <code>awk</code>. The <code>awk</code> program <code>'{print "-f "$0}'</code> adds a prefix to the input, for example:</p>
<pre><code>% echo "blah" | awk '{print "-f "$0}'      
-f blah
</code></pre>
<p>I wanted that trailing backslash too, so I used this:</p>
<div class="highlight highlight-source-awk"><pre>{<span class="pl-k">print</span> <span class="pl-s"><span class="pl-pds">"</span>-f <span class="pl-pds">"</span></span><span class="pl-c1">$0</span> <span class="pl-s"><span class="pl-pds">"</span> <span class="pl-cce">\\</span><span class="pl-pds">"</span></span>}</pre></div>
<p>Piping to <code>awk</code> works, so I combined that with <code>ls ../*.json</code> like so:</p>
<pre><code>% ls /tmp/pypi-datasette-packages/packages/*.json | awk '{print "-f "$0 " \\"}' 
-f /tmp/pypi-datasette-packages/packages/airtable-export.json \
-f /tmp/pypi-datasette-packages/packages/csv-diff.json \
-f /tmp/pypi-datasette-packages/packages/csvs-to-sqlite.json \
</code></pre>
<p>Then I used <code>eval</code> to execute the command. The full recipe looks like this:</p>
<div class="highlight highlight-source-shell"><pre>args=<span class="pl-s"><span class="pl-pds">$(</span>ls /tmp/pypi-datasette-packages/packages/<span class="pl-k">*</span>.json <span class="pl-k">|</span> awk <span class="pl-s"><span class="pl-pds">'</span>{print "-f "$0 " \\"}<span class="pl-pds">'</span></span><span class="pl-pds">)</span></span>
<span class="pl-c1">eval</span> <span class="pl-s"><span class="pl-pds">"</span>pypi-to-sqlite content.db <span class="pl-smi">$args</span></span>
<span class="pl-s">--prefix pypi_<span class="pl-pds">"</span></span></pre></div>
<p>Full details in <a href="https://github.com/simonw/datasette.io/issues/98">datasette.io issue 98</a>.</p>


path: spatialite_gunion-to-combine-geometries.md
topic: spatialite
title: GUnion to combine geometries in SpatiaLite
url: https://github.com/simonw/til/blob/main/spatialite/gunion-to-combine-geometries.md
created: 2022-04-12T11:58:17-07:00
created_utc: 2022-04-12T18:58:17+00:00
updated: 2022-04-12T12:49:15-07:00
updated_utc: 2022-04-12T19:49:15+00:00
shot_hash: 336867d880c40a7f359fd4cb8854ccf3
slug: gunion-to-combine-geometries
body:
I was playing around with [datanews/amtrak-geojson](https://github.com/datanews/amtrak-geojson), which provides GeoJSON for Amtrak stations and track segments.

I loaded it into SQLite using [geojson-to-sqlite](https://datasette.io/tools/geojson-to-sqlite) like this:

    curl "https://raw.githubusercontent.com/datanews/amtrak-geojson/master/amtrak-stations.geojson" | \
      geojson-to-sqlite /tmp/amtrak.db stations - --spatialite
    curl "https://raw.githubusercontent.com/datanews/amtrak-geojson/master/amtrak-track.geojson" | \
      geojson-to-sqlite /tmp/amtrak.db track - --spatialite

Then I installed [datasette-geojson-map](https://datasette.io/plugins/datasette-geojson-map) in order to visualize it, and opened it in Datasette:

    datasette install datasette-geojson-map
    datasette /tmp/amtrak.db --load-extension spatialite

The stations table rendered a map just fine. I added `?_size=1000` to the URL to see all of the stations rather than just the first 100:

    http://127.0.0.1:8001/amtrak/stations?_size=1000

<img width="1200" alt="The table view includes a map of the USA showing all 500 stations as markers" src="https://user-images.githubusercontent.com/9599/163033038-e02fe482-21df-445e-874f-94236072883b.png">

But the track page was less useful, even with the `?_size=1000` parameter:

    http://127.0.0.1:8001/amtrak/track?_size=1000

![The track page shows the USA map but with only a small number of little black squiggles, not a full track map for the country](https://user-images.githubusercontent.com/9599/163033296-bfa915f6-d0de-4219-b89c-b6384c826f59.png)

This is because there are 10,768 segments of track in the database, so even showing 1,000 at a time results in a very spotty map.

## Using GUnion

The solution was to combine the track segments together using the [SpatiaLite GUnion function](http://www.gaia-gis.it/gaia-sins/spatialite-sql-4.2.0.html#p14). I used the following custom SQL query:
```sql
select GUnion(geometry) as geometry from track
```
The `as geometry` is required because the mapping plugin looks for a column of that name.

Here's the result:

<img width="1202" alt="This time there is a map showing Amtrak track across the whole of the USA" src="https://user-images.githubusercontent.com/9599/163033681-527a0158-cded-40c5-9efa-9d912eeaa04a.png">

This also works for queries that pull out a subset of the data. Here's the combination of every track in FRAREGIONS 7:

<img width="1198" alt="A map just of California showing Amtrak railways there" src="https://user-images.githubusercontent.com/9599/163033991-c82dafbf-8348-496f-8300-2ae8d35eacd5.png">

Using this query:

```sql
select GUnion(geometry) as geometry from track where "FRAREGIONS" = :p0
```

## Different colours for different sections

Thanks to faceting I noticed there are 8 different FRAREGIONS. `datasette-geojson-map` supports [styled map features](https://datasette.io/plugins/datasette-geojson-map#user-content-styling-map-features), so I decided to try and show the different regions in different colours.

This query did th trick:

```sql
select
  'FRA Region ' || FRAREGIONS as title,
  case
    FRAREGIONS
    when 1 then "#dfff00"
    when 2 then "#ffbf00"
    when 3 then "#ff7f50"
    when 4 then "#de3163"
    when 5 then "#9fe2bf"
    when 6 then "#40e0d0"
    when 7 then "#6495ed"
    when 8 then "#ccccff"
    else "#000000"
  end as stroke,
  GUnion(geometry) as geometry
from
  track
group by
  FRAREGIONS
```

<img width="1197" alt="Each region now shows as a different coloured line" src="https://user-images.githubusercontent.com/9599/163035751-f0d465df-bc44-4822-82c6-24715680663f.png">

## Using sqlite-colorbrewer

`datasette-geojson-map` creator Chris Amico [suggested](https://twitter.com/eyeseast/status/1513961522392883205) using his [sqlite-colorbrewer](https://datasette.io/plugins/sqlite-colorbrewer) plugin to generate the colours. Here's the query using that instead:

```sql
select
  'FRA Region ' || FRAREGIONS as title,
  colorbrewer('Paired', 9, FRAREGIONS) as stroke,
  GUnion(geometry) as geometry
from
  track
group by
  FRAREGIONS
```

<img width="1200" alt="The same map again but with a more pleasant colour scheme" src="https://user-images.githubusercontent.com/9599/163041882-e9a1500b-ca11-4059-a779-b1f3e1f51035.png">

html:
<p>I was playing around with <a href="https://github.com/datanews/amtrak-geojson">datanews/amtrak-geojson</a>, which provides GeoJSON for Amtrak stations and track segments.</p>
<p>I loaded it into SQLite using <a href="https://datasette.io/tools/geojson-to-sqlite" rel="nofollow">geojson-to-sqlite</a> like this:</p>
<pre><code>curl "https://raw.githubusercontent.com/datanews/amtrak-geojson/master/amtrak-stations.geojson" | \
  geojson-to-sqlite /tmp/amtrak.db stations - --spatialite
curl "https://raw.githubusercontent.com/datanews/amtrak-geojson/master/amtrak-track.geojson" | \
  geojson-to-sqlite /tmp/amtrak.db track - --spatialite
</code></pre>
<p>Then I installed <a href="https://datasette.io/plugins/datasette-geojson-map" rel="nofollow">datasette-geojson-map</a> in order to visualize it, and opened it in Datasette:</p>
<pre><code>datasette install datasette-geojson-map
datasette /tmp/amtrak.db --load-extension spatialite
</code></pre>
<p>The stations table rendered a map just fine. I added <code>?_size=1000</code> to the URL to see all of the stations rather than just the first 100:</p>
<pre><code>http://127.0.0.1:8001/amtrak/stations?_size=1000
</code></pre>
<p><a href="https://user-images.githubusercontent.com/9599/163033038-e02fe482-21df-445e-874f-94236072883b.png" target="_blank" rel="nofollow"><img width="1200" alt="The table view includes a map of the USA showing all 500 stations as markers" src="https://user-images.githubusercontent.com/9599/163033038-e02fe482-21df-445e-874f-94236072883b.png" style="max-width:100%;"></a></p>
<p>But the track page was less useful, even with the <code>?_size=1000</code> parameter:</p>
<pre><code>http://127.0.0.1:8001/amtrak/track?_size=1000
</code></pre>
<p><a href="https://user-images.githubusercontent.com/9599/163033296-bfa915f6-d0de-4219-b89c-b6384c826f59.png" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/163033296-bfa915f6-d0de-4219-b89c-b6384c826f59.png" alt="The track page shows the USA map but with only a small number of little black squiggles, not a full track map for the country" style="max-width:100%;"></a></p>
<p>This is because there are 10,768 segments of track in the database, so even showing 1,000 at a time results in a very spotty map.</p>
<h2>
<a id="user-content-using-gunion" class="anchor" href="#using-gunion" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using GUnion</h2>
<p>The solution was to combine the track segments together using the <a href="http://www.gaia-gis.it/gaia-sins/spatialite-sql-4.2.0.html#p14" rel="nofollow">SpatiaLite GUnion function</a>. I used the following custom SQL query:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> GUnion(geometry) <span class="pl-k">as</span> geometry <span class="pl-k">from</span> track</pre></div>
<p>The <code>as geometry</code> is required because the mapping plugin looks for a column of that name.</p>
<p>Here's the result:</p>
<p><a href="https://user-images.githubusercontent.com/9599/163033681-527a0158-cded-40c5-9efa-9d912eeaa04a.png" target="_blank" rel="nofollow"><img width="1202" alt="This time there is a map showing Amtrak track across the whole of the USA" src="https://user-images.githubusercontent.com/9599/163033681-527a0158-cded-40c5-9efa-9d912eeaa04a.png" style="max-width:100%;"></a></p>
<p>This also works for queries that pull out a subset of the data. Here's the combination of every track in FRAREGIONS 7:</p>
<p><a href="https://user-images.githubusercontent.com/9599/163033991-c82dafbf-8348-496f-8300-2ae8d35eacd5.png" target="_blank" rel="nofollow"><img width="1198" alt="A map just of California showing Amtrak railways there" src="https://user-images.githubusercontent.com/9599/163033991-c82dafbf-8348-496f-8300-2ae8d35eacd5.png" style="max-width:100%;"></a></p>
<p>Using this query:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> GUnion(geometry) <span class="pl-k">as</span> geometry <span class="pl-k">from</span> track <span class="pl-k">where</span> <span class="pl-s"><span class="pl-pds">"</span>FRAREGIONS<span class="pl-pds">"</span></span> <span class="pl-k">=</span> :p0</pre></div>
<h2>
<a id="user-content-different-colours-for-different-sections" class="anchor" href="#different-colours-for-different-sections" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Different colours for different sections</h2>
<p>Thanks to faceting I noticed there are 8 different FRAREGIONS. <code>datasette-geojson-map</code> supports <a href="https://datasette.io/plugins/datasette-geojson-map#user-content-styling-map-features" rel="nofollow">styled map features</a>, so I decided to try and show the different regions in different colours.</p>
<p>This query did th trick:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>FRA Region <span class="pl-pds">'</span></span> <span class="pl-k">||</span> FRAREGIONS <span class="pl-k">as</span> title,
  case
    FRAREGIONS
    when <span class="pl-c1">1</span> then <span class="pl-s"><span class="pl-pds">"</span>#dfff00<span class="pl-pds">"</span></span>
    when <span class="pl-c1">2</span> then <span class="pl-s"><span class="pl-pds">"</span>#ffbf00<span class="pl-pds">"</span></span>
    when <span class="pl-c1">3</span> then <span class="pl-s"><span class="pl-pds">"</span>#ff7f50<span class="pl-pds">"</span></span>
    when <span class="pl-c1">4</span> then <span class="pl-s"><span class="pl-pds">"</span>#de3163<span class="pl-pds">"</span></span>
    when <span class="pl-c1">5</span> then <span class="pl-s"><span class="pl-pds">"</span>#9fe2bf<span class="pl-pds">"</span></span>
    when <span class="pl-c1">6</span> then <span class="pl-s"><span class="pl-pds">"</span>#40e0d0<span class="pl-pds">"</span></span>
    when <span class="pl-c1">7</span> then <span class="pl-s"><span class="pl-pds">"</span>#6495ed<span class="pl-pds">"</span></span>
    when <span class="pl-c1">8</span> then <span class="pl-s"><span class="pl-pds">"</span>#ccccff<span class="pl-pds">"</span></span>
    else <span class="pl-s"><span class="pl-pds">"</span>#000000<span class="pl-pds">"</span></span>
  end <span class="pl-k">as</span> stroke,
  GUnion(geometry) <span class="pl-k">as</span> geometry
<span class="pl-k">from</span>
  track
<span class="pl-k">group by</span>
  FRAREGIONS</pre></div>
<p><a href="https://user-images.githubusercontent.com/9599/163035751-f0d465df-bc44-4822-82c6-24715680663f.png" target="_blank" rel="nofollow"><img width="1197" alt="Each region now shows as a different coloured line" src="https://user-images.githubusercontent.com/9599/163035751-f0d465df-bc44-4822-82c6-24715680663f.png" style="max-width:100%;"></a></p>
<h2>
<a id="user-content-using-sqlite-colorbrewer" class="anchor" href="#using-sqlite-colorbrewer" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using sqlite-colorbrewer</h2>
<p><code>datasette-geojson-map</code> creator Chris Amico <a href="https://twitter.com/eyeseast/status/1513961522392883205" rel="nofollow">suggested</a> using his <a href="https://datasette.io/plugins/sqlite-colorbrewer" rel="nofollow">sqlite-colorbrewer</a> plugin to generate the colours. Here's the query using that instead:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  <span class="pl-s"><span class="pl-pds">'</span>FRA Region <span class="pl-pds">'</span></span> <span class="pl-k">||</span> FRAREGIONS <span class="pl-k">as</span> title,
  colorbrewer(<span class="pl-s"><span class="pl-pds">'</span>Paired<span class="pl-pds">'</span></span>, <span class="pl-c1">9</span>, FRAREGIONS) <span class="pl-k">as</span> stroke,
  GUnion(geometry) <span class="pl-k">as</span> geometry
<span class="pl-k">from</span>
  track
<span class="pl-k">group by</span>
  FRAREGIONS</pre></div>
<p><a href="https://user-images.githubusercontent.com/9599/163041882-e9a1500b-ca11-4059-a779-b1f3e1f51035.png" target="_blank" rel="nofollow"><img width="1200" alt="The same map again but with a more pleasant colour scheme" src="https://user-images.githubusercontent.com/9599/163041882-e9a1500b-ca11-4059-a779-b1f3e1f51035.png" style="max-width:100%;"></a></p>


path: macos_apple-photos-large-files.md
topic: macos
title: Trick Apple Photos into letting you access your video files
url: https://github.com/simonw/til/blob/main/macos/apple-photos-large-files.md
created: 2022-04-12T16:35:57-07:00
created_utc: 2022-04-12T23:35:57+00:00
updated: 2022-04-12T16:35:57-07:00
updated_utc: 2022-04-12T23:35:57+00:00
shot_hash: 1d56fa360f056d9e0e30902fff69b322
slug: apple-photos-large-files
body:
I had an 11GB movie in Apple Photos (sync'd from my iPhone) and I wanted to upload it to YouTube (actually via AirDrop to another laptop first).

The "export" options in Apple Photos provided no visual indicator of what they were doing - as far as I could tell they were broken.

I wanted to deal with the actual file on disk. I figured out how to get access to that like so:

1. Right click on the movie in Apple Photes
2. Select Edit With -> QuickTime Player
3. In QuickTime Player, command-click on the filename in the QuickTime window to get a list of folders
4. Click on the parent folder of the file to get a Finder window

Now you can treat the .MOV like a regular file! Right click and AirDrop and suchlike should work fine.

The command-click menu in QuickTime should look like this:

<img width="368" alt="IMG_7790.mov
In a folder with a big complex UUID name
In a folder called ExternalEditSessions
In a folder called com.apple.Photos
in a folder called private
In a folder called Photos Library.photoslibrary" src="https://user-images.githubusercontent.com/9599/163071235-6ff80bc1-374c-4905-93d6-cd3c5486c6a8.png">

Then when I went to quit the Apple Photos app later it told me I had export jobs in progress, the jobs that I hadn't been able to see in the first place!

html:
<p>I had an 11GB movie in Apple Photos (sync'd from my iPhone) and I wanted to upload it to YouTube (actually via AirDrop to another laptop first).</p>
<p>The "export" options in Apple Photos provided no visual indicator of what they were doing - as far as I could tell they were broken.</p>
<p>I wanted to deal with the actual file on disk. I figured out how to get access to that like so:</p>
<ol>
<li>Right click on the movie in Apple Photes</li>
<li>Select Edit With -&gt; QuickTime Player</li>
<li>In QuickTime Player, command-click on the filename in the QuickTime window to get a list of folders</li>
<li>Click on the parent folder of the file to get a Finder window</li>
</ol>
<p>Now you can treat the .MOV like a regular file! Right click and AirDrop and suchlike should work fine.</p>
<p>The command-click menu in QuickTime should look like this:</p>
<p><a href="https://user-images.githubusercontent.com/9599/163071235-6ff80bc1-374c-4905-93d6-cd3c5486c6a8.png" target="_blank" rel="nofollow"><img width="368" alt="IMG_7790.mov
In a folder with a big complex UUID name
In a folder called ExternalEditSessions
In a folder called com.apple.Photos
in a folder called private
In a folder called Photos Library.photoslibrary" src="https://user-images.githubusercontent.com/9599/163071235-6ff80bc1-374c-4905-93d6-cd3c5486c6a8.png" style="max-width:100%;"></a></p>
<p>Then when I went to quit the Apple Photos app later it told me I had export jobs in progress, the jobs that I hadn't been able to see in the first place!</p>


path: twitter_credentials-twitter-bot.md
topic: twitter
title: How to get credentials for a new Twitter bot
url: https://github.com/simonw/til/blob/main/twitter/credentials-twitter-bot.md
created: 2022-04-17T16:03:22-07:00
created_utc: 2022-04-17T23:03:22+00:00
updated: 2022-04-17T16:22:35-07:00
updated_utc: 2022-04-17T23:22:35+00:00
shot_hash: 2a85ddf87ae75e140e45c2729293ee89
slug: credentials-twitter-bot
body:
I wanted to build a Twitter bot that would tweet Covid sewage graphs from [this page](https://covid19.sccgov.org/dashboard-wastewater) every day.

To do that, I needed the following four credentials for a new [@covidsewage](https://twitter.com/covidsewage) Twitter account.

- consumer key (for a new Twitter application)
- consumer secret
- access token key (for my specific Twitter account)
- access token secret

## Applying for a Developer account

I created my new Twitter account, then applied to set that up as a Twitter Developer on https://developer.twitter.com/

When I filled in the form there I said I wanted to create a bot. I think this is one of the things that triggers a manual review flow.

I filled out the application form and submitted it.

I then got an email requesting further details - I replied to that email with pretty much a copy of the data I had entered in the application form early on.

A few days later my application was approved.

## Configuring the app

Since the goal here is to get credentials that can be used to write to the account (in order to Tweet), the application needs to be configured to support that.

It turns out you need to turn on "OAuth 1.0a" in order to generate a read-write token for the account (thanks Igor Brigadir for [the tip](https://twitter.com/IgorBrigadir/status/1515766922742272004)).

The following settings worked for me:

<img width="475" alt="Screenshot showing settings - I set it to read-write permissions with OAuth 1.0 and filled out the website and callback URI fields" src="https://user-images.githubusercontent.com/9599/163735286-d5e64f63-6d70-4a41-9a6f-975e1e918c1c.png">

Since callback URI and website URI were required (even though I'm not going to be using them) I set them to the Twitter profile page, which seemed to work.

## Generating the credentials

Having configured the app, the "Keys and tokens" page for that application gave me the option to generate an access token and secret for the account:

<img width="475" alt="Screenshot indicating the revoke / regenerate token button" src="https://user-images.githubusercontent.com/9599/163735341-44fae6ca-5c16-4798-a15b-53112750b54e.png">

## Building the bot

Having generated the four credentials I needed, I built the rest of the bot using GitHub Actions, see https://github.com/simonw/covidsewage-bot

html:
<p>I wanted to build a Twitter bot that would tweet Covid sewage graphs from <a href="https://covid19.sccgov.org/dashboard-wastewater" rel="nofollow">this page</a> every day.</p>
<p>To do that, I needed the following four credentials for a new <a href="https://twitter.com/covidsewage" rel="nofollow">@covidsewage</a> Twitter account.</p>
<ul>
<li>consumer key (for a new Twitter application)</li>
<li>consumer secret</li>
<li>access token key (for my specific Twitter account)</li>
<li>access token secret</li>
</ul>
<h2>
<a id="user-content-applying-for-a-developer-account" class="anchor" href="#applying-for-a-developer-account" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Applying for a Developer account</h2>
<p>I created my new Twitter account, then applied to set that up as a Twitter Developer on <a href="https://developer.twitter.com/" rel="nofollow">https://developer.twitter.com/</a></p>
<p>When I filled in the form there I said I wanted to create a bot. I think this is one of the things that triggers a manual review flow.</p>
<p>I filled out the application form and submitted it.</p>
<p>I then got an email requesting further details - I replied to that email with pretty much a copy of the data I had entered in the application form early on.</p>
<p>A few days later my application was approved.</p>
<h2>
<a id="user-content-configuring-the-app" class="anchor" href="#configuring-the-app" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Configuring the app</h2>
<p>Since the goal here is to get credentials that can be used to write to the account (in order to Tweet), the application needs to be configured to support that.</p>
<p>It turns out you need to turn on "OAuth 1.0a" in order to generate a read-write token for the account (thanks Igor Brigadir for <a href="https://twitter.com/IgorBrigadir/status/1515766922742272004" rel="nofollow">the tip</a>).</p>
<p>The following settings worked for me:</p>
<p><a href="https://user-images.githubusercontent.com/9599/163735286-d5e64f63-6d70-4a41-9a6f-975e1e918c1c.png" target="_blank" rel="nofollow"><img width="475" alt="Screenshot showing settings - I set it to read-write permissions with OAuth 1.0 and filled out the website and callback URI fields" src="https://user-images.githubusercontent.com/9599/163735286-d5e64f63-6d70-4a41-9a6f-975e1e918c1c.png" style="max-width:100%;"></a></p>
<p>Since callback URI and website URI were required (even though I'm not going to be using them) I set them to the Twitter profile page, which seemed to work.</p>
<h2>
<a id="user-content-generating-the-credentials" class="anchor" href="#generating-the-credentials" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Generating the credentials</h2>
<p>Having configured the app, the "Keys and tokens" page for that application gave me the option to generate an access token and secret for the account:</p>
<p><a href="https://user-images.githubusercontent.com/9599/163735341-44fae6ca-5c16-4798-a15b-53112750b54e.png" target="_blank" rel="nofollow"><img width="475" alt="Screenshot indicating the revoke / regenerate token button" src="https://user-images.githubusercontent.com/9599/163735341-44fae6ca-5c16-4798-a15b-53112750b54e.png" style="max-width:100%;"></a></p>
<h2>
<a id="user-content-building-the-bot" class="anchor" href="#building-the-bot" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Building the bot</h2>
<p>Having generated the four credentials I needed, I built the rest of the bot using GitHub Actions, see <a href="https://github.com/simonw/covidsewage-bot">https://github.com/simonw/covidsewage-bot</a></p>


path: sphinx_blacken-docs.md
topic: sphinx
title: Format code examples in documentation with blacken-docs
url: https://github.com/simonw/til/blob/main/sphinx/blacken-docs.md
created: 2022-04-24T09:16:56-07:00
created_utc: 2022-04-24T16:16:56+00:00
updated: 2022-04-24T09:38:25-07:00
updated_utc: 2022-04-24T16:38:25+00:00
shot_hash: cb40c30a8a0a3ca73baa65fe2bd67f49
slug: blacken-docs
body:
I decided to enforce that all code examples in the [Datasette documentation](https://docs.datasette.io/) be formatted using [Black](https://github.com/psf/black). Here's [issue 1718](https://github.com/simonw/datasette/issues/1718) where I researched the options for doing this.

I found  the [blacken-docs](https://pypi.org/project/blacken-docs/) tool. Here's how to run it against a folder full of reStructuredText files:

    pip install blacken-docs
    blacken-docs docs/*.rst

This modifies the files in place.

## Setting a different line length

I read most documentation on my phone, so when I'm writing code examples I tend to try to keep the line lengths a little bit shorter to avoid having to scroll sideways when reading.

`blacken-docs` has a `-l` option for changing the length (Black defaults to 88 characters) which can be used like this:

    blacken-docs -l 60 docs/*.rst

## Missing function bodies with ...

I was getting errors with some of my code examples that looked like this:

```python
@pytest.fixture
def datasette(tmp_path_factory):
    # This fixture will be executed repeatedly for every test
```
This is because of the missing function body. It turns out adding `...` (which looks prettier than `pass`) fixes this issue:
```python
@pytest.fixture
def datasette(tmp_path_factory):
    # This fixture will be executed repeatedly for every test
    ...
```

## Running this in CI

The `blacken-docs` command outputs errors if it finds any Python examples it cannot parse. I actually found a couple of bugs in my examples using this, so it's a handy feature.

This also causes the tool to exit with a status code of 1:
```
% blacken-docs -l 60 docs/*.rst                        
docs/authentication.rst: Rewriting...
docs/internals.rst:196: code block parse error Cannot parse: 14:0: <line number missing in source>
docs/json_api.rst:449: code block parse error Cannot parse: 1:0: <link rel="alternate"
docs/plugin_hooks.rst:250: code block parse error Cannot parse: 6:4:     ]
docs/plugin_hooks.rst:311: code block parse error Cannot parse: 38:0: <line number missing in source>
% echo $?
1
```
I also wanted my CI to fail if the author had forgotten to run `blacken-docs` against the repository before pushing the commit.

I filed [a feature request](https://github.com/asottile/blacken-docs/issues/161) asking for an equivalent of the `black . --check` option, but it turns out that feature isn't necessary - `blacken-docs` returns a non-zero exit code if it makes any changes. So just running the following in CI works for checking if it should have been applied:

```yaml
    - name: Check if blacken-docs needs to be run
      run: |
        blacken-docs -l 60 docs/*.rst
```

html:
<p>I decided to enforce that all code examples in the <a href="https://docs.datasette.io/" rel="nofollow">Datasette documentation</a> be formatted using <a href="https://github.com/psf/black">Black</a>. Here's <a href="https://github.com/simonw/datasette/issues/1718">issue 1718</a> where I researched the options for doing this.</p>
<p>I found  the <a href="https://pypi.org/project/blacken-docs/" rel="nofollow">blacken-docs</a> tool. Here's how to run it against a folder full of reStructuredText files:</p>
<pre><code>pip install blacken-docs
blacken-docs docs/*.rst
</code></pre>
<p>This modifies the files in place.</p>
<h2>
<a id="user-content-setting-a-different-line-length" class="anchor" href="#setting-a-different-line-length" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Setting a different line length</h2>
<p>I read most documentation on my phone, so when I'm writing code examples I tend to try to keep the line lengths a little bit shorter to avoid having to scroll sideways when reading.</p>
<p><code>blacken-docs</code> has a <code>-l</code> option for changing the length (Black defaults to 88 characters) which can be used like this:</p>
<pre><code>blacken-docs -l 60 docs/*.rst
</code></pre>
<h2>
<a id="user-content-missing-function-bodies-with-" class="anchor" href="#missing-function-bodies-with-" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Missing function bodies with ...</h2>
<p>I was getting errors with some of my code examples that looked like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-s1">fixture</span></span>
<span class="pl-k">def</span> <span class="pl-en">datasette</span>(<span class="pl-s1">tmp_path_factory</span>):
    <span class="pl-c"># This fixture will be executed repeatedly for every test</span></pre></div>
<p>This is because of the missing function body. It turns out adding <code>...</code> (which looks prettier than <code>pass</code>) fixes this issue:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-s1">fixture</span></span>
<span class="pl-k">def</span> <span class="pl-en">datasette</span>(<span class="pl-s1">tmp_path_factory</span>):
    <span class="pl-c"># This fixture will be executed repeatedly for every test</span>
    ...</pre></div>
<h2>
<a id="user-content-running-this-in-ci" class="anchor" href="#running-this-in-ci" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running this in CI</h2>
<p>The <code>blacken-docs</code> command outputs errors if it finds any Python examples it cannot parse. I actually found a couple of bugs in my examples using this, so it's a handy feature.</p>
<p>This also causes the tool to exit with a status code of 1:</p>
<pre><code>% blacken-docs -l 60 docs/*.rst                        
docs/authentication.rst: Rewriting...
docs/internals.rst:196: code block parse error Cannot parse: 14:0: &lt;line number missing in source&gt;
docs/json_api.rst:449: code block parse error Cannot parse: 1:0: &lt;link rel="alternate"
docs/plugin_hooks.rst:250: code block parse error Cannot parse: 6:4:     ]
docs/plugin_hooks.rst:311: code block parse error Cannot parse: 38:0: &lt;line number missing in source&gt;
% echo $?
1
</code></pre>
<p>I also wanted my CI to fail if the author had forgotten to run <code>blacken-docs</code> against the repository before pushing the commit.</p>
<p>I filed <a href="https://github.com/asottile/blacken-docs/issues/161">a feature request</a> asking for an equivalent of the <code>black . --check</code> option, but it turns out that feature isn't necessary - <code>blacken-docs</code> returns a non-zero exit code if it makes any changes. So just running the following in CI works for checking if it should have been applied:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Check if blacken-docs needs to be run</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        blacken-docs -l 60 docs/*.rst</span></pre></div>


path: macos_open-files-with-opensnoop.md
topic: macos
title: Seeing files opened by a process using opensnoop
url: https://github.com/simonw/til/blob/main/macos/open-files-with-opensnoop.md
created: 2022-04-26T10:17:50-07:00
created_utc: 2022-04-26T17:17:50+00:00
updated: 2022-04-26T11:57:30-07:00
updated_utc: 2022-04-26T18:57:30+00:00
shot_hash: e37f443bd24ebae647055a7822ac2617
slug: open-files-with-opensnoop
body:
I decided to try out [atuin](https://github.com/ellie/atuin?utm_source=tldrnewsletter), a shell extension that writes your history to a SQLite database.

It's really neat. I wanted to see where the SQLite database lived on disk so I could poke around inside it with [Datasette](https://datasette.io/).

The documentation didn't mention the location of the database file, so I decided to figure that out myself.

I worked out a recipe using `opensnoop`, which comes pre-installed on macOS.

In one terminal window, run this:

    sudo opensnoop 2>/dev/null | grep atuin

Then run the `atuin history` command in another terminal - and the files that it accesses will be dumped out by `opensnoop`:

```
  501  51725 atuin          4 /dev/dtracehelper    
  501  51725 atuin         -1 /etc/.mdns_debug     
  501  51725 atuin          4 /usr/local/Cellar/atuin/0.9.1/bin 
  501  51725 atuin         -1 /usr/local/Cellar/atuin/0.9.1/bin/Info.plist 
  501  51725 atuin          4 /dev/autofs_nowait   
  501  51725 atuin          5 /Users/simon/.CFUserTextEncoding 
  501  51725 atuin          4 /dev/autofs_nowait   
  501  51725 atuin          5 /Users/simon/.CFUserTextEncoding 
  501  51725 atuin         10 .                    
  501  51725 atuin         10 /Users/simon/.config/atuin/config.toml 
  501  51725 atuin         10 /Users/simon/.local/share/atuin/history.db 
  501  51725 atuin         11 /Users/simon/.local/share/atuin/history.db-wal 
  501  51725 atuin         12 /Users/simon/.local/share/atuin/history.db-shm 
```
Then I ran `open /Users/simon/.local/share/atuin/history.db` (because I have [Datasette Desktop](https://datasette.io/desktop) installed) and could start exploring that database:

![Screenshot of Datasette showing the history table from the atuin database](https://user-images.githubusercontent.com/9599/165356208-4546e23a-47e6-47f1-a759-f1d849131aa0.png)

The `2>/dev/null` bit redirects standard error for `opensnoop` to `/dev/null` - without this it spews out a noisy volume of `dtrace: error ...` warnings.

## Alternative solutions

My [Twitter thread](https://twitter.com/simonw/status/1518999393415811073) asking about this resulted in a bunch of leads that I've not fully investigated yet, including:

- [FileMonitor](https://objective-see.com/products/utilities.html#FileMonitor)
- [FSMonitor](https://fsmonitor.com/)
- `iosnoop`
- `fs_usage`
- `sudo dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'`
- `strace` (Linux, not macOS)

html:
<p>I decided to try out <a href="https://github.com/ellie/atuin?utm_source=tldrnewsletter">atuin</a>, a shell extension that writes your history to a SQLite database.</p>
<p>It's really neat. I wanted to see where the SQLite database lived on disk so I could poke around inside it with <a href="https://datasette.io/" rel="nofollow">Datasette</a>.</p>
<p>The documentation didn't mention the location of the database file, so I decided to figure that out myself.</p>
<p>I worked out a recipe using <code>opensnoop</code>, which comes pre-installed on macOS.</p>
<p>In one terminal window, run this:</p>
<pre><code>sudo opensnoop 2&gt;/dev/null | grep atuin
</code></pre>
<p>Then run the <code>atuin history</code> command in another terminal - and the files that it accesses will be dumped out by <code>opensnoop</code>:</p>
<pre><code>  501  51725 atuin          4 /dev/dtracehelper    
  501  51725 atuin         -1 /etc/.mdns_debug     
  501  51725 atuin          4 /usr/local/Cellar/atuin/0.9.1/bin 
  501  51725 atuin         -1 /usr/local/Cellar/atuin/0.9.1/bin/Info.plist 
  501  51725 atuin          4 /dev/autofs_nowait   
  501  51725 atuin          5 /Users/simon/.CFUserTextEncoding 
  501  51725 atuin          4 /dev/autofs_nowait   
  501  51725 atuin          5 /Users/simon/.CFUserTextEncoding 
  501  51725 atuin         10 .                    
  501  51725 atuin         10 /Users/simon/.config/atuin/config.toml 
  501  51725 atuin         10 /Users/simon/.local/share/atuin/history.db 
  501  51725 atuin         11 /Users/simon/.local/share/atuin/history.db-wal 
  501  51725 atuin         12 /Users/simon/.local/share/atuin/history.db-shm 
</code></pre>
<p>Then I ran <code>open /Users/simon/.local/share/atuin/history.db</code> (because I have <a href="https://datasette.io/desktop" rel="nofollow">Datasette Desktop</a> installed) and could start exploring that database:</p>
<p><a href="https://user-images.githubusercontent.com/9599/165356208-4546e23a-47e6-47f1-a759-f1d849131aa0.png" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/165356208-4546e23a-47e6-47f1-a759-f1d849131aa0.png" alt="Screenshot of Datasette showing the history table from the atuin database" style="max-width:100%;"></a></p>
<p>The <code>2&gt;/dev/null</code> bit redirects standard error for <code>opensnoop</code> to <code>/dev/null</code> - without this it spews out a noisy volume of <code>dtrace: error ...</code> warnings.</p>
<h2>
<a id="user-content-alternative-solutions" class="anchor" href="#alternative-solutions" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Alternative solutions</h2>
<p>My <a href="https://twitter.com/simonw/status/1518999393415811073" rel="nofollow">Twitter thread</a> asking about this resulted in a bunch of leads that I've not fully investigated yet, including:</p>
<ul>
<li><a href="https://objective-see.com/products/utilities.html#FileMonitor" rel="nofollow">FileMonitor</a></li>
<li><a href="https://fsmonitor.com/" rel="nofollow">FSMonitor</a></li>
<li><code>iosnoop</code></li>
<li><code>fs_usage</code></li>
<li><code>sudo dtrace -n 'syscall::open*:entry { printf("%s %s",execname,copyinstr(arg0)); }'</code></li>
<li>
<code>strace</code> (Linux, not macOS)</li>
</ul>


path: macos_atuin.md
topic: macos
title: Atuin for zsh shell history in SQLite
url: https://github.com/simonw/til/blob/main/macos/atuin.md
created: 2022-04-26T11:53:21-07:00
created_utc: 2022-04-26T18:53:21+00:00
updated: 2022-04-26T12:26:03-07:00
updated_utc: 2022-04-26T19:26:03+00:00
shot_hash: d8c23db264659ca0e2cf3d383d3a75fd
slug: atuin
body:
[Atuin](https://github.com/ellie/atuin) (via [Rhet Turnbull](https://twitter.com/RhetTurnbull/status/1518942324004319232)) "replaces your existing shell history with a SQLite database". Obviously I had to try this out!

I installed it with Homebrew:

    brew install atuin

Then ran this to add it to my `.zshrc` config:

    echo 'eval "$(atuin init zsh)"' >> ~/.zshrc

I restarted my terminal... and now I have a SQLite database full of commands I have run!

The database lives at `~/.local/share/atuin/history.db` (I figured that out with [this TIL](https://til.simonwillison.net/macos/open-files-with-opensnoop)), and can be opened in Datasette.

![Screenshot of Datasette showing the history table from the atuin database](https://user-images.githubusercontent.com/9599/165356208-4546e23a-47e6-47f1-a759-f1d849131aa0.png)

## Removing the "up" key binding

Atuin hijacks the `Ctrl+R` shortcut to provide an interactive search window, which I really like:

![Screenshot of my terminal history in Atuin](https://user-images.githubusercontent.com/9599/165371428-69f60fa5-0ad6-4d7a-a476-79d88c8b6959.png)

But it also hijacks the Up arrow key, which I didn't like - it interfered with how I usually use that (quickly getting back my previous command and editing it).

[This thread](https://github.com/ellie/atuin/issues/51) in the Atuin issues lead me to a solution - though the fix there put my cursor at the start of the line when I wanted it at the end of the line instead. [This StackExchange](https://unix.stackexchange.com/a/97844) showed me the fix for that.

I edited my `~/.zshrc` file and replaced the `atuin` bit at the bottom with this:

```zsh
eval "$(atuin init zsh)"
# Restore original up binding: https://til.simonwillison.net/macos/atuin
autoload -U history-search-end
zle -N history-beginning-search-backward-end history-search-end
bindkey "^[[A" history-beginning-search-backward-end
```

html:
<p><a href="https://github.com/ellie/atuin">Atuin</a> (via <a href="https://twitter.com/RhetTurnbull/status/1518942324004319232" rel="nofollow">Rhet Turnbull</a>) "replaces your existing shell history with a SQLite database". Obviously I had to try this out!</p>
<p>I installed it with Homebrew:</p>
<pre><code>brew install atuin
</code></pre>
<p>Then ran this to add it to my <code>.zshrc</code> config:</p>
<pre><code>echo 'eval "$(atuin init zsh)"' &gt;&gt; ~/.zshrc
</code></pre>
<p>I restarted my terminal... and now I have a SQLite database full of commands I have run!</p>
<p>The database lives at <code>~/.local/share/atuin/history.db</code> (I figured that out with <a href="https://til.simonwillison.net/macos/open-files-with-opensnoop" rel="nofollow">this TIL</a>), and can be opened in Datasette.</p>
<p><a href="https://user-images.githubusercontent.com/9599/165356208-4546e23a-47e6-47f1-a759-f1d849131aa0.png" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/165356208-4546e23a-47e6-47f1-a759-f1d849131aa0.png" alt="Screenshot of Datasette showing the history table from the atuin database" style="max-width:100%;"></a></p>
<h2>
<a id="user-content-removing-the-up-key-binding" class="anchor" href="#removing-the-up-key-binding" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Removing the "up" key binding</h2>
<p>Atuin hijacks the <code>Ctrl+R</code> shortcut to provide an interactive search window, which I really like:</p>
<p><a href="https://user-images.githubusercontent.com/9599/165371428-69f60fa5-0ad6-4d7a-a476-79d88c8b6959.png" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/165371428-69f60fa5-0ad6-4d7a-a476-79d88c8b6959.png" alt="Screenshot of my terminal history in Atuin" style="max-width:100%;"></a></p>
<p>But it also hijacks the Up arrow key, which I didn't like - it interfered with how I usually use that (quickly getting back my previous command and editing it).</p>
<p><a href="https://github.com/ellie/atuin/issues/51">This thread</a> in the Atuin issues lead me to a solution - though the fix there put my cursor at the start of the line when I wanted it at the end of the line instead. <a href="https://unix.stackexchange.com/a/97844" rel="nofollow">This StackExchange</a> showed me the fix for that.</p>
<p>I edited my <code>~/.zshrc</code> file and replaced the <code>atuin</code> bit at the bottom with this:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c1">eval</span> <span class="pl-s"><span class="pl-pds">"</span><span class="pl-s"><span class="pl-pds">$(</span>atuin init zsh<span class="pl-pds">)</span></span><span class="pl-pds">"</span></span>
<span class="pl-c"><span class="pl-c">#</span> Restore original up binding: https://til.simonwillison.net/macos/atuin</span>
autoload -U history-search-end
zle -N history-beginning-search-backward-end history-search-end
bindkey <span class="pl-s"><span class="pl-pds">"</span>^[[A<span class="pl-pds">"</span></span> history-beginning-search-backward-end</pre></div>


path: service-workers_intercept-fetch.md
topic: service-workers
title: Intercepting fetch in a service worker
url: https://github.com/simonw/til/blob/main/service-workers/intercept-fetch.md
created: 2022-04-30T15:56:28-07:00
created_utc: 2022-04-30T22:56:28+00:00
updated: 2022-04-30T15:56:28-07:00
updated_utc: 2022-04-30T22:56:28+00:00
shot_hash: fc9158f6ee3decc08a8cecb519d6f477
slug: intercept-fetch
body:
I'm learning service workers. I wanted to start with one that intercepts calls to a `/path` and returns "Hello World".

Here's the initial recipe I came up with.

`index.html` contained this:

```html
<h1>Service worker demo</h1>

<script>
const registerServiceWorker = async () => {
  if ('serviceWorker' in navigator) {
    try {
      const registration = await navigator.serviceWorker.register(
        '/sw.js',
        {
          scope: '/',
        }
      );
      if (registration.installing) {
        console.log('Service worker installing');
      } else if (registration.waiting) {
        console.log('Service worker installed');
      } else if (registration.active) {
        console.log('Service worker active');
      }
    } catch (error) {
      console.error(`Registration failed with ${error}`);
    }
  }
};

registerServiceWorker();
</script>
```
This is using the service worker registration boilerplate [from MDN](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API/Using_Service_Workers#registering_your_worker).

Then my service worker script itself in `sw.js` just does this:

```javascript
self.addEventListener('fetch', (event) => {
  const request = event.request;
  const url = (new URL(request.url));
  if (url.pathname == "/") {
    // Don't intercept hits to the homepage
    return;
  }
  const params = new URLSearchParams(url.search);
  const info = {
    url: request.url,
    method: request.method,
    path: url.pathname,
    params: Array.from(params.entries())
  };
  event.respondWith(new Response(
    `<p>Hello world! Request was: <pre>${JSON.stringify(info, null, 4)}</p>`, {
    headers: { 'Content-Type': 'text/html' }
  }));
});
```
You have to run service workers with a real web server - you can't serve them directly from disk.

I used the Python 3 one-liner recipe for that:

    python3 -m http.server 8009

Then I visited `http://localhost:8009/` to load the service worker.

Then I visited `http://localhost:8009/foo/bar?a=1&b=2` and got this:

<img alt="Hello world! Request was: {
    &quot;url&quot;: &quot;http://localhost:8009/foo/bar?a=1&b=2&quot;,
    &quot;method&quot;: &quot;GET&quot;,
    &quot;path&quot;: &quot;/foo/bar&quot;,
    &quot;params&quot;: [
        [
            &quot;a&quot;,
            &quot;1&quot;
        ],
        [
            &quot;b&quot;,
            &quot;2&quot;
        ]
    ]
}" src="https://user-images.githubusercontent.com/9599/166125219-7820133e-4c9f-4ea2-898b-87b126f07115.png">

html:
<p>I'm learning service workers. I wanted to start with one that intercepts calls to a <code>/path</code> and returns "Hello World".</p>
<p>Here's the initial recipe I came up with.</p>
<p><code>index.html</code> contained this:</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">h1</span><span class="pl-kos">&gt;</span>Service worker demo<span class="pl-kos">&lt;/</span><span class="pl-ent">h1</span><span class="pl-kos">&gt;</span>

<span class="pl-kos">&lt;</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span>
<span class="pl-k">const</span> <span class="pl-en">registerServiceWorker</span> <span class="pl-c1">=</span> <span class="pl-k">async</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
  <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s">'serviceWorker'</span> <span class="pl-k">in</span> <span class="pl-s1">navigator</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-k">try</span> <span class="pl-kos">{</span>
      <span class="pl-k">const</span> <span class="pl-s1">registration</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">navigator</span><span class="pl-kos">.</span><span class="pl-c1">serviceWorker</span><span class="pl-kos">.</span><span class="pl-en">register</span><span class="pl-kos">(</span>
        <span class="pl-s">'/sw.js'</span><span class="pl-kos">,</span>
        <span class="pl-kos">{</span>
          <span class="pl-c1">scope</span>: <span class="pl-s">'/'</span><span class="pl-kos">,</span>
        <span class="pl-kos">}</span>
      <span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">registration</span><span class="pl-kos">.</span><span class="pl-c1">installing</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s">'Service worker installing'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-kos">}</span> <span class="pl-k">else</span> <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">registration</span><span class="pl-kos">.</span><span class="pl-c1">waiting</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s">'Service worker installed'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-kos">}</span> <span class="pl-k">else</span> <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">registration</span><span class="pl-kos">.</span><span class="pl-c1">active</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s">'Service worker active'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-kos">}</span>
    <span class="pl-kos">}</span> <span class="pl-k">catch</span> <span class="pl-kos">(</span><span class="pl-s1">error</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
      <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">error</span><span class="pl-kos">(</span><span class="pl-s">`Registration failed with <span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">error</span><span class="pl-kos">}</span></span>`</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span>
  <span class="pl-kos">}</span>
<span class="pl-kos">}</span><span class="pl-kos">;</span>

<span class="pl-en">registerServiceWorker</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">&lt;/</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span></pre></div>
<p>This is using the service worker registration boilerplate <a href="https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API/Using_Service_Workers#registering_your_worker" rel="nofollow">from MDN</a>.</p>
<p>Then my service worker script itself in <code>sw.js</code> just does this:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-s1">self</span><span class="pl-kos">.</span><span class="pl-en">addEventListener</span><span class="pl-kos">(</span><span class="pl-s">'fetch'</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-s1">event</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
  <span class="pl-k">const</span> <span class="pl-s1">request</span> <span class="pl-c1">=</span> <span class="pl-s1">event</span><span class="pl-kos">.</span><span class="pl-c1">request</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">url</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-k">new</span> <span class="pl-c1">URL</span><span class="pl-kos">(</span><span class="pl-s1">request</span><span class="pl-kos">.</span><span class="pl-c1">url</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">url</span><span class="pl-kos">.</span><span class="pl-c1">pathname</span> <span class="pl-c1">==</span> <span class="pl-s">"/"</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-c">// Don't intercept hits to the homepage</span>
    <span class="pl-k">return</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span>
  <span class="pl-k">const</span> <span class="pl-s1">params</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">URLSearchParams</span><span class="pl-kos">(</span><span class="pl-s1">url</span><span class="pl-kos">.</span><span class="pl-c1">search</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">info</span> <span class="pl-c1">=</span> <span class="pl-kos">{</span>
    <span class="pl-c1">url</span>: <span class="pl-s1">request</span><span class="pl-kos">.</span><span class="pl-c1">url</span><span class="pl-kos">,</span>
    <span class="pl-c1">method</span>: <span class="pl-s1">request</span><span class="pl-kos">.</span><span class="pl-c1">method</span><span class="pl-kos">,</span>
    <span class="pl-c1">path</span>: <span class="pl-s1">url</span><span class="pl-kos">.</span><span class="pl-c1">pathname</span><span class="pl-kos">,</span>
    <span class="pl-c1">params</span>: <span class="pl-v">Array</span><span class="pl-kos">.</span><span class="pl-en">from</span><span class="pl-kos">(</span><span class="pl-s1">params</span><span class="pl-kos">.</span><span class="pl-en">entries</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">)</span>
  <span class="pl-kos">}</span><span class="pl-kos">;</span>
  <span class="pl-s1">event</span><span class="pl-kos">.</span><span class="pl-en">respondWith</span><span class="pl-kos">(</span><span class="pl-k">new</span> <span class="pl-v">Response</span><span class="pl-kos">(</span>
    <span class="pl-s">`&lt;p&gt;Hello world! Request was: &lt;pre&gt;<span class="pl-s1"><span class="pl-kos">${</span><span class="pl-c1">JSON</span><span class="pl-kos">.</span><span class="pl-en">stringify</span><span class="pl-kos">(</span><span class="pl-s1">info</span><span class="pl-kos">,</span> <span class="pl-c1">null</span><span class="pl-kos">,</span> <span class="pl-c1">4</span><span class="pl-kos">)</span><span class="pl-kos">}</span></span>&lt;/p&gt;`</span><span class="pl-kos">,</span> <span class="pl-kos">{</span>
    <span class="pl-c1">headers</span>: <span class="pl-kos">{</span> <span class="pl-s">'Content-Type'</span>: <span class="pl-s">'text/html'</span> <span class="pl-kos">}</span>
  <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<p>You have to run service workers with a real web server - you can't serve them directly from disk.</p>
<p>I used the Python 3 one-liner recipe for that:</p>
<pre><code>python3 -m http.server 8009
</code></pre>
<p>Then I visited <code>http://localhost:8009/</code> to load the service worker.</p>
<p>Then I visited <code>http://localhost:8009/foo/bar?a=1&amp;b=2</code> and got this:</p>
<p><a href="https://user-images.githubusercontent.com/9599/166125219-7820133e-4c9f-4ea2-898b-87b126f07115.png" target="_blank" rel="nofollow"><img alt='Hello world! Request was: {
"url": "http://localhost:8009/foo/bar?a=1&amp;b=2",
"method": "GET",
"path": "/foo/bar",
"params": [
[
"a",
"1"
],
[
"b",
"2"
]
]
}' src="https://user-images.githubusercontent.com/9599/166125219-7820133e-4c9f-4ea2-898b-87b126f07115.png" style="max-width:100%;"></a></p>


path: github_custom-subdomain-github-pages.md
topic: github
title: Setting up a custom subdomain for a GitHub Pages site
url: https://github.com/simonw/til/blob/main/github/custom-subdomain-github-pages.md
created: 2022-05-04T15:25:38-07:00
created_utc: 2022-05-04T22:25:38+00:00
updated: 2022-08-16T22:11:30-07:00
updated_utc: 2022-08-17T05:11:30+00:00
shot_hash: c46722ed339b73d0daf30462338cbcd7
slug: custom-subdomain-github-pages
body:
This is so much easier than I was expecting it to be.

I launched my new [Datasette Lite](https://github.com/simonw/datasette-lite) project ([background](https://simonwillison.net/2022/May/4/datasette-lite/)) using GitHub Pages - originally hosted here:

https://simonw.github.io/datasette-lite/

Shortly after launching I realized that the project deserved its own vanity subdomain. I wanted to use `lite.datasette.io` for this.

GitHub Pages makes this really easy.

I started by pointing the `CNAME` for `lite.datasette.io` at `simonw.github.io` - I use Vercel for the DNS for that domain, so I set this up in their interface.

Then I used the Pages tab in the "settings" interface for the GitHub repository and told it about my new domain.

GitHub wrote a file called [CNAME](https://github.com/simonw/datasette-lite/blob/main/CNAME) directly into the root of my repository for me containing just `lite.datasette.io`

This started working over HTTP straight away - it took a few more minutes for the Lets Encrypt certificate to be issued for it, at which point https://lite.datasette.io/ started working too.

I then used the "Enforce HTTPS" option to ensure any hits to `http://...` were redirected to `https://`

<img width="781" alt="Screenshot of the GitHub interface for setting the domain, showing the Enforce HTTPS checkbox" src="https://user-images.githubusercontent.com/9599/166835190-3ca925a8-3fbf-4656-aea5-1b67bb984a24.png">

## Redirects

GitHub Pages set up redirects from the old https://simonw.github.io/datasette-lite/ site to the new one without me even asking them to.

The redirects preserve both query string parameters AND fragment hashes, which means that this old URL continues to work after the move:

https://simonw.github.io/datasette-lite/#/fixtures?sql=select+sqlite_version%28%29

It redirects to:

https://lite.datasette.io/#/fixtures?sql=select+sqlite_version%28%29

html:
<p>This is so much easier than I was expecting it to be.</p>
<p>I launched my new <a href="https://github.com/simonw/datasette-lite">Datasette Lite</a> project (<a href="https://simonwillison.net/2022/May/4/datasette-lite/" rel="nofollow">background</a>) using GitHub Pages - originally hosted here:</p>
<p><a href="https://simonw.github.io/datasette-lite/" rel="nofollow">https://simonw.github.io/datasette-lite/</a></p>
<p>Shortly after launching I realized that the project deserved its own vanity subdomain. I wanted to use <code>lite.datasette.io</code> for this.</p>
<p>GitHub Pages makes this really easy.</p>
<p>I started by pointing the <code>CNAME</code> for <code>lite.datasette.io</code> at <code>simonw.github.io</code> - I use Vercel for the DNS for that domain, so I set this up in their interface.</p>
<p>Then I used the Pages tab in the "settings" interface for the GitHub repository and told it about my new domain.</p>
<p>GitHub wrote a file called <a href="https://github.com/simonw/datasette-lite/blob/main/CNAME">CNAME</a> directly into the root of my repository for me containing just <code>lite.datasette.io</code></p>
<p>This started working over HTTP straight away - it took a few more minutes for the Lets Encrypt certificate to be issued for it, at which point <a href="https://lite.datasette.io/" rel="nofollow">https://lite.datasette.io/</a> started working too.</p>
<p>I then used the "Enforce HTTPS" option to ensure any hits to <code>http://...</code> were redirected to <code>https://</code></p>
<p><a target="_blank" rel="noopener noreferrer" href="https://user-images.githubusercontent.com/9599/166835190-3ca925a8-3fbf-4656-aea5-1b67bb984a24.png"><img width="781" alt="Screenshot of the GitHub interface for setting the domain, showing the Enforce HTTPS checkbox" src="https://user-images.githubusercontent.com/9599/166835190-3ca925a8-3fbf-4656-aea5-1b67bb984a24.png" style="max-width: 100%;"></a></p>
<h2><a id="user-content-redirects" class="anchor" aria-hidden="true" href="#redirects"><span aria-hidden="true" class="octicon octicon-link"></span></a>Redirects</h2>
<p>GitHub Pages set up redirects from the old <a href="https://simonw.github.io/datasette-lite/" rel="nofollow">https://simonw.github.io/datasette-lite/</a> site to the new one without me even asking them to.</p>
<p>The redirects preserve both query string parameters AND fragment hashes, which means that this old URL continues to work after the move:</p>
<p><a href="https://simonw.github.io/datasette-lite/#/fixtures?sql=select+sqlite_version%28%29" rel="nofollow">https://simonw.github.io/datasette-lite/#/fixtures?sql=select+sqlite_version%28%29</a></p>
<p>It redirects to:</p>
<p><a href="https://lite.datasette.io/#/fixtures?sql=select+sqlite_version%28%29" rel="nofollow">https://lite.datasette.io/#/fixtures?sql=select+sqlite_version%28%29</a></p>


path: python_copy-file.md
topic: python
title: Efficiently copying a file
url: https://github.com/simonw/til/blob/main/python/copy-file.md
created: 2022-05-13T17:22:54-07:00
created_utc: 2022-05-14T00:22:54+00:00
updated: 2022-05-13T17:22:54-07:00
updated_utc: 2022-05-14T00:22:54+00:00
shot_hash: 4cdf7582c54a98b53272c1c062b63b8f
slug: copy-file
body:
**TLDR:** Use `shutil.copyfileobj(fsrc, fdst)`

I'm writing a Datasette plugin that handles an uploaded file, borrowing the Starlette mechanism for handling file uploads, [documented here](https://www.starlette.io/requests/#request-files).

Starlette uploads result in a [SpooledTemporaryFile](https://docs.python.org/3/library/tempfile.html#tempfile.SpooledTemporaryFile) file-like object. These look very much like a file, with one frustrating limitation: they don't have a defined, stable path on disk.

I thought that this meant that you couldn't easily copy it, and ended up coming up with this recipe based on [this code](https://github.com/abersheeran/baize/blob/23791841f30ca92775e50a544a8606d1d4deac93/baize/datastructures.py#L543-L561) I spotted in the [BáiZé](https://github.com/abersheeran/baize) framework:

```python
from shutil import COPY_BUFSIZE

with open(new_filepath, "wb+") as target_file:
    source_file.seek(0)
    source_read = source_file.read
    target_write = target_file.write
    while True:
        buf = source_read(COPY_BUFSIZE)
        if not buf:
            break
        target_write(buf)
```
`COPY_BUFSIZE` defined by Python [here](https://github.com/python/cpython/blob/v3.10.4/Lib/shutil.py#L42) - it handles the difference in ideal buffer size between Windows and other operating systems:
```python
COPY_BUFSIZE = 1024 * 1024 if _WINDOWS else 64 * 1024
```
But then I sat down to write this TIL, and stumbled across [shutil.copyfileobj(fsrc, fdst)](https://docs.python.org/3/library/shutil.html#shutil.copyfileobj) in the standard library which implements the [exact same pattern](https://github.com/python/cpython/blob/v3.10.4/Lib/shutil.py#L187-L198)!

```python
def copyfileobj(fsrc, fdst, length=0):
    """copy data from file-like object fsrc to file-like object fdst"""
    # Localize variable access to minimize overhead.
    if not length:
        length = COPY_BUFSIZE
    fsrc_read = fsrc.read
    fdst_write = fdst.write
    while True:
        buf = fsrc_read(length)
        if not buf:
            break
        fdst_write(buf)
```
So you should use that instead.

html:
<p><strong>TLDR:</strong> Use <code>shutil.copyfileobj(fsrc, fdst)</code></p>
<p>I'm writing a Datasette plugin that handles an uploaded file, borrowing the Starlette mechanism for handling file uploads, <a href="https://www.starlette.io/requests/#request-files" rel="nofollow">documented here</a>.</p>
<p>Starlette uploads result in a <a href="https://docs.python.org/3/library/tempfile.html#tempfile.SpooledTemporaryFile" rel="nofollow">SpooledTemporaryFile</a> file-like object. These look very much like a file, with one frustrating limitation: they don't have a defined, stable path on disk.</p>
<p>I thought that this meant that you couldn't easily copy it, and ended up coming up with this recipe based on <a href="https://github.com/abersheeran/baize/blob/23791841f30ca92775e50a544a8606d1d4deac93/baize/datastructures.py#L543-L561">this code</a> I spotted in the <a href="https://github.com/abersheeran/baize">BáiZé</a> framework:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">shutil</span> <span class="pl-k">import</span> <span class="pl-v">COPY_BUFSIZE</span>

<span class="pl-k">with</span> <span class="pl-en">open</span>(<span class="pl-s1">new_filepath</span>, <span class="pl-s">"wb+"</span>) <span class="pl-k">as</span> <span class="pl-s1">target_file</span>:
    <span class="pl-s1">source_file</span>.<span class="pl-en">seek</span>(<span class="pl-c1">0</span>)
    <span class="pl-s1">source_read</span> <span class="pl-c1">=</span> <span class="pl-s1">source_file</span>.<span class="pl-s1">read</span>
    <span class="pl-s1">target_write</span> <span class="pl-c1">=</span> <span class="pl-s1">target_file</span>.<span class="pl-s1">write</span>
    <span class="pl-k">while</span> <span class="pl-c1">True</span>:
        <span class="pl-s1">buf</span> <span class="pl-c1">=</span> <span class="pl-en">source_read</span>(<span class="pl-v">COPY_BUFSIZE</span>)
        <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">buf</span>:
            <span class="pl-k">break</span>
        <span class="pl-en">target_write</span>(<span class="pl-s1">buf</span>)</pre></div>
<p><code>COPY_BUFSIZE</code> defined by Python <a href="https://github.com/python/cpython/blob/v3.10.4/Lib/shutil.py#L42">here</a> - it handles the difference in ideal buffer size between Windows and other operating systems:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-v">COPY_BUFSIZE</span> <span class="pl-c1">=</span> <span class="pl-c1">1024</span> <span class="pl-c1">*</span> <span class="pl-c1">1024</span> <span class="pl-k">if</span> <span class="pl-s1">_WINDOWS</span> <span class="pl-k">else</span> <span class="pl-c1">64</span> <span class="pl-c1">*</span> <span class="pl-c1">1024</span></pre></div>
<p>But then I sat down to write this TIL, and stumbled across <a href="https://docs.python.org/3/library/shutil.html#shutil.copyfileobj" rel="nofollow">shutil.copyfileobj(fsrc, fdst)</a> in the standard library which implements the <a href="https://github.com/python/cpython/blob/v3.10.4/Lib/shutil.py#L187-L198">exact same pattern</a>!</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">copyfileobj</span>(<span class="pl-s1">fsrc</span>, <span class="pl-s1">fdst</span>, <span class="pl-s1">length</span><span class="pl-c1">=</span><span class="pl-c1">0</span>):
    <span class="pl-s">"""copy data from file-like object fsrc to file-like object fdst"""</span>
    <span class="pl-c"># Localize variable access to minimize overhead.</span>
    <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">length</span>:
        <span class="pl-s1">length</span> <span class="pl-c1">=</span> <span class="pl-v">COPY_BUFSIZE</span>
    <span class="pl-s1">fsrc_read</span> <span class="pl-c1">=</span> <span class="pl-s1">fsrc</span>.<span class="pl-s1">read</span>
    <span class="pl-s1">fdst_write</span> <span class="pl-c1">=</span> <span class="pl-s1">fdst</span>.<span class="pl-s1">write</span>
    <span class="pl-k">while</span> <span class="pl-c1">True</span>:
        <span class="pl-s1">buf</span> <span class="pl-c1">=</span> <span class="pl-en">fsrc_read</span>(<span class="pl-s1">length</span>)
        <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">buf</span>:
            <span class="pl-k">break</span>
        <span class="pl-en">fdst_write</span>(<span class="pl-s1">buf</span>)</pre></div>
<p>So you should use that instead.</p>


path: github-actions_job-summaries.md
topic: github-actions
title: GitHub Actions job summaries
url: https://github.com/simonw/til/blob/main/github-actions/job-summaries.md
created: 2022-05-17T10:28:21-07:00
created_utc: 2022-05-17T17:28:21+00:00
updated: 2022-05-17T10:49:39-07:00
updated_utc: 2022-05-17T17:49:39+00:00
shot_hash: 4626096cbdbf784228ec31127d5ac199
slug: job-summaries
body:
New feature [announced here](https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/). Here's the [full documentation](https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#adding-a-job-summary).

These are incredibly easy to use. GitHub creates a file in your workspace and puts the filename in `$GITHUB_STEP_SUMMARY`, so you can build the summary markdown over multiple steps like this:

```bash
echo "{markdown content}" >> $GITHUB_STEP_SUMMARY
```
I decided to try this out in my [simonw/pypi-datasette-packages](https://github.com/simonw/pypi-datasette-packages/) repo, which runs a daily Git scraper that records a copy of the PyPI JSON for packages within the Datasette ecosystem.

I ended up mixing it with the Git commit code, so the step [now looks like this](https://github.com/simonw/pypi-datasette-packages/blob/54d43180a97d30011149d1e7ae3aaafed2ad7818/.github/workflows/fetch.yml#L20-L32):

```yaml
    - name: Commit and push
      run: |-
        git config user.name "Automated"
        git config user.email "actions@users.noreply.github.com"
        git add -A
        timestamp=$(date -u)
        git commit -m "${timestamp}" || exit 0
        echo '### Changed files' >> $GITHUB_STEP_SUMMARY
        echo '```' >> $GITHUB_STEP_SUMMARY
        git show --name-only --format=tformat: >> $GITHUB_STEP_SUMMARY
        echo '```' >> $GITHUB_STEP_SUMMARY
        git pull --rebase
        git push
```
This produces [a summary](https://github.com/simonw/pypi-datasette-packages/actions/runs/2336190331) that looks like this:

<img width="657" alt="Screenshot of the summary" src="https://user-images.githubusercontent.com/9599/168874059-b08afb20-c9f3-4c6d-9224-311f21696bfd.png">

Two things I had to figure out here. First, the backtick needs escaping if used in double quotes but does not in single quotes:

```bash
echo '```' >> $GITHUB_STEP_SUMMARY
```
I wanted to show just the list of affected filenames from the most recent Git commit. That's what this does:

    git show --name-only --format=tformat:

Without the `--format=tformat` bit this shows the full commit message and header in addition to the list of files.

I'm running this in the same block as the other `git` commands so that this line will terminate the step early without writing to the summary file if there are no changes to be committed:

```bash
git commit -m "${timestamp}" || exit 0
```

html:
<p>New feature <a href="https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/" rel="nofollow">announced here</a>. Here's the <a href="https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#adding-a-job-summary">full documentation</a>.</p>
<p>These are incredibly easy to use. GitHub creates a file in your workspace and puts the filename in <code>$GITHUB_STEP_SUMMARY</code>, so you can build the summary markdown over multiple steps like this:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>{markdown content}<span class="pl-pds">"</span></span> <span class="pl-k">&gt;&gt;</span> <span class="pl-smi">$GITHUB_STEP_SUMMARY</span></pre></div>
<p>I decided to try this out in my <a href="https://github.com/simonw/pypi-datasette-packages/">simonw/pypi-datasette-packages</a> repo, which runs a daily Git scraper that records a copy of the PyPI JSON for packages within the Datasette ecosystem.</p>
<p>I ended up mixing it with the Git commit code, so the step <a href="https://github.com/simonw/pypi-datasette-packages/blob/54d43180a97d30011149d1e7ae3aaafed2ad7818/.github/workflows/fetch.yml#L20-L32">now looks like this</a>:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Commit and push</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        git config user.name "Automated"</span>
<span class="pl-s">        git config user.email "actions@users.noreply.github.com"</span>
<span class="pl-s">        git add -A</span>
<span class="pl-s">        timestamp=$(date -u)</span>
<span class="pl-s">        git commit -m "${timestamp}" || exit 0</span>
<span class="pl-s">        echo '### Changed files' &gt;&gt; $GITHUB_STEP_SUMMARY</span>
<span class="pl-s">        echo '```' &gt;&gt; $GITHUB_STEP_SUMMARY</span>
<span class="pl-s">        git show --name-only --format=tformat: &gt;&gt; $GITHUB_STEP_SUMMARY</span>
<span class="pl-s">        echo '```' &gt;&gt; $GITHUB_STEP_SUMMARY</span>
<span class="pl-s">        git pull --rebase</span>
<span class="pl-s">        git push</span></pre></div>
<p>This produces <a href="https://github.com/simonw/pypi-datasette-packages/actions/runs/2336190331">a summary</a> that looks like this:</p>
<p><a href="https://user-images.githubusercontent.com/9599/168874059-b08afb20-c9f3-4c6d-9224-311f21696bfd.png" target="_blank" rel="nofollow"><img width="657" alt="Screenshot of the summary" src="https://user-images.githubusercontent.com/9599/168874059-b08afb20-c9f3-4c6d-9224-311f21696bfd.png" style="max-width:100%;"></a></p>
<p>Two things I had to figure out here. First, the backtick needs escaping if used in double quotes but does not in single quotes:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">'</span>```<span class="pl-pds">'</span></span> <span class="pl-k">&gt;&gt;</span> <span class="pl-smi">$GITHUB_STEP_SUMMARY</span></pre></div>
<p>I wanted to show just the list of affected filenames from the most recent Git commit. That's what this does:</p>
<pre><code>git show --name-only --format=tformat:
</code></pre>
<p>Without the <code>--format=tformat</code> bit this shows the full commit message and header in addition to the list of files.</p>
<p>I'm running this in the same block as the other <code>git</code> commands so that this line will terminate the step early without writing to the summary file if there are no changes to be committed:</p>
<div class="highlight highlight-source-shell"><pre>git commit -m <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">${timestamp}</span><span class="pl-pds">"</span></span> <span class="pl-k">||</span> <span class="pl-c1">exit</span> 0</pre></div>


path: github-actions_oxipng.md
topic: github-actions
title: Optimizing PNGs in GitHub Actions using Oxipng
url: https://github.com/simonw/til/blob/main/github-actions/oxipng.md
created: 2022-05-18T18:45:23-07:00
created_utc: 2022-05-19T01:45:23+00:00
updated: 2022-09-12T14:22:14-07:00
updated_utc: 2022-09-12T21:22:14+00:00
shot_hash: c158add7f728071f4c7f99b983716947
slug: oxipng
body:
My [datasette-screenshots](https://github.com/simonw/datasette-screenshots) repository generates screenshots of [Datasette](https://datasette.io/) using my [shot-scraper](https://github.com/simonw/shot-scraper) tool, for people who need them for articles or similar.

Jacob Weisz [suggested](https://github.com/simonw/datasette-screenshots/issues/1) optimizing these images as they were quite big. I want them to be as high quality as possible (I even take them using `--retina` mode), but that didn't mean I couldn't use lossless compression on them.

I often use [squoosh.app](https://squoosh.app/) to run a version of [Oxipng](https://github.com/shssoichiro/oxipng) compiled to WebAssembly in my browser. I decided to figure out how to run that same program in GitHub Actions.

## Installing Rust apps using Cargo

Surprisingly there [isn't yet a packaged version](https://github.com/shssoichiro/oxipng/issues/69) of Oxipng for Ubuntu - so I needed another way of installing it.

The [project README suggests](https://github.com/shssoichiro/oxipng/blob/master/README.md#installing) installing it using `cargo install oxipng`.

I used the [tmate trick](https://til.simonwillison.net/github-actions/debug-tmate) to try that out in a GitHub Actions worker - the `cargo` command is available by default but it took over a minute to fetch and compile all of the dependencies.

I didn't want to do this on every run, so I looked into ways to cache the built program. Thankfully the `actions/cache` action documents [how to use it with Rust](https://github.com/actions/cache/blob/main/examples.md#rust---cargo).

The full recipe for installing Oxipng in GitHub Actions looks like this:

```yaml
    - name: Cache Oxipng
      uses: actions/cache@v3
      with:
        path: ~/.cargo/
        key: ${{ runner.os }}-cargo
    - name: Install Oxipng
      run: |
        which oxipng || cargo install oxipng
```

The first time the action runs it does a full compile of Oxipng - but on subsequent runs the `which oxipng` command succeeds and skips the `cargo install` step entirely.

## Running Oxipng in an Action

All of the PNGs that I wanted to optimize were in the root of my checkout, so I added this step:

```yaml
    - name: Optimize PNGs
      run: |-
        oxipng -o 4 -i 0 --strip safe *.png
```
The `-o 4` is the highest recommended level of optimization.

`-i 0` causes it to remove interlacing - "Interlacing can add 25-50% to the size of an optimized image" according to the README.

`--strip safe` strips out any image metadata that is guaranteed not to affect how the image is rendered.

Oxipng updates the specified images in place, hence the `*.png` at the end.

## Testing this in a branch first

I tested this all in a branch first so that I could see if it was working correctly.

Since my workflow usually pushes any changed files back to the same GitHub repository, I added a check to that step which caused it to only run on pushes to the `main` branch:

```yaml
    - name: Commit and push
      if: github.ref == 'refs/heads/main'
      run: |-
        git config user.name "Automated"
        ...
```
But I wanted to preview the generated images - so I added this step in the branch to save them to an artifact zip file that I could then inspect:

```yaml
    - name: Artifacts
      uses: actions/upload-artifact@v3
      with:
        name: screenshots
        path: "*.png"
```

Once I got it working, I squash-merged [this pull request](https://github.com/simonw/datasette-screenshots/pull/2) back into `main`.

## The result

Oxipng worked really well!

It reduced the size of all three of my screenshots:

- [faceting.png](https://github.com/simonw/datasette-screenshots/blob/main/faceting.png) from 305KB to 211KB
- [global-power-plants.png](https://github.com/simonw/datasette-screenshots/blob/main/global-power-plants.png) from 1.14MB to 770KB
- [sql-query.png](https://github.com/simonw/datasette-screenshots/blob/main/sql-query.png) from 235KB to 169KB

[This commit](https://github.com/simonw/datasette-screenshots/commit/0b7ccd8fe1c0fdc3714d768659a88bae1fa69ca4) shows the difference and lets you compare both images.

html:
<p>My <a href="https://github.com/simonw/datasette-screenshots">datasette-screenshots</a> repository generates screenshots of <a href="https://datasette.io/" rel="nofollow">Datasette</a> using my <a href="https://github.com/simonw/shot-scraper">shot-scraper</a> tool, for people who need them for articles or similar.</p>
<p>Jacob Weisz <a href="https://github.com/simonw/datasette-screenshots/issues/1">suggested</a> optimizing these images as they were quite big. I want them to be as high quality as possible (I even take them using <code>--retina</code> mode), but that didn't mean I couldn't use lossless compression on them.</p>
<p>I often use <a href="https://squoosh.app/" rel="nofollow">squoosh.app</a> to run a version of <a href="https://github.com/shssoichiro/oxipng">Oxipng</a> compiled to WebAssembly in my browser. I decided to figure out how to run that same program in GitHub Actions.</p>
<h2><a id="user-content-installing-rust-apps-using-cargo" class="anchor" aria-hidden="true" href="#installing-rust-apps-using-cargo"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing Rust apps using Cargo</h2>
<p>Surprisingly there <a href="https://github.com/shssoichiro/oxipng/issues/69">isn't yet a packaged version</a> of Oxipng for Ubuntu - so I needed another way of installing it.</p>
<p>The <a href="https://github.com/shssoichiro/oxipng/blob/master/README.md#installing">project README suggests</a> installing it using <code>cargo install oxipng</code>.</p>
<p>I used the <a href="https://til.simonwillison.net/github-actions/debug-tmate" rel="nofollow">tmate trick</a> to try that out in a GitHub Actions worker - the <code>cargo</code> command is available by default but it took over a minute to fetch and compile all of the dependencies.</p>
<p>I didn't want to do this on every run, so I looked into ways to cache the built program. Thankfully the <code>actions/cache</code> action documents <a href="https://github.com/actions/cache/blob/main/examples.md#rust---cargo">how to use it with Rust</a>.</p>
<p>The full recipe for installing Oxipng in GitHub Actions looks like this:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Cache Oxipng</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">actions/cache@v3</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">path</span>: <span class="pl-s">~/.cargo/</span>
        <span class="pl-ent">key</span>: <span class="pl-s">${{ runner.os }}-cargo</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Install Oxipng</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        which oxipng || cargo install oxipng</span></pre></div>
<p>The first time the action runs it does a full compile of Oxipng - but on subsequent runs the <code>which oxipng</code> command succeeds and skips the <code>cargo install</code> step entirely.</p>
<h2><a id="user-content-running-oxipng-in-an-action" class="anchor" aria-hidden="true" href="#running-oxipng-in-an-action"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running Oxipng in an Action</h2>
<p>All of the PNGs that I wanted to optimize were in the root of my checkout, so I added this step:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Optimize PNGs</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        oxipng -o 4 -i 0 --strip safe *.png</span></pre></div>
<p>The <code>-o 4</code> is the highest recommended level of optimization.</p>
<p><code>-i 0</code> causes it to remove interlacing - "Interlacing can add 25-50% to the size of an optimized image" according to the README.</p>
<p><code>--strip safe</code> strips out any image metadata that is guaranteed not to affect how the image is rendered.</p>
<p>Oxipng updates the specified images in place, hence the <code>*.png</code> at the end.</p>
<h2><a id="user-content-testing-this-in-a-branch-first" class="anchor" aria-hidden="true" href="#testing-this-in-a-branch-first"><span aria-hidden="true" class="octicon octicon-link"></span></a>Testing this in a branch first</h2>
<p>I tested this all in a branch first so that I could see if it was working correctly.</p>
<p>Since my workflow usually pushes any changed files back to the same GitHub repository, I added a check to that step which caused it to only run on pushes to the <code>main</code> branch:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Commit and push</span>
      <span class="pl-ent">if</span>: <span class="pl-s">github.ref == 'refs/heads/main'</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        git config user.name "Automated"</span>
<span class="pl-s">        ...</span></pre></div>
<p>But I wanted to preview the generated images - so I added this step in the branch to save them to an artifact zip file that I could then inspect:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Artifacts</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">actions/upload-artifact@v3</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">name</span>: <span class="pl-s">screenshots</span>
        <span class="pl-ent">path</span>: <span class="pl-s"><span class="pl-pds">"</span>*.png<span class="pl-pds">"</span></span></pre></div>
<p>Once I got it working, I squash-merged <a href="https://github.com/simonw/datasette-screenshots/pull/2">this pull request</a> back into <code>main</code>.</p>
<h2><a id="user-content-the-result" class="anchor" aria-hidden="true" href="#the-result"><span aria-hidden="true" class="octicon octicon-link"></span></a>The result</h2>
<p>Oxipng worked really well!</p>
<p>It reduced the size of all three of my screenshots:</p>
<ul>
<li>
<a href="https://github.com/simonw/datasette-screenshots/blob/main/faceting.png">faceting.png</a> from 305KB to 211KB</li>
<li>
<a href="https://github.com/simonw/datasette-screenshots/blob/main/global-power-plants.png">global-power-plants.png</a> from 1.14MB to 770KB</li>
<li>
<a href="https://github.com/simonw/datasette-screenshots/blob/main/sql-query.png">sql-query.png</a> from 235KB to 169KB</li>
</ul>
<p><a href="https://github.com/simonw/datasette-screenshots/commit/0b7ccd8fe1c0fdc3714d768659a88bae1fa69ca4">This commit</a> shows the difference and lets you compare both images.</p>


path: fly_fly-docker-registry.md
topic: fly
title: Using the Fly Docker registry
url: https://github.com/simonw/til/blob/main/fly/fly-docker-registry.md
created: 2022-05-21T19:33:19-07:00
created_utc: 2022-05-22T02:33:19+00:00
updated: 2022-05-21T19:33:19-07:00
updated_utc: 2022-05-22T02:33:19+00:00
shot_hash: 437f94106c0a14fe25514ad91ba6da7d
slug: fly-docker-registry
body:
[Fly.io](https://fly.io/) lets you deploy Docker containers that will be compiled as a Firecracker VM and run in locations around the world.

Fly offer [a number of ways](https://fly.io/docs/reference/builders/) to build and deploy apps. For many frameworks you can run `fly launch` and it will detect the framework and configure a container for you. For others you can pass it a `Dockerfile` which will be built and deployed. But you can also push your own images to a Docker registry and deploy them to Fly.

Today I figured out how to use Fly's own registry to deploy an app.

## Tagging images for the Fly registry

Fly's registry is called `registry.fly.io`. To use it, you need to tag your Docker images with a tag that begins with that string.

Every Fly app gets its own registry subdomain. You can create apps in a number of ways, but the easiest is to use the Fly CLI:

    flyctl apps create datasette-demo

Fly app names must be globally unique across all of Fly - you will get an error if the app name is already taken.

You can create an app with a random, freely available name using the `--generate-name` option:

```
~ % flyctl apps create --generate-name
? Select Organization: Simon Willison (personal)
New app created: rough-dew-1296
```

Now that you have an app name, you can tag your Docker image using:

    registry.fly.io/your-app-name:unique-tag-for-your-image

If you are building an image using Docker on your machine, you can run this command in the same directory as your `Dockerfile`:

    docker build -t registry.fly.io/datasette-demo:datasette-demo-v0 .

## Pushing images to the registry

In order to push your image to Fly, you will first need to [authenticate](https://fly.io/docs/flyctl/auth-docker/).

The `flyctl auth docker` command will do this for you:
```
~ % flyctl auth docker
Authentication successful. You can now tag and push images to registry.fly.io/{your-app}
```
This works by hooking into Docker's own authentication mechanism. You can see what it has done by looking at your `~/.docker/config.json` file. Mine looks like this:

```json
{
  "auths": {
    "registry.fly.io": {
      "auth": "... secret token here ..."
    }
  ,
  "experimental": "disabled",
  "stackOrchestrator": "swarm"
}
```
Now you can push your image to the registry like this:

    docker push registry.fly.io/datasette-demo:datasette-demo-v0

## Deploying an image

Now that your image is pushed, you can deploy an instance of it like this:

    flyctl deploy --app datasette-demo \
      --image registry.fly.io/datasette-demo:datasette-demo-v0

A few seconds later your app will be running at:

    https://name-of-your-app.fly.dev/

html:
<p><a href="https://fly.io/" rel="nofollow">Fly.io</a> lets you deploy Docker containers that will be compiled as a Firecracker VM and run in locations around the world.</p>
<p>Fly offer <a href="https://fly.io/docs/reference/builders/" rel="nofollow">a number of ways</a> to build and deploy apps. For many frameworks you can run <code>fly launch</code> and it will detect the framework and configure a container for you. For others you can pass it a <code>Dockerfile</code> which will be built and deployed. But you can also push your own images to a Docker registry and deploy them to Fly.</p>
<p>Today I figured out how to use Fly's own registry to deploy an app.</p>
<h2>
<a id="user-content-tagging-images-for-the-fly-registry" class="anchor" href="#tagging-images-for-the-fly-registry" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Tagging images for the Fly registry</h2>
<p>Fly's registry is called <code>registry.fly.io</code>. To use it, you need to tag your Docker images with a tag that begins with that string.</p>
<p>Every Fly app gets its own registry subdomain. You can create apps in a number of ways, but the easiest is to use the Fly CLI:</p>
<pre><code>flyctl apps create datasette-demo
</code></pre>
<p>Fly app names must be globally unique across all of Fly - you will get an error if the app name is already taken.</p>
<p>You can create an app with a random, freely available name using the <code>--generate-name</code> option:</p>
<pre><code>~ % flyctl apps create --generate-name
? Select Organization: Simon Willison (personal)
New app created: rough-dew-1296
</code></pre>
<p>Now that you have an app name, you can tag your Docker image using:</p>
<pre><code>registry.fly.io/your-app-name:unique-tag-for-your-image
</code></pre>
<p>If you are building an image using Docker on your machine, you can run this command in the same directory as your <code>Dockerfile</code>:</p>
<pre><code>docker build -t registry.fly.io/datasette-demo:datasette-demo-v0 .
</code></pre>
<h2>
<a id="user-content-pushing-images-to-the-registry" class="anchor" href="#pushing-images-to-the-registry" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Pushing images to the registry</h2>
<p>In order to push your image to Fly, you will first need to <a href="https://fly.io/docs/flyctl/auth-docker/" rel="nofollow">authenticate</a>.</p>
<p>The <code>flyctl auth docker</code> command will do this for you:</p>
<pre><code>~ % flyctl auth docker
Authentication successful. You can now tag and push images to registry.fly.io/{your-app}
</code></pre>
<p>This works by hooking into Docker's own authentication mechanism. You can see what it has done by looking at your <code>~/.docker/config.json</code> file. Mine looks like this:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-ent">"auths"</span>: {
    <span class="pl-ent">"registry.fly.io"</span>: {
      <span class="pl-ent">"auth"</span>: <span class="pl-s"><span class="pl-pds">"</span>... secret token here ...<span class="pl-pds">"</span></span>
    }
  ,
  <span class="pl-ent">"experimental"</span>: <span class="pl-s"><span class="pl-pds">"</span>disabled<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"stackOrchestrator"</span>: <span class="pl-s"><span class="pl-pds">"</span>swarm<span class="pl-pds">"</span></span>
}</pre></div>
<p>Now you can push your image to the registry like this:</p>
<pre><code>docker push registry.fly.io/datasette-demo:datasette-demo-v0
</code></pre>
<h2>
<a id="user-content-deploying-an-image" class="anchor" href="#deploying-an-image" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Deploying an image</h2>
<p>Now that your image is pushed, you can deploy an instance of it like this:</p>
<pre><code>flyctl deploy --app datasette-demo \
  --image registry.fly.io/datasette-demo:datasette-demo-v0
</code></pre>
<p>A few seconds later your app will be running at:</p>
<pre><code>https://name-of-your-app.fly.dev/
</code></pre>


path: fly_fly-logs-to-s3.md
topic: fly
title: Writing Fly logs to S3
url: https://github.com/simonw/til/blob/main/fly/fly-logs-to-s3.md
created: 2022-05-25T12:47:40-07:00
created_utc: 2022-05-25T19:47:40+00:00
updated: 2022-09-27T20:59:18-07:00
updated_utc: 2022-09-28T03:59:18+00:00
shot_hash: 3ac2c543a1de8af7c9207aa5896ba423
slug: fly-logs-to-s3
body:
[Fly](https://fly.io/) offers [fly-log-shipper](https://github.com/superfly/fly-log-shipper) as a container you can run in a Fly application to send all of the logs from your other applications to a logging provider.

Several providers are supported. I decided to write them to an S3 bucket.

## Bucket credentials

I used my [s3-credentials](https://github.com/simonw/s3-credentials) tool to generate an access key and secret locked down to just one newly created bucket:

```
s3-credentials create my-project-fly-logs \
  --format ini \
  --bucket-region us-west-1 \
  --create-bucket \
  > logging-credentials.txt
```
I chose `us-west-1` or Northern California as the region, as it is closest to me.

That command output the following:

```
Created bucket: my-project-fly-logs in region: us-west-1
Created  user: 's3.read-write.my-project-fly-logs' with permissions boundary: 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
Attached policy s3.read-write.my-project-fly-logs to user s3.read-write.my-project-fly-logs
Created access key for user: s3.read-write.my-project-fly-logs
```

The full set of configuration needed by `fly-log-shipper` for S3 is:

- `ORG` - Fly organisation slug
- `ACCESS_TOKEN` - Fly personal access token
- `AWS_ACCESS_KEY_ID`	- AWS Access key with access to the log bucket
- `AWS_SECRET_ACCESS_KEY` - AWS secret access key
- `AWS_BUCKET` - AWS S3 bucket to store logs in
- `AWS_REGION` - Region for the bucket

I created a personal access token at https://fly.io/user/personal_access_tokens

## Creating the app

I created a new Fly application to run the container like so:

    fly apps create --name my-project-log-shipper --org my-project-org

## Setting the secrets

I set all of the configuration variables as secrets in one go like this:
```
fly secrets set \
  ORG="my-project-org" \
  ACCESS_TOKEN="..." \
  AWS_ACCESS_KEY_ID="AKIAWXFXAIOZIPTTHMBQ" \
  AWS_SECRET_ACCESS_KEY="..." \
  AWS_BUCKET="my-project-fly-logs" \
  AWS_REGION="us-west-1" \
  -a my-project-shipper
```

## Deploying the app

It turns out you still need a `fly.toml` file to successfully deploy - I tried without and got healthcheck errors.

Here's the TOML file I used:

```toml
app = "my-project-log-shipper"

[metrics]
  port = 9598
  path = "/metrics"
```

Then I passed that to the `fly deploy` command like this:

```
fly deploy --image ghcr.io/superfly/fly-log-shipper:v0.0.1 -a my-project-log-shipper -c fly.toml
```
Output:
```
==> Verifying app config
--> Verified app config
==> Building image
Searching for image 'ghcr.io/superfly/fly-log-shipper:v0.0.1' locally...
Searching for image 'ghcr.io/superfly/fly-log-shipper:v0.0.1' remotely...
image found: img_19gm46rqwn14x0jk
==> Creating release
--> release v1 created

--> You can detach the terminal anytime without stopping the deployment
==> Monitoring deployment

 1 desired, 1 placed, 1 healthy, 0 unhealthy
--> v1 deployed successfully
```
I used [Transmit](https://panic.com/transmit/) to inspect the bucket and my logs are showing up there as intended.

I can also now list my log files using `s3-credentials list-bucket` and the credentials I saved earlier:

```
% s3-credentials list-bucket -a logging-credentials.txt my-project-fly-logs --csv 
Key,LastModified,ETag,Size,StorageClass,Owner
my-project-admin/2022-05-25//1653507279-b9534d49-546a-47cb-bd93-36e08b8457ee.log.gz,2022-05-25 19:34:40+00:00,"""8b499f523bd2f5e9438cf2c0d42eab8c""",335,STANDARD,
my-project-admin/2022-05-25//1653507625-fc3c0f32-b3e8-4681-98ce-a4fe2960d1b2.log.gz,2022-05-25 19:40:26+00:00,"""5fe1f6fbf747706930edcfd00e3dd4fe""",444,STANDARD,
my-project-admin/2022-05-25//1653508693-7f4da568-e462-46a2-9904-e3ac801c7fee.log.gz,2022-05-25 19:58:14+00:00,"""794c57787a692a17d1fbfb78df442d58""",353,STANDARD,
my-project-postgresql/2022-05-25//1653507222-60e072e2-fd78-437f-a097-c063df424224.log.gz,2022-05-25 19:33:43+00:00,"""5d9965b79260e6a0f27cff5f02eee71d""",3376,STANDARD,
my-project-postgresql/2022-05-25//1653507525-b3a04523-8732-4072-92c0-57dd06a680b1.log.gz,2022-05-25 
```
**Update:** My initial attempt at doing this had a small bug: those `//` folder separators in that output turn out [to break AWS Athena](https://til.simonwillison.net/aws/athena-key-does-not-exist).

Deploying the latest version of the `fly-log-shipper` directly from their repository seems to fix that though (with [this fix](https://github.com/superfly/fly-log-shipper/pull/29) applied too).

html:
<p><a href="https://fly.io/" rel="nofollow">Fly</a> offers <a href="https://github.com/superfly/fly-log-shipper">fly-log-shipper</a> as a container you can run in a Fly application to send all of the logs from your other applications to a logging provider.</p>
<p>Several providers are supported. I decided to write them to an S3 bucket.</p>
<h2><a id="user-content-bucket-credentials" class="anchor" aria-hidden="true" href="#bucket-credentials"><span aria-hidden="true" class="octicon octicon-link"></span></a>Bucket credentials</h2>
<p>I used my <a href="https://github.com/simonw/s3-credentials">s3-credentials</a> tool to generate an access key and secret locked down to just one newly created bucket:</p>
<pre><code>s3-credentials create my-project-fly-logs \
  --format ini \
  --bucket-region us-west-1 \
  --create-bucket \
  &gt; logging-credentials.txt
</code></pre>
<p>I chose <code>us-west-1</code> or Northern California as the region, as it is closest to me.</p>
<p>That command output the following:</p>
<pre><code>Created bucket: my-project-fly-logs in region: us-west-1
Created  user: 's3.read-write.my-project-fly-logs' with permissions boundary: 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
Attached policy s3.read-write.my-project-fly-logs to user s3.read-write.my-project-fly-logs
Created access key for user: s3.read-write.my-project-fly-logs
</code></pre>
<p>The full set of configuration needed by <code>fly-log-shipper</code> for S3 is:</p>
<ul>
<li>
<code>ORG</code> - Fly organisation slug</li>
<li>
<code>ACCESS_TOKEN</code> - Fly personal access token</li>
<li>
<code>AWS_ACCESS_KEY_ID</code>	- AWS Access key with access to the log bucket</li>
<li>
<code>AWS_SECRET_ACCESS_KEY</code> - AWS secret access key</li>
<li>
<code>AWS_BUCKET</code> - AWS S3 bucket to store logs in</li>
<li>
<code>AWS_REGION</code> - Region for the bucket</li>
</ul>
<p>I created a personal access token at <a href="https://fly.io/user/personal_access_tokens" rel="nofollow">https://fly.io/user/personal_access_tokens</a></p>
<h2><a id="user-content-creating-the-app" class="anchor" aria-hidden="true" href="#creating-the-app"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating the app</h2>
<p>I created a new Fly application to run the container like so:</p>
<pre><code>fly apps create --name my-project-log-shipper --org my-project-org
</code></pre>
<h2><a id="user-content-setting-the-secrets" class="anchor" aria-hidden="true" href="#setting-the-secrets"><span aria-hidden="true" class="octicon octicon-link"></span></a>Setting the secrets</h2>
<p>I set all of the configuration variables as secrets in one go like this:</p>
<pre><code>fly secrets set \
  ORG="my-project-org" \
  ACCESS_TOKEN="..." \
  AWS_ACCESS_KEY_ID="AKIAWXFXAIOZIPTTHMBQ" \
  AWS_SECRET_ACCESS_KEY="..." \
  AWS_BUCKET="my-project-fly-logs" \
  AWS_REGION="us-west-1" \
  -a my-project-shipper
</code></pre>
<h2><a id="user-content-deploying-the-app" class="anchor" aria-hidden="true" href="#deploying-the-app"><span aria-hidden="true" class="octicon octicon-link"></span></a>Deploying the app</h2>
<p>It turns out you still need a <code>fly.toml</code> file to successfully deploy - I tried without and got healthcheck errors.</p>
<p>Here's the TOML file I used:</p>
<div class="highlight highlight-source-toml"><pre><span class="pl-smi">app</span> = <span class="pl-s"><span class="pl-pds">"</span>my-project-log-shipper<span class="pl-pds">"</span></span>

[<span class="pl-en">metrics</span>]
  <span class="pl-smi">port</span> = <span class="pl-c1">9598</span>
  <span class="pl-smi">path</span> = <span class="pl-s"><span class="pl-pds">"</span>/metrics<span class="pl-pds">"</span></span></pre></div>
<p>Then I passed that to the <code>fly deploy</code> command like this:</p>
<pre><code>fly deploy --image ghcr.io/superfly/fly-log-shipper:v0.0.1 -a my-project-log-shipper -c fly.toml
</code></pre>
<p>Output:</p>
<pre><code>==&gt; Verifying app config
--&gt; Verified app config
==&gt; Building image
Searching for image 'ghcr.io/superfly/fly-log-shipper:v0.0.1' locally...
Searching for image 'ghcr.io/superfly/fly-log-shipper:v0.0.1' remotely...
image found: img_19gm46rqwn14x0jk
==&gt; Creating release
--&gt; release v1 created

--&gt; You can detach the terminal anytime without stopping the deployment
==&gt; Monitoring deployment

 1 desired, 1 placed, 1 healthy, 0 unhealthy
--&gt; v1 deployed successfully
</code></pre>
<p>I used <a href="https://panic.com/transmit/" rel="nofollow">Transmit</a> to inspect the bucket and my logs are showing up there as intended.</p>
<p>I can also now list my log files using <code>s3-credentials list-bucket</code> and the credentials I saved earlier:</p>
<pre><code>% s3-credentials list-bucket -a logging-credentials.txt my-project-fly-logs --csv 
Key,LastModified,ETag,Size,StorageClass,Owner
my-project-admin/2022-05-25//1653507279-b9534d49-546a-47cb-bd93-36e08b8457ee.log.gz,2022-05-25 19:34:40+00:00,"""8b499f523bd2f5e9438cf2c0d42eab8c""",335,STANDARD,
my-project-admin/2022-05-25//1653507625-fc3c0f32-b3e8-4681-98ce-a4fe2960d1b2.log.gz,2022-05-25 19:40:26+00:00,"""5fe1f6fbf747706930edcfd00e3dd4fe""",444,STANDARD,
my-project-admin/2022-05-25//1653508693-7f4da568-e462-46a2-9904-e3ac801c7fee.log.gz,2022-05-25 19:58:14+00:00,"""794c57787a692a17d1fbfb78df442d58""",353,STANDARD,
my-project-postgresql/2022-05-25//1653507222-60e072e2-fd78-437f-a097-c063df424224.log.gz,2022-05-25 19:33:43+00:00,"""5d9965b79260e6a0f27cff5f02eee71d""",3376,STANDARD,
my-project-postgresql/2022-05-25//1653507525-b3a04523-8732-4072-92c0-57dd06a680b1.log.gz,2022-05-25 
</code></pre>
<p><strong>Update:</strong> My initial attempt at doing this had a small bug: those <code>//</code> folder separators in that output turn out <a href="https://til.simonwillison.net/aws/athena-key-does-not-exist" rel="nofollow">to break AWS Athena</a>.</p>
<p>Deploying the latest version of the <code>fly-log-shipper</code> directly from their repository seems to fix that though (with <a href="https://github.com/superfly/fly-log-shipper/pull/29">this fix</a> applied too).</p>


path: fly_wildcard-dns-ssl.md
topic: fly
title: Wildcard DNS and SSL on Fly
url: https://github.com/simonw/til/blob/main/fly/wildcard-dns-ssl.md
created: 2022-05-25T19:46:47-07:00
created_utc: 2022-05-26T02:46:47+00:00
updated: 2022-05-25T19:52:07-07:00
updated_utc: 2022-05-26T02:52:07+00:00
shot_hash: ebbaa42f14cb55fd6009d8707f2d482d
slug: wildcard-dns-ssl
body:
[Fly](https://fly.io/) makes it surprisingly easy to configure wildcard DNS, such that `anything.your-new-domain.dev` is served by a single Fly application (which can include multiple instances in multiple regions with global load-balancing).

Their documentation is at [SSL for Custom Domains](https://fly.io/docs/app-guides/custom-domains-with-fly). Here's how I set it up.

## Register the domain

I'm using `your-new-domain.dev` in this example, which is not a domain I have registered. `.dev` is interesting here because it requires SSL (or TLS if you want to be pedantic about it).

## Create an application with an IPv4 and IPv6 IP address

First, create an application:

    fly apps create --name your-wildcard-dns-app

Then create both an IPv4 and an IPv6 address for the application:
```
fly ips allocate-v4 -a your-wildcard-dns-app
TYPE ADDRESS      REGION CREATED AT 
v4   37.16.10.138 global 7s ago     

fly ips allocate-v6 -a your-wildcard-dns-app
TYPE ADDRESS             REGION CREATED AT 
v6   2a09:8280:1::1:3e99 global 4s ago     
```
The IPv4 address is so you can serve traffic.

The IPv6 address is needed as part of Fly's scheme to protect against subdomain takeover - see [How CDNs Generate Certificates: A Note About a Related Problem](https://fly.io/blog/how-cdns-generate-certificates/#a-note-about-a-related-problem) for details.

## Configuring DNS

Now setup the following DNS records:
```
your-new-domain.dev   A: 37.16.10.138
your-new-domain.dev   AAAA: 2a09:8280:1::1:3e99
*.your-new-domain.dev CNAME: your-wildcard-dns-app.fly.dev.
```

That `CNAME` record does the real magic here.

## Issue the certificate

You can ask Fly to issue the certificate (which uses LetsEncrypt under the hood) by running this:
```
fly certs create "*.your-new-domain.dev" \
  -a your-wildcard-dns-app
```

## Verifying the certificate

There's one last step: you need ta add an additional DNS record to verify the certificate.

Instructions for doing this can be found at:

    https://fly.io/apps/your-wildcard-dns-app/certificates

Click to view your unverified certificate and you should get a set of information that includes the extra DNS entry you need to add.

It will look something like this:

    _acme-challenge.your-new-domain.dev CNAME your-new-domain.dev.6g0mj.flydns.net.

Add that DNS record, then click the "Check again" button on the certificate screen and your certificate should be verified shortly afterwards.

## Deploy an app

With the certificate in place, you can deploy a Fly app (using regular Fly or the new [Fly Machines](https://fly.io/blog/fly-machines/)). Traffic to any subdomain of your domain, with or without `https://`, will be served by your new application.

## Configuring ports for your app

One last detail which caught me out: you need to configure your Fly app to accept traffic on both port 80 AND port 443 for this to work.

This is needed even if your app itself only serves on port 80 (or 8000 or whatever). The configuration here tells Fly's routing proxy what to do.

Using `fly.toml` this means there should be two `services.ports` sections - here's an example that includes those:

```
app = "your-wildcard-dns-app"

kill_signal = "SIGINT"
kill_timeout = 5

[[services]]
  internal_port = 8000
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["http"]
    port = "80"

  [[services.ports]]
    handlers = ["tls", "http"]
    port = "443"

  [[services.tcp_checks]]
    interval = 10000
    timeout = 2000
    grace_period = "10s"
```
If you are using Fly Machines you'll need to include the following in the JSON that is used to launch the machine:

```json
    "services": [
      {
        "ports": [
          {
            "port": 80,
            "handlers": [
              "http"
            ]
          },
          {
            "port": 443,
            "handlers": [
              "tls",
              "http"
            ]
          }
        ],
        "protocol": "tcp",
        "internal_port": 8001
      }
```

html:
<p><a href="https://fly.io/" rel="nofollow">Fly</a> makes it surprisingly easy to configure wildcard DNS, such that <code>anything.your-new-domain.dev</code> is served by a single Fly application (which can include multiple instances in multiple regions with global load-balancing).</p>
<p>Their documentation is at <a href="https://fly.io/docs/app-guides/custom-domains-with-fly" rel="nofollow">SSL for Custom Domains</a>. Here's how I set it up.</p>
<h2>
<a id="user-content-register-the-domain" class="anchor" href="#register-the-domain" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Register the domain</h2>
<p>I'm using <code>your-new-domain.dev</code> in this example, which is not a domain I have registered. <code>.dev</code> is interesting here because it requires SSL (or TLS if you want to be pedantic about it).</p>
<h2>
<a id="user-content-create-an-application-with-an-ipv4-and-ipv6-ip-address" class="anchor" href="#create-an-application-with-an-ipv4-and-ipv6-ip-address" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Create an application with an IPv4 and IPv6 IP address</h2>
<p>First, create an application:</p>
<pre><code>fly apps create --name your-wildcard-dns-app
</code></pre>
<p>Then create both an IPv4 and an IPv6 address for the application:</p>
<pre><code>fly ips allocate-v4 -a your-wildcard-dns-app
TYPE ADDRESS      REGION CREATED AT 
v4   37.16.10.138 global 7s ago     

fly ips allocate-v6 -a your-wildcard-dns-app
TYPE ADDRESS             REGION CREATED AT 
v6   2a09:8280:1::1:3e99 global 4s ago     
</code></pre>
<p>The IPv4 address is so you can serve traffic.</p>
<p>The IPv6 address is needed as part of Fly's scheme to protect against subdomain takeover - see <a href="https://fly.io/blog/how-cdns-generate-certificates/#a-note-about-a-related-problem" rel="nofollow">How CDNs Generate Certificates: A Note About a Related Problem</a> for details.</p>
<h2>
<a id="user-content-configuring-dns" class="anchor" href="#configuring-dns" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Configuring DNS</h2>
<p>Now setup the following DNS records:</p>
<pre><code>your-new-domain.dev   A: 37.16.10.138
your-new-domain.dev   AAAA: 2a09:8280:1::1:3e99
*.your-new-domain.dev CNAME: your-wildcard-dns-app.fly.dev.
</code></pre>
<p>That <code>CNAME</code> record does the real magic here.</p>
<h2>
<a id="user-content-issue-the-certificate" class="anchor" href="#issue-the-certificate" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Issue the certificate</h2>
<p>You can ask Fly to issue the certificate (which uses LetsEncrypt under the hood) by running this:</p>
<pre><code>fly certs create "*.your-new-domain.dev" \
  -a your-wildcard-dns-app
</code></pre>
<h2>
<a id="user-content-verifying-the-certificate" class="anchor" href="#verifying-the-certificate" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Verifying the certificate</h2>
<p>There's one last step: you need ta add an additional DNS record to verify the certificate.</p>
<p>Instructions for doing this can be found at:</p>
<pre><code>https://fly.io/apps/your-wildcard-dns-app/certificates
</code></pre>
<p>Click to view your unverified certificate and you should get a set of information that includes the extra DNS entry you need to add.</p>
<p>It will look something like this:</p>
<pre><code>_acme-challenge.your-new-domain.dev CNAME your-new-domain.dev.6g0mj.flydns.net.
</code></pre>
<p>Add that DNS record, then click the "Check again" button on the certificate screen and your certificate should be verified shortly afterwards.</p>
<h2>
<a id="user-content-deploy-an-app" class="anchor" href="#deploy-an-app" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Deploy an app</h2>
<p>With the certificate in place, you can deploy a Fly app (using regular Fly or the new <a href="https://fly.io/blog/fly-machines/" rel="nofollow">Fly Machines</a>). Traffic to any subdomain of your domain, with or without <code>https://</code>, will be served by your new application.</p>
<h2>
<a id="user-content-configuring-ports-for-your-app" class="anchor" href="#configuring-ports-for-your-app" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Configuring ports for your app</h2>
<p>One last detail which caught me out: you need to configure your Fly app to accept traffic on both port 80 AND port 443 for this to work.</p>
<p>This is needed even if your app itself only serves on port 80 (or 8000 or whatever). The configuration here tells Fly's routing proxy what to do.</p>
<p>Using <code>fly.toml</code> this means there should be two <code>services.ports</code> sections - here's an example that includes those:</p>
<pre><code>app = "your-wildcard-dns-app"

kill_signal = "SIGINT"
kill_timeout = 5

[[services]]
  internal_port = 8000
  protocol = "tcp"

  [services.concurrency]
    hard_limit = 25
    soft_limit = 20

  [[services.ports]]
    handlers = ["http"]
    port = "80"

  [[services.ports]]
    handlers = ["tls", "http"]
    port = "443"

  [[services.tcp_checks]]
    interval = 10000
    timeout = 2000
    grace_period = "10s"
</code></pre>
<p>If you are using Fly Machines you'll need to include the following in the JSON that is used to launch the machine:</p>
<div class="highlight highlight-source-json"><pre>    <span class="pl-ent">"services"</span>: [
      {
        <span class="pl-ent">"ports"</span>: [
          {
            <span class="pl-ent">"port"</span>: <span class="pl-c1">80</span>,
            <span class="pl-ent">"handlers"</span>: [
              <span class="pl-s"><span class="pl-pds">"</span>http<span class="pl-pds">"</span></span>
            ]
          },
          {
            <span class="pl-ent">"port"</span>: <span class="pl-c1">443</span>,
            <span class="pl-ent">"handlers"</span>: [
              <span class="pl-s"><span class="pl-pds">"</span>tls<span class="pl-pds">"</span></span>,
              <span class="pl-s"><span class="pl-pds">"</span>http<span class="pl-pds">"</span></span>
            ]
          }
        ],
        <span class="pl-ent">"protocol"</span>: <span class="pl-s"><span class="pl-pds">"</span>tcp<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"internal_port"</span>: <span class="pl-c1">8001</span>
      }</pre></div>


path: django_just-with-django.md
topic: django
title: Using just with Django
url: https://github.com/simonw/til/blob/main/django/just-with-django.md
created: 2022-06-06T14:24:37-07:00
created_utc: 2022-06-06T21:24:37+00:00
updated: 2022-06-06T14:24:37-07:00
updated_utc: 2022-06-06T21:24:37+00:00
shot_hash: 27d773c9510b97c00f6ed94bbc061e52
slug: just-with-django
body:
Jeff Triplett [convinced me](https://twitter.com/webology/status/1532860591307726851) to take a look at [just](https://github.com/casey/just) as a command automation tool - sort of an alternative to Make, except with a focus on commands rather than managing build dependencies.

I really like it, and I've started using it for my own Django projects.

## Installing with Homebrew

Installing just on my Mac was easy:

    brew install just

The tool is written in Rust and provides binaries for basically everything - there are [plenty more ways](https://github.com/casey/just/blob/master/README.md#installation) to install it.

## Writing a Justfile

The `Justfile` defines which commands `just` makes available. When you run `just` it looks for a `Justfile` in the current or any parent directory.

Commands in that file are run as if the working directory was the directory containing the `Justfile`.

Here's the file I've built so far for my current Django project. The project already uses `pipenv` and has some slightly convoluted environment requirements - just is a perfect tool for patching over those so I don't have to think about them any more. 

I added some comments to help explain what's going on:

```
# Using export here causes this DATABASE_URL to be made available as an
# environment variable for any command run by Just
export DATABASE_URL := "postgresql://localhost/myproject"

# The first command is the default if you run "just" with no options.
# I used *options to allow this to accept options, which means I can run:
#
#    just test -k auth --pdb
#
# To pass the "-k auth --pdb" options to pytest

@test *options:
  pipenv run pytest {{options}}

# This starts the Django development server with an extra environment variable
# I also print out a URL to the console so I can click on it without
# remembering which extra item I configured in /etc/hosts for this project
@server:
  echo "Starting http://myapp.local:8000/"
  DJANGO_SETTINGS_MODULE="config.localhost" pipenv run ./manage.py runserver

# I added this so I could run things like "just manage migrate" to run migrations
@manage *options:
  DJANGO_SETTINGS_MODULE="config.localhost" pipenv run ./manage.py {{options}}
```
The `@` before a command name causes that command to NOT output each line to the console before it is executed, which I found is usually what I wanted.

Here's what that example looks like without the comments: it's pretty succinct.

```
export DATABASE_URL := "postgresql://localhost/myproject"

@test *options:
  pipenv run pytest {{options}}

@server:
  echo "Starting http://myapp.local:8000/"
  DJANGO_SETTINGS_MODULE="config.localhost" pipenv run ./manage.py runserver

@manage *options:
  DJANGO_SETTINGS_MODULE="config.localhost" pipenv run ./manage.py {{options}}
```

## What this lets me do

To list all options:
```
% just --list
Available recipes:
    manage *options
    server
    test *options
```

To run all of my tests:

    just

To run specific tests

    just test -k name

To run tests, stopping at the first error and opening a debugger:

    just test -x --pdb

To start my development server running:

    just server

To run migrations:

    just manage migrate

To run `./manage.py shell`:

    just manage shell

## More examples

I found Jeff's [scripts-to-rule-them-all](https://github.com/jefftriplett/scripts-to-rule-them-all) example really helpful in learning how to use `just`.

The [official README](https://github.com/casey/just/blob/master/README.md) is excellent.

html:
<p>Jeff Triplett <a href="https://twitter.com/webology/status/1532860591307726851" rel="nofollow">convinced me</a> to take a look at <a href="https://github.com/casey/just">just</a> as a command automation tool - sort of an alternative to Make, except with a focus on commands rather than managing build dependencies.</p>
<p>I really like it, and I've started using it for my own Django projects.</p>
<h2>
<a id="user-content-installing-with-homebrew" class="anchor" href="#installing-with-homebrew" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing with Homebrew</h2>
<p>Installing just on my Mac was easy:</p>
<pre><code>brew install just
</code></pre>
<p>The tool is written in Rust and provides binaries for basically everything - there are <a href="https://github.com/casey/just/blob/master/README.md#installation">plenty more ways</a> to install it.</p>
<h2>
<a id="user-content-writing-a-justfile" class="anchor" href="#writing-a-justfile" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Writing a Justfile</h2>
<p>The <code>Justfile</code> defines which commands <code>just</code> makes available. When you run <code>just</code> it looks for a <code>Justfile</code> in the current or any parent directory.</p>
<p>Commands in that file are run as if the working directory was the directory containing the <code>Justfile</code>.</p>
<p>Here's the file I've built so far for my current Django project. The project already uses <code>pipenv</code> and has some slightly convoluted environment requirements - just is a perfect tool for patching over those so I don't have to think about them any more.</p>
<p>I added some comments to help explain what's going on:</p>
<pre><code># Using export here causes this DATABASE_URL to be made available as an
# environment variable for any command run by Just
export DATABASE_URL := "postgresql://localhost/myproject"

# The first command is the default if you run "just" with no options.
# I used *options to allow this to accept options, which means I can run:
#
#    just test -k auth --pdb
#
# To pass the "-k auth --pdb" options to pytest

@test *options:
  pipenv run pytest {{options}}

# This starts the Django development server with an extra environment variable
# I also print out a URL to the console so I can click on it without
# remembering which extra item I configured in /etc/hosts for this project
@server:
  echo "Starting http://myapp.local:8000/"
  DJANGO_SETTINGS_MODULE="config.localhost" pipenv run ./manage.py runserver

# I added this so I could run things like "just manage migrate" to run migrations
@manage *options:
  DJANGO_SETTINGS_MODULE="config.localhost" pipenv run ./manage.py {{options}}
</code></pre>
<p>The <code>@</code> before a command name causes that command to NOT output each line to the console before it is executed, which I found is usually what I wanted.</p>
<p>Here's what that example looks like without the comments: it's pretty succinct.</p>
<pre><code>export DATABASE_URL := "postgresql://localhost/myproject"

@test *options:
  pipenv run pytest {{options}}

@server:
  echo "Starting http://myapp.local:8000/"
  DJANGO_SETTINGS_MODULE="config.localhost" pipenv run ./manage.py runserver

@manage *options:
  DJANGO_SETTINGS_MODULE="config.localhost" pipenv run ./manage.py {{options}}
</code></pre>
<h2>
<a id="user-content-what-this-lets-me-do" class="anchor" href="#what-this-lets-me-do" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>What this lets me do</h2>
<p>To list all options:</p>
<pre><code>% just --list
Available recipes:
    manage *options
    server
    test *options
</code></pre>
<p>To run all of my tests:</p>
<pre><code>just
</code></pre>
<p>To run specific tests</p>
<pre><code>just test -k name
</code></pre>
<p>To run tests, stopping at the first error and opening a debugger:</p>
<pre><code>just test -x --pdb
</code></pre>
<p>To start my development server running:</p>
<pre><code>just server
</code></pre>
<p>To run migrations:</p>
<pre><code>just manage migrate
</code></pre>
<p>To run <code>./manage.py shell</code>:</p>
<pre><code>just manage shell
</code></pre>
<h2>
<a id="user-content-more-examples" class="anchor" href="#more-examples" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>More examples</h2>
<p>I found Jeff's <a href="https://github.com/jefftriplett/scripts-to-rule-them-all">scripts-to-rule-them-all</a> example really helpful in learning how to use <code>just</code>.</p>
<p>The <a href="https://github.com/casey/just/blob/master/README.md">official README</a> is excellent.</p>


path: networking_http-ipv6.md
topic: networking
title: Making HTTP calls using IPv6
url: https://github.com/simonw/til/blob/main/networking/http-ipv6.md
created: 2022-06-08T15:58:28-07:00
created_utc: 2022-06-08T22:58:28+00:00
updated: 2022-06-08T15:58:28-07:00
updated_utc: 2022-06-08T22:58:28+00:00
shot_hash: 0f53be67c4318676d822e214e1f18b9a
slug: http-ipv6
body:
Tiny TIL today: I learned how to make an HTTP call to an IPv6 address. The trick is to enclose the address in the URL in square braces:

    http://[2a09:8280:1::1:2741]

Here's that working as a request to www.pillarpointstewards.com (hosted on [Fly.io](https://fly.io/) which issues IPv6 addresses) using the [httpx](https://www.python-httpx.org/) Python library:
```pycon
>>> httpx.get("http://[2a09:8280:1::1:2741]", headers={"host": "www.pillarpointstewards.com"}).text
'<!DOCTYPE html>\n<html lang="en">\n<head>\n<title>Pillar Point Tidepool Stewards</title>\n<meta
```

html:
<p>Tiny TIL today: I learned how to make an HTTP call to an IPv6 address. The trick is to enclose the address in the URL in square braces:</p>
<pre><code>http://[2a09:8280:1::1:2741]
</code></pre>
<p>Here's that working as a request to <a href="http://www.pillarpointstewards.com" rel="nofollow">www.pillarpointstewards.com</a> (hosted on <a href="https://fly.io/" rel="nofollow">Fly.io</a> which issues IPv6 addresses) using the <a href="https://www.python-httpx.org/" rel="nofollow">httpx</a> Python library:</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; httpx.get(<span class="pl-s"><span class="pl-pds">"</span>http://[2a09:8280:1::1:2741]<span class="pl-pds">"</span></span>, <span class="pl-v">headers</span><span class="pl-k">=</span>{<span class="pl-s"><span class="pl-pds">"</span>host<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>www.pillarpointstewards.com<span class="pl-pds">"</span></span>}).text
'&lt;!DOCTYPE html&gt;\n&lt;html lang="en"&gt;\n&lt;head&gt;\n&lt;title&gt;Pillar Point Tidepool Stewards&lt;/title&gt;\n&lt;meta</pre></div>


path: jinja_format-thousands.md
topic: jinja
title: Formatting thousands in Jinja
url: https://github.com/simonw/til/blob/main/jinja/format-thousands.md
created: 2022-06-08T16:34:49-07:00
created_utc: 2022-06-08T23:34:49+00:00
updated: 2022-07-14T22:46:55-07:00
updated_utc: 2022-07-15T05:46:55+00:00
shot_hash: 7ec19a72b10ccc8448f28f33d7b6c31b
slug: format-thousands
body:
Here's how to format a number in Jinja with commas for thousands, without needing any custom filters or template functions:

    {{ "{:,}".format(row_count) }}

Output looks like this:

    179,119

Bonus: here's how to display a different pluralization of "row" if there is a single row:

    {{ "{:,}".format(row_count) }} row{{ "" if row_count == 1 else "s" }}

Outputs:

    179,119 rows

html:
<p>Here's how to format a number in Jinja with commas for thousands, without needing any custom filters or template functions:</p>
<pre><code>{{ "{:,}".format(row_count) }}
</code></pre>
<p>Output looks like this:</p>
<pre><code>179,119
</code></pre>
<p>Bonus: here's how to display a different pluralization of "row" if there is a single row:</p>
<pre><code>{{ "{:,}".format(row_count) }} row{{ "" if row_count == 1 else "s" }}
</code></pre>
<p>Outputs:</p>
<pre><code>179,119 rows
</code></pre>


path: linux_iconv.md
topic: linux
title: Using iconv to convert the text encoding of a file
url: https://github.com/simonw/til/blob/main/linux/iconv.md
created: 2022-06-14T15:42:39-07:00
created_utc: 2022-06-14T22:42:39+00:00
updated: 2022-06-14T15:42:39-07:00
updated_utc: 2022-06-14T22:42:39+00:00
shot_hash: 5b16f04576506dbbb84b9b7587b02321
slug: iconv
body:
In [sqlite-utils issue 439](https://github.com/simonw/sqlite-utils/issues/439) I was testing against a CSV file that used UTF16 little endian encoding, also known as `utf-16-le`.

I converted it to UTF-8 using `iconv` like this:

    iconv -f UTF-16LE -t UTF-8 file-in-utf16le.csv > file-in-utf8.csv

The `-f` argument here is the input encoding and `-t` is the desired output encoding.

I figured out the `-f` argument should be `UTF-16LE` (after first trying and failing with `utf-16-le`) by running:

    iconv -l

This outputs all of the available encoding options. It's a pretty long list so I filtered it like this:
```
% iconv -l | grep UTF
UTF-8 UTF8
UTF-8-MAC UTF8-MAC
UTF-16
UTF-16BE
UTF-16LE
UTF-32
UTF-32BE
UTF-32LE
UNICODE-1-1-UTF-7 UTF-7 CSUNICODE11UTF7
```

html:
<p>In <a href="https://github.com/simonw/sqlite-utils/issues/439">sqlite-utils issue 439</a> I was testing against a CSV file that used UTF16 little endian encoding, also known as <code>utf-16-le</code>.</p>
<p>I converted it to UTF-8 using <code>iconv</code> like this:</p>
<pre><code>iconv -f UTF-16LE -t UTF-8 file-in-utf16le.csv &gt; file-in-utf8.csv
</code></pre>
<p>The <code>-f</code> argument here is the input encoding and <code>-t</code> is the desired output encoding.</p>
<p>I figured out the <code>-f</code> argument should be <code>UTF-16LE</code> (after first trying and failing with <code>utf-16-le</code>) by running:</p>
<pre><code>iconv -l
</code></pre>
<p>This outputs all of the available encoding options. It's a pretty long list so I filtered it like this:</p>
<pre><code>% iconv -l | grep UTF
UTF-8 UTF8
UTF-8-MAC UTF8-MAC
UTF-16
UTF-16BE
UTF-16LE
UTF-32
UTF-32BE
UTF-32LE
UNICODE-1-1-UTF-7 UTF-7 CSUNICODE11UTF7
</code></pre>


path: datasette_register-new-plugin-hooks.md
topic: datasette
title: Registering new Datasette plugin hooks by defining them in other plugins
url: https://github.com/simonw/til/blob/main/datasette/register-new-plugin-hooks.md
created: 2022-06-17T13:04:35-07:00
created_utc: 2022-06-17T20:04:35+00:00
updated: 2022-06-17T13:04:35-07:00
updated_utc: 2022-06-17T20:04:35+00:00
shot_hash: 38c23e5679fa88c44f7e14038ea1b4ae
slug: register-new-plugin-hooks
body:
I'm experimenting with a Datasette plugin that itself adds new plugin hooks which other plugins can then interact with.

It's called [datasette-low-disk-space-hook](https://github.com/simonw/datasette-low-disk-space-hook), and it adds a new plugin hook called `low_disk_space(datasette)`, defined in the [datasette_low_disk_space_hook/hookspecs.py](https://github.com/simonw/datasette-low-disk-space-hook/blob/0.1a0/datasette_low_disk_space_hook/hookspecs.py) module.

The hook is registered by this code in [datasette_low_disk_space_hook/\_\_init\_\_.py](https://github.com/simonw/datasette-low-disk-space-hook/blob/0.1a0/datasette_low_disk_space_hook/__init__.py)

```python
from datasette.utils import await_me_maybe
from datasette.plugins import pm
from . import hookspecs

pm.add_hookspecs(hookspecs)
```
This imports the plugin manager directly from Datasette and uses it to add the new hooks.

I was worried that the `pm.add_hookspects(hookspecs)` line was not guaranteed to be executed if that module had not been imported.

It turns out that having this `entrpoints=` line in [setup.py](https://github.com/simonw/datasette-low-disk-space-hook/blob/0.1a0/setup.py) is enough to ensure that the module is imported and the `pm.add_hookspecs()` line is executed:

```python
from setuptools import setup

setup(
    name="datasette-low-disk-space-hook",
    # ...
    entry_points={"datasette": ["low_disk_space_hook = datasette_low_disk_space_hook"]},
    # ...
)
```

html:
<p>I'm experimenting with a Datasette plugin that itself adds new plugin hooks which other plugins can then interact with.</p>
<p>It's called <a href="https://github.com/simonw/datasette-low-disk-space-hook">datasette-low-disk-space-hook</a>, and it adds a new plugin hook called <code>low_disk_space(datasette)</code>, defined in the <a href="https://github.com/simonw/datasette-low-disk-space-hook/blob/0.1a0/datasette_low_disk_space_hook/hookspecs.py">datasette_low_disk_space_hook/hookspecs.py</a> module.</p>
<p>The hook is registered by this code in <a href="https://github.com/simonw/datasette-low-disk-space-hook/blob/0.1a0/datasette_low_disk_space_hook/__init__.py">datasette_low_disk_space_hook/__init__.py</a></p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">datasette</span>.<span class="pl-s1">utils</span> <span class="pl-k">import</span> <span class="pl-s1">await_me_maybe</span>
<span class="pl-k">from</span> <span class="pl-s1">datasette</span>.<span class="pl-s1">plugins</span> <span class="pl-k">import</span> <span class="pl-s1">pm</span>
<span class="pl-k">from</span> . <span class="pl-k">import</span> <span class="pl-s1">hookspecs</span>

<span class="pl-s1">pm</span>.<span class="pl-en">add_hookspecs</span>(<span class="pl-s1">hookspecs</span>)</pre></div>
<p>This imports the plugin manager directly from Datasette and uses it to add the new hooks.</p>
<p>I was worried that the <code>pm.add_hookspects(hookspecs)</code> line was not guaranteed to be executed if that module had not been imported.</p>
<p>It turns out that having this <code>entrpoints=</code> line in <a href="https://github.com/simonw/datasette-low-disk-space-hook/blob/0.1a0/setup.py">setup.py</a> is enough to ensure that the module is imported and the <code>pm.add_hookspecs()</code> line is executed:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">setuptools</span> <span class="pl-k">import</span> <span class="pl-s1">setup</span>

<span class="pl-en">setup</span>(
    <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"datasette-low-disk-space-hook"</span>,
    <span class="pl-c"># ...</span>
    <span class="pl-s1">entry_points</span><span class="pl-c1">=</span>{<span class="pl-s">"datasette"</span>: [<span class="pl-s">"low_disk_space_hook = datasette_low_disk_space_hook"</span>]},
    <span class="pl-c"># ...</span>
)</pre></div>


path: sqlite_one-line-csv-operations.md
topic: sqlite
title: One-liner for running queries against CSV files with SQLite
url: https://github.com/simonw/til/blob/main/sqlite/one-line-csv-operations.md
created: 2022-06-20T16:00:51-07:00
created_utc: 2022-06-20T23:00:51+00:00
updated: 2022-06-22T09:32:00-07:00
updated_utc: 2022-06-22T16:32:00+00:00
shot_hash: d62fd583ede4cddc95edf3c1c5febe1d
slug: one-line-csv-operations
body:
I figured out how to run a SQL query directly against a CSV file using the `sqlite3` command-line utility:

    sqlite3 :memory: -cmd '.mode csv' -cmd '.import taxi.csv taxi' \
      'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count'

This uses the special `:memory:` filename to open an in-memory database. Then it uses two `-cmd` options to turn on CSV mode and import the `taxi.csv` file into a table called `taxi`. Then it runs the SQL query.

Instead of setting the mode with `.mode` you can use `.import -csv` like this (thanks, [Mark Lawrence](https://sqlite.org/forum/forumpost/ad9d1a8f3e9feb8b)):

    sqlite3 :memory: -cmd '.import -csv taxi.csv taxi' \
      'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count'

You can get `taxi.csv` by downloading the compressed file from [here](https://github.com/multiprocessio/dsq/blob/43e72ff1d2c871082fed0ae401dd59e2ff9f6cfe/testdata/taxi.csv.7z) and running:

    7z e -aos taxi.csv.7z

I figured this out while commenting on [this issue](https://github.com/multiprocessio/dsq/issues/70).

The output looks like this:

```
"",128020,32.2371511482553
0,42228,17.0214016766151
1,1533197,17.6418833067999
2,286461,18.0975870711456
3,72852,17.9153958710923
4,25510,18.452774990196
5,50291,17.2709248175672
6,32623,17.6002964166367
7,2,87.17
8,2,95.705
9,1,113.6
```

Add `-cmd '.mode column'` to output in columns instead:
```
$ sqlite3 :memory: -cmd '.mode csv' -cmd '.import taxi.csv taxi' -cmd '.mode column' \
    'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count'
passenger_count  COUNT(*)  AVG(total_amount)
---------------  --------  -----------------
                 128020    32.2371511482553 
0                42228     17.0214016766151 
1                1533197   17.6418833067999 
2                286461    18.0975870711456 
3                72852     17.9153958710923 
4                25510     18.452774990196  
5                50291     17.2709248175672 
6                32623     17.6002964166367 
7                2         87.17            
8                2         95.705           
9                1         113.6            
```
Or use `-cmd '.mode markdown'` to get a Markdown table:

| passenger_count | COUNT(*) | AVG(total_amount) |
|-----------------|----------|-------------------|
|                 | 128020   | 32.2371511482553  |
| 0               | 42228    | 17.0214016766151  |
| 1               | 1533197  | 17.6418833067999  |
| 2               | 286461   | 18.0975870711456  |
| 3               | 72852    | 17.9153958710923  |
| 4               | 25510    | 18.452774990196   |
| 5               | 50291    | 17.2709248175672  |
| 6               | 32623    | 17.6002964166367  |
| 7               | 2        | 87.17             |
| 8               | 2        | 95.705            |
| 9               | 1        | 113.6             |

A full list of output modes can be seen like this:

```
% sqlite3 -cmd '.help mode'
.mode MODE ?TABLE?       Set output mode
   MODE is one of:
     ascii     Columns/rows delimited by 0x1F and 0x1E
     box       Tables using unicode box-drawing characters
     csv       Comma-separated values
     column    Output in columns.  (See .width)
     html      HTML <table> code
     insert    SQL insert statements for TABLE
     json      Results in a JSON array
     line      One value per line
     list      Values delimited by "|"
     markdown  Markdown table format
     quote     Escape answers as for SQL
     table     ASCII-art table
     tabs      Tab-separated values
     tcl       TCL list elements
```
## Other options

There are a whole bunch of other tools that can be used for this kind of thing!

My own [sqlite-utils memory](https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/) command can load data from JSON, CSV or TSV into an in-memory database and run a query against it. It's a LOT slower than using `sqlite3` directly though.

[dsq](https://github.com/multiprocessio/dsq) is a tool that does this kind of thing (and a lot more). Author Phil Eaton compiled [a collection of benchmarks](https://github.com/multiprocessio/dsq#benchmark) of other similar tools, and his [benchmarking script](https://github.com/multiprocessio/dsq/blob/43e72ff1d2c871082fed0ae401dd59e2ff9f6cfe/scripts/benchmark.sh) demonstrates how to use each one of them.

html:
<p>I figured out how to run a SQL query directly against a CSV file using the <code>sqlite3</code> command-line utility:</p>
<pre><code>sqlite3 :memory: -cmd '.mode csv' -cmd '.import taxi.csv taxi' \
  'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count'
</code></pre>
<p>This uses the special <code>:memory:</code> filename to open an in-memory database. Then it uses two <code>-cmd</code> options to turn on CSV mode and import the <code>taxi.csv</code> file into a table called <code>taxi</code>. Then it runs the SQL query.</p>
<p>Instead of setting the mode with <code>.mode</code> you can use <code>.import -csv</code> like this (thanks, <a href="https://sqlite.org/forum/forumpost/ad9d1a8f3e9feb8b" rel="nofollow">Mark Lawrence</a>):</p>
<pre><code>sqlite3 :memory: -cmd '.import -csv taxi.csv taxi' \
  'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count'
</code></pre>
<p>You can get <code>taxi.csv</code> by downloading the compressed file from <a href="https://github.com/multiprocessio/dsq/blob/43e72ff1d2c871082fed0ae401dd59e2ff9f6cfe/testdata/taxi.csv.7z">here</a> and running:</p>
<pre><code>7z e -aos taxi.csv.7z
</code></pre>
<p>I figured this out while commenting on <a href="https://github.com/multiprocessio/dsq/issues/70">this issue</a>.</p>
<p>The output looks like this:</p>
<pre><code>"",128020,32.2371511482553
0,42228,17.0214016766151
1,1533197,17.6418833067999
2,286461,18.0975870711456
3,72852,17.9153958710923
4,25510,18.452774990196
5,50291,17.2709248175672
6,32623,17.6002964166367
7,2,87.17
8,2,95.705
9,1,113.6
</code></pre>
<p>Add <code>-cmd '.mode column'</code> to output in columns instead:</p>
<pre><code>$ sqlite3 :memory: -cmd '.mode csv' -cmd '.import taxi.csv taxi' -cmd '.mode column' \
    'SELECT passenger_count, COUNT(*), AVG(total_amount) FROM taxi GROUP BY passenger_count'
passenger_count  COUNT(*)  AVG(total_amount)
---------------  --------  -----------------
                 128020    32.2371511482553 
0                42228     17.0214016766151 
1                1533197   17.6418833067999 
2                286461    18.0975870711456 
3                72852     17.9153958710923 
4                25510     18.452774990196  
5                50291     17.2709248175672 
6                32623     17.6002964166367 
7                2         87.17            
8                2         95.705           
9                1         113.6            
</code></pre>
<p>Or use <code>-cmd '.mode markdown'</code> to get a Markdown table:</p>
<table>
<thead>
<tr>
<th>passenger_count</th>
<th>COUNT(*)</th>
<th>AVG(total_amount)</th>
</tr>
</thead>
<tbody>
<tr>
<td></td>
<td>128020</td>
<td>32.2371511482553</td>
</tr>
<tr>
<td>0</td>
<td>42228</td>
<td>17.0214016766151</td>
</tr>
<tr>
<td>1</td>
<td>1533197</td>
<td>17.6418833067999</td>
</tr>
<tr>
<td>2</td>
<td>286461</td>
<td>18.0975870711456</td>
</tr>
<tr>
<td>3</td>
<td>72852</td>
<td>17.9153958710923</td>
</tr>
<tr>
<td>4</td>
<td>25510</td>
<td>18.452774990196</td>
</tr>
<tr>
<td>5</td>
<td>50291</td>
<td>17.2709248175672</td>
</tr>
<tr>
<td>6</td>
<td>32623</td>
<td>17.6002964166367</td>
</tr>
<tr>
<td>7</td>
<td>2</td>
<td>87.17</td>
</tr>
<tr>
<td>8</td>
<td>2</td>
<td>95.705</td>
</tr>
<tr>
<td>9</td>
<td>1</td>
<td>113.6</td>
</tr>
</tbody>
</table>
<p>A full list of output modes can be seen like this:</p>
<pre><code>% sqlite3 -cmd '.help mode'
.mode MODE ?TABLE?       Set output mode
   MODE is one of:
     ascii     Columns/rows delimited by 0x1F and 0x1E
     box       Tables using unicode box-drawing characters
     csv       Comma-separated values
     column    Output in columns.  (See .width)
     html      HTML &lt;table&gt; code
     insert    SQL insert statements for TABLE
     json      Results in a JSON array
     line      One value per line
     list      Values delimited by "|"
     markdown  Markdown table format
     quote     Escape answers as for SQL
     table     ASCII-art table
     tabs      Tab-separated values
     tcl       TCL list elements
</code></pre>
<h2>
<a id="user-content-other-options" class="anchor" href="#other-options" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Other options</h2>
<p>There are a whole bunch of other tools that can be used for this kind of thing!</p>
<p>My own <a href="https://simonwillison.net/2021/Jun/19/sqlite-utils-memory/" rel="nofollow">sqlite-utils memory</a> command can load data from JSON, CSV or TSV into an in-memory database and run a query against it. It's a LOT slower than using <code>sqlite3</code> directly though.</p>
<p><a href="https://github.com/multiprocessio/dsq">dsq</a> is a tool that does this kind of thing (and a lot more). Author Phil Eaton compiled <a href="https://github.com/multiprocessio/dsq#benchmark">a collection of benchmarks</a> of other similar tools, and his <a href="https://github.com/multiprocessio/dsq/blob/43e72ff1d2c871082fed0ae401dd59e2ff9f6cfe/scripts/benchmark.sh">benchmarking script</a> demonstrates how to use each one of them.</p>


path: bash_ignore-errors.md
topic: bash
title: Ignoring errors in a section of a Bash script
url: https://github.com/simonw/til/blob/main/bash/ignore-errors.md
created: 2022-06-27T17:24:42-07:00
created_utc: 2022-06-28T00:24:42+00:00
updated: 2022-06-27T17:24:42-07:00
updated_utc: 2022-06-28T00:24:42+00:00
shot_hash: d8c7cdf1528485991e86832fb6951377
slug: ignore-errors
body:
For [simonw/museums#32](https://github.com/simonw/museums/issues/32) I wanted to have certain lines in my Bash script ignore any errors: lines that used `sqlite-utils` to add columns and configure FTS, but that might fail with an error if the column already existed or FTS had already been configured.

[This tip](https://stackoverflow.com/a/60362732) on StackOverflow lead me to the [following recipe](https://github.com/simonw/museums/blob/d94410440a5c81a5cb3a0f0b886a8cd30941b8a9/build.sh):

```bash
#!/bin/bash
set -euo pipefail

yaml-to-sqlite browse.db museums museums.yaml --pk=id
python annotate_nominatum.py browse.db
python annotate_timestamps.py
# Ignore errors in following block until set -e:
set +e
sqlite-utils add-column browse.db museums country 2>/dev/null
sqlite3 browse.db < set-country.sql
sqlite-utils disable-fts browse.db museums 2>/dev/null
sqlite-utils enable-fts browse.db museums \
  name description country osm_city \
  --tokenize porter --create-triggers 2>/dev/null
set -e
```
Everything between the `set +e` and the `set -e` lines can now error without the Bash script itself failing.

The failing lines were still showing a bunch of Python tracebacks. I fixed that by redirecting their standard error output to `/dev/null` like this:
```bash
sqlite-utils disable-fts browse.db museums 2>/dev/null
```

html:
<p>For <a href="https://github.com/simonw/museums/issues/32">simonw/museums#32</a> I wanted to have certain lines in my Bash script ignore any errors: lines that used <code>sqlite-utils</code> to add columns and configure FTS, but that might fail with an error if the column already existed or FTS had already been configured.</p>
<p><a href="https://stackoverflow.com/a/60362732" rel="nofollow">This tip</a> on StackOverflow lead me to the <a href="https://github.com/simonw/museums/blob/d94410440a5c81a5cb3a0f0b886a8cd30941b8a9/build.sh">following recipe</a>:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c"><span class="pl-c">#!</span>/bin/bash</span>
<span class="pl-c1">set</span> -euo pipefail

yaml-to-sqlite browse.db museums museums.yaml --pk=id
python annotate_nominatum.py browse.db
python annotate_timestamps.py
<span class="pl-c"><span class="pl-c">#</span> Ignore errors in following block until set -e:</span>
<span class="pl-c1">set</span> +e
sqlite-utils add-column browse.db museums country <span class="pl-k">2&gt;</span>/dev/null
sqlite3 browse.db <span class="pl-k">&lt;</span> set-country.sql
sqlite-utils disable-fts browse.db museums <span class="pl-k">2&gt;</span>/dev/null
sqlite-utils enable-fts browse.db museums \
  name description country osm_city \
  --tokenize porter --create-triggers <span class="pl-k">2&gt;</span>/dev/null
<span class="pl-c1">set</span> -e</pre></div>
<p>Everything between the <code>set +e</code> and the <code>set -e</code> lines can now error without the Bash script itself failing.</p>
<p>The failing lines were still showing a bunch of Python tracebacks. I fixed that by redirecting their standard error output to <code>/dev/null</code> like this:</p>
<div class="highlight highlight-source-shell"><pre>sqlite-utils disable-fts browse.db museums <span class="pl-k">2&gt;</span>/dev/null</pre></div>


