# til

path: sql_cumulative-total-over-time.md
topic: sql
title: Cumulative total over time in SQL
url: https://github.com/simonw/til/blob/main/sql/cumulative-total-over-time.md
created: 2021-09-13T08:33:52-07:00
created_utc: 2021-09-13T15:33:52+00:00
updated: 2021-12-24T15:20:40-08:00
updated_utc: 2021-12-24T23:20:40+00:00
shot_hash: cb4693fa2c413b627aff81fad2f2383c
slug: cumulative-total-over-time
body:
This is a quick trick for creating a cumulative chart of the total number of items created over time based just on their creation date.

- [Example below](https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++created_at%2C%0D%0A++%28%0D%0A++++select%0D%0A++++++count%28*%29%0D%0A++++from%0D%0A++++++repos+repos2%0D%0A++++where%0D%0A++++++repos2.owner+%3D+%3Ap0%0D%0A++++++and+repos2.created_at+%3C%3D+repos.created_at%0D%0A++%29+as+cumulative%0D%0Afrom%0D%0A++repos%0D%0Awhere%0D%0A++%22owner%22+%3D+%3Ap0%0D%0Aorder+by%0D%0A++created_at+desc&p0=9599#g.mark=line&g.x_column=created_at&g.x_type=temporal&g.y_column=cumulative&g.y_type=quantitative) which shows number of repositories I have created over time
- [Cumulative stars for a GitHub repository over time](https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++starred_at%2C%0D%0A++(%0D%0A++++select%0D%0A++++++count(*)%0D%0A++++from%0D%0A++++++stars+stars2%0D%0A++++where%0D%0A++++++stars2.repo+%3D+stars.repo%0D%0A++++++and+stars2.starred_at+%3C%3D+stars.starred_at%0D%0A++)+as+cumulative%0D%0Afrom%0D%0A++stars%0D%0Awhere%0D%0A++repo+%3D+(select+id+from+repos+where+full_name+%3D+%3Afull_name)%0D%0Aorder+by%0D%0A++starred_at&full_name=dogsheep%2Fgithub-to-sqlite#g.mark=line&g.x_column=starred_at&g.x_type=temporal&g.y_column=cumulative&g.y_type=quantitative)

```sql
select
  created_at,
  (
    select
      count(*)
    from
      repos repos2
    where
      repos2.owner = 9599
      and repos2.created_at <= repos.created_at
  ) as cumulative
from
  repos
where
  "owner" = 9599
order by
  created_at desc
```
I imagine there's a more elegant way to do this using a window function but this works fine.

html:
<p>This is a quick trick for creating a cumulative chart of the total number of items created over time based just on their creation date.</p>
<ul>
<li>
<a href="https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++created_at%2C%0D%0A++%28%0D%0A++++select%0D%0A++++++count%28*%29%0D%0A++++from%0D%0A++++++repos+repos2%0D%0A++++where%0D%0A++++++repos2.owner+%3D+%3Ap0%0D%0A++++++and+repos2.created_at+%3C%3D+repos.created_at%0D%0A++%29+as+cumulative%0D%0Afrom%0D%0A++repos%0D%0Awhere%0D%0A++%22owner%22+%3D+%3Ap0%0D%0Aorder+by%0D%0A++created_at+desc&amp;p0=9599#g.mark=line&amp;g.x_column=created_at&amp;g.x_type=temporal&amp;g.y_column=cumulative&amp;g.y_type=quantitative" rel="nofollow">Example below</a> which shows number of repositories I have created over time</li>
<li><a href="https://github-to-sqlite.dogsheep.net/github?sql=select%0D%0A++starred_at%2C%0D%0A++(%0D%0A++++select%0D%0A++++++count(*)%0D%0A++++from%0D%0A++++++stars+stars2%0D%0A++++where%0D%0A++++++stars2.repo+%3D+stars.repo%0D%0A++++++and+stars2.starred_at+%3C%3D+stars.starred_at%0D%0A++)+as+cumulative%0D%0Afrom%0D%0A++stars%0D%0Awhere%0D%0A++repo+%3D+(select+id+from+repos+where+full_name+%3D+%3Afull_name)%0D%0Aorder+by%0D%0A++starred_at&amp;full_name=dogsheep%2Fgithub-to-sqlite#g.mark=line&amp;g.x_column=starred_at&amp;g.x_type=temporal&amp;g.y_column=cumulative&amp;g.y_type=quantitative" rel="nofollow">Cumulative stars for a GitHub repository over time</a></li>
</ul>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  created_at,
  (
    <span class="pl-k">select</span>
      <span class="pl-c1">count</span>(<span class="pl-k">*</span>)
    <span class="pl-k">from</span>
      repos repos2
    <span class="pl-k">where</span>
      <span class="pl-c1">repos2</span>.<span class="pl-c1">owner</span> <span class="pl-k">=</span> <span class="pl-c1">9599</span>
      <span class="pl-k">and</span> <span class="pl-c1">repos2</span>.<span class="pl-c1">created_at</span> <span class="pl-k">&lt;=</span> <span class="pl-c1">repos</span>.<span class="pl-c1">created_at</span>
  ) <span class="pl-k">as</span> cumulative
<span class="pl-k">from</span>
  repos
<span class="pl-k">where</span>
  <span class="pl-s"><span class="pl-pds">"</span>owner<span class="pl-pds">"</span></span> <span class="pl-k">=</span> <span class="pl-c1">9599</span>
<span class="pl-k">order by</span>
  created_at <span class="pl-k">desc</span></pre></div>
<p>I imagine there's a more elegant way to do this using a window function but this works fine.</p>


path: electron_electrion-auto-update.md
topic: electron
title: Configuring auto-update for an Electron app
url: https://github.com/simonw/til/blob/main/electron/electrion-auto-update.md
created: 2021-09-13T20:57:03-07:00
created_utc: 2021-09-14T03:57:03+00:00
updated: 2021-09-13T21:11:18-07:00
updated_utc: 2021-09-14T04:11:18+00:00
shot_hash: b3d1f989fac8930682b18b7c984199ea
slug: electrion-auto-update
body:
This is _almost_ really simple. I used [electron/update-electron-app](https://github.com/electron/update-electron-app) for it, the instructions for which are:

- Add it to `packages.json` with `npm i update-electron-app`
- Make sure your `"repository"` field in that file points to your GitHub repository
- Use GitHub releases to release signed versions of your application
- Add `require('update-electron-app')()` somewhere in your `main.js`

I added this... and it didn't work ([#106](https://github.com/simonw/datasette-app/issues/106)).

Then I spotted [this recipe](https://github.com/electron/update.electronjs.org#manual-setup) in the manual setup instructions for the `update.electronjs.org` server that it uses:

```javascript
const server = 'https://update.electronjs.org'
const feed = `${server}/OWNER/REPO/${process.platform}-${process.arch}/${app.getVersion()}`
```
I ran that in the Electron debugger, swapping in `simonw/datasette-app` as the `OWNER/REPO` and got this URL:

`https://update.electronjs.org/simonw/datasette-app/darwin-x64/0.2.0`

Which returned this:

> `No updates found (needs asset matching *{mac,darwin,osx}*.zip in public repository)`

It turns out your asset filename needs to match that pattern!

I renamed the asset I was attaching to the release to `Datasette-mac.app.zip` and the auto-update mechanism started working instantly.

## How it works

That update URL is interesting. If you hit it with the most recent version of the software (`0.2.1` at time of writing) you get this:

```
~ % curl -i 'https://update.electronjs.org/simonw/datasette-app/darwin-x64/0.2.1'
HTTP/1.1 204 No Content
Server: Cowboy
Content-Length: 0
Connection: keep-alive
Date: Tue, 14 Sep 2021 03:54:47 GMT
Via: 1.1 vegur
```
But if you tell it you are running a previous version you get this instead:

```
~ % curl -i 'https://update.electronjs.org/simonw/datasette-app/darwin-x64/0.2.0'
HTTP/1.1 200 OK
Server: Cowboy
Connection: keep-alive
Content-Type: application/json
Date: Tue, 14 Sep 2021 03:55:19 GMT
Content-Length: 740
Via: 1.1 vegur

{"name":"0.2.1","notes":"- Fixed bug where application would not start without a working internet connection. [#115](https://github.com/simonw/datasette-app/issues/115)\r\n- The \"Debug -> Open Chromium DevTools\" menu item no longer shows an error if no windows are focused. [#113](https://github.com/simonw/datasette-app/issues/113)\r\n- Fixed bug where the `datasette-leaflet` plugin could be uninstalled despite being automatically re-installed. [#118](https://github.com/simonw/datasette-app/issues/118)\r\n- Time limit for facet calculations increased from 1 second to 3 seconds. [#114](https://github.com/simonw/datasette-app/issues/114)","url":"https://github.com/simonw/datasette-app/releases/download/0.2.1/Datasette-mac.app.zip"}
```
Which pretty-prints to:

```json
{
  "name": "0.2.1",
  "notes": "- Fixed bug where application would not start without a working internet connection. [#115](https://github.com/simonw/datasette-app/issues/115)\r\n- The \"Debug -> Open Chromium DevTools\" menu item no longer shows an error if no windows are focused. [#113](https://github.com/simonw/datasette-app/issues/113)\r\n- Fixed bug where the `datasette-leaflet` plugin could be uninstalled despite being automatically re-installed. [#118](https://github.com/simonw/datasette-app/issues/118)\r\n- Time limit for facet calculations increased from 1 second to 3 seconds. [#114](https://github.com/simonw/datasette-app/issues/114)",
  "url": "https://github.com/simonw/datasette-app/releases/download/0.2.1/Datasette-mac.app.zip"
}
```

html:
<p>This is <em>almost</em> really simple. I used <a href="https://github.com/electron/update-electron-app">electron/update-electron-app</a> for it, the instructions for which are:</p>
<ul>
<li>Add it to <code>packages.json</code> with <code>npm i update-electron-app</code>
</li>
<li>Make sure your <code>"repository"</code> field in that file points to your GitHub repository</li>
<li>Use GitHub releases to release signed versions of your application</li>
<li>Add <code>require('update-electron-app')()</code> somewhere in your <code>main.js</code>
</li>
</ul>
<p>I added this... and it didn't work (<a href="https://github.com/simonw/datasette-app/issues/106">#106</a>).</p>
<p>Then I spotted <a href="https://github.com/electron/update.electronjs.org#manual-setup">this recipe</a> in the manual setup instructions for the <code>update.electronjs.org</code> server that it uses:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">const</span> <span class="pl-s1">server</span> <span class="pl-c1">=</span> <span class="pl-s">'https://update.electronjs.org'</span>
<span class="pl-k">const</span> <span class="pl-s1">feed</span> <span class="pl-c1">=</span> <span class="pl-s">`<span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">server</span><span class="pl-kos">}</span></span>/OWNER/REPO/<span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">process</span><span class="pl-kos">.</span><span class="pl-c1">platform</span><span class="pl-kos">}</span></span>-<span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">process</span><span class="pl-kos">.</span><span class="pl-c1">arch</span><span class="pl-kos">}</span></span>/<span class="pl-s1"><span class="pl-kos">${</span><span class="pl-s1">app</span><span class="pl-kos">.</span><span class="pl-en">getVersion</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">}</span></span>`</span></pre></div>
<p>I ran that in the Electron debugger, swapping in <code>simonw/datasette-app</code> as the <code>OWNER/REPO</code> and got this URL:</p>
<p><code>https://update.electronjs.org/simonw/datasette-app/darwin-x64/0.2.0</code></p>
<p>Which returned this:</p>
<blockquote>
<p><code>No updates found (needs asset matching *{mac,darwin,osx}*.zip in public repository)</code></p>
</blockquote>
<p>It turns out your asset filename needs to match that pattern!</p>
<p>I renamed the asset I was attaching to the release to <code>Datasette-mac.app.zip</code> and the auto-update mechanism started working instantly.</p>
<h2>
<a id="user-content-how-it-works" class="anchor" href="#how-it-works" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>How it works</h2>
<p>That update URL is interesting. If you hit it with the most recent version of the software (<code>0.2.1</code> at time of writing) you get this:</p>
<pre><code>~ % curl -i 'https://update.electronjs.org/simonw/datasette-app/darwin-x64/0.2.1'
HTTP/1.1 204 No Content
Server: Cowboy
Content-Length: 0
Connection: keep-alive
Date: Tue, 14 Sep 2021 03:54:47 GMT
Via: 1.1 vegur
</code></pre>
<p>But if you tell it you are running a previous version you get this instead:</p>
<pre><code>~ % curl -i 'https://update.electronjs.org/simonw/datasette-app/darwin-x64/0.2.0'
HTTP/1.1 200 OK
Server: Cowboy
Connection: keep-alive
Content-Type: application/json
Date: Tue, 14 Sep 2021 03:55:19 GMT
Content-Length: 740
Via: 1.1 vegur

{"name":"0.2.1","notes":"- Fixed bug where application would not start without a working internet connection. [#115](https://github.com/simonw/datasette-app/issues/115)\r\n- The \"Debug -&gt; Open Chromium DevTools\" menu item no longer shows an error if no windows are focused. [#113](https://github.com/simonw/datasette-app/issues/113)\r\n- Fixed bug where the `datasette-leaflet` plugin could be uninstalled despite being automatically re-installed. [#118](https://github.com/simonw/datasette-app/issues/118)\r\n- Time limit for facet calculations increased from 1 second to 3 seconds. [#114](https://github.com/simonw/datasette-app/issues/114)","url":"https://github.com/simonw/datasette-app/releases/download/0.2.1/Datasette-mac.app.zip"}
</code></pre>
<p>Which pretty-prints to:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>0.2.1<span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>notes<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>- Fixed bug where application would not start without a working internet connection. [#115](https://github.com/simonw/datasette-app/issues/115)<span class="pl-cce">\r\n</span>- The <span class="pl-cce">\"</span>Debug -&gt; Open Chromium DevTools<span class="pl-cce">\"</span> menu item no longer shows an error if no windows are focused. [#113](https://github.com/simonw/datasette-app/issues/113)<span class="pl-cce">\r\n</span>- Fixed bug where the `datasette-leaflet` plugin could be uninstalled despite being automatically re-installed. [#118](https://github.com/simonw/datasette-app/issues/118)<span class="pl-cce">\r\n</span>- Time limit for facet calculations increased from 1 second to 3 seconds. [#114](https://github.com/simonw/datasette-app/issues/114)<span class="pl-pds">"</span></span>,
  <span class="pl-s"><span class="pl-pds">"</span>url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://github.com/simonw/datasette-app/releases/download/0.2.1/Datasette-mac.app.zip<span class="pl-pds">"</span></span>
}</pre></div>


path: googlecloud_gsutil-bucket.md
topic: googlecloud
title: Publishing to a public Google Cloud bucket with gsutil
url: https://github.com/simonw/til/blob/main/googlecloud/gsutil-bucket.md
created: 2021-09-20T13:13:52-07:00
created_utc: 2021-09-20T20:13:52+00:00
updated: 2021-09-20T13:13:52-07:00
updated_utc: 2021-09-20T20:13:52+00:00
shot_hash: f40fa7435ffed4b08cdc3f702f8b35c6
slug: gsutil-bucket
body:
I decided to publish static CSV files to accompany my https://cdc-vaccination-history.datasette.io/ project, using a Google Cloud bucket (see [cdc-vaccination-history issue #9](https://github.com/simonw/cdc-vaccination-history/issues/9)).

The Google Cloud tutorial on [https://cloud.google.com/storage/docs/hosting-static-website-http#gsutil](https://cloud.google.com/storage/docs/hosting-static-website-http#gsutil) was very helpful.

## Creating the bucket

I used an authenticated `gsutil` session that I already had from my work with Google Cloud Run.

To create a new bucket:

    gsutil mb gs://cdc-vaccination-history-csv.datasette.io/

`mb` is the [make bucket](https://cloud.google.com/storage/docs/gsutil/commands/mb) command.

I had already verified my `datasette.io` bucket with Google, otherwise this step would not have worked.

## Uploading files

    gsutil cp *.csv gs://cdc-vaccination-history-csv.datasette.io

Using the [gsutil cp command](https://cloud.google.com/storage/docs/gsutil/commands/cp).

## Making them available to the public

This command allows anyone to download from the bucket:

    gsutil iam ch allUsers:objectViewer gs://cdc-vaccination-history-csv.datasette.io

## DNS

I configured `cdc-vaccination-history-csv` as a `CNAME` pointing to `c.storage.googleapis.com.`

https://cdc-vaccination-history-csv.datasette.io/ now shows an XML directory listing.

html:
<p>I decided to publish static CSV files to accompany my <a href="https://cdc-vaccination-history.datasette.io/" rel="nofollow">https://cdc-vaccination-history.datasette.io/</a> project, using a Google Cloud bucket (see <a href="https://github.com/simonw/cdc-vaccination-history/issues/9">cdc-vaccination-history issue #9</a>).</p>
<p>The Google Cloud tutorial on <a href="https://cloud.google.com/storage/docs/hosting-static-website-http#gsutil" rel="nofollow">https://cloud.google.com/storage/docs/hosting-static-website-http#gsutil</a> was very helpful.</p>
<h2>
<a id="user-content-creating-the-bucket" class="anchor" href="#creating-the-bucket" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating the bucket</h2>
<p>I used an authenticated <code>gsutil</code> session that I already had from my work with Google Cloud Run.</p>
<p>To create a new bucket:</p>
<pre><code>gsutil mb gs://cdc-vaccination-history-csv.datasette.io/
</code></pre>
<p><code>mb</code> is the <a href="https://cloud.google.com/storage/docs/gsutil/commands/mb" rel="nofollow">make bucket</a> command.</p>
<p>I had already verified my <code>datasette.io</code> bucket with Google, otherwise this step would not have worked.</p>
<h2>
<a id="user-content-uploading-files" class="anchor" href="#uploading-files" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Uploading files</h2>
<pre><code>gsutil cp *.csv gs://cdc-vaccination-history-csv.datasette.io
</code></pre>
<p>Using the <a href="https://cloud.google.com/storage/docs/gsutil/commands/cp" rel="nofollow">gsutil cp command</a>.</p>
<h2>
<a id="user-content-making-them-available-to-the-public" class="anchor" href="#making-them-available-to-the-public" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Making them available to the public</h2>
<p>This command allows anyone to download from the bucket:</p>
<pre><code>gsutil iam ch allUsers:objectViewer gs://cdc-vaccination-history-csv.datasette.io
</code></pre>
<h2>
<a id="user-content-dns" class="anchor" href="#dns" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>DNS</h2>
<p>I configured <code>cdc-vaccination-history-csv</code> as a <code>CNAME</code> pointing to <code>c.storage.googleapis.com.</code></p>
<p><a href="https://cdc-vaccination-history-csv.datasette.io/" rel="nofollow">https://cdc-vaccination-history-csv.datasette.io/</a> now shows an XML directory listing.</p>


path: javascript_lit-with-skypack.md
topic: javascript
title: Loading lit from Skypack
url: https://github.com/simonw/til/blob/main/javascript/lit-with-skypack.md
created: 2021-09-21T18:03:21-07:00
created_utc: 2021-09-22T01:03:21+00:00
updated: 2021-11-21T16:44:18-08:00
updated_utc: 2021-11-22T00:44:18+00:00
shot_hash: 8fc7db1f83bf1c22bdb7d8a098bc4110
slug: lit-with-skypack
body:
[Lit 2](https://lit.dev/blog/2021-09-21-announcing-lit-2/) stable was released today, offering a tiny, feature-full framework for constructing web components using modern JavaScript.

The [getting started documentation](https://lit.dev/docs/getting-started/) involves a whole lot of `npm` usage, but I wanted to just drop something into an HTML page and start trying out the library, without any kind of build step.

After [some discussion](https://twitter.com/simonw/status/1440462801630208001) on Twitter and with [the help of @WestbrookJ](https://twitter.com/WestbrookJ/status/1440477115741130757) I figured out the following pattern, which loads code from [Skypack](https://www.skypack.dev/):

```html
<script type="module">
import { LitElement, html } from 'https://cdn.skypack.dev/lit';
  
class MyEl extends LitElement {
  render() {
    return html`Hello world!`
  }
}
customElements.define('my-el',MyEl);
</script>
<my-el></my-el>
```

Also relevant: [lit-dist](https://github.com/fserb/lit-dist).

html:
<p><a href="https://lit.dev/blog/2021-09-21-announcing-lit-2/" rel="nofollow">Lit 2</a> stable was released today, offering a tiny, feature-full framework for constructing web components using modern JavaScript.</p>
<p>The <a href="https://lit.dev/docs/getting-started/" rel="nofollow">getting started documentation</a> involves a whole lot of <code>npm</code> usage, but I wanted to just drop something into an HTML page and start trying out the library, without any kind of build step.</p>
<p>After <a href="https://twitter.com/simonw/status/1440462801630208001" rel="nofollow">some discussion</a> on Twitter and with <a href="https://twitter.com/WestbrookJ/status/1440477115741130757" rel="nofollow">the help of @WestbrookJ</a> I figured out the following pattern, which loads code from <a href="https://www.skypack.dev/" rel="nofollow">Skypack</a>:</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">script</span> <span class="pl-c1">type</span>="<span class="pl-s">module</span>"<span class="pl-kos">&gt;</span>
<span class="pl-k">import</span> <span class="pl-kos">{</span> <span class="pl-v">LitElement</span><span class="pl-kos">,</span> <span class="pl-s1">html</span> <span class="pl-kos">}</span> <span class="pl-k">from</span> <span class="pl-s">'https://cdn.skypack.dev/lit'</span><span class="pl-kos">;</span>
  
<span class="pl-k">class</span> <span class="pl-v">MyEl</span> <span class="pl-k">extends</span> <span class="pl-v">LitElement</span> <span class="pl-kos">{</span>
  <span class="pl-en">render</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-k">return</span> <span class="pl-en">html</span><span class="pl-s">`Hello world!`</span>
  <span class="pl-kos">}</span>
<span class="pl-kos">}</span>
<span class="pl-s1">customElements</span><span class="pl-kos">.</span><span class="pl-en">define</span><span class="pl-kos">(</span><span class="pl-s">'my-el'</span><span class="pl-kos">,</span><span class="pl-v">MyEl</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">&lt;/</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;</span><span class="pl-ent">my-el</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">my-el</span><span class="pl-kos">&gt;</span></pre></div>
<p>Also relevant: <a href="https://github.com/fserb/lit-dist">lit-dist</a>.</p>


path: python_fabric-ssh-key.md
topic: python
title: Using Fabric with an SSH public key
url: https://github.com/simonw/til/blob/main/python/fabric-ssh-key.md
created: 2021-10-06T11:22:47-07:00
created_utc: 2021-10-06T18:22:47+00:00
updated: 2021-10-06T11:22:47-07:00
updated_utc: 2021-10-06T18:22:47+00:00
shot_hash: cb177781f1ca7017f4caa724b9c9c1af
slug: fabric-ssh-key
body:
Inspired by [this tweet](https://twitter.com/driscollis/status/1445772718507376646) by Mike Driscoll I decided to try using Fabric to run commands over SSH from a Python script, using a public key for authentication.

This worked:

```python
from fabric import Connection

c = Connection(
    host="my-server.simonwillison.net",
    user="root",
    connect_kwargs={
        "key_filename": "/path/to/id_rsa.pub",
    },
)
output = c.run("pwd && ls -lah")
# This outputs to the console but also lets me retrieve the result like so:
string_output = output.stdout
```

Mike's Tweet recipe looks like this:
```python
with Connection(f"{username}@{host}:{port}", connect_kwargs={"password: "pw"}) as conn:
    output = conn.run("docker images")
    print(output.stdout)
```

html:
<p>Inspired by <a href="https://twitter.com/driscollis/status/1445772718507376646" rel="nofollow">this tweet</a> by Mike Driscoll I decided to try using Fabric to run commands over SSH from a Python script, using a public key for authentication.</p>
<p>This worked:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">fabric</span> <span class="pl-k">import</span> <span class="pl-v">Connection</span>

<span class="pl-s1">c</span> <span class="pl-c1">=</span> <span class="pl-v">Connection</span>(
    <span class="pl-s1">host</span><span class="pl-c1">=</span><span class="pl-s">"my-server.simonwillison.net"</span>,
    <span class="pl-s1">user</span><span class="pl-c1">=</span><span class="pl-s">"root"</span>,
    <span class="pl-s1">connect_kwargs</span><span class="pl-c1">=</span>{
        <span class="pl-s">"key_filename"</span>: <span class="pl-s">"/path/to/id_rsa.pub"</span>,
    },
)
<span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-s1">c</span>.<span class="pl-en">run</span>(<span class="pl-s">"pwd &amp;&amp; ls -lah"</span>)
<span class="pl-c"># This outputs to the console but also lets me retrieve the result like so:</span>
<span class="pl-s1">string_output</span> <span class="pl-c1">=</span> <span class="pl-s1">output</span>.<span class="pl-s1">stdout</span></pre></div>
<p>Mike's Tweet recipe looks like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">with</span> <span class="pl-v">Connection</span>(<span class="pl-s">f"<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">username</span><span class="pl-kos">}</span></span>@<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">host</span><span class="pl-kos">}</span></span>:<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">port</span><span class="pl-kos">}</span></span>"</span>, <span class="pl-s1">connect_kwargs</span><span class="pl-c1">=</span>{<span class="pl-s">"password: "</span><span class="pl-s1">pw</span>"}) <span class="pl-k">as</span> <span class="pl-s1">conn</span>:
    <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-s1">conn</span>.<span class="pl-en">run</span>(<span class="pl-s">"docker images"</span>)
    <span class="pl-en">print</span>(<span class="pl-s1">output</span>.<span class="pl-s1">stdout</span>)</pre></div>


path: python_sqlite-in-pyodide.md
topic: python
title: Using the sqlite3 Python module in Pyodide - Python WebAssembly
url: https://github.com/simonw/til/blob/main/python/sqlite-in-pyodide.md
created: 2021-10-18T11:04:21-07:00
created_utc: 2021-10-18T18:04:21+00:00
updated: 2021-10-18T11:40:37-07:00
updated_utc: 2021-10-18T18:40:37+00:00
shot_hash: 1f1d015671c98280512f2ae549d9389b
slug: sqlite-in-pyodide
body:
[Pyodide](https://github.com/pyodide/pyodide) provides "Python with the scientific stack, compiled to WebAssembly" - it's an incredible project which lets you run a full working Jupyter notebook, complete with complex packages such as numpy and pandas, entirely in your browser without any server-side Python component running at all.

It turns out it also now includes [a working version](https://github.com/pyodide/pyodide/issues/345) of the standard library `sqlite3` module, by bundling a WebAssembly compiled version of SQLite!

## Trying this in the REPL

[pyodide.org/en/stable/console.html](https://pyodide.org/en/stable/console.html) provides an interactive REPL for trying eut Pyodide. You can run a one-liner to demonstrate the available SQLite version like this:

```pycon
Welcome to the Pyodide terminal emulator 🐍
Python 3.9.5 (default, Sep 16 2021 11:22:45) on WebAssembly VM
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
>>> sqlite3.connect(":memory:").execute("select sqlite_version()").fetchall()
[('3.27.2',)]
```

## Querying an existing database file from JupyterLite

[JupyterLite](https://blog.jupyter.org/jupyterlite-jupyter-%EF%B8%8F-webassembly-%EF%B8%8F-python-f6e2e41ab3fa) is "a JupyterLab distribution that runs entirely in the web browser, backed by in-browser language kernels."

Their online demo is at [jupyterlite.github.io/demo/lab/index.html](https://jupyterlite.github.io/demo/lab/index.html). I opened that demo and created a new Pyolite notebook there, then used the bridge to the JavaScript `fetch()` function to download the 11MB [power plants database file](https://global-power-plants.datasettes.com/global-power-plants) from this URL:

`https://global-power-plants.datasettes.com/global-power-plants.db`

(Downloading this via `fetch()` works because Datasette includes CORS headers for these files.)

```python
from js import fetch

res = await fetch("https://global-power-plants.datasettes.com/global-power-plants.db")
buffer = await res.arrayBuffer()

# Now write that to the in-memory simulated filesystem:
open("tmp/power.db", "wb").write(bytes(buffer.valueOf().to_py()))

# And run some queries against it:
import sqlite3
c = sqlite3.connect("tmp/power.db")
c.execute('select * from "global-power-plants" limit 10').fetchall()
```
This works!

<img width="1546" alt="Screenshot of JupyterLite running my example code" src="https://user-images.githubusercontent.com/9599/137788071-c39bbdbb-7d8d-464e-9729-ad1c63bb8be8.png">

html:
<p><a href="https://github.com/pyodide/pyodide">Pyodide</a> provides "Python with the scientific stack, compiled to WebAssembly" - it's an incredible project which lets you run a full working Jupyter notebook, complete with complex packages such as numpy and pandas, entirely in your browser without any server-side Python component running at all.</p>
<p>It turns out it also now includes <a href="https://github.com/pyodide/pyodide/issues/345">a working version</a> of the standard library <code>sqlite3</code> module, by bundling a WebAssembly compiled version of SQLite!</p>
<h2>
<a id="user-content-trying-this-in-the-repl" class="anchor" href="#trying-this-in-the-repl" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Trying this in the REPL</h2>
<p><a href="https://pyodide.org/en/stable/console.html" rel="nofollow">pyodide.org/en/stable/console.html</a> provides an interactive REPL for trying eut Pyodide. You can run a one-liner to demonstrate the available SQLite version like this:</p>
<div class="highlight highlight-text-python-console"><pre>Welcome to the Pyodide terminal emulator 🐍
Python 3.9.5 (default, Sep 16 2021 11:22:45) on WebAssembly VM
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; <span class="pl-k">import</span> sqlite3
&gt;&gt;&gt; sqlite3.connect(<span class="pl-s"><span class="pl-pds">"</span>:memory:<span class="pl-pds">"</span></span>).execute(<span class="pl-s"><span class="pl-pds">"</span>select sqlite_version()<span class="pl-pds">"</span></span>).fetchall()
[('3.27.2',)]</pre></div>
<h2>
<a id="user-content-querying-an-existing-database-file-from-jupyterlite" class="anchor" href="#querying-an-existing-database-file-from-jupyterlite" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Querying an existing database file from JupyterLite</h2>
<p><a href="https://blog.jupyter.org/jupyterlite-jupyter-%EF%B8%8F-webassembly-%EF%B8%8F-python-f6e2e41ab3fa" rel="nofollow">JupyterLite</a> is "a JupyterLab distribution that runs entirely in the web browser, backed by in-browser language kernels."</p>
<p>Their online demo is at <a href="https://jupyterlite.github.io/demo/lab/index.html" rel="nofollow">jupyterlite.github.io/demo/lab/index.html</a>. I opened that demo and created a new Pyolite notebook there, then used the bridge to the JavaScript <code>fetch()</code> function to download the 11MB <a href="https://global-power-plants.datasettes.com/global-power-plants" rel="nofollow">power plants database file</a> from this URL:</p>
<p><code>https://global-power-plants.datasettes.com/global-power-plants.db</code></p>
<p>(Downloading this via <code>fetch()</code> works because Datasette includes CORS headers for these files.)</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">js</span> <span class="pl-k">import</span> <span class="pl-s1">fetch</span>

<span class="pl-s1">res</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-en">fetch</span>(<span class="pl-s">"https://global-power-plants.datasettes.com/global-power-plants.db"</span>)
<span class="pl-s1">buffer</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">res</span>.<span class="pl-en">arrayBuffer</span>()

<span class="pl-c"># Now write that to the in-memory simulated filesystem:</span>
<span class="pl-en">open</span>(<span class="pl-s">"tmp/power.db"</span>, <span class="pl-s">"wb"</span>).<span class="pl-en">write</span>(<span class="pl-en">bytes</span>(<span class="pl-s1">buffer</span>.<span class="pl-en">valueOf</span>().<span class="pl-en">to_py</span>()))

<span class="pl-c"># And run some queries against it:</span>
<span class="pl-k">import</span> <span class="pl-s1">sqlite3</span>
<span class="pl-s1">c</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite3</span>.<span class="pl-en">connect</span>(<span class="pl-s">"tmp/power.db"</span>)
<span class="pl-s1">c</span>.<span class="pl-en">execute</span>(<span class="pl-s">'select * from "global-power-plants" limit 10'</span>).<span class="pl-en">fetchall</span>()</pre></div>
<p>This works!</p>
<p><a href="https://user-images.githubusercontent.com/9599/137788071-c39bbdbb-7d8d-464e-9729-ad1c63bb8be8.png" target="_blank" rel="nofollow"><img width="1546" alt="Screenshot of JupyterLite running my example code" src="https://user-images.githubusercontent.com/9599/137788071-c39bbdbb-7d8d-464e-9729-ad1c63bb8be8.png" style="max-width:100%;"></a></p>


path: git_remove-commit-and-force-push.md
topic: git
title: Removing a git commit and force pushing to remove it from history
url: https://github.com/simonw/til/blob/main/git/remove-commit-and-force-push.md
created: 2021-10-22T12:25:57-07:00
created_utc: 2021-10-22T19:25:57+00:00
updated: 2021-10-22T12:25:57-07:00
updated_utc: 2021-10-22T19:25:57+00:00
shot_hash: 0885470afde1e0e022cfd0757da982a4
slug: remove-commit-and-force-push
body:
I accidentally triggered a commit which added a big chunk of unwanted data to my repository. I didn't want this to stick around in the history forever, and no-one else was pulling from the repo, so I decided to use force push to remove the rogue commit entirely.

I figured out the commit hash of the previous version that I wanted to restore and ran:

    git reset --hard 1909f93

Then I ran the force push like this:

    git push --force origin main

See https://github.com/simonw/sf-tree-history/issues/1

html:
<p>I accidentally triggered a commit which added a big chunk of unwanted data to my repository. I didn't want this to stick around in the history forever, and no-one else was pulling from the repo, so I decided to use force push to remove the rogue commit entirely.</p>
<p>I figured out the commit hash of the previous version that I wanted to restore and ran:</p>
<pre><code>git reset --hard 1909f93
</code></pre>
<p>Then I ran the force push like this:</p>
<pre><code>git push --force origin main
</code></pre>
<p>See <a href="https://github.com/simonw/sf-tree-history/issues/1">https://github.com/simonw/sf-tree-history/issues/1</a></p>


path: web-components_understanding-single-file-web-component.md
topic: web-components
title: Understanding Kristofer Joseph's Single File Web Component
url: https://github.com/simonw/til/blob/main/web-components/understanding-single-file-web-component.md
created: 2021-10-27T17:33:08-07:00
created_utc: 2021-10-28T00:33:08+00:00
updated: 2021-10-27T19:48:23-07:00
updated_utc: 2021-10-28T02:48:23+00:00
shot_hash: fe72e966d484aad94f52ae001a501d2c
slug: understanding-single-file-web-component
body:
[Via Brian LeRoux](https://twitter.com/brianleroux/status/1453472609518034944) I found [single-file-web-component.html](https://gist.github.com/kristoferjoseph/c4e47389ae0f0447db175b914e471628) by Kristofer Joseph. It's really clever! It demonstrates how to build a `<hello-world></hello-world>` custom Web Component in a single HTML file, using some neat tricks.

For my own education I spent some time picking it apart and built my own annotated version of the code showing what I learned.

Justin Fagnani [provided useful feedback](https://twitter.com/justinfagnani/status/1453531485240102916) on this on Twitter.

```html
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <title>Single File Web Component</title>
    <style>
      body {
        background-color: #eee;
        font-family: Helvetica, sans-serif;
      }
      h1 {
        color: blue;
        background-color: pink;
      }
    </style>
  </head>
  <body>
    <template id="single-file">
      <style>
        /*
          These styles affect only content inside the shadow DOM.

          Styles in the outside document mostly do not affect stuff
          here, but there are some exceptions: font-family affects
          this <h1> for example. I don't understand the rules here.
        */
        h1 {
          color: red;
        }
      </style>

      <h1>Hello world (web component)</h1>
      <!--
      This code still works if you remove the type="module" parameter.

      Using that parameter enables features such as 'import ... from'

      More importantly it stops variables inside the script tag from
      leaking out to the global scope - if you remove type="module"
      from here then the HelloWorld class becomes visible.
      -->
      <script type="module">
        class HelloWorld extends HTMLElement {
          constructor() {
            /*
              If you remove the call to super() here Firefox shows an error:
              "Uncaught ReferenceError: must call super constructor before
              using 'this' in derived class constructor'"
            */
            super();
            const template = document.getElementById("single-file");
            /*
              mode: 'open' means you are allowed to access
              document.querySelector('hello-world').shadowRoot to get
              at the DOM inside. Set to 'closed' and the .shadowRoot
              property will return null.
            */
            this.attachShadow({ mode: "open" }).appendChild(
              template.content.cloneNode(true)
            );
            /* 
              template.content is a 'DocumentFragment' array.

              template.content.cloneNode() without the true performs
              a shallow clone, which provides a empty DocumentFragment
              array.

              template.content.cloneNode(true) provides one with 6 items
            */
          }
          connectedCallback() {
            // https://developer.mozilla.org/en-US/docs/Web/Web_Components/Using_custom_elements#using_the_lifecycle_callbacks
            console.log("Why hello there 👋");
          }
        }
        customElements.define("hello-world", HelloWorld);
      </script>
    </template>

    <h1>This is not a web component</h1>

    <hello-world></hello-world>

    <script>
      const sf = document.getElementById("single-file");
      /*
        Before executing this line, sf.content.lastElementChild
        is the <script type="module"> element hidden away inside
        the <template> - we remove it from the template here and
        append it to the document.body, causing it to execute in
        the main document and activate the <hello-world> tag.
      */
      document.body.appendChild(sf.content.lastElementChild);
    </script>
  </body>
</html>
```

html:
<p><a href="https://twitter.com/brianleroux/status/1453472609518034944" rel="nofollow">Via Brian LeRoux</a> I found <a href="https://gist.github.com/kristoferjoseph/c4e47389ae0f0447db175b914e471628">single-file-web-component.html</a> by Kristofer Joseph. It's really clever! It demonstrates how to build a <code>&lt;hello-world&gt;&lt;/hello-world&gt;</code> custom Web Component in a single HTML file, using some neat tricks.</p>
<p>For my own education I spent some time picking it apart and built my own annotated version of the code showing what I learned.</p>
<p>Justin Fagnani <a href="https://twitter.com/justinfagnani/status/1453531485240102916" rel="nofollow">provided useful feedback</a> on this on Twitter.</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-c1">&lt;!DOCTYPE html<span class="pl-kos">&gt;</span></span>
<span class="pl-kos">&lt;</span><span class="pl-ent">html</span> <span class="pl-c1">lang</span>="<span class="pl-s">en</span>"<span class="pl-kos">&gt;</span>
  <span class="pl-kos">&lt;</span><span class="pl-ent">head</span><span class="pl-kos">&gt;</span>
    <span class="pl-kos">&lt;</span><span class="pl-ent">meta</span> <span class="pl-c1">charset</span>="<span class="pl-s">UTF-8</span>" /&gt;
    <span class="pl-kos">&lt;</span><span class="pl-ent">title</span><span class="pl-kos">&gt;</span>Single File Web Component<span class="pl-kos">&lt;/</span><span class="pl-ent">title</span><span class="pl-kos">&gt;</span>
    <span class="pl-kos">&lt;</span><span class="pl-ent">style</span><span class="pl-kos">&gt;</span>
      <span class="pl-ent">body</span> {
        <span class="pl-c1">background-color</span><span class="pl-kos">:</span> <span class="pl-pds"><span class="pl-kos">#</span>eee</span>;
        <span class="pl-c1">font-family</span><span class="pl-kos">:</span> Helvetica<span class="pl-kos">,</span> sans-serif;
      }
      <span class="pl-ent">h1</span> {
        <span class="pl-c1">color</span><span class="pl-kos">:</span> blue;
        <span class="pl-c1">background-color</span><span class="pl-kos">:</span> pink;
      }
    <span class="pl-kos">&lt;/</span><span class="pl-ent">style</span><span class="pl-kos">&gt;</span>
  <span class="pl-kos">&lt;/</span><span class="pl-ent">head</span><span class="pl-kos">&gt;</span>
  <span class="pl-kos">&lt;</span><span class="pl-ent">body</span><span class="pl-kos">&gt;</span>
    <span class="pl-kos">&lt;</span><span class="pl-ent">template</span> <span class="pl-c1">id</span>="<span class="pl-s">single-file</span>"<span class="pl-kos">&gt;</span>
      <span class="pl-kos">&lt;</span><span class="pl-ent">style</span><span class="pl-kos">&gt;</span>
        <span class="pl-c">/*</span>
<span class="pl-c">          These styles affect only content inside the shadow DOM.</span>
<span class="pl-c"></span>
<span class="pl-c">          Styles in the outside document mostly do not affect stuff</span>
<span class="pl-c">          here, but there are some exceptions: font-family affects</span>
<span class="pl-c">          this &lt;h1&gt; for example. I don't understand the rules here.</span>
<span class="pl-c">        */</span>
        <span class="pl-ent">h1</span> {
          <span class="pl-c1">color</span><span class="pl-kos">:</span> red;
        }
      <span class="pl-kos">&lt;/</span><span class="pl-ent">style</span><span class="pl-kos">&gt;</span>

      <span class="pl-kos">&lt;</span><span class="pl-ent">h1</span><span class="pl-kos">&gt;</span>Hello world (web component)<span class="pl-kos">&lt;/</span><span class="pl-ent">h1</span><span class="pl-kos">&gt;</span>
      <span class="pl-c">&lt;!--</span>
<span class="pl-c">      This code still works if you remove the type="module" parameter.</span>
<span class="pl-c"></span>
<span class="pl-c">      Using that parameter enables features such as 'import ... from'</span>
<span class="pl-c"></span>
<span class="pl-c">      More importantly it stops variables inside the script tag from</span>
<span class="pl-c">      leaking out to the global scope - if you remove type="module"</span>
<span class="pl-c">      from here then the HelloWorld class becomes visible.</span>
<span class="pl-c">      --&gt;</span>
      <span class="pl-kos">&lt;</span><span class="pl-ent">script</span> <span class="pl-c1">type</span>="<span class="pl-s">module</span>"<span class="pl-kos">&gt;</span>
        <span class="pl-k">class</span> <span class="pl-v">HelloWorld</span> <span class="pl-k">extends</span> <span class="pl-v">HTMLElement</span> <span class="pl-kos">{</span>
          <span class="pl-en">constructor</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
            <span class="pl-c">/*</span>
<span class="pl-c">              If you remove the call to super() here Firefox shows an error:</span>
<span class="pl-c">              "Uncaught ReferenceError: must call super constructor before</span>
<span class="pl-c">              using 'this' in derived class constructor'"</span>
<span class="pl-c">            */</span>
            <span class="pl-smi">super</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
            <span class="pl-k">const</span> <span class="pl-s1">template</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">getElementById</span><span class="pl-kos">(</span><span class="pl-s">"single-file"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
            <span class="pl-c">/*</span>
<span class="pl-c">              mode: 'open' means you are allowed to access</span>
<span class="pl-c">              document.querySelector('hello-world').shadowRoot to get</span>
<span class="pl-c">              at the DOM inside. Set to 'closed' and the .shadowRoot</span>
<span class="pl-c">              property will return null.</span>
<span class="pl-c">            */</span>
            <span class="pl-smi">this</span><span class="pl-kos">.</span><span class="pl-en">attachShadow</span><span class="pl-kos">(</span><span class="pl-kos">{</span> <span class="pl-c1">mode</span>: <span class="pl-s">"open"</span> <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">appendChild</span><span class="pl-kos">(</span>
              <span class="pl-s1">template</span><span class="pl-kos">.</span><span class="pl-c1">content</span><span class="pl-kos">.</span><span class="pl-en">cloneNode</span><span class="pl-kos">(</span><span class="pl-c1">true</span><span class="pl-kos">)</span>
            <span class="pl-kos">)</span><span class="pl-kos">;</span>
            <span class="pl-c">/* </span>
<span class="pl-c">              template.content is a 'DocumentFragment' array.</span>
<span class="pl-c"></span>
<span class="pl-c">              template.content.cloneNode() without the true performs</span>
<span class="pl-c">              a shallow clone, which provides a empty DocumentFragment</span>
<span class="pl-c">              array.</span>
<span class="pl-c"></span>
<span class="pl-c">              template.content.cloneNode(true) provides one with 6 items</span>
<span class="pl-c">            */</span>
          <span class="pl-kos">}</span>
          <span class="pl-en">connectedCallback</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
            <span class="pl-c">// https://developer.mozilla.org/en-US/docs/Web/Web_Components/Using_custom_elements#using_the_lifecycle_callbacks</span>
            <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s">"Why hello there 👋"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
          <span class="pl-kos">}</span>
        <span class="pl-kos">}</span>
        <span class="pl-s1">customElements</span><span class="pl-kos">.</span><span class="pl-en">define</span><span class="pl-kos">(</span><span class="pl-s">"hello-world"</span><span class="pl-kos">,</span> <span class="pl-v">HelloWorld</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-kos">&lt;/</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span>
    <span class="pl-kos">&lt;/</span><span class="pl-ent">template</span><span class="pl-kos">&gt;</span>

    <span class="pl-kos">&lt;</span><span class="pl-ent">h1</span><span class="pl-kos">&gt;</span>This is not a web component<span class="pl-kos">&lt;/</span><span class="pl-ent">h1</span><span class="pl-kos">&gt;</span>

    <span class="pl-kos">&lt;</span><span class="pl-ent">hello-world</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">hello-world</span><span class="pl-kos">&gt;</span>

    <span class="pl-kos">&lt;</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span>
      <span class="pl-k">const</span> <span class="pl-s1">sf</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">getElementById</span><span class="pl-kos">(</span><span class="pl-s">"single-file"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
      <span class="pl-c">/*</span>
<span class="pl-c">        Before executing this line, sf.content.lastElementChild</span>
<span class="pl-c">        is the &lt;script type="module"&gt; element hidden away inside</span>
<span class="pl-c">        the &lt;template&gt; - we remove it from the template here and</span>
<span class="pl-c">        append it to the document.body, causing it to execute in</span>
<span class="pl-c">        the main document and activate the &lt;hello-world&gt; tag.</span>
<span class="pl-c">      */</span>
      <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-c1">body</span><span class="pl-kos">.</span><span class="pl-en">appendChild</span><span class="pl-kos">(</span><span class="pl-s1">sf</span><span class="pl-kos">.</span><span class="pl-c1">content</span><span class="pl-kos">.</span><span class="pl-c1">lastElementChild</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-kos">&lt;/</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span>
  <span class="pl-kos">&lt;/</span><span class="pl-ent">body</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;/</span><span class="pl-ent">html</span><span class="pl-kos">&gt;</span></pre></div>


path: pytest_pytest-recording-vcr.md
topic: pytest
title: Using VCR and pytest with pytest-recording
url: https://github.com/simonw/til/blob/main/pytest/pytest-recording-vcr.md
created: 2021-11-02T17:51:51-07:00
created_utc: 2021-11-03T00:51:51+00:00
updated: 2021-11-02T17:51:51-07:00
updated_utc: 2021-11-03T00:51:51+00:00
shot_hash: 245dd34079c6789b4d4af17061a25f77
slug: pytest-recording-vcr
body:
[pytest-recording](https://github.com/kiwicom/pytest-recording) is a neat pytest plugin that makes it easy to use the [VCR library](https://vcrpy.readthedocs.io/), which helps write tests against HTTP resources by automatically capturing responses and baking them into a YAML file to be replayed during the tests.

It even works with [boto3](https://aws.amazon.com/sdk-for-python/)!

To use it, first install it with `pip install pytest-recording` and then add the `@pytest.mark.vcr` decorator to a test that makes HTTP calls:

```python
@pytest.mark.vcr
def test_create():
    runner = CliRunner()
    with runner.isolated_filesystem():
        result = runner.invoke(cli, ["create", "pytest-bucket-simonw-1", "-c"])
        assert result.exit_code == 0
```

The first time you run the tests, use the `--record-mode=once` option:

    pytest -k test_create --record-mode=once

This defaults to creating a YAML file in `tests/cassettes/test_s3_credentials/test_create.yaml`.

Subsequent runs of `pytest -k test_create` will reuse those recorded HTTP requests and will not make any network requests - I confirmed this by turning off my laptop's WiFi.

html:
<p><a href="https://github.com/kiwicom/pytest-recording">pytest-recording</a> is a neat pytest plugin that makes it easy to use the <a href="https://vcrpy.readthedocs.io/" rel="nofollow">VCR library</a>, which helps write tests against HTTP resources by automatically capturing responses and baking them into a YAML file to be replayed during the tests.</p>
<p>It even works with <a href="https://aws.amazon.com/sdk-for-python/" rel="nofollow">boto3</a>!</p>
<p>To use it, first install it with <code>pip install pytest-recording</code> and then add the <code>@pytest.mark.vcr</code> decorator to a test that makes HTTP calls:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-s1">mark</span>.<span class="pl-s1">vcr</span></span>
<span class="pl-k">def</span> <span class="pl-en">test_create</span>():
    <span class="pl-s1">runner</span> <span class="pl-c1">=</span> <span class="pl-v">CliRunner</span>()
    <span class="pl-k">with</span> <span class="pl-s1">runner</span>.<span class="pl-en">isolated_filesystem</span>():
        <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-s1">runner</span>.<span class="pl-en">invoke</span>(<span class="pl-s1">cli</span>, [<span class="pl-s">"create"</span>, <span class="pl-s">"pytest-bucket-simonw-1"</span>, <span class="pl-s">"-c"</span>])
        <span class="pl-k">assert</span> <span class="pl-s1">result</span>.<span class="pl-s1">exit_code</span> <span class="pl-c1">==</span> <span class="pl-c1">0</span></pre></div>
<p>The first time you run the tests, use the <code>--record-mode=once</code> option:</p>
<pre><code>pytest -k test_create --record-mode=once
</code></pre>
<p>This defaults to creating a YAML file in <code>tests/cassettes/test_s3_credentials/test_create.yaml</code>.</p>
<p>Subsequent runs of <code>pytest -k test_create</code> will reuse those recorded HTTP requests and will not make any network requests - I confirmed this by turning off my laptop's WiFi.</p>


path: pytest_pytest-mock-calls.md
topic: pytest
title: Quick and dirty mock testing with mock_calls
url: https://github.com/simonw/til/blob/main/pytest/pytest-mock-calls.md
created: 2021-11-02T18:35:01-07:00
created_utc: 2021-11-03T01:35:01+00:00
updated: 2022-06-29T15:00:48-07:00
updated_utc: 2022-06-29T22:00:48+00:00
shot_hash: 6cbe835b12510df4bce1905420230b32
slug: pytest-mock-calls
body:
I needed to write a test that checked for a really complex sequence of mock calls for [s3-credentials#3](https://github.com/simonw/s3-credentials/issues/3).

I ended up using the following trick, using [pytest-mock](https://pypi.org/project/pytest-mock/):

```python
def test_create(mocker):
    boto3 = mocker.patch("boto3.client")
    runner = CliRunner()
    with runner.isolated_filesystem():
        result = runner.invoke(cli, ["create", "pytest-bucket-simonw-1", "-c"])
        assert [str(c) for c in boto3.mock_calls] == [
            "call('s3')",
            "call('iam')",
            "call().head_bucket(Bucket='pytest-bucket-simonw-1')",
            "call().get_user(UserName='s3.read-write.pytest-bucket-simonw-1')",
            'call().put_user_policy(PolicyDocument=\'{"Version": "2012-10-17", "Statement": [{"Sid": "ListObjectsInBucket", "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::pytest-bucket-simonw-1"]}, {"Sid": "AllObjectActions", "Effect": "Allow", "Action": "s3:*Object", "Resource": ["arn:aws:s3:::pytest-bucket-simonw-1/*"]}]}\', PolicyName=\'s3.read-write.pytest-bucket-simonw-1\', UserName=\'s3.read-write.pytest-bucket-simonw-1\')',
            "call().create_access_key(UserName='s3.read-write.pytest-bucket-simonw-1')",
            "call().create_access_key().__getitem__('AccessKey')",
            "call().create_access_key().__getitem__().__str__()",
        ]
```
I used the trick I describe in [How to cheat at unit tests with pytest and Black](https://simonwillison.net/2020/Feb/11/cheating-at-unit-tests-pytest-black/) where I run that comparison against an empty `[]` list, then use `pytest --pdb` to drop into a debugger and copy and paste the output of `[str(c) for c in boto3.mock_calls]` into my test code.

Initially I used a comparison directly against `boto3.mock_calls` - but this threw a surprising error. The calls sequence I baked into my tests looked like this:

```python
from unittest.mock import call

# ...

        assert boto3.mock_calls == [
            call("s3"),
            call("iam"),
            call().head_bucket(Bucket="pytest-bucket-simonw-1"),
            call().get_user(UserName="s3.read-write.pytest-bucket-simonw-1"),
            call().put_user_policy(
                PolicyDocument='{"Version": "2012-10-17", "Statement": [{"Sid": "ListObjectsInBucket", "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::pytest-bucket-simonw-1"]}, {"Sid": "AllObjectActions", "Effect": "Allow", "Action": "s3:*Object", "Resource": ["arn:aws:s3:::pytest-bucket-simonw-1/*"]}]}',
                PolicyName="s3.read-write.pytest-bucket-simonw-1",
                UserName="s3.read-write.pytest-bucket-simonw-1",
            ),
            call().create_access_key(UserName="s3.read-write.pytest-bucket-simonw-1"),
            call().create_access_key().__getitem__("AccessKey"),
            call().create_access_key().__getitem__().__str__(),
        ]
```
But when I ran `pytest` that last one failed:
```
E             -  'call().create_access_key().__getitem__()',
E             ?  -                                        ^
E             +  call().create_access_key().__getitem__().__str__(),
E             ?                                          ^^^^^^^^^^
```
It turns out `__str__()` calls do not play well with the `call()` constructor - see [this StackOverflow question](https://stackoverflow.com/questions/61926147/how-to-represent-unittest-mock-call-str).

My solution was to cast them all to `str()` using a list comprehension, which ended up fixing that problem.

## Gotcha: parameter ordering

There's one major flaw to the `str()` trick I'm using here: the order in which parameters are displayed in the string representation of `call()` may differ between Python versions. I had to undo this trick in one place I was using it ([see here](https://github.com/simonw/s3-credentials/issues/8)) as a result due to the following test failure:

```
E  At index 4 diff:
     "call().get_user_policy(PolicyName='policy-one', UserName='one')"
  != "call().get_user_policy(UserName='one', PolicyName='policy-one')"
```

html:
<p>I needed to write a test that checked for a really complex sequence of mock calls for <a href="https://github.com/simonw/s3-credentials/issues/3">s3-credentials#3</a>.</p>
<p>I ended up using the following trick, using <a href="https://pypi.org/project/pytest-mock/" rel="nofollow">pytest-mock</a>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">test_create</span>(<span class="pl-s1">mocker</span>):
    <span class="pl-s1">boto3</span> <span class="pl-c1">=</span> <span class="pl-s1">mocker</span>.<span class="pl-en">patch</span>(<span class="pl-s">"boto3.client"</span>)
    <span class="pl-s1">runner</span> <span class="pl-c1">=</span> <span class="pl-v">CliRunner</span>()
    <span class="pl-k">with</span> <span class="pl-s1">runner</span>.<span class="pl-en">isolated_filesystem</span>():
        <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-s1">runner</span>.<span class="pl-en">invoke</span>(<span class="pl-s1">cli</span>, [<span class="pl-s">"create"</span>, <span class="pl-s">"pytest-bucket-simonw-1"</span>, <span class="pl-s">"-c"</span>])
        <span class="pl-k">assert</span> [<span class="pl-en">str</span>(<span class="pl-s1">c</span>) <span class="pl-k">for</span> <span class="pl-s1">c</span> <span class="pl-c1">in</span> <span class="pl-s1">boto3</span>.<span class="pl-s1">mock_calls</span>] <span class="pl-c1">==</span> [
            <span class="pl-s">"call('s3')"</span>,
            <span class="pl-s">"call('iam')"</span>,
            <span class="pl-s">"call().head_bucket(Bucket='pytest-bucket-simonw-1')"</span>,
            <span class="pl-s">"call().get_user(UserName='s3.read-write.pytest-bucket-simonw-1')"</span>,
            <span class="pl-s">'call().put_user_policy(PolicyDocument=<span class="pl-cce">\'</span>{"Version": "2012-10-17", "Statement": [{"Sid": "ListObjectsInBucket", "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::pytest-bucket-simonw-1"]}, {"Sid": "AllObjectActions", "Effect": "Allow", "Action": "s3:*Object", "Resource": ["arn:aws:s3:::pytest-bucket-simonw-1/*"]}]}<span class="pl-cce">\'</span>, PolicyName=<span class="pl-cce">\'</span>s3.read-write.pytest-bucket-simonw-1<span class="pl-cce">\'</span>, UserName=<span class="pl-cce">\'</span>s3.read-write.pytest-bucket-simonw-1<span class="pl-cce">\'</span>)'</span>,
            <span class="pl-s">"call().create_access_key(UserName='s3.read-write.pytest-bucket-simonw-1')"</span>,
            <span class="pl-s">"call().create_access_key().__getitem__('AccessKey')"</span>,
            <span class="pl-s">"call().create_access_key().__getitem__().__str__()"</span>,
        ]</pre></div>
<p>I used the trick I describe in <a href="https://simonwillison.net/2020/Feb/11/cheating-at-unit-tests-pytest-black/" rel="nofollow">How to cheat at unit tests with pytest and Black</a> where I run that comparison against an empty <code>[]</code> list, then use <code>pytest --pdb</code> to drop into a debugger and copy and paste the output of <code>[str(c) for c in boto3.mock_calls]</code> into my test code.</p>
<p>Initially I used a comparison directly against <code>boto3.mock_calls</code> - but this threw a surprising error. The calls sequence I baked into my tests looked like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">unittest</span>.<span class="pl-s1">mock</span> <span class="pl-k">import</span> <span class="pl-s1">call</span>

<span class="pl-c"># ...</span>

        <span class="pl-k">assert</span> <span class="pl-s1">boto3</span>.<span class="pl-s1">mock_calls</span> <span class="pl-c1">==</span> [
            <span class="pl-en">call</span>(<span class="pl-s">"s3"</span>),
            <span class="pl-en">call</span>(<span class="pl-s">"iam"</span>),
            <span class="pl-en">call</span>().<span class="pl-en">head_bucket</span>(<span class="pl-v">Bucket</span><span class="pl-c1">=</span><span class="pl-s">"pytest-bucket-simonw-1"</span>),
            <span class="pl-en">call</span>().<span class="pl-en">get_user</span>(<span class="pl-v">UserName</span><span class="pl-c1">=</span><span class="pl-s">"s3.read-write.pytest-bucket-simonw-1"</span>),
            <span class="pl-en">call</span>().<span class="pl-en">put_user_policy</span>(
                <span class="pl-v">PolicyDocument</span><span class="pl-c1">=</span><span class="pl-s">'{"Version": "2012-10-17", "Statement": [{"Sid": "ListObjectsInBucket", "Effect": "Allow", "Action": ["s3:ListBucket"], "Resource": ["arn:aws:s3:::pytest-bucket-simonw-1"]}, {"Sid": "AllObjectActions", "Effect": "Allow", "Action": "s3:*Object", "Resource": ["arn:aws:s3:::pytest-bucket-simonw-1/*"]}]}'</span>,
                <span class="pl-v">PolicyName</span><span class="pl-c1">=</span><span class="pl-s">"s3.read-write.pytest-bucket-simonw-1"</span>,
                <span class="pl-v">UserName</span><span class="pl-c1">=</span><span class="pl-s">"s3.read-write.pytest-bucket-simonw-1"</span>,
            ),
            <span class="pl-en">call</span>().<span class="pl-en">create_access_key</span>(<span class="pl-v">UserName</span><span class="pl-c1">=</span><span class="pl-s">"s3.read-write.pytest-bucket-simonw-1"</span>),
            <span class="pl-en">call</span>().<span class="pl-en">create_access_key</span>().<span class="pl-en">__getitem__</span>(<span class="pl-s">"AccessKey"</span>),
            <span class="pl-en">call</span>().<span class="pl-en">create_access_key</span>().<span class="pl-en">__getitem__</span>().<span class="pl-en">__str__</span>(),
        ]</pre></div>
<p>But when I ran <code>pytest</code> that last one failed:</p>
<pre><code>E             -  'call().create_access_key().__getitem__()',
E             ?  -                                        ^
E             +  call().create_access_key().__getitem__().__str__(),
E             ?                                          ^^^^^^^^^^
</code></pre>
<p>It turns out <code>__str__()</code> calls do not play well with the <code>call()</code> constructor - see <a href="https://stackoverflow.com/questions/61926147/how-to-represent-unittest-mock-call-str" rel="nofollow">this StackOverflow question</a>.</p>
<p>My solution was to cast them all to <code>str()</code> using a list comprehension, which ended up fixing that problem.</p>
<h2>
<a id="user-content-gotcha-parameter-ordering" class="anchor" href="#gotcha-parameter-ordering" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Gotcha: parameter ordering</h2>
<p>There's one major flaw to the <code>str()</code> trick I'm using here: the order in which parameters are displayed in the string representation of <code>call()</code> may differ between Python versions. I had to undo this trick in one place I was using it (<a href="https://github.com/simonw/s3-credentials/issues/8">see here</a>) as a result due to the following test failure:</p>
<pre><code>E  At index 4 diff:
     "call().get_user_policy(PolicyName='policy-one', UserName='one')"
  != "call().get_user_policy(UserName='one', PolicyName='policy-one')"
</code></pre>


path: kubernetes_basic-datasette-in-kubernetes.md
topic: kubernetes
title: Basic Datasette in Kubernetes
url: https://github.com/simonw/til/blob/main/kubernetes/basic-datasette-in-kubernetes.md
created: 2021-11-05T13:20:01-07:00
created_utc: 2021-11-05T20:20:01+00:00
updated: 2021-11-05T13:20:01-07:00
updated_utc: 2021-11-05T20:20:01+00:00
shot_hash: 9f478cfc8ae7a09088042a73cdb3a89d
slug: basic-datasette-in-kubernetes
body:
This recipe for deploying the official `datasetteproject/datasette` container in Kubernetes just worked for me. It uses an interesting (possibly nasty?) trick to install plugins and download a SQLite database file on container startup, without needing to bake a brand new container image.

I'm running 2 replicas, to experiment with the cluster.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: datasette-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: datasette
  template:
    metadata:
      labels:
        app: datasette
    spec:
      containers:
      - name: datasette
        image: datasetteproject/datasette
        command:
        - sh
        - -c
        args:
        - |-
          # Install some plugins
          pip install \
            datasette-debug-asgi \
            datasette-cluster-map \
            datasette-psutil
          # Download a DB (using Python because curl/wget are not available)
          python -c 'import urllib.request; urllib.request.urlretrieve("https://global-power-plants.datasettes.com/global-power-plants.db", "/home/global-power-plants.db")'
          # Start Datasette, on 0.0.0.0 to allow external traffic
          datasette -h 0.0.0.0 /home/global-power-plants.db
        ports:
        - containerPort: 8001
          protocol: TCP
```
The devious trick here is to use `command` and `args` to specify a multi-line shell script to run on startup, which installs some plugins with `pip install`, then downloads a database file using `urllib.request.urlretrieve()` in Python (because `wget` and `curl` are not included in the official image), then starts Datasette against that downloaded file.

html:
<p>This recipe for deploying the official <code>datasetteproject/datasette</code> container in Kubernetes just worked for me. It uses an interesting (possibly nasty?) trick to install plugins and download a SQLite database file on container startup, without needing to bake a brand new container image.</p>
<p>I'm running 2 replicas, to experiment with the cluster.</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">apiVersion</span>: <span class="pl-s">apps/v1</span>
<span class="pl-ent">kind</span>: <span class="pl-s">Deployment</span>
<span class="pl-ent">metadata</span>:
  <span class="pl-ent">name</span>: <span class="pl-s">datasette-deployment</span>
<span class="pl-ent">spec</span>:
  <span class="pl-ent">replicas</span>: <span class="pl-c1">2</span>
  <span class="pl-ent">selector</span>:
    <span class="pl-ent">matchLabels</span>:
      <span class="pl-ent">app</span>: <span class="pl-s">datasette</span>
  <span class="pl-ent">template</span>:
    <span class="pl-ent">metadata</span>:
      <span class="pl-ent">labels</span>:
        <span class="pl-ent">app</span>: <span class="pl-s">datasette</span>
    <span class="pl-ent">spec</span>:
      <span class="pl-ent">containers</span>:
      - <span class="pl-ent">name</span>: <span class="pl-s">datasette</span>
        <span class="pl-ent">image</span>: <span class="pl-s">datasetteproject/datasette</span>
        <span class="pl-ent">command</span>:
        - <span class="pl-s">sh</span>
        - <span class="pl-s">-c</span>
        <span class="pl-ent">args</span>:
        - <span class="pl-s">|-</span>
<span class="pl-s">          # Install some plugins</span>
<span class="pl-s">          pip install \</span>
<span class="pl-s">            datasette-debug-asgi \</span>
<span class="pl-s">            datasette-cluster-map \</span>
<span class="pl-s">            datasette-psutil</span>
<span class="pl-s">          # Download a DB (using Python because curl/wget are not available)</span>
<span class="pl-s">          python -c 'import urllib.request; urllib.request.urlretrieve("https://global-power-plants.datasettes.com/global-power-plants.db", "/home/global-power-plants.db")'</span>
<span class="pl-s">          # Start Datasette, on 0.0.0.0 to allow external traffic</span>
<span class="pl-s">          datasette -h 0.0.0.0 /home/global-power-plants.db</span>
<span class="pl-s"></span>        <span class="pl-ent">ports</span>:
        - <span class="pl-ent">containerPort</span>: <span class="pl-c1">8001</span>
          <span class="pl-ent">protocol</span>: <span class="pl-s">TCP</span></pre></div>
<p>The devious trick here is to use <code>command</code> and <code>args</code> to specify a multi-line shell script to run on startup, which installs some plugins with <code>pip install</code>, then downloads a database file using <code>urllib.request.urlretrieve()</code> in Python (because <code>wget</code> and <code>curl</code> are not included in the official image), then starts Datasette against that downloaded file.</p>


path: deno_annotated-deno-deploy-demo.md
topic: deno
title: Annotated code for a demo of WebSocket chat in Deno Deploy
url: https://github.com/simonw/til/blob/main/deno/annotated-deno-deploy-demo.md
created: 2021-11-06T18:34:17-07:00
created_utc: 2021-11-07T01:34:17+00:00
updated: 2021-11-07T09:01:47-08:00
updated_utc: 2021-11-07T17:01:47+00:00
shot_hash: cd72f542e30595301089ca728b6be770
slug: annotated-deno-deploy-demo
body:
Deno Deploy is a hosted Deno service that promises [a multi-tenant JavaScript engine running in 25 data centers across the world](https://deno.com/blog/deploy-beta1/).

Today [this demo](https://dash.deno.com/playground/mini-ws-chat) by [Ondřej Žára](https://twitter.com/0ndras/status/1457027832404713479) showed up [on Hacker News](https://news.ycombinator.com/item?id=29131751), which implements "a multi-datacenter chat, client+server in 23 lines of TS".

Here's my annotated copy of the code, which I wrote while figuring out how it works.

```typescript
// listenAndServe is the Deno standard mechanism for creating an HTTP server
// https://deno.land/manual/examples/http_server#using-the-codestdhttpcode-library
import { listenAndServe } from "https://deno.land/std/http/server.ts"

// Set of all of the currently open WebSocket connections from browsers
const sockets = new Set<WebSocket>(),
/*
BroadcastChannel is a concept that is unique to the Deno Deploy environment.

https://deno.com/deploy/docs/runtime-broadcast-channel/

It is modelled after the browser API of the same name.

It sets up a channel between ALL instances of the server-side script running
in every one of the Deno Deploy global network of data centers.

The argument is the name of the channel, which apparently can be an empty string.
*/
    channel = new BroadcastChannel(""),
    headers = {"Content-type": "text/html"},
/*
This is the bare-bones HTML for the browser side of the application

It creates a WebSocket connection back to the host, and sets it up so any
message that arrives via that WebSocket will be appended to the textContent
of the pre element on the page.

The input element has an onkeyup that checks for the Enter key and sends
the value of that element over the WebSocket channel to the server.
*/
    html = `<script>let ws = new WebSocket("wss://"+location.host)
ws.onmessage = e => pre.textContent += e.data+"\\n"</script>
<input onkeyup="event.key=='Enter'&&ws.send(this.value)"><pre id=pre>`

/*
This bit does the broadcast work: any time a message is received from the
BroadcastChannel it is forwarded on to every single one of the currently
attached WebSocket connections, using the data in that "sockets" set.

Additionally, this covers the case of messages coming from a client connected
to THIS instance - these are also sent to the channel (see code below), but
here it spots that the message event's e.target is NOT the current instance
and sends the message to that channel so it broadcast to the other data centers.
*/
channel.onmessage = e => {
    (e.target != channel) && channel.postMessage(e.data)
    sockets.forEach(s => s.send(e.data))
}

/*
I tried removing the await here and the demo still worked.

But https://deno.land/std@0.113.0/http/server.ts#L224 shows that this function
is indeed an async that returns a Promise.
*/
await listenAndServe(":8080", (r: Request) => {
    try {
        /*
        Deno.upgradeWebSocket is a relatively new feature, added in Deno v1.12
        in July 2021:
        https://deno.com/blog/v1.12#server-side-websocket-support-in-native-http
        
        It gives you back a response that you should return to the client in order
        to finish establishing the WebSocket connection, and a socket object which
        you can then use for further WebSocket communication.
        */
        const { socket, response } = Deno.upgradeWebSocket(r)
        // Add it to the set so we can send to all of them later
        sockets.add(socket)
        /*
        This is a sneaky hack: when a message arrives from the WebSocket we pass it
        directly to the BroadcastChannel - then use the e.target != channel check
        above to broadcast it on to every other global instance.
        */
        socket.onmessage = channel.onmessage
        // When browser disconnects, remove the socket from the set of sockets
        socket.onclose = _ => sockets.delete(socket)
        return response
    } catch {
    /*
    I added code here to catch(e) and display e.toString() which showed me
    that the exception caught here is:

      exception: TypeError: Invalid Header: 'upgrade' header must be 'websocket'

    This is an exception thrown by Deno.upgradeWebSocket(r) if the incoming
    request does not include the "upgrade: websocket" HTTP header, which
    is added by browsers when using new WebSocket("wss://...")
    
    So here we return the HTML and headers for the application itself.
    */
    return new Response(html, {headers})
  }
})
```

Relevant links:

- [Deno listenAndServe documentation](https://deno.land/manual/examples/http_server#using-the-codestdhttpcode-library)
- [Deno Deploy BroadcastChannel documentation](https://deno.com/deploy/docs/runtime-broadcast-channel/)
- [MDN documentation of the related BroadcastChannel browser API](https://developer.mozilla.org/en-US/docs/Web/API/Broadcast_Channel_API)

html:
<p>Deno Deploy is a hosted Deno service that promises <a href="https://deno.com/blog/deploy-beta1/" rel="nofollow">a multi-tenant JavaScript engine running in 25 data centers across the world</a>.</p>
<p>Today <a href="https://dash.deno.com/playground/mini-ws-chat" rel="nofollow">this demo</a> by <a href="https://twitter.com/0ndras/status/1457027832404713479" rel="nofollow">Ondřej Žára</a> showed up <a href="https://news.ycombinator.com/item?id=29131751" rel="nofollow">on Hacker News</a>, which implements "a multi-datacenter chat, client+server in 23 lines of TS".</p>
<p>Here's my annotated copy of the code, which I wrote while figuring out how it works.</p>
<div class="highlight highlight-source-ts"><pre><span class="pl-c">// listenAndServe is the Deno standard mechanism for creating an HTTP server</span>
<span class="pl-c">// https://deno.land/manual/examples/http_server#using-the-codestdhttpcode-library</span>
<span class="pl-k">import</span> <span class="pl-kos">{</span> <span class="pl-s1">listenAndServe</span> <span class="pl-kos">}</span> <span class="pl-k">from</span> <span class="pl-s">"https://deno.land/std/http/server.ts"</span>

<span class="pl-c">// Set of all of the currently open WebSocket connections from browsers</span>
<span class="pl-k">const</span> <span class="pl-s1">sockets</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-smi">Set</span><span class="pl-kos">&lt;</span><span class="pl-smi">WebSocket</span><span class="pl-kos">&gt;</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">,</span>
<span class="pl-c">/*</span>
<span class="pl-c">BroadcastChannel is a concept that is unique to the Deno Deploy environment.</span>
<span class="pl-c"></span>
<span class="pl-c">https://deno.com/deploy/docs/runtime-broadcast-channel/</span>
<span class="pl-c"></span>
<span class="pl-c">It is modelled after the browser API of the same name.</span>
<span class="pl-c"></span>
<span class="pl-c">It sets up a channel between ALL instances of the server-side script running</span>
<span class="pl-c">in every one of the Deno Deploy global network of data centers.</span>
<span class="pl-c"></span>
<span class="pl-c">The argument is the name of the channel, which apparently can be an empty string.</span>
<span class="pl-c">*/</span>
    <span class="pl-s1">channel</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-smi">BroadcastChannel</span><span class="pl-kos">(</span><span class="pl-s">""</span><span class="pl-kos">)</span><span class="pl-kos">,</span>
    <span class="pl-s1">headers</span> <span class="pl-c1">=</span> <span class="pl-kos">{</span><span class="pl-s">"Content-type"</span>: <span class="pl-s">"text/html"</span><span class="pl-kos">}</span><span class="pl-kos">,</span>
<span class="pl-c">/*</span>
<span class="pl-c">This is the bare-bones HTML for the browser side of the application</span>
<span class="pl-c"></span>
<span class="pl-c">It creates a WebSocket connection back to the host, and sets it up so any</span>
<span class="pl-c">message that arrives via that WebSocket will be appended to the textContent</span>
<span class="pl-c">of the pre element on the page.</span>
<span class="pl-c"></span>
<span class="pl-c">The input element has an onkeyup that checks for the Enter key and sends</span>
<span class="pl-c">the value of that element over the WebSocket channel to the server.</span>
<span class="pl-c">*/</span>
    <span class="pl-s1">html</span> <span class="pl-c1">=</span> <span class="pl-s">`&lt;script&gt;let ws = new WebSocket("wss://"+location.host)</span>
<span class="pl-s">ws.onmessage = e =&gt; pre.textContent += e.data+"\\n"&lt;/script&gt;</span>
<span class="pl-s">&lt;input onkeyup="event.key=='Enter'&amp;&amp;ws.send(this.value)"&gt;&lt;pre id=pre&gt;`</span>

<span class="pl-c">/*</span>
<span class="pl-c">This bit does the broadcast work: any time a message is received from the</span>
<span class="pl-c">BroadcastChannel it is forwarded on to every single one of the currently</span>
<span class="pl-c">attached WebSocket connections, using the data in that "sockets" set.</span>
<span class="pl-c"></span>
<span class="pl-c">Additionally, this covers the case of messages coming from a client connected</span>
<span class="pl-c">to THIS instance - these are also sent to the channel (see code below), but</span>
<span class="pl-c">here it spots that the message event's e.target is NOT the current instance</span>
<span class="pl-c">and sends the message to that channel so it broadcast to the other data centers.</span>
<span class="pl-c">*/</span>
<span class="pl-s1">channel</span><span class="pl-kos">.</span><span class="pl-en">onmessage</span> <span class="pl-c1">=</span> <span class="pl-s1">e</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-kos">(</span><span class="pl-s1">e</span><span class="pl-kos">.</span><span class="pl-c1">target</span> <span class="pl-c1">!=</span> <span class="pl-s1">channel</span><span class="pl-kos">)</span> <span class="pl-c1">&amp;&amp;</span> <span class="pl-s1">channel</span><span class="pl-kos">.</span><span class="pl-en">postMessage</span><span class="pl-kos">(</span><span class="pl-s1">e</span><span class="pl-kos">.</span><span class="pl-c1">data</span><span class="pl-kos">)</span>
    <span class="pl-s1">sockets</span><span class="pl-kos">.</span><span class="pl-en">forEach</span><span class="pl-kos">(</span><span class="pl-s1">s</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">s</span><span class="pl-kos">.</span><span class="pl-en">send</span><span class="pl-kos">(</span><span class="pl-s1">e</span><span class="pl-kos">.</span><span class="pl-c1">data</span><span class="pl-kos">)</span><span class="pl-kos">)</span>
<span class="pl-kos">}</span>

<span class="pl-c">/*</span>
<span class="pl-c">I tried removing the await here and the demo still worked.</span>
<span class="pl-c"></span>
<span class="pl-c">But https://deno.land/std@0.113.0/http/server.ts#L224 shows that this function</span>
<span class="pl-c">is indeed an async that returns a Promise.</span>
<span class="pl-c">*/</span>
<span class="pl-k">await</span> <span class="pl-en">listenAndServe</span><span class="pl-kos">(</span><span class="pl-s">":8080"</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-s1">r</span>: <span class="pl-smi">Request</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-k">try</span> <span class="pl-kos">{</span>
        <span class="pl-c">/*</span>
<span class="pl-c">        Deno.upgradeWebSocket is a relatively new feature, added in Deno v1.12</span>
<span class="pl-c">        in July 2021:</span>
<span class="pl-c">        https://deno.com/blog/v1.12#server-side-websocket-support-in-native-http</span>
<span class="pl-c">        </span>
<span class="pl-c">        It gives you back a response that you should return to the client in order</span>
<span class="pl-c">        to finish establishing the WebSocket connection, and a socket object which</span>
<span class="pl-c">        you can then use for further WebSocket communication.</span>
<span class="pl-c">        */</span>
        <span class="pl-k">const</span> <span class="pl-kos">{</span> socket<span class="pl-kos">,</span> response <span class="pl-kos">}</span> <span class="pl-c1">=</span> <span class="pl-smi">Deno</span><span class="pl-kos">.</span><span class="pl-en">upgradeWebSocket</span><span class="pl-kos">(</span><span class="pl-s1">r</span><span class="pl-kos">)</span>
        <span class="pl-c">// Add it to the set so we can send to all of them later</span>
        <span class="pl-s1">sockets</span><span class="pl-kos">.</span><span class="pl-en">add</span><span class="pl-kos">(</span><span class="pl-s1">socket</span><span class="pl-kos">)</span>
        <span class="pl-c">/*</span>
<span class="pl-c">        This is a sneaky hack: when a message arrives from the WebSocket we pass it</span>
<span class="pl-c">        directly to the BroadcastChannel - then use the e.target != channel check</span>
<span class="pl-c">        above to broadcast it on to every other global instance.</span>
<span class="pl-c">        */</span>
        <span class="pl-s1">socket</span><span class="pl-kos">.</span><span class="pl-c1">onmessage</span> <span class="pl-c1">=</span> <span class="pl-s1">channel</span><span class="pl-kos">.</span><span class="pl-c1">onmessage</span>
        <span class="pl-c">// When browser disconnects, remove the socket from the set of sockets</span>
        <span class="pl-s1">socket</span><span class="pl-kos">.</span><span class="pl-en">onclose</span> <span class="pl-c1">=</span> <span class="pl-s1">_</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">sockets</span><span class="pl-kos">.</span><span class="pl-en">delete</span><span class="pl-kos">(</span><span class="pl-s1">socket</span><span class="pl-kos">)</span>
        <span class="pl-k">return</span> <span class="pl-s1">response</span>
    <span class="pl-kos">}</span> <span class="pl-k">catch</span> <span class="pl-kos">{</span>
    <span class="pl-c">/*</span>
<span class="pl-c">    I added code here to catch(e) and display e.toString() which showed me</span>
<span class="pl-c">    that the exception caught here is:</span>
<span class="pl-c"></span>
<span class="pl-c">      exception: TypeError: Invalid Header: 'upgrade' header must be 'websocket'</span>
<span class="pl-c"></span>
<span class="pl-c">    This is an exception thrown by Deno.upgradeWebSocket(r) if the incoming</span>
<span class="pl-c">    request does not include the "upgrade: websocket" HTTP header, which</span>
<span class="pl-c">    is added by browsers when using new WebSocket("wss://...")</span>
<span class="pl-c">    </span>
<span class="pl-c">    So here we return the HTML and headers for the application itself.</span>
<span class="pl-c">    */</span>
    <span class="pl-k">return</span> <span class="pl-k">new</span> <span class="pl-smi">Response</span><span class="pl-kos">(</span><span class="pl-s1">html</span><span class="pl-kos">,</span> <span class="pl-kos">{</span>headers<span class="pl-kos">}</span><span class="pl-kos">)</span>
  <span class="pl-kos">}</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span></pre></div>
<p>Relevant links:</p>
<ul>
<li><a href="https://deno.land/manual/examples/http_server#using-the-codestdhttpcode-library" rel="nofollow">Deno listenAndServe documentation</a></li>
<li><a href="https://deno.com/deploy/docs/runtime-broadcast-channel/" rel="nofollow">Deno Deploy BroadcastChannel documentation</a></li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/API/Broadcast_Channel_API" rel="nofollow">MDN documentation of the related BroadcastChannel browser API</a></li>
</ul>


path: javascript_tesseract-ocr-javascript.md
topic: javascript
title: Using Tesseract.js to OCR every image on a page
url: https://github.com/simonw/til/blob/main/javascript/tesseract-ocr-javascript.md
created: 2021-11-09T12:27:11-08:00
created_utc: 2021-11-09T20:27:11+00:00
updated: 2021-11-09T12:27:11-08:00
updated_utc: 2021-11-09T20:27:11+00:00
shot_hash: 8a7123ed8b33a46717bdbe838dff93d1
slug: tesseract-ocr-javascript
body:
Pasting this code into a DevTools console should load [Tesseract.js](https://github.com/naptha/tesseract.js) from a CDN, loop through every image loaded by that page (every PNG, GIF, JPG or JPEG), run OCR on them and output the result to the DevTools console.
                                                                     
There's one major catch: the images need to be served in a context that allows JavaScript to read their content - either from the same domain, or from a separate domain with a permissive CORS policy.
                                                                     
Very few sites do this! It worked on www.google.com for me, where it successfully OCRs the Google logo as containing the text "Google".
                                                                     
```javascript
var s = document.createElement("script")
s.src = "https://unpkg.com/tesseract.js@v2.1.0/dist/tesseract.min.js";
document.head.appendChild(s);
s.onload = (async () => {
  const imageUrls = performance.getEntries().map(f => f.name).filter(
    n => n.includes('.jpg') || n.includes('.gif') || n.includes('.png')  || n.includes('.jpeg')
  );
  const worker = Tesseract.createWorker();
  await worker.load();
  await worker.loadLanguage('eng');
  await worker.initialize('eng');
  for (const url of imageUrls) {
    console.log(url);
    var { data: { text } } = await worker.recognize(url);
    console.log(text);
  }
});
```

html:
<p>Pasting this code into a DevTools console should load <a href="https://github.com/naptha/tesseract.js">Tesseract.js</a> from a CDN, loop through every image loaded by that page (every PNG, GIF, JPG or JPEG), run OCR on them and output the result to the DevTools console.</p>
<p>There's one major catch: the images need to be served in a context that allows JavaScript to read their content - either from the same domain, or from a separate domain with a permissive CORS policy.</p>
<p>Very few sites do this! It worked on <a href="http://www.google.com" rel="nofollow">www.google.com</a> for me, where it successfully OCRs the Google logo as containing the text "Google".</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">var</span> <span class="pl-s1">s</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">createElement</span><span class="pl-kos">(</span><span class="pl-s">"script"</span><span class="pl-kos">)</span>
<span class="pl-s1">s</span><span class="pl-kos">.</span><span class="pl-c1">src</span> <span class="pl-c1">=</span> <span class="pl-s">"https://unpkg.com/tesseract.js@v2.1.0/dist/tesseract.min.js"</span><span class="pl-kos">;</span>
<span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-c1">head</span><span class="pl-kos">.</span><span class="pl-en">appendChild</span><span class="pl-kos">(</span><span class="pl-s1">s</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-s1">s</span><span class="pl-kos">.</span><span class="pl-c1">onload</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-k">async</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
  <span class="pl-k">const</span> <span class="pl-s1">imageUrls</span> <span class="pl-c1">=</span> <span class="pl-s1">performance</span><span class="pl-kos">.</span><span class="pl-en">getEntries</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">map</span><span class="pl-kos">(</span><span class="pl-s1">f</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">f</span><span class="pl-kos">.</span><span class="pl-c1">name</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">filter</span><span class="pl-kos">(</span>
    <span class="pl-s1">n</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">n</span><span class="pl-kos">.</span><span class="pl-en">includes</span><span class="pl-kos">(</span><span class="pl-s">'.jpg'</span><span class="pl-kos">)</span> <span class="pl-c1">||</span> <span class="pl-s1">n</span><span class="pl-kos">.</span><span class="pl-en">includes</span><span class="pl-kos">(</span><span class="pl-s">'.gif'</span><span class="pl-kos">)</span> <span class="pl-c1">||</span> <span class="pl-s1">n</span><span class="pl-kos">.</span><span class="pl-en">includes</span><span class="pl-kos">(</span><span class="pl-s">'.png'</span><span class="pl-kos">)</span>  <span class="pl-c1">||</span> <span class="pl-s1">n</span><span class="pl-kos">.</span><span class="pl-en">includes</span><span class="pl-kos">(</span><span class="pl-s">'.jpeg'</span><span class="pl-kos">)</span>
  <span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">const</span> <span class="pl-s1">worker</span> <span class="pl-c1">=</span> <span class="pl-v">Tesseract</span><span class="pl-kos">.</span><span class="pl-en">createWorker</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">await</span> <span class="pl-s1">worker</span><span class="pl-kos">.</span><span class="pl-en">load</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">await</span> <span class="pl-s1">worker</span><span class="pl-kos">.</span><span class="pl-en">loadLanguage</span><span class="pl-kos">(</span><span class="pl-s">'eng'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">await</span> <span class="pl-s1">worker</span><span class="pl-kos">.</span><span class="pl-en">initialize</span><span class="pl-kos">(</span><span class="pl-s">'eng'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">for</span> <span class="pl-kos">(</span><span class="pl-k">const</span> <span class="pl-s1">url</span> <span class="pl-k">of</span> <span class="pl-s1">imageUrls</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">url</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-k">var</span> <span class="pl-kos">{</span> <span class="pl-c1">data</span>: <span class="pl-kos">{</span> text <span class="pl-kos">}</span> <span class="pl-kos">}</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">worker</span><span class="pl-kos">.</span><span class="pl-en">recognize</span><span class="pl-kos">(</span><span class="pl-s1">url</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">text</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>


path: python_graphlib-topologicalsorter.md
topic: python
title: Planning parallel downloads with TopologicalSorter
url: https://github.com/simonw/til/blob/main/python/graphlib-topologicalsorter.md
created: 2021-11-16T13:41:07-08:00
created_utc: 2021-11-16T21:41:07+00:00
updated: 2021-11-16T15:48:37-08:00
updated_utc: 2021-11-16T23:48:37+00:00
shot_hash: 9c9962a62c4ead13970bc556e6c9dd3c
slug: graphlib-topologicalsorter
body:
For [complicated reasons](https://github.com/simonw/datasette/issues/878) I found myself wanting to write Python code to resolve a graph of dependencies and produce a plan for efficiently executing them, in parallel where possible.

I figured out how to do it using [graphlib.TopologicalSorter](https://docs.python.org/3/library/graphlib.html#graphlib.TopologicalSorter), which was added to the Python standard library [in Python 3.9](https://docs.python.org/3/whatsnew/3.9.html#graphlib).

Since I want my code to work on older versions of Python, I was pleased to see that [the graphlib.py module](https://github.com/python/cpython/blob/3.10/Lib/graphlib.py) is standalone and can be easily vendored (though it uses the walrus `:=` operator so I had to [modify it](https://github.com/simonw/datasette/blob/8f757da0750fe7f27b4ed3839bc3ef3650832ad9/datasette/utils/vendored_graphlib.py) for compatibility with Python 3.6 and 3.7).

For this example: imagine I know the dependencies of some packages, and I want to fire off some downloads - running them in parallel where possible.

```python
from graphlib import TopologicalSorter

graph = {
    "datasette": {"httpx", "sqlite-utils", "click"},
    "sqlite-utils": {"click", "httpx"}
}

ts = TopologicalSorter(graph)
ts.prepare()
plan = []
while ts.is_active():
    nodes = ts.get_ready()
    plan.append(nodes)
    ts.done(*nodes)

print(plan)
```
This outputs:
```
[('click', 'httpx'), ('sqlite-utils',), ('datasette',)]
```
So the plan would be to download `click` and `httpx` in parallel, then `sqlite-utils`, then `datasette`.

In this usage of the module we're passing the full `graph` as an argument to `TopologicalSorter`. It's also possible to build it up a step at a time like this:

```python
ts = TopologicalSorter()
# add(node, *predecessors)
ts.add("datasette", "httpx", "sqlite-utils", "click")
ts.add("sqlite-utils", "click", "httpx")
ts.prepare()
plan = []
while ts.is_active():
    nodes = ts.get_ready()
    plan.append(nodes)
    ts.done(*nodes)
```
Having called `.prepare()` no further `.add()` calls are allowed.

If you just want the nodes ordered such that none of them come before a dependency you can use `.static_order()`:

```pycon
>>> graph = {
...     "datasette": {"httpx", "sqlite-utils", "click"},
...     "sqlite-utils": {"click", "httpx"}
... }
>>> ts = TopologicalSorter(graph)
>>> list(ts.static_order())
['click', 'httpx', 'sqlite-utils', 'datasette']
```

The `.prepare()` method raises `graphlib.CycleError` if it detects a cycle:

```pycon
>>> TopologicalSorter({"datasette": {"sqlite-utils"}, "sqlite-utils": {"datasette"}}).prepare()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/simon/.pyenv/versions/3.10.0/lib/python3.10/graphlib.py", line 104, in prepare
    raise CycleError(f"nodes are in a cycle", cycle)
graphlib.CycleError: ('nodes are in a cycle', ['datasette', 'sqlite-utils', 'datasette'])
```

html:
<p>For <a href="https://github.com/simonw/datasette/issues/878">complicated reasons</a> I found myself wanting to write Python code to resolve a graph of dependencies and produce a plan for efficiently executing them, in parallel where possible.</p>
<p>I figured out how to do it using <a href="https://docs.python.org/3/library/graphlib.html#graphlib.TopologicalSorter" rel="nofollow">graphlib.TopologicalSorter</a>, which was added to the Python standard library <a href="https://docs.python.org/3/whatsnew/3.9.html#graphlib" rel="nofollow">in Python 3.9</a>.</p>
<p>Since I want my code to work on older versions of Python, I was pleased to see that <a href="https://github.com/python/cpython/blob/3.10/Lib/graphlib.py">the graphlib.py module</a> is standalone and can be easily vendored (though it uses the walrus <code>:=</code> operator so I had to <a href="https://github.com/simonw/datasette/blob/8f757da0750fe7f27b4ed3839bc3ef3650832ad9/datasette/utils/vendored_graphlib.py">modify it</a> for compatibility with Python 3.6 and 3.7).</p>
<p>For this example: imagine I know the dependencies of some packages, and I want to fire off some downloads - running them in parallel where possible.</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">graphlib</span> <span class="pl-k">import</span> <span class="pl-v">TopologicalSorter</span>

<span class="pl-s1">graph</span> <span class="pl-c1">=</span> {
    <span class="pl-s">"datasette"</span>: {<span class="pl-s">"httpx"</span>, <span class="pl-s">"sqlite-utils"</span>, <span class="pl-s">"click"</span>},
    <span class="pl-s">"sqlite-utils"</span>: {<span class="pl-s">"click"</span>, <span class="pl-s">"httpx"</span>}
}

<span class="pl-s1">ts</span> <span class="pl-c1">=</span> <span class="pl-v">TopologicalSorter</span>(<span class="pl-s1">graph</span>)
<span class="pl-s1">ts</span>.<span class="pl-en">prepare</span>()
<span class="pl-s1">plan</span> <span class="pl-c1">=</span> []
<span class="pl-k">while</span> <span class="pl-s1">ts</span>.<span class="pl-en">is_active</span>():
    <span class="pl-s1">nodes</span> <span class="pl-c1">=</span> <span class="pl-s1">ts</span>.<span class="pl-en">get_ready</span>()
    <span class="pl-s1">plan</span>.<span class="pl-en">append</span>(<span class="pl-s1">nodes</span>)
    <span class="pl-s1">ts</span>.<span class="pl-en">done</span>(<span class="pl-c1">*</span><span class="pl-s1">nodes</span>)

<span class="pl-en">print</span>(<span class="pl-s1">plan</span>)</pre></div>
<p>This outputs:</p>
<pre><code>[('click', 'httpx'), ('sqlite-utils',), ('datasette',)]
</code></pre>
<p>So the plan would be to download <code>click</code> and <code>httpx</code> in parallel, then <code>sqlite-utils</code>, then <code>datasette</code>.</p>
<p>In this usage of the module we're passing the full <code>graph</code> as an argument to <code>TopologicalSorter</code>. It's also possible to build it up a step at a time like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">ts</span> <span class="pl-c1">=</span> <span class="pl-v">TopologicalSorter</span>()
<span class="pl-c"># add(node, *predecessors)</span>
<span class="pl-s1">ts</span>.<span class="pl-en">add</span>(<span class="pl-s">"datasette"</span>, <span class="pl-s">"httpx"</span>, <span class="pl-s">"sqlite-utils"</span>, <span class="pl-s">"click"</span>)
<span class="pl-s1">ts</span>.<span class="pl-en">add</span>(<span class="pl-s">"sqlite-utils"</span>, <span class="pl-s">"click"</span>, <span class="pl-s">"httpx"</span>)
<span class="pl-s1">ts</span>.<span class="pl-en">prepare</span>()
<span class="pl-s1">plan</span> <span class="pl-c1">=</span> []
<span class="pl-k">while</span> <span class="pl-s1">ts</span>.<span class="pl-en">is_active</span>():
    <span class="pl-s1">nodes</span> <span class="pl-c1">=</span> <span class="pl-s1">ts</span>.<span class="pl-en">get_ready</span>()
    <span class="pl-s1">plan</span>.<span class="pl-en">append</span>(<span class="pl-s1">nodes</span>)
    <span class="pl-s1">ts</span>.<span class="pl-en">done</span>(<span class="pl-c1">*</span><span class="pl-s1">nodes</span>)</pre></div>
<p>Having called <code>.prepare()</code> no further <code>.add()</code> calls are allowed.</p>
<p>If you just want the nodes ordered such that none of them come before a dependency you can use <code>.static_order()</code>:</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; graph <span class="pl-k">=</span> {
...     <span class="pl-s"><span class="pl-pds">"</span>datasette<span class="pl-pds">"</span></span>: {<span class="pl-s"><span class="pl-pds">"</span>httpx<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>sqlite-utils<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>click<span class="pl-pds">"</span></span>},
...     <span class="pl-s"><span class="pl-pds">"</span>sqlite-utils<span class="pl-pds">"</span></span>: {<span class="pl-s"><span class="pl-pds">"</span>click<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>httpx<span class="pl-pds">"</span></span>}
... }
&gt;&gt;&gt; ts <span class="pl-k">=</span> TopologicalSorter(graph)
&gt;&gt;&gt; <span class="pl-c1">list</span>(ts.static_order())
['click', 'httpx', 'sqlite-utils', 'datasette']</pre></div>
<p>The <code>.prepare()</code> method raises <code>graphlib.CycleError</code> if it detects a cycle:</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; TopologicalSorter({<span class="pl-s"><span class="pl-pds">"</span>datasette<span class="pl-pds">"</span></span>: {<span class="pl-s"><span class="pl-pds">"</span>sqlite-utils<span class="pl-pds">"</span></span>}, <span class="pl-s"><span class="pl-pds">"</span>sqlite-utils<span class="pl-pds">"</span></span>: {<span class="pl-s"><span class="pl-pds">"</span>datasette<span class="pl-pds">"</span></span>}}).prepare()
Traceback (most recent call last):
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;
  File "/Users/simon/.pyenv/versions/3.10.0/lib/python3.10/graphlib.py", line 104, in prepare
    raise CycleError(f"nodes are in a cycle", cycle)
graphlib.CycleError: ('nodes are in a cycle', ['datasette', 'sqlite-utils', 'datasette'])</pre></div>


path: python_cog-to-update-help-in-readme.md
topic: python
title: Using cog to update --help in a Markdown README file
url: https://github.com/simonw/til/blob/main/python/cog-to-update-help-in-readme.md
created: 2021-11-18T08:27:19-08:00
created_utc: 2021-11-18T16:27:19+00:00
updated: 2022-02-12T08:07:17-08:00
updated_utc: 2022-02-12T16:07:17+00:00
shot_hash: 2f77ee496b7eb94a863e0c61cd5f23e8
slug: cog-to-update-help-in-readme
body:
My [csvs-to-sqlite README](https://github.com/simonw/csvs-to-sqlite/blob/main/README.md) includes a section that shows the output of the `csvs-to-sqlite --help` command ([relevant issue](https://github.com/simonw/csvs-to-sqlite/issues/82)).

I had been manually copying this in, but I decided to try using [cog](https://nedbatchelder.com/code/cog) to automate the process.

Here's what I came up with:

````markdown
<!-- [[[cog
import cog
from csvs_to_sqlite import cli
from click.testing import CliRunner
runner = CliRunner()
result = runner.invoke(cli.cli, ["--help"])
help = result.output.replace("Usage: cli", "Usage: csvs-to-sqlite")
cog.out(
    "```\n{}\n```".format(help)
)
]]] -->
```
Usage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME
...
```
<!-- [[[end]]] -->
````
Then to update the README file, run this:

    cog -r README.md

The `-r` option causes it to modify that file in place.

Cog works by scanning for a `[[[cog ... ]]]` section, executing the code there, capturing the `cog.out()` output and using that to replace everything from the end of the code block up to the line containing the `[[[end]]]` marker.

It's designed to interact well with comments - in this case HTML comments - such that the `cog` generation code can be hidden.

## Testing with cog --check

A version of Cog released after I first wrote this TIL added a new `--check` option, so you can run a test in CI to check if the file needs to be updated using:

    cog --check README.md

## Writing a test (before cog --check)

Any time I generate content like this in a repo I like to include a test that will fail if I forget to update the content.

`cog` clearly isn't designed to be used as an independent library, but I came up with the following pattern `pytest` test which works well, in my `tests/test_csvs_to_sqlite.py` module:

```python
from cogapp import Cog
import sys
from io import StringIO
import pathlib


def test_if_cog_needs_to_be_run():
    _stdout = sys.stdout
    sys.stdout = StringIO()
    readme = pathlib.Path(__file__).parent.parent / "README.md"
    result = Cog().main(["cog", str(readme)])
    output = sys.stdout.getvalue()
    sys.stdout = _stdout
    assert (
        output == readme.read_text()
    ), "Run 'cog -r README.md' to update help in README"
```
The key line here is this one:
```python
result = Cog().main(["cog", str(readme)])
```
In cog's implementation, that code is called like this:
```python
Cog().main(sys.argv)
```
Here I'm faking the command-line arguments to pass in just the path to my `README.md` file.

Cog then writes the generated output to `stdout` - which I capture with that `sys.stdout` trick.

Finally, I compare the generated output to the current file content and fail the test with a reminder to run `cog -r` if they do not match.

## Cog for reStructuredText

Here's an example of `cog` in a `.rst` file:

```rst
.. [[[cog
    import tabulate
    cog.out("\n" + "\n".join('- ``{}``'.format(t) for t in tabulate.tabulate_formats) + "\n\n")
.. ]]]

- ``fancy_grid``
- ``fancy_outline``
- ``github``

.. [[[end]]]
```
The trailing and leading newlines are important to avoid a warning about "Explicit markup ends without a blank line; unexpected unindent".

## CLI reference pages

I added these to both Datasette and `sqlite-utils`: full pages that list the help for every command provided by those tools.

- [Datasette CLI reference](https://docs.datasette.io/en/stable/cli-reference.html) - underlying [source code](https://raw.githubusercontent.com/simonw/datasette/main/docs/cli-reference.rst)
- [sqlite-utils CLI reference](https://sqlite-utils.datasette.io/en/stable/cli-reference.html) - underlying [source code](https://raw.githubusercontent.com/simonw/sqlite-utils/main/docs/cli-reference.rst)

html:
<p>My <a href="https://github.com/simonw/csvs-to-sqlite/blob/main/README.md">csvs-to-sqlite README</a> includes a section that shows the output of the <code>csvs-to-sqlite --help</code> command (<a href="https://github.com/simonw/csvs-to-sqlite/issues/82">relevant issue</a>).</p>
<p>I had been manually copying this in, but I decided to try using <a href="https://nedbatchelder.com/code/cog" rel="nofollow">cog</a> to automate the process.</p>
<p>Here's what I came up with:</p>
<div class="highlight highlight-source-gfm"><pre><span class="pl-c"><span class="pl-c">&lt;!--</span> [[[cog</span>
<span class="pl-c">import cog</span>
<span class="pl-c">from csvs_to_sqlite import cli</span>
<span class="pl-c">from click.testing import CliRunner</span>
<span class="pl-c">runner = CliRunner()</span>
<span class="pl-c">result = runner.invoke(cli.cli, ["--help"])</span>
<span class="pl-c">help = result.output.replace("Usage: cli", "Usage: csvs-to-sqlite")</span>
<span class="pl-c">cog.out(</span>
<span class="pl-c">    "```\n{}\n```".format(help)</span>
<span class="pl-c">)</span>
<span class="pl-c">]]] <span class="pl-c">--&gt;</span></span>
<span class="pl-c1"><span class="pl-c1">```</span></span>
<span class="pl-c1">Usage: csvs-to-sqlite [OPTIONS] PATHS... DBNAME</span>
<span class="pl-c1">...</span>
<span class="pl-c1"><span class="pl-c1">```</span></span>
<span class="pl-c"><span class="pl-c">&lt;!--</span> [[[end]]] <span class="pl-c">--&gt;</span></span></pre></div>
<p>Then to update the README file, run this:</p>
<pre><code>cog -r README.md
</code></pre>
<p>The <code>-r</code> option causes it to modify that file in place.</p>
<p>Cog works by scanning for a <code>[[[cog ... ]]]</code> section, executing the code there, capturing the <code>cog.out()</code> output and using that to replace everything from the end of the code block up to the line containing the <code>[[[end]]]</code> marker.</p>
<p>It's designed to interact well with comments - in this case HTML comments - such that the <code>cog</code> generation code can be hidden.</p>
<h2>
<a id="user-content-testing-with-cog---check" class="anchor" href="#testing-with-cog---check" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Testing with cog --check</h2>
<p>A version of Cog released after I first wrote this TIL added a new <code>--check</code> option, so you can run a test in CI to check if the file needs to be updated using:</p>
<pre><code>cog --check README.md
</code></pre>
<h2>
<a id="user-content-writing-a-test-before-cog---check" class="anchor" href="#writing-a-test-before-cog---check" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Writing a test (before cog --check)</h2>
<p>Any time I generate content like this in a repo I like to include a test that will fail if I forget to update the content.</p>
<p><code>cog</code> clearly isn't designed to be used as an independent library, but I came up with the following pattern <code>pytest</code> test which works well, in my <code>tests/test_csvs_to_sqlite.py</code> module:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">cogapp</span> <span class="pl-k">import</span> <span class="pl-v">Cog</span>
<span class="pl-k">import</span> <span class="pl-s1">sys</span>
<span class="pl-k">from</span> <span class="pl-s1">io</span> <span class="pl-k">import</span> <span class="pl-v">StringIO</span>
<span class="pl-k">import</span> <span class="pl-s1">pathlib</span>


<span class="pl-k">def</span> <span class="pl-en">test_if_cog_needs_to_be_run</span>():
    <span class="pl-s1">_stdout</span> <span class="pl-c1">=</span> <span class="pl-s1">sys</span>.<span class="pl-s1">stdout</span>
    <span class="pl-s1">sys</span>.<span class="pl-s1">stdout</span> <span class="pl-c1">=</span> <span class="pl-v">StringIO</span>()
    <span class="pl-s1">readme</span> <span class="pl-c1">=</span> <span class="pl-s1">pathlib</span>.<span class="pl-v">Path</span>(<span class="pl-s1">__file__</span>).<span class="pl-s1">parent</span>.<span class="pl-s1">parent</span> <span class="pl-c1">/</span> <span class="pl-s">"README.md"</span>
    <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-v">Cog</span>().<span class="pl-en">main</span>([<span class="pl-s">"cog"</span>, <span class="pl-en">str</span>(<span class="pl-s1">readme</span>)])
    <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-s1">sys</span>.<span class="pl-s1">stdout</span>.<span class="pl-en">getvalue</span>()
    <span class="pl-s1">sys</span>.<span class="pl-s1">stdout</span> <span class="pl-c1">=</span> <span class="pl-s1">_stdout</span>
    <span class="pl-k">assert</span> (
        <span class="pl-s1">output</span> <span class="pl-c1">==</span> <span class="pl-s1">readme</span>.<span class="pl-en">read_text</span>()
    ), <span class="pl-s">"Run 'cog -r README.md' to update help in README"</span></pre></div>
<p>The key line here is this one:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-v">Cog</span>().<span class="pl-en">main</span>([<span class="pl-s">"cog"</span>, <span class="pl-en">str</span>(<span class="pl-s1">readme</span>)])</pre></div>
<p>In cog's implementation, that code is called like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-v">Cog</span>().<span class="pl-en">main</span>(<span class="pl-s1">sys</span>.<span class="pl-s1">argv</span>)</pre></div>
<p>Here I'm faking the command-line arguments to pass in just the path to my <code>README.md</code> file.</p>
<p>Cog then writes the generated output to <code>stdout</code> - which I capture with that <code>sys.stdout</code> trick.</p>
<p>Finally, I compare the generated output to the current file content and fail the test with a reminder to run <code>cog -r</code> if they do not match.</p>
<h2>
<a id="user-content-cog-for-restructuredtext" class="anchor" href="#cog-for-restructuredtext" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Cog for reStructuredText</h2>
<p>Here's an example of <code>cog</code> in a <code>.rst</code> file:</p>
<div class="highlight highlight-text-restructuredtext"><pre><span class="pl-c"><span class="pl-c">..</span> [[[cog</span>
<span class="pl-c">    import tabulate</span>
<span class="pl-c">    cog.out("\n" + "\n".join('- ``{}``'.format(t) for t in tabulate.tabulate_formats) + "\n\n")</span>
<span class="pl-c"><span class="pl-c">..</span> ]]]</span>
<span class="pl-c"></span>
<span class="pl-c"></span>- <span class="pl-c1">``fancy_grid``</span>
- <span class="pl-c1">``fancy_outline``</span>
- <span class="pl-c1">``github``</span>

<span class="pl-c"><span class="pl-c">..</span> [[[end]]]</span></pre></div>
<p>The trailing and leading newlines are important to avoid a warning about "Explicit markup ends without a blank line; unexpected unindent".</p>
<h2>
<a id="user-content-cli-reference-pages" class="anchor" href="#cli-reference-pages" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CLI reference pages</h2>
<p>I added these to both Datasette and <code>sqlite-utils</code>: full pages that list the help for every command provided by those tools.</p>
<ul>
<li>
<a href="https://docs.datasette.io/en/stable/cli-reference.html" rel="nofollow">Datasette CLI reference</a> - underlying <a href="https://raw.githubusercontent.com/simonw/datasette/main/docs/cli-reference.rst" rel="nofollow">source code</a>
</li>
<li>
<a href="https://sqlite-utils.datasette.io/en/stable/cli-reference.html" rel="nofollow">sqlite-utils CLI reference</a> - underlying <a href="https://raw.githubusercontent.com/simonw/sqlite-utils/main/docs/cli-reference.rst" rel="nofollow">source code</a>
</li>
</ul>


path: cloudrun_using-build-args-with-cloud-run.md
topic: cloudrun
title: Using build-arg variables with Cloud Run deployments
url: https://github.com/simonw/til/blob/main/cloudrun/using-build-args-with-cloud-run.md
created: 2021-11-19T16:24:56-08:00
created_utc: 2021-11-20T00:24:56+00:00
updated: 2021-11-19T16:32:57-08:00
updated_utc: 2021-11-20T00:32:57+00:00
shot_hash: 328fd8809d3af0f5cdde5c937c7085bd
slug: using-build-args-with-cloud-run
body:
For [datasette/issues/1522](https://github.com/simonw/datasette/issues/1522) I wanted to use a Docker build argument in a `Dockerfile` that would then be deployed to Cloud Run.

I needed this to be able to control the version of Datasette that was deployed. Here's my simplified `Dockerfile`:

```dockerfile
FROM python:3-alpine

ARG DATASETTE_REF
# Copy to environment variable for use in CMD later
ENV VERSION_NOTE=$DATASETTE_REF

RUN pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip

# Need to use "shell form" here to get variable substition:
CMD datasette -h 0.0.0.0 -p 8080 --version-note $VERSION_NOTE
```
I can build this on my laptop like so:

    docker build -t datasette-build-arg-demo . \
      --build-arg DATASETTE_REF=c617e1769ea27e045b0f2907ef49a9a1244e577d

Then run it like this:

    docker run -p 5000:8080 --rm datasette-build-arg-demo

And visit `http://localhost:5000/-/versions` to see the version number to confirm it worked.

I wanted to deploy this to Cloud Run, using [this recipe](https://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-run).

Unfortunately, the `gcloud builds submit` command doesn't have a mechanism for specifying `--build-arg`.

Instead, you need to use a YAML file and pass it with the `gcloud builds submit --config cloudbuild.yml` option. The YAML should look like this:

```yaml
steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'gc.io/MY-PROJECT/MY-NAME', '.', '--build-arg', 'DATASETTE_REF=c617e1769ea27e045b0f2907ef49a9a1244e577d']
- name: 'gcr.io/cloud-builders/docker'
  args: ['push', $IMAGE]
```

Since I want to dynamically populate my YAML file, I ended up using the following pattern in a `./deploy.sh` script:

```bash
#!/bin/bash
# https://til.simonwillison.net/cloudrun/using-build-args-with-cloud-run

if [[ -z "$DATASETTE_REF" ]]; then
    echo "Must provide DATASETTE_REF environment variable" 1>&2
    exit 1
fi

NAME="datasette-apache-proxy-demo"
PROJECT=$(gcloud config get-value project)
IMAGE="gcr.io/$PROJECT/$NAME"

# Need YAML so we can set --build-arg
echo "
steps:
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', '$IMAGE', '.', '--build-arg', 'DATASETTE_REF=$DATASETTE_REF']
- name: 'gcr.io/cloud-builders/docker'
  args: ['push', '$IMAGE']
" > /tmp/cloudbuild.yml

gcloud builds submit --config /tmp/cloudbuild.yml

rm /tmp/cloudbuild.yml

gcloud run deploy $NAME \
  --allow-unauthenticated \
  --platform=managed \
  --image $IMAGE \
  --port 80
```

html:
<p>For <a href="https://github.com/simonw/datasette/issues/1522">datasette/issues/1522</a> I wanted to use a Docker build argument in a <code>Dockerfile</code> that would then be deployed to Cloud Run.</p>
<p>I needed this to be able to control the version of Datasette that was deployed. Here's my simplified <code>Dockerfile</code>:</p>
<div class="highlight highlight-source-dockerfile"><pre><span class="pl-k">FROM</span> python:3-alpine

<span class="pl-k">ARG</span> DATASETTE_REF
<span class="pl-c"><span class="pl-c">#</span> Copy to environment variable for use in CMD later</span>
<span class="pl-k">ENV</span> VERSION_NOTE=$DATASETTE_REF

<span class="pl-k">RUN</span> pip install https://github.com/simonw/datasette/archive/${DATASETTE_REF}.zip

<span class="pl-c"><span class="pl-c">#</span> Need to use "shell form" here to get variable substition:</span>
<span class="pl-k">CMD</span> datasette -h 0.0.0.0 -p 8080 --version-note $VERSION_NOTE</pre></div>
<p>I can build this on my laptop like so:</p>
<pre><code>docker build -t datasette-build-arg-demo . \
  --build-arg DATASETTE_REF=c617e1769ea27e045b0f2907ef49a9a1244e577d
</code></pre>
<p>Then run it like this:</p>
<pre><code>docker run -p 5000:8080 --rm datasette-build-arg-demo
</code></pre>
<p>And visit <code>http://localhost:5000/-/versions</code> to see the version number to confirm it worked.</p>
<p>I wanted to deploy this to Cloud Run, using <a href="https://til.simonwillison.net/cloudrun/ship-dockerfile-to-cloud-run" rel="nofollow">this recipe</a>.</p>
<p>Unfortunately, the <code>gcloud builds submit</code> command doesn't have a mechanism for specifying <code>--build-arg</code>.</p>
<p>Instead, you need to use a YAML file and pass it with the <code>gcloud builds submit --config cloudbuild.yml</code> option. The YAML should look like this:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">steps</span>:
- <span class="pl-ent">name</span>: <span class="pl-s"><span class="pl-pds">'</span>gcr.io/cloud-builders/docker<span class="pl-pds">'</span></span>
  <span class="pl-ent">args</span>: <span class="pl-s">['build', '-t', 'gc.io/MY-PROJECT/MY-NAME', '.', '--build-arg', 'DATASETTE_REF=c617e1769ea27e045b0f2907ef49a9a1244e577d']</span>
- <span class="pl-ent">name</span>: <span class="pl-s"><span class="pl-pds">'</span>gcr.io/cloud-builders/docker<span class="pl-pds">'</span></span>
  <span class="pl-ent">args</span>: <span class="pl-s">['push', $IMAGE]</span></pre></div>
<p>Since I want to dynamically populate my YAML file, I ended up using the following pattern in a <code>./deploy.sh</code> script:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c"><span class="pl-c">#!</span>/bin/bash</span>
<span class="pl-c"><span class="pl-c">#</span> https://til.simonwillison.net/cloudrun/using-build-args-with-cloud-run</span>

<span class="pl-k">if</span> [[ <span class="pl-k">-z</span> <span class="pl-s"><span class="pl-pds">"</span><span class="pl-smi">$DATASETTE_REF</span><span class="pl-pds">"</span></span> ]]<span class="pl-k">;</span> <span class="pl-k">then</span>
    <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>Must provide DATASETTE_REF environment variable<span class="pl-pds">"</span></span> <span class="pl-k">1&gt;&amp;2</span>
    <span class="pl-c1">exit</span> 1
<span class="pl-k">fi</span>

NAME=<span class="pl-s"><span class="pl-pds">"</span>datasette-apache-proxy-demo<span class="pl-pds">"</span></span>
PROJECT=<span class="pl-s"><span class="pl-pds">$(</span>gcloud config get-value project<span class="pl-pds">)</span></span>
IMAGE=<span class="pl-s"><span class="pl-pds">"</span>gcr.io/<span class="pl-smi">$PROJECT</span>/<span class="pl-smi">$NAME</span><span class="pl-pds">"</span></span>

<span class="pl-c"><span class="pl-c">#</span> Need YAML so we can set --build-arg</span>
<span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span></span>
<span class="pl-s">steps:</span>
<span class="pl-s">- name: 'gcr.io/cloud-builders/docker'</span>
<span class="pl-s">  args: ['build', '-t', '<span class="pl-smi">$IMAGE</span>', '.', '--build-arg', 'DATASETTE_REF=<span class="pl-smi">$DATASETTE_REF</span>']</span>
<span class="pl-s">- name: 'gcr.io/cloud-builders/docker'</span>
<span class="pl-s">  args: ['push', '<span class="pl-smi">$IMAGE</span>']</span>
<span class="pl-s"><span class="pl-pds">"</span></span> <span class="pl-k">&gt;</span> /tmp/cloudbuild.yml

gcloud builds submit --config /tmp/cloudbuild.yml

rm /tmp/cloudbuild.yml

gcloud run deploy <span class="pl-smi">$NAME</span> \
  --allow-unauthenticated \
  --platform=managed \
  --image <span class="pl-smi">$IMAGE</span> \
  --port 80</pre></div>


path: fly_custom-subdomain-fly.md
topic: fly
title: Assigning a custom subdomain to a Fly app
url: https://github.com/simonw/til/blob/main/fly/custom-subdomain-fly.md
created: 2021-11-20T12:46:43-08:00
created_utc: 2021-11-20T20:46:43+00:00
updated: 2021-11-20T12:46:43-08:00
updated_utc: 2021-11-20T20:46:43+00:00
shot_hash: f70fecfe6cfda21a079a753a1b96d491
slug: custom-subdomain-fly
body:
I deployed an app to [Fly](https://fly.io/) and decided to point a custom subdomain to it.

My fly app is https://datasette-apache-proxy-demo.fly.dev/

I wanted the URL to be https://datasette-apache-proxy-demo.datasette.io/ (see [issue #1524](https://github.com/simonw/datasette/issues/1524)).

Relevant documentation: [SSL for Custom Domains](https://fly.io/docs/app-guides/custom-domains-with-fly/).

## Assign a CNAME

First step was to add a CNAME to my `datasette.io` domain.

I pointed `CNAME` of `datasette-apache-proxy-demo.datasette.io` at `datasette-apache-proxy-demo.fly.dev.` using Vercel DNS:

<img width="586" alt="image" src="https://user-images.githubusercontent.com/9599/142740008-942f180b-bedb-4a44-b6ef-1b0e7fd32416.png">

## Issuing a certificate

Fly started serving from `http://datasette-apache-proxy-demo.datasette.io/` as soon as the DNS change propagated. To get `https://` to work I had to run this:

```
% flyctl certs create datasette-apache-proxy-demo.datasette.io 
Your certificate for datasette-apache-proxy-demo.datasette.io is being issued. Status is Awaiting certificates.
```
I could then run this command periodically to see if it had been issued, which happened about 53 seconds later:
```
apache-proxy % flyctl certs show datasette-apache-proxy-demo.datasette.io
The certificate for datasette-apache-proxy-demo.datasette.io has been issued.

Hostname                  = datasette-apache-proxy-demo.datasette.io

DNS Provider              = constellix

Certificate Authority     = Let's Encrypt

Issued                    = ecdsa,rsa

Added to App              = 53 seconds ago

Source                    = fly
```

html:
<p>I deployed an app to <a href="https://fly.io/" rel="nofollow">Fly</a> and decided to point a custom subdomain to it.</p>
<p>My fly app is <a href="https://datasette-apache-proxy-demo.fly.dev/" rel="nofollow">https://datasette-apache-proxy-demo.fly.dev/</a></p>
<p>I wanted the URL to be <a href="https://datasette-apache-proxy-demo.datasette.io/" rel="nofollow">https://datasette-apache-proxy-demo.datasette.io/</a> (see <a href="https://github.com/simonw/datasette/issues/1524">issue #1524</a>).</p>
<p>Relevant documentation: <a href="https://fly.io/docs/app-guides/custom-domains-with-fly/" rel="nofollow">SSL for Custom Domains</a>.</p>
<h2>
<a id="user-content-assign-a-cname" class="anchor" href="#assign-a-cname" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Assign a CNAME</h2>
<p>First step was to add a CNAME to my <code>datasette.io</code> domain.</p>
<p>I pointed <code>CNAME</code> of <code>datasette-apache-proxy-demo.datasette.io</code> at <code>datasette-apache-proxy-demo.fly.dev.</code> using Vercel DNS:</p>
<p><a href="https://user-images.githubusercontent.com/9599/142740008-942f180b-bedb-4a44-b6ef-1b0e7fd32416.png" target="_blank" rel="nofollow"><img width="586" alt="image" src="https://user-images.githubusercontent.com/9599/142740008-942f180b-bedb-4a44-b6ef-1b0e7fd32416.png" style="max-width:100%;"></a></p>
<h2>
<a id="user-content-issuing-a-certificate" class="anchor" href="#issuing-a-certificate" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Issuing a certificate</h2>
<p>Fly started serving from <code>http://datasette-apache-proxy-demo.datasette.io/</code> as soon as the DNS change propagated. To get <code>https://</code> to work I had to run this:</p>
<pre><code>% flyctl certs create datasette-apache-proxy-demo.datasette.io 
Your certificate for datasette-apache-proxy-demo.datasette.io is being issued. Status is Awaiting certificates.
</code></pre>
<p>I could then run this command periodically to see if it had been issued, which happened about 53 seconds later:</p>
<pre><code>apache-proxy % flyctl certs show datasette-apache-proxy-demo.datasette.io
The certificate for datasette-apache-proxy-demo.datasette.io has been issued.

Hostname                  = datasette-apache-proxy-demo.datasette.io

DNS Provider              = constellix

Certificate Authority     = Let's Encrypt

Issued                    = ecdsa,rsa

Added to App              = 53 seconds ago

Source                    = fly
</code></pre>


path: caddy_pause-retry-traffic.md
topic: caddy
title: Pausing traffic and retrying in Caddy
url: https://github.com/simonw/til/blob/main/caddy/pause-retry-traffic.md
created: 2021-11-24T17:18:24-08:00
created_utc: 2021-11-25T01:18:24+00:00
updated: 2021-11-24T18:37:21-08:00
updated_utc: 2021-11-25T02:37:21+00:00
shot_hash: 9abe5cc7d3baf71f2b3a28b0c12b8bbe
slug: pause-retry-traffic
body:
A pattern I really like for zero-downtime deploys is the ability to "pause" HTTP traffic at the load balancer, such that incoming requests from browsers appear to take a few extra seconds to return, but under the hood they've actually been held in a queue while a backend server is swapped out or upgraded in some way.

I first heard about this pattern [from Braintree](https://simonwillison.net/2011/Jun/30/braintree/), and a [conversation on Twitter](https://twitter.com/simonw/status/1463652411365494791) today brought up a few more examples, including [this NGINX Lua config](https://github.com/basecamp/intermission) from Basecamp.

[Caddy](https://caddyserver.com/) creator Matt Holt [pointed me](https://twitter.com/mholt6/status/1463656086360051714) to [lb_try_duration and lb_try_interval](https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#lb_try_duration) in Caddy, which can hold requests for up to a specific number of seconds, retrying the backend to see if it has become available again.

I decided to try this out. This was my first time using Caddy and I'm really impressed with both the design of the software and the quality of the [getting started documentation](https://caddyserver.com/docs/getting-started).

I installed Caddy using Homebrew:

    brew install caddy

## The Caddyfile

You can configure Caddy in a bunch of different ways - the two main options are using JSON via the Caddy API or using their own custom Caddyfile format.

Here's the `Caddyfile` I created:

```
{
    auto_https off
}
:80 {
    reverse_proxy localhost:8003 {
        lb_try_duration 30s
        lb_try_interval 1s
    }
}
```
Caddy defaults to `https`, even on `localhost`, which is very cool but not what I wanted for this demo - hence the first block.

The next block listens on port 80 and proxies to `localhost:8003` - with a 30s duration during which incoming requests will "pause" if the backend is not available, and a polling interval of 1s.

## Running Caddy

I started Caddy in the same directory as my `Caddyfile` using:

    caddy run

Then I hit `http://localhost/` in my browser. The browser hung, waiting for the response.

Then I started [Datasette](https://datasette.io/) on port 8003 like this:

    datasette -p 8003

And about a second later my browser returned a response showing the Datasette homepage!

Quitting and restarting Datasette while executing new requests confirmed that traffic was being paused while the backend was unavailable.

## Trying to reconfigure the proxy

My second experiment didn't work, sadly. I wanted to see if I could reconfigure the backend to use `localhost:8004` instead, and then reload Caddy such that paused traffic would resume against the new backend.

I edited the `Caddyfile` to use port `8004` and ran this to hot-reload the configuration:

    caddy reload

New requests did indeed get served from the new backend, but sadly requests that I had already started (and were paused awaiting the backend) did not automatically get served from the new backend - I had to hit refresh in my browser instead.

I [filed a Caddy issue](https://github.com/caddyserver/caddy/issues/4442) about this.

html:
<p>A pattern I really like for zero-downtime deploys is the ability to "pause" HTTP traffic at the load balancer, such that incoming requests from browsers appear to take a few extra seconds to return, but under the hood they've actually been held in a queue while a backend server is swapped out or upgraded in some way.</p>
<p>I first heard about this pattern <a href="https://simonwillison.net/2011/Jun/30/braintree/" rel="nofollow">from Braintree</a>, and a <a href="https://twitter.com/simonw/status/1463652411365494791" rel="nofollow">conversation on Twitter</a> today brought up a few more examples, including <a href="https://github.com/basecamp/intermission">this NGINX Lua config</a> from Basecamp.</p>
<p><a href="https://caddyserver.com/" rel="nofollow">Caddy</a> creator Matt Holt <a href="https://twitter.com/mholt6/status/1463656086360051714" rel="nofollow">pointed me</a> to <a href="https://caddyserver.com/docs/caddyfile/directives/reverse_proxy#lb_try_duration" rel="nofollow">lb_try_duration and lb_try_interval</a> in Caddy, which can hold requests for up to a specific number of seconds, retrying the backend to see if it has become available again.</p>
<p>I decided to try this out. This was my first time using Caddy and I'm really impressed with both the design of the software and the quality of the <a href="https://caddyserver.com/docs/getting-started" rel="nofollow">getting started documentation</a>.</p>
<p>I installed Caddy using Homebrew:</p>
<pre><code>brew install caddy
</code></pre>
<h2>
<a id="user-content-the-caddyfile" class="anchor" href="#the-caddyfile" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>The Caddyfile</h2>
<p>You can configure Caddy in a bunch of different ways - the two main options are using JSON via the Caddy API or using their own custom Caddyfile format.</p>
<p>Here's the <code>Caddyfile</code> I created:</p>
<pre><code>{
    auto_https off
}
:80 {
    reverse_proxy localhost:8003 {
        lb_try_duration 30s
        lb_try_interval 1s
    }
}
</code></pre>
<p>Caddy defaults to <code>https</code>, even on <code>localhost</code>, which is very cool but not what I wanted for this demo - hence the first block.</p>
<p>The next block listens on port 80 and proxies to <code>localhost:8003</code> - with a 30s duration during which incoming requests will "pause" if the backend is not available, and a polling interval of 1s.</p>
<h2>
<a id="user-content-running-caddy" class="anchor" href="#running-caddy" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running Caddy</h2>
<p>I started Caddy in the same directory as my <code>Caddyfile</code> using:</p>
<pre><code>caddy run
</code></pre>
<p>Then I hit <code>http://localhost/</code> in my browser. The browser hung, waiting for the response.</p>
<p>Then I started <a href="https://datasette.io/" rel="nofollow">Datasette</a> on port 8003 like this:</p>
<pre><code>datasette -p 8003
</code></pre>
<p>And about a second later my browser returned a response showing the Datasette homepage!</p>
<p>Quitting and restarting Datasette while executing new requests confirmed that traffic was being paused while the backend was unavailable.</p>
<h2>
<a id="user-content-trying-to-reconfigure-the-proxy" class="anchor" href="#trying-to-reconfigure-the-proxy" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Trying to reconfigure the proxy</h2>
<p>My second experiment didn't work, sadly. I wanted to see if I could reconfigure the backend to use <code>localhost:8004</code> instead, and then reload Caddy such that paused traffic would resume against the new backend.</p>
<p>I edited the <code>Caddyfile</code> to use port <code>8004</code> and ran this to hot-reload the configuration:</p>
<pre><code>caddy reload
</code></pre>
<p>New requests did indeed get served from the new backend, but sadly requests that I had already started (and were paused awaiting the backend) did not automatically get served from the new backend - I had to hit refresh in my browser instead.</p>
<p>I <a href="https://github.com/caddyserver/caddy/issues/4442">filed a Caddy issue</a> about this.</p>


path: npm_publish-web-component.md
topic: npm
title: Publishing a Web Component to npm
url: https://github.com/simonw/til/blob/main/npm/publish-web-component.md
created: 2021-11-28T10:47:55-08:00
created_utc: 2021-11-28T18:47:55+00:00
updated: 2022-07-13T07:48:37-07:00
updated_utc: 2022-07-13T14:48:37+00:00
shot_hash: 2a7f1e13bcc44fa4406e4cdbadd117b6
slug: publish-web-component
body:
I tried this for the first time today with my highly experimental [datasette-table](https://www.npmjs.com/package/datasette-table) Web Component. Here's [the source code](https://github.com/simonw/datasette-table/tree/0.1.0) for version 0.1.0.

I mainly followed the guidelines in Justin Fagnani's guide [How to Publish Web Components to NPM](https://justinfagnani.com/2019/11/01/how-to-publish-web-components-to-npm/).

## Creating the Web Component

See [the source code](https://github.com/simonw/datasette-table/blob/0.1.0/datasette-table.js) to see the full code. It's a single file, `datasette-table.js`, which is structured like this:

```javascript
import {LitElement, html, css} from 'lit';
export class DatasetteTable extends LitElement {
    ...
}
customElements.define("datasette-table", DatasetteTable);
```

## The initial `package.json`

I created my `package.json` using the `npm init` command, which asks questions and generates the file - then I manually edited it to fit Justin's suggestions. It ended up looking like this:

```json
{
  "author": "Simon Willison (https://simonwillison.net/)",
  "name": "datasette-table",
  "version": "0.1.0",
  "type": "module",
  "description": "A Web Component for embedding Datasette tables on a page",
  "main": "datasette-table.js",
  "module": "datasette-table.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
  },
  "dependencies": {
    "lit": "^2.0.0"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/simonw/datasette-table.git"
  },
  "license": "Apache-2.0",
  "bugs": {
    "url": "https://github.com/simonw/datasette-table/issues"
  },
  "homepage": "https://github.com/simonw/datasette-table#readme"
}
```
## Using Vite to load the `lit` dependency

My first version of this ran directly in my browser because I imported `lit` directly like so:

```javascript
import {LitElement, html, css} from 'https://cdn.skypack.dev/lit@2.0.2?min';
```
When I switched to `from 'lit'` I had to add a build tool. I decided to try [Vite](https://vitejs.dev/guide/).

I added the following section to `package.json`:

```
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "dev": "vite",
    "build": "vite build",
    "serve": "vite preview"
  },
  ...
  "devDependencies": {
    "vite": "^2.6.4"
  },
```
Then I ran `npm install` to install Vite and Lit.

Vite needs an `index.html` file, so I created one like so:
```html
<script type="module" src="/datasette-table.js"></script>

<h1>Covid in Greene county, Mississippi</h1>

<datasette-table
  url="https://covid-19.datasettes.com/covid/ny_times_us_counties.json?_size=1000&county=Greene&state=Mississippi"
></datasette-table>

<h1>Power plants</h1>

<datasette-table
  url="https://global-power-plants.datasettes.com/global-power-plants/global-power-plants.json"
></datasette-table>
```

Then I ran `npm run dev` to start the Vite server, then navigated to `http://localhost:3000/` to try out the app.

Vite works by scanning the `index.html` for any JavaScript references, then doing magic on the files that it finds. View source on the running dev server showed that it adds this line to the top of `index.html` when it serves it:

```html
<script type="module" src="/@vite/client"></script>
```
And when it serves the `datasette-table.js` file it rewrites the first import line to look like this:
```script
import {LitElement, html, css} from '/node_modules/.vite/lit.js?v=bf29c8a9';
```

## Publishing to npm

Publishing to npm was almost as easy as running `npm login` - entering my username and password - and then running `npm publish` in that directory. Except this happened:

```
datasette-table % npm publish
...
npm ERR! code E403
npm ERR! 403 403 Forbidden - PUT https://registry.npmjs.org/datasette-table - Forbidden
npm ERR! 403 In most cases, you or one of your dependencies are requesting
npm ERR! 403 a package version that is forbidden by your security policy, or
npm ERR! 403 on a server you do not have access to.
```
That error message was entirely misleading: the issue was that I hadn't yet clicked the "Verify" link in the email they sent me when I created my account! Once I did that re-running `npm publish` worked... and my package showed up live here:

https://www.npmjs.com/package/datasette-table

## Now that it's on npm unpkg and Skypack work

Now that it's live, the following works as a quick way of trying out the Web Component loaded from [unpkg](https://unpkg.com):
```html
<script type="module" src="https://unpkg.com/datasette-table?module"></script>

<datasette-table
    url="https://global-power-plants.datasettes.com/global-power-plants/global-power-plants.json"
></datasette-table>
```
[Skypack](https://www.skypack.dev/) works the same way:
```html
<script type="module" src="https://cdn.skypack.dev/datasette-table"></script>

<datasette-table
    url="https://global-power-plants.datasettes.com/global-power-plants/global-power-plants.json"
></datasette-table>
```

html:
<p>I tried this for the first time today with my highly experimental <a href="https://www.npmjs.com/package/datasette-table" rel="nofollow">datasette-table</a> Web Component. Here's <a href="https://github.com/simonw/datasette-table/tree/0.1.0">the source code</a> for version 0.1.0.</p>
<p>I mainly followed the guidelines in Justin Fagnani's guide <a href="https://justinfagnani.com/2019/11/01/how-to-publish-web-components-to-npm/" rel="nofollow">How to Publish Web Components to NPM</a>.</p>
<h2>
<a id="user-content-creating-the-web-component" class="anchor" href="#creating-the-web-component" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating the Web Component</h2>
<p>See <a href="https://github.com/simonw/datasette-table/blob/0.1.0/datasette-table.js">the source code</a> to see the full code. It's a single file, <code>datasette-table.js</code>, which is structured like this:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">import</span> <span class="pl-kos">{</span><span class="pl-v">LitElement</span><span class="pl-kos">,</span> <span class="pl-s1">html</span><span class="pl-kos">,</span> <span class="pl-s1">css</span><span class="pl-kos">}</span> <span class="pl-k">from</span> <span class="pl-s">'lit'</span><span class="pl-kos">;</span>
<span class="pl-k">export</span> <span class="pl-k">class</span> <span class="pl-v">DatasetteTable</span> <span class="pl-k">extends</span> <span class="pl-v">LitElement</span> <span class="pl-kos">{</span>
    ...
<span class="pl-kos">}</span>
<span class="pl-s1">customElements</span><span class="pl-kos">.</span><span class="pl-en">define</span><span class="pl-kos">(</span><span class="pl-s">"datasette-table"</span><span class="pl-kos">,</span> <span class="pl-v">DatasetteTable</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<h2>
<a id="user-content-the-initial-packagejson" class="anchor" href="#the-initial-packagejson" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>The initial <code>package.json</code>
</h2>
<p>I created my <code>package.json</code> using the <code>npm init</code> command, which asks questions and generates the file - then I manually edited it to fit Justin's suggestions. It ended up looking like this:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-ent">"author"</span>: <span class="pl-s"><span class="pl-pds">"</span>Simon Willison (https://simonwillison.net/)<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>datasette-table<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"version"</span>: <span class="pl-s"><span class="pl-pds">"</span>0.1.0<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>module<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"description"</span>: <span class="pl-s"><span class="pl-pds">"</span>A Web Component for embedding Datasette tables on a page<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"main"</span>: <span class="pl-s"><span class="pl-pds">"</span>datasette-table.js<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"module"</span>: <span class="pl-s"><span class="pl-pds">"</span>datasette-table.js<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"scripts"</span>: {
    <span class="pl-ent">"test"</span>: <span class="pl-s"><span class="pl-pds">"</span>echo <span class="pl-cce">\"</span>Error: no test specified<span class="pl-cce">\"</span> &amp;&amp; exit 1<span class="pl-pds">"</span></span>,
  },
  <span class="pl-ent">"dependencies"</span>: {
    <span class="pl-ent">"lit"</span>: <span class="pl-s"><span class="pl-pds">"</span>^2.0.0<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"repository"</span>: {
    <span class="pl-ent">"type"</span>: <span class="pl-s"><span class="pl-pds">"</span>git<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"url"</span>: <span class="pl-s"><span class="pl-pds">"</span>git+https://github.com/simonw/datasette-table.git<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"license"</span>: <span class="pl-s"><span class="pl-pds">"</span>Apache-2.0<span class="pl-pds">"</span></span>,
  <span class="pl-ent">"bugs"</span>: {
    <span class="pl-ent">"url"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://github.com/simonw/datasette-table/issues<span class="pl-pds">"</span></span>
  },
  <span class="pl-ent">"homepage"</span>: <span class="pl-s"><span class="pl-pds">"</span>https://github.com/simonw/datasette-table#readme<span class="pl-pds">"</span></span>
}</pre></div>
<h2>
<a id="user-content-using-vite-to-load-the-lit-dependency" class="anchor" href="#using-vite-to-load-the-lit-dependency" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using Vite to load the <code>lit</code> dependency</h2>
<p>My first version of this ran directly in my browser because I imported <code>lit</code> directly like so:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">import</span> <span class="pl-kos">{</span><span class="pl-v">LitElement</span><span class="pl-kos">,</span> <span class="pl-s1">html</span><span class="pl-kos">,</span> <span class="pl-s1">css</span><span class="pl-kos">}</span> <span class="pl-k">from</span> <span class="pl-s">'https://cdn.skypack.dev/lit@2.0.2?min'</span><span class="pl-kos">;</span></pre></div>
<p>When I switched to <code>from 'lit'</code> I had to add a build tool. I decided to try <a href="https://vitejs.dev/guide/" rel="nofollow">Vite</a>.</p>
<p>I added the following section to <code>package.json</code>:</p>
<pre><code>  "scripts": {
    "test": "echo \"Error: no test specified\" &amp;&amp; exit 1",
    "dev": "vite",
    "build": "vite build",
    "serve": "vite preview"
  },
  ...
  "devDependencies": {
    "vite": "^2.6.4"
  },
</code></pre>
<p>Then I ran <code>npm install</code> to install Vite and Lit.</p>
<p>Vite needs an <code>index.html</code> file, so I created one like so:</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">script</span> <span class="pl-c1">type</span>="<span class="pl-s">module</span>" <span class="pl-c1">src</span>="<span class="pl-s">/datasette-table.js</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span>

<span class="pl-kos">&lt;</span><span class="pl-ent">h1</span><span class="pl-kos">&gt;</span>Covid in Greene county, Mississippi<span class="pl-kos">&lt;/</span><span class="pl-ent">h1</span><span class="pl-kos">&gt;</span>

<span class="pl-kos">&lt;</span><span class="pl-ent">datasette-table</span>
  <span class="pl-c1">url</span>="<span class="pl-s">https://covid-19.datasettes.com/covid/ny_times_us_counties.json?_size=1000&amp;county=Greene&amp;state=Mississippi</span>"
<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">datasette-table</span><span class="pl-kos">&gt;</span>

<span class="pl-kos">&lt;</span><span class="pl-ent">h1</span><span class="pl-kos">&gt;</span>Power plants<span class="pl-kos">&lt;/</span><span class="pl-ent">h1</span><span class="pl-kos">&gt;</span>

<span class="pl-kos">&lt;</span><span class="pl-ent">datasette-table</span>
  <span class="pl-c1">url</span>="<span class="pl-s">https://global-power-plants.datasettes.com/global-power-plants/global-power-plants.json</span>"
<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">datasette-table</span><span class="pl-kos">&gt;</span></pre></div>
<p>Then I ran <code>npm run dev</code> to start the Vite server, then navigated to <code>http://localhost:3000/</code> to try out the app.</p>
<p>Vite works by scanning the <code>index.html</code> for any JavaScript references, then doing magic on the files that it finds. View source on the running dev server showed that it adds this line to the top of <code>index.html</code> when it serves it:</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">script</span> <span class="pl-c1">type</span>="<span class="pl-s">module</span>" <span class="pl-c1">src</span>="<span class="pl-s">/@vite/client</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span></pre></div>
<p>And when it serves the <code>datasette-table.js</code> file it rewrites the first import line to look like this:</p>
<pre lang="script"><code>import {LitElement, html, css} from '/node_modules/.vite/lit.js?v=bf29c8a9';
</code></pre>
<h2>
<a id="user-content-publishing-to-npm" class="anchor" href="#publishing-to-npm" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Publishing to npm</h2>
<p>Publishing to npm was almost as easy as running <code>npm login</code> - entering my username and password - and then running <code>npm publish</code> in that directory. Except this happened:</p>
<pre><code>datasette-table % npm publish
...
npm ERR! code E403
npm ERR! 403 403 Forbidden - PUT https://registry.npmjs.org/datasette-table - Forbidden
npm ERR! 403 In most cases, you or one of your dependencies are requesting
npm ERR! 403 a package version that is forbidden by your security policy, or
npm ERR! 403 on a server you do not have access to.
</code></pre>
<p>That error message was entirely misleading: the issue was that I hadn't yet clicked the "Verify" link in the email they sent me when I created my account! Once I did that re-running <code>npm publish</code> worked... and my package showed up live here:</p>
<p><a href="https://www.npmjs.com/package/datasette-table" rel="nofollow">https://www.npmjs.com/package/datasette-table</a></p>
<h2>
<a id="user-content-now-that-its-on-npm-unpkg-and-skypack-work" class="anchor" href="#now-that-its-on-npm-unpkg-and-skypack-work" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Now that it's on npm unpkg and Skypack work</h2>
<p>Now that it's live, the following works as a quick way of trying out the Web Component loaded from <a href="https://unpkg.com" rel="nofollow">unpkg</a>:</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">script</span> <span class="pl-c1">type</span>="<span class="pl-s">module</span>" <span class="pl-c1">src</span>="<span class="pl-s">https://unpkg.com/datasette-table?module</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span>

<span class="pl-kos">&lt;</span><span class="pl-ent">datasette-table</span>
    <span class="pl-c1">url</span>="<span class="pl-s">https://global-power-plants.datasettes.com/global-power-plants/global-power-plants.json</span>"
<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">datasette-table</span><span class="pl-kos">&gt;</span></pre></div>
<p><a href="https://www.skypack.dev/" rel="nofollow">Skypack</a> works the same way:</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">script</span> <span class="pl-c1">type</span>="<span class="pl-s">module</span>" <span class="pl-c1">src</span>="<span class="pl-s">https://cdn.skypack.dev/datasette-table</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span>

<span class="pl-kos">&lt;</span><span class="pl-ent">datasette-table</span>
    <span class="pl-c1">url</span>="<span class="pl-s">https://global-power-plants.datasettes.com/global-power-plants/global-power-plants.json</span>"
<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">datasette-table</span><span class="pl-kos">&gt;</span></pre></div>


path: datasette_reuse-click-for-register-commands.md
topic: datasette
title: Reusing an existing Click tool with register_commands
url: https://github.com/simonw/til/blob/main/datasette/reuse-click-for-register-commands.md
created: 2021-11-29T09:32:07-08:00
created_utc: 2021-11-29T17:32:07+00:00
updated: 2021-11-29T09:32:07-08:00
updated_utc: 2021-11-29T17:32:07+00:00
shot_hash: 3322ac874006f755fec92e2caa623e21
slug: reuse-click-for-register-commands
body:
The [register_commands](https://docs.datasette.io/en/stable/plugin_hooks.html#register-commands-cli) plugin hook lets you add extra sub-commands to the `datasette` CLI tool.

I have a lot of existing tools that I'd like to also make available as plugins. I figured out this pattern for my [git-history](https://datasette.io/tools/git-history) tool today:

```python
from datasette import hookimpl
from git_history.cli import cli as git_history_cli

@hookimpl
def register_commands(cli):
    cli.add_command(git_history_cli, name="git-history")
```
Now I can run the following:

```
% datasette git-history --help
Usage: datasette git-history [OPTIONS] COMMAND [ARGS]...

  Tools for analyzing Git history using SQLite

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  file  Analyze the history of a specific file and write it to SQLite
```

I initially tried doing this:

```python
@hookimpl
def register_commands(cli):
    cli.command(name="git-history")(git_history_file)
```
But got the following error:

    TypeError: Attempted to convert a callback into a command twice.

Using [cli.add_command()](https://click.palletsprojects.com/en/8.0.x/api/?highlight=add_command#click.Group.add_command) turns out to be the right way to do this.

Research issue for this: [datasette#1538](https://github.com/simonw/datasette/issues/1538).

html:
<p>The <a href="https://docs.datasette.io/en/stable/plugin_hooks.html#register-commands-cli" rel="nofollow">register_commands</a> plugin hook lets you add extra sub-commands to the <code>datasette</code> CLI tool.</p>
<p>I have a lot of existing tools that I'd like to also make available as plugins. I figured out this pattern for my <a href="https://datasette.io/tools/git-history" rel="nofollow">git-history</a> tool today:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">datasette</span> <span class="pl-k">import</span> <span class="pl-s1">hookimpl</span>
<span class="pl-k">from</span> <span class="pl-s1">git_history</span>.<span class="pl-s1">cli</span> <span class="pl-k">import</span> <span class="pl-s1">cli</span> <span class="pl-k">as</span> <span class="pl-s1">git_history_cli</span>

<span class="pl-en">@<span class="pl-s1">hookimpl</span></span>
<span class="pl-k">def</span> <span class="pl-en">register_commands</span>(<span class="pl-s1">cli</span>):
    <span class="pl-s1">cli</span>.<span class="pl-en">add_command</span>(<span class="pl-s1">git_history_cli</span>, <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"git-history"</span>)</pre></div>
<p>Now I can run the following:</p>
<pre><code>% datasette git-history --help
Usage: datasette git-history [OPTIONS] COMMAND [ARGS]...

  Tools for analyzing Git history using SQLite

Options:
  --version  Show the version and exit.
  --help     Show this message and exit.

Commands:
  file  Analyze the history of a specific file and write it to SQLite
</code></pre>
<p>I initially tried doing this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">hookimpl</span></span>
<span class="pl-k">def</span> <span class="pl-en">register_commands</span>(<span class="pl-s1">cli</span>):
    <span class="pl-s1">cli</span>.<span class="pl-en">command</span>(<span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"git-history"</span>)(<span class="pl-s1">git_history_file</span>)</pre></div>
<p>But got the following error:</p>
<pre><code>TypeError: Attempted to convert a callback into a command twice.
</code></pre>
<p>Using <a href="https://click.palletsprojects.com/en/8.0.x/api/?highlight=add_command#click.Group.add_command" rel="nofollow">cli.add_command()</a> turns out to be the right way to do this.</p>
<p>Research issue for this: <a href="https://github.com/simonw/datasette/issues/1538">datasette#1538</a>.</p>


path: python_ignore-both-flake8-and-mypy.md
topic: python
title: Ignoring a line in both flake8 and mypy
url: https://github.com/simonw/til/blob/main/python/ignore-both-flake8-and-mypy.md
created: 2021-11-30T18:16:19-08:00
created_utc: 2021-12-01T02:16:19+00:00
updated: 2021-11-30T18:16:19-08:00
updated_utc: 2021-12-01T02:16:19+00:00
shot_hash: 8ec51ba5f36ac411a6966d138a342ad5
slug: ignore-both-flake8-and-mypy
body:
I [needed to tell](https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982133970) both `flake8` and `mypy` to ignore the same line of code.

This worked:

```python
from sqlite3.dump import _iterdump as iterdump # type: ignore # noqa: F401
```

The order here mattered. This did not get picked up by both tools:

    # noqa: F401 # type: ignore

But this did:

    # type: ignore # noqa: F401

html:
<p>I <a href="https://github.com/simonw/sqlite-utils/pull/347#issuecomment-982133970">needed to tell</a> both <code>flake8</code> and <code>mypy</code> to ignore the same line of code.</p>
<p>This worked:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">sqlite3</span>.<span class="pl-s1">dump</span> <span class="pl-k">import</span> <span class="pl-s1">_iterdump</span> <span class="pl-k">as</span> <span class="pl-s1">iterdump</span> <span class="pl-c"># type: ignore # noqa: F401</span></pre></div>
<p>The order here mattered. This did not get picked up by both tools:</p>
<pre><code># noqa: F401 # type: ignore
</code></pre>
<p>But this did:</p>
<pre><code># type: ignore # noqa: F401
</code></pre>


path: python_init-subclass.md
topic: python
title: __init_subclass__
url: https://github.com/simonw/til/blob/main/python/init-subclass.md
created: 2021-12-03T11:19:51-08:00
created_utc: 2021-12-03T19:19:51+00:00
updated: 2021-12-03T11:22:59-08:00
updated_utc: 2021-12-03T19:22:59+00:00
shot_hash: 298580095fd089e62c9ab73324582885
slug: init-subclass
body:
David Beazley [on Twitter](https://twitter.com/dabeaz/status/1466731368956809219) said:

> I think 95% of the problems once solved by a metaclass can be solved by `__init_subclass__` instead

This inspired me to finally learn how to use it! I used [my asyncinject project](https://github.com/simonw/asyncinject/issues/2) as an experimental playground.

The `__init_subclass__` class method is called when the class itself is being constructed. It gets passed the `cls` and can make modifications to it.

Here's the pattern I used:

```python
class AsyncInject:
    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        # Decorate any items that are 'async def' methods
        cls._registry = {}
        inject_all = getattr(cls, "_inject_all", False)
        for name in dir(cls):
            value = getattr(cls, name)
            if inspect.iscoroutinefunction(value) and (
                inject_all or getattr(value, "_inject", None)
            ):
                setattr(cls, name, _make_method(value))
                cls._registry[name] = getattr(cls, name)
        # Gather graph for later dependency resolution
        graph = {
            key: {
                p
                for p in inspect.signature(method).parameters.keys()
                if p != "self" and not p.startswith("_")
            }
            for key, method in cls._registry.items()
        }
        cls._graph = graph
```
As you can see, it's using `getattr()` and `setattr()` against the `cls` object to make modifications to the class - in this case it's running a decorator against various methods and adding two new class properties, `_registry` and `_graph`.

The `**kwargs` thing there is interesting: you can define keyword arguments and use them when you subclass, like this:

```python
class MySubClass(AsyncInject, inject_all=True):
    ...
```
This doesn't work with my above example, but I could change it to start like this instead:
```python
class AsyncInject:
    def __init_subclass__(cls, inject_all=False, **kwargs):
        super().__init_subclass__(**kwargs)
        # Decorate any items that are 'async def' methods
        cls._registry = {}
        for name in dir(cls):
            value = getattr(cls, name)
            if inspect.iscoroutinefunction(value) and (
                inject_all or getattr(value, "_inject", None)
            ):
                setattr(cls, name, _make_method(value))
                cls._registry[name] = getattr(cls, name)
```
## Further reading


- [Use \_\_init_subclass\_\_ hook to validate subclasses](https://github.com/rednafi/notes/blob/master/notes/python/use_init_subclass_hook_to_validate_subclasses.md) [via Redowan Delowar](https://twitter.com/rednafi/status/1466833693993930755)
- [PEP 487 -- Simpler customisation of class creation](https://www.python.org/dev/peps/pep-0487/)

html:
<p>David Beazley <a href="https://twitter.com/dabeaz/status/1466731368956809219" rel="nofollow">on Twitter</a> said:</p>
<blockquote>
<p>I think 95% of the problems once solved by a metaclass can be solved by <code>__init_subclass__</code> instead</p>
</blockquote>
<p>This inspired me to finally learn how to use it! I used <a href="https://github.com/simonw/asyncinject/issues/2">my asyncinject project</a> as an experimental playground.</p>
<p>The <code>__init_subclass__</code> class method is called when the class itself is being constructed. It gets passed the <code>cls</code> and can make modifications to it.</p>
<p>Here's the pattern I used:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">class</span> <span class="pl-v">AsyncInject</span>:
    <span class="pl-k">def</span> <span class="pl-en">__init_subclass__</span>(<span class="pl-s1">cls</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
        <span class="pl-en">super</span>().<span class="pl-en">__init_subclass__</span>(<span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
        <span class="pl-c"># Decorate any items that are 'async def' methods</span>
        <span class="pl-s1">cls</span>.<span class="pl-s1">_registry</span> <span class="pl-c1">=</span> {}
        <span class="pl-s1">inject_all</span> <span class="pl-c1">=</span> <span class="pl-en">getattr</span>(<span class="pl-s1">cls</span>, <span class="pl-s">"_inject_all"</span>, <span class="pl-c1">False</span>)
        <span class="pl-k">for</span> <span class="pl-s1">name</span> <span class="pl-c1">in</span> <span class="pl-en">dir</span>(<span class="pl-s1">cls</span>):
            <span class="pl-s1">value</span> <span class="pl-c1">=</span> <span class="pl-en">getattr</span>(<span class="pl-s1">cls</span>, <span class="pl-s1">name</span>)
            <span class="pl-k">if</span> <span class="pl-s1">inspect</span>.<span class="pl-en">iscoroutinefunction</span>(<span class="pl-s1">value</span>) <span class="pl-c1">and</span> (
                <span class="pl-s1">inject_all</span> <span class="pl-c1">or</span> <span class="pl-en">getattr</span>(<span class="pl-s1">value</span>, <span class="pl-s">"_inject"</span>, <span class="pl-c1">None</span>)
            ):
                <span class="pl-en">setattr</span>(<span class="pl-s1">cls</span>, <span class="pl-s1">name</span>, <span class="pl-en">_make_method</span>(<span class="pl-s1">value</span>))
                <span class="pl-s1">cls</span>.<span class="pl-s1">_registry</span>[<span class="pl-s1">name</span>] <span class="pl-c1">=</span> <span class="pl-en">getattr</span>(<span class="pl-s1">cls</span>, <span class="pl-s1">name</span>)
        <span class="pl-c"># Gather graph for later dependency resolution</span>
        <span class="pl-s1">graph</span> <span class="pl-c1">=</span> {
            <span class="pl-s1">key</span>: {
                <span class="pl-s1">p</span>
                <span class="pl-k">for</span> <span class="pl-s1">p</span> <span class="pl-c1">in</span> <span class="pl-s1">inspect</span>.<span class="pl-en">signature</span>(<span class="pl-s1">method</span>).<span class="pl-s1">parameters</span>.<span class="pl-en">keys</span>()
                <span class="pl-k">if</span> <span class="pl-s1">p</span> <span class="pl-c1">!=</span> <span class="pl-s">"self"</span> <span class="pl-c1">and</span> <span class="pl-c1">not</span> <span class="pl-s1">p</span>.<span class="pl-en">startswith</span>(<span class="pl-s">"_"</span>)
            }
            <span class="pl-k">for</span> <span class="pl-s1">key</span>, <span class="pl-s1">method</span> <span class="pl-c1">in</span> <span class="pl-s1">cls</span>.<span class="pl-s1">_registry</span>.<span class="pl-en">items</span>()
        }
        <span class="pl-s1">cls</span>.<span class="pl-s1">_graph</span> <span class="pl-c1">=</span> <span class="pl-s1">graph</span></pre></div>
<p>As you can see, it's using <code>getattr()</code> and <code>setattr()</code> against the <code>cls</code> object to make modifications to the class - in this case it's running a decorator against various methods and adding two new class properties, <code>_registry</code> and <code>_graph</code>.</p>
<p>The <code>**kwargs</code> thing there is interesting: you can define keyword arguments and use them when you subclass, like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">class</span> <span class="pl-v">MySubClass</span>(<span class="pl-v">AsyncInject</span>, <span class="pl-s1">inject_all</span><span class="pl-c1">=</span><span class="pl-c1">True</span>):
    ...</pre></div>
<p>This doesn't work with my above example, but I could change it to start like this instead:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">class</span> <span class="pl-v">AsyncInject</span>:
    <span class="pl-k">def</span> <span class="pl-en">__init_subclass__</span>(<span class="pl-s1">cls</span>, <span class="pl-s1">inject_all</span><span class="pl-c1">=</span><span class="pl-c1">False</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
        <span class="pl-en">super</span>().<span class="pl-en">__init_subclass__</span>(<span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
        <span class="pl-c"># Decorate any items that are 'async def' methods</span>
        <span class="pl-s1">cls</span>.<span class="pl-s1">_registry</span> <span class="pl-c1">=</span> {}
        <span class="pl-k">for</span> <span class="pl-s1">name</span> <span class="pl-c1">in</span> <span class="pl-en">dir</span>(<span class="pl-s1">cls</span>):
            <span class="pl-s1">value</span> <span class="pl-c1">=</span> <span class="pl-en">getattr</span>(<span class="pl-s1">cls</span>, <span class="pl-s1">name</span>)
            <span class="pl-k">if</span> <span class="pl-s1">inspect</span>.<span class="pl-en">iscoroutinefunction</span>(<span class="pl-s1">value</span>) <span class="pl-c1">and</span> (
                <span class="pl-s1">inject_all</span> <span class="pl-c1">or</span> <span class="pl-en">getattr</span>(<span class="pl-s1">value</span>, <span class="pl-s">"_inject"</span>, <span class="pl-c1">None</span>)
            ):
                <span class="pl-en">setattr</span>(<span class="pl-s1">cls</span>, <span class="pl-s1">name</span>, <span class="pl-en">_make_method</span>(<span class="pl-s1">value</span>))
                <span class="pl-s1">cls</span>.<span class="pl-s1">_registry</span>[<span class="pl-s1">name</span>] <span class="pl-c1">=</span> <span class="pl-en">getattr</span>(<span class="pl-s1">cls</span>, <span class="pl-s1">name</span>)</pre></div>
<h2>
<a id="user-content-further-reading" class="anchor" href="#further-reading" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Further reading</h2>
<ul>
<li>
<a href="https://github.com/rednafi/notes/blob/master/notes/python/use_init_subclass_hook_to_validate_subclasses.md">Use __init_subclass__ hook to validate subclasses</a> <a href="https://twitter.com/rednafi/status/1466833693993930755" rel="nofollow">via Redowan Delowar</a>
</li>
<li><a href="https://www.python.org/dev/peps/pep-0487/" rel="nofollow">PEP 487 -- Simpler customisation of class creation</a></li>
</ul>


path: github-actions_s3-bucket-github-actions.md
topic: github-actions
title: Storing files in an S3 bucket between GitHub Actions runs
url: https://github.com/simonw/til/blob/main/github-actions/s3-bucket-github-actions.md
created: 2021-12-07T11:52:18-08:00
created_utc: 2021-12-07T19:52:18+00:00
updated: 2021-12-07T13:17:24-08:00
updated_utc: 2021-12-07T21:17:24+00:00
shot_hash: 97320e848a480af2db589c6954bc508d
slug: s3-bucket-github-actions
body:
For my [git-history live demos](https://github.com/simonw/git-history/issues/30) I needed to store quite large files (~200MB SQLite databases) in between GitHub Actions runs, to avoid having to recreate the entire file from scratch every time.

## Creating the bucket and credentials

I used my [s3-credentials](https://github.com/simonw/s3-credentials) tool to create an S3 bucket with permanent, locked-down credentials.

```
~ % s3-credentials create git-history-demos --public --create-bucket
Created bucket: git-history-demos
Attached bucket policy allowing public access
Created  user: 's3.read-write.git-history-demos' with permissions boundary: 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
Attached policy s3.read-write.git-history-demos to user s3.read-write.git-history-demos
Created access key for user: s3.read-write.git-history-demos
{
    "UserName": "s3.read-write.git-history-demos",
    "AccessKeyId": "AKIAWXFXAIOZOLWKY4FP",
    "Status": "Active",
    "SecretAccessKey": “…”,;
    "CreateDate": "2021-12-07 07:11:48+00:00"
}
```
I saved the new access key and secret key to `SECRETS` in my repository called `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`.

## Downloading the files as part of the Actions workflow

Since I used `--public` to create the bucket, I could download files inside the action like this:

    curl -o ca-fires.db https://s3.amazonaws.com/git-history-demos/ca-fires.db

If I had not used `--public` I could use `s3-credentials get-object` instead:

    s3-credentials get-object git-history-demos ca-fires.db -o ca-fires.db

I would need to expose the secrets as environment variables first, see below.

I actually wanted to attempt the download but keep running if the file was not there, because in this case my scripts will generate the file from scratch if it's not already in the bucket. I used this pattern for that:

```yaml
    - name: Download ca-fires.db
      run: curl --fail -o ca-fires.db https://s3.amazonaws.com/git-history-demos/ca-fires.db
      continue-on-error: true
```

## Uploading the files to the bucket

My step to upload the generated files to the bucket looks like this:

```yaml
    - name: Upload databases to S3
      env:
        AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
        AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
      run: |-
        s3-credentials put-object git-history-demos pge-outages.db pge-outages.db
        s3-credentials put-object git-history-demos ca-fires.db ca-fires.db
        s3-credentials put-object git-history-demos sf-bay-511.db sf-bay-511.db
```
Here I'm passing through the repository secrets as environment variables that `s3-credentials put-object` can then use.

Full workflow is here: https://github.com/simonw/git-history/blob/main/.github/workflows/deploy-demos.yml

html:
<p>For my <a href="https://github.com/simonw/git-history/issues/30">git-history live demos</a> I needed to store quite large files (~200MB SQLite databases) in between GitHub Actions runs, to avoid having to recreate the entire file from scratch every time.</p>
<h2>
<a id="user-content-creating-the-bucket-and-credentials" class="anchor" href="#creating-the-bucket-and-credentials" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating the bucket and credentials</h2>
<p>I used my <a href="https://github.com/simonw/s3-credentials">s3-credentials</a> tool to create an S3 bucket with permanent, locked-down credentials.</p>
<pre><code>~ % s3-credentials create git-history-demos --public --create-bucket
Created bucket: git-history-demos
Attached bucket policy allowing public access
Created  user: 's3.read-write.git-history-demos' with permissions boundary: 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
Attached policy s3.read-write.git-history-demos to user s3.read-write.git-history-demos
Created access key for user: s3.read-write.git-history-demos
{
    "UserName": "s3.read-write.git-history-demos",
    "AccessKeyId": "AKIAWXFXAIOZOLWKY4FP",
    "Status": "Active",
    "SecretAccessKey": “…”,;
    "CreateDate": "2021-12-07 07:11:48+00:00"
}
</code></pre>
<p>I saved the new access key and secret key to <code>SECRETS</code> in my repository called <code>AWS_ACCESS_KEY_ID</code> and <code>AWS_SECRET_ACCESS_KEY</code>.</p>
<h2>
<a id="user-content-downloading-the-files-as-part-of-the-actions-workflow" class="anchor" href="#downloading-the-files-as-part-of-the-actions-workflow" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Downloading the files as part of the Actions workflow</h2>
<p>Since I used <code>--public</code> to create the bucket, I could download files inside the action like this:</p>
<pre><code>curl -o ca-fires.db https://s3.amazonaws.com/git-history-demos/ca-fires.db
</code></pre>
<p>If I had not used <code>--public</code> I could use <code>s3-credentials get-object</code> instead:</p>
<pre><code>s3-credentials get-object git-history-demos ca-fires.db -o ca-fires.db
</code></pre>
<p>I would need to expose the secrets as environment variables first, see below.</p>
<p>I actually wanted to attempt the download but keep running if the file was not there, because in this case my scripts will generate the file from scratch if it's not already in the bucket. I used this pattern for that:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Download ca-fires.db</span>
      <span class="pl-ent">run</span>: <span class="pl-s">curl --fail -o ca-fires.db https://s3.amazonaws.com/git-history-demos/ca-fires.db</span>
      <span class="pl-ent">continue-on-error</span>: <span class="pl-c1">true</span></pre></div>
<h2>
<a id="user-content-uploading-the-files-to-the-bucket" class="anchor" href="#uploading-the-files-to-the-bucket" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Uploading the files to the bucket</h2>
<p>My step to upload the generated files to the bucket looks like this:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Upload databases to S3</span>
      <span class="pl-ent">env</span>:
        <span class="pl-ent">AWS_ACCESS_KEY_ID</span>: <span class="pl-s">${{ secrets.AWS_ACCESS_KEY_ID }}</span>
        <span class="pl-ent">AWS_SECRET_ACCESS_KEY</span>: <span class="pl-s">${{ secrets.AWS_SECRET_ACCESS_KEY }}</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        s3-credentials put-object git-history-demos pge-outages.db pge-outages.db</span>
<span class="pl-s">        s3-credentials put-object git-history-demos ca-fires.db ca-fires.db</span>
<span class="pl-s">        s3-credentials put-object git-history-demos sf-bay-511.db sf-bay-511.db</span></pre></div>
<p>Here I'm passing through the repository secrets as environment variables that <code>s3-credentials put-object</code> can then use.</p>
<p>Full workflow is here: <a href="https://github.com/simonw/git-history/blob/main/.github/workflows/deploy-demos.yml">https://github.com/simonw/git-history/blob/main/.github/workflows/deploy-demos.yml</a></p>


path: python_using-c-include-path-to-install-python-packages.md
topic: python
title: Using C_INCLUDE_PATH to install Python packages
url: https://github.com/simonw/til/blob/main/python/using-c-include-path-to-install-python-packages.md
created: 2021-12-09T14:27:16-08:00
created_utc: 2021-12-09T22:27:16+00:00
updated: 2021-12-09T14:27:16-08:00
updated_utc: 2021-12-09T22:27:16+00:00
shot_hash: c562d9cc3716a445db1adcd73f7d9b6e
slug: using-c-include-path-to-install-python-packages
body:
I tried to install my [datasette-bplist](https://github.com/simonw/datasette-bplist) plugin today in a fresh Python 3.10 virtual environment on macOS and got this error:

```
% pip install datasette-bplist
...
    src/bplist.c:2:10: fatal error: 'pytime.h' file not found
    #include <pytime.h>
             ^~~~~~~~~~
    1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/simon/.local/share/virtualenvs/datasette-nska-phcxS5VC/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pip-install-8vbksayc/bpylist_af584b1da32f4f7ea8f6eb8548355128/setup.py'"'"'; __file__='"'"'/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pip-install-8vbksayc/bpylist_af584b1da32f4f7ea8f6eb8548355128/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pip-record-2m2ya37u/install-record.txt --single-version-externally-managed --compile --install-headers /Users/simon/.local/share/virtualenvs/datasette-nska-phcxS5VC/include/site/python3.10/bpylist Check the logs for full command output.
```
Thanks [to StackOverflow](https://stackoverflow.com/questions/35778495/fatal-error-python-h-file-not-found-while-installing-opencv/35778751) I figured out a workaround. The problem was that the build script didn't know where to find the `pytime.h` header - the solution was to use the `C_INCLUDE_PATH` environment variable to feed it the parent directory of that file.

I found the missing `pytime.h` header file using Spotlight (`mdfind`) like this:
```
% mdfind pytime.h
/usr/local/Cellar/python@3.7/3.7.12/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pytime.h
/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/include/python3.10/cpython/pytime.h
/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/include/python3.9/pytime.h
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/Headers/pytime.h
/usr/local/Cellar/python@3.8/3.8.12/Frameworks/Python.framework/Versions/3.8/include/python3.8/pytime.h
/Users/simon/Dropbox/Development/datasette-app/python/include/python3.9/pytime.h
/Users/simon/Dropbox/Development/datasette-app/python/lib/python3.9/config-3.9-darwin/Makefile
/Users/simon/Dropbox/Development/datasette-app/python/include/python3.9/Python.h
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/Headers/Python.h
```
Since `python --version` in this environment returned `Python 3.10.0` I decided to go with that second one.

`C_INCLUDE_PATH` accepts one or more directory paths (not file paths), separated by a `:`. I used the following one-liner to successfully install the package:

`C_INCLUDE_PATH=/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/include/python3.10/cpython pip install datasette-bplist`

html:
<p>I tried to install my <a href="https://github.com/simonw/datasette-bplist">datasette-bplist</a> plugin today in a fresh Python 3.10 virtual environment on macOS and got this error:</p>
<pre><code>% pip install datasette-bplist
...
    src/bplist.c:2:10: fatal error: 'pytime.h' file not found
    #include &lt;pytime.h&gt;
             ^~~~~~~~~~
    1 error generated.
    error: command '/usr/bin/clang' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /Users/simon/.local/share/virtualenvs/datasette-nska-phcxS5VC/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pip-install-8vbksayc/bpylist_af584b1da32f4f7ea8f6eb8548355128/setup.py'"'"'; __file__='"'"'/private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pip-install-8vbksayc/bpylist_af584b1da32f4f7ea8f6eb8548355128/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/wr/hn3206rs1yzgq3r49bz8nvnh0000gn/T/pip-record-2m2ya37u/install-record.txt --single-version-externally-managed --compile --install-headers /Users/simon/.local/share/virtualenvs/datasette-nska-phcxS5VC/include/site/python3.10/bpylist Check the logs for full command output.
</code></pre>
<p>Thanks <a href="https://stackoverflow.com/questions/35778495/fatal-error-python-h-file-not-found-while-installing-opencv/35778751" rel="nofollow">to StackOverflow</a> I figured out a workaround. The problem was that the build script didn't know where to find the <code>pytime.h</code> header - the solution was to use the <code>C_INCLUDE_PATH</code> environment variable to feed it the parent directory of that file.</p>
<p>I found the missing <code>pytime.h</code> header file using Spotlight (<code>mdfind</code>) like this:</p>
<pre><code>% mdfind pytime.h
/usr/local/Cellar/python@3.7/3.7.12/Frameworks/Python.framework/Versions/3.7/include/python3.7m/pytime.h
/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/include/python3.10/cpython/pytime.h
/usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/include/python3.9/pytime.h
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/Headers/pytime.h
/usr/local/Cellar/python@3.8/3.8.12/Frameworks/Python.framework/Versions/3.8/include/python3.8/pytime.h
/Users/simon/Dropbox/Development/datasette-app/python/include/python3.9/pytime.h
/Users/simon/Dropbox/Development/datasette-app/python/lib/python3.9/config-3.9-darwin/Makefile
/Users/simon/Dropbox/Development/datasette-app/python/include/python3.9/Python.h
/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/Headers/Python.h
</code></pre>
<p>Since <code>python --version</code> in this environment returned <code>Python 3.10.0</code> I decided to go with that second one.</p>
<p><code>C_INCLUDE_PATH</code> accepts one or more directory paths (not file paths), separated by a <code>:</code>. I used the following one-liner to successfully install the package:</p>
<p><code>C_INCLUDE_PATH=/usr/local/Cellar/python@3.10/3.10.0_1/Frameworks/Python.framework/Versions/3.10/include/python3.10/cpython pip install datasette-bplist</code></p>


path: macos_lsof-macos.md
topic: macos
title: Using lsof on macOS
url: https://github.com/simonw/til/blob/main/macos/lsof-macos.md
created: 2021-12-11T14:37:48-08:00
created_utc: 2021-12-11T22:37:48+00:00
updated: 2022-06-09T08:17:38-07:00
updated_utc: 2022-06-09T15:17:38+00:00
shot_hash: 3a39b1aebd1a183035c4b25134f7c893
slug: lsof-macos
body:
`lsof` stands for "list open files". Here are some very basic usage notes for the version that ships with macOS.

## Listing processes that have a specific file open
```
~ % lsof /tmp/covid-smaller.db
COMMAND   PID  USER   FD   TYPE DEVICE  SIZE/OFF      NODE NAME
Python  17990 simon    6r   REG    1,5 747413504 187550734 /private/tmp/covid-smaller.db
Python  17990 simon   12r   REG    1,5 747413504 187550734 /private/tmp/covid-smaller.db
```
Note that this file is opened twice by the same process (Datasette maintains multiple connections to the same SQLite database). The `PID` column is useful.

In the `FD` column `6r` means "file descriptor 6, open for read access". `w` would mean write access, `u` would mean both.

## Files for a specific process

I removed some of the output here trying to leave the most interesting bits:

```
~ % lsof -p 17990
COMMAND   PID  USER   FD     TYPE             DEVICE  SIZE/OFF                NODE NAME
Python  17990 simon  cwd      DIR                1,5      1344           174186151 /private/tmp
Python  17990 simon  txt      REG                1,5     49400           163507251 /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python
Python  17990 simon  txt      REG                1,5   3632512           163507083 /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/Python
...
Python  17990 simon  txt      REG                1,5     50840           152019254 /usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/markupsafe/_speedups.cpython-39-darwin.so
...
Python  17990 simon  txt      REG                1,5   1069536           148927587 /usr/local/Cellar/sqlite/3.36.0/lib/libsqlite3.0.dylib
Python  17990 simon  txt      REG                1,5     24552           179047672 /usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/bpylist/bplist.cpython-39-darwin.so
Python  17990 simon  txt      REG                1,5   1572480 1152921500312757159 /usr/lib/dyld
Python  17990 simon    0u     CHR               16,5   0t94656                6931 /dev/ttys005
Python  17990 simon    1u     CHR               16,5   0t94656                6931 /dev/ttys005
Python  17990 simon    2u     CHR               16,5   0t94656                6931 /dev/ttys005
Python  17990 simon    3u  KQUEUE                                                  count=0, state=0xa
Python  17990 simon    4u    unix 0x67a396d5b54b61cb       0t0                     ->0x67a396d5b54b522b
Python  17990 simon    5u    unix 0x67a396d5b54b522b       0t0                     ->0x67a396d5b54b61cb
Python  17990 simon    6r     REG                1,5 747413504           187550734 /private/tmp/covid-smaller.db
Python  17990 simon    7u  KQUEUE                                                  count=0, state=0xa
Python  17990 simon    8u    unix 0x67a396d5b54b68d3       0t0                     ->0x67a396d5b54b5613
Python  17990 simon    9u    unix 0x67a396d5b54b5613       0t0                     ->0x67a396d5b54b68d3
Python  17990 simon   10u    IPv4 0x67a396d59c718b2b       0t0                 TCP localhost:8422 (LISTEN)
Python  17990 simon   12r     REG                1,5 747413504           187550734 /private/tmp/covid-smaller.db
```
The column with `FD=cwd`, `TYPE=DIR` shows the working directory for the process.

The `TYPE=IPv4` one is interesting, it shows that we are listening on `localhost:8422`.

html:
<p><code>lsof</code> stands for "list open files". Here are some very basic usage notes for the version that ships with macOS.</p>
<h2>
<a id="user-content-listing-processes-that-have-a-specific-file-open" class="anchor" href="#listing-processes-that-have-a-specific-file-open" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Listing processes that have a specific file open</h2>
<pre><code>~ % lsof /tmp/covid-smaller.db
COMMAND   PID  USER   FD   TYPE DEVICE  SIZE/OFF      NODE NAME
Python  17990 simon    6r   REG    1,5 747413504 187550734 /private/tmp/covid-smaller.db
Python  17990 simon   12r   REG    1,5 747413504 187550734 /private/tmp/covid-smaller.db
</code></pre>
<p>Note that this file is opened twice by the same process (Datasette maintains multiple connections to the same SQLite database). The <code>PID</code> column is useful.</p>
<p>In the <code>FD</code> column <code>6r</code> means "file descriptor 6, open for read access". <code>w</code> would mean write access, <code>u</code> would mean both.</p>
<h2>
<a id="user-content-files-for-a-specific-process" class="anchor" href="#files-for-a-specific-process" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Files for a specific process</h2>
<p>I removed some of the output here trying to leave the most interesting bits:</p>
<pre><code>~ % lsof -p 17990
COMMAND   PID  USER   FD     TYPE             DEVICE  SIZE/OFF                NODE NAME
Python  17990 simon  cwd      DIR                1,5      1344           174186151 /private/tmp
Python  17990 simon  txt      REG                1,5     49400           163507251 /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/Resources/Python.app/Contents/MacOS/Python
Python  17990 simon  txt      REG                1,5   3632512           163507083 /usr/local/Cellar/python@3.9/3.9.7/Frameworks/Python.framework/Versions/3.9/Python
...
Python  17990 simon  txt      REG                1,5     50840           152019254 /usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/markupsafe/_speedups.cpython-39-darwin.so
...
Python  17990 simon  txt      REG                1,5   1069536           148927587 /usr/local/Cellar/sqlite/3.36.0/lib/libsqlite3.0.dylib
Python  17990 simon  txt      REG                1,5     24552           179047672 /usr/local/Cellar/datasette/0.58.1/libexec/lib/python3.9/site-packages/bpylist/bplist.cpython-39-darwin.so
Python  17990 simon  txt      REG                1,5   1572480 1152921500312757159 /usr/lib/dyld
Python  17990 simon    0u     CHR               16,5   0t94656                6931 /dev/ttys005
Python  17990 simon    1u     CHR               16,5   0t94656                6931 /dev/ttys005
Python  17990 simon    2u     CHR               16,5   0t94656                6931 /dev/ttys005
Python  17990 simon    3u  KQUEUE                                                  count=0, state=0xa
Python  17990 simon    4u    unix 0x67a396d5b54b61cb       0t0                     -&gt;0x67a396d5b54b522b
Python  17990 simon    5u    unix 0x67a396d5b54b522b       0t0                     -&gt;0x67a396d5b54b61cb
Python  17990 simon    6r     REG                1,5 747413504           187550734 /private/tmp/covid-smaller.db
Python  17990 simon    7u  KQUEUE                                                  count=0, state=0xa
Python  17990 simon    8u    unix 0x67a396d5b54b68d3       0t0                     -&gt;0x67a396d5b54b5613
Python  17990 simon    9u    unix 0x67a396d5b54b5613       0t0                     -&gt;0x67a396d5b54b68d3
Python  17990 simon   10u    IPv4 0x67a396d59c718b2b       0t0                 TCP localhost:8422 (LISTEN)
Python  17990 simon   12r     REG                1,5 747413504           187550734 /private/tmp/covid-smaller.db
</code></pre>
<p>The column with <code>FD=cwd</code>, <code>TYPE=DIR</code> shows the working directory for the process.</p>
<p>The <code>TYPE=IPv4</code> one is interesting, it shows that we are listening on <code>localhost:8422</code>.</p>


path: pluggy_multiple-hooks-same-file.md
topic: pluggy
title: Registering the same Pluggy hook multiple times in a single file
url: https://github.com/simonw/til/blob/main/pluggy/multiple-hooks-same-file.md
created: 2021-12-16T15:16:45-08:00
created_utc: 2021-12-16T23:16:45+00:00
updated: 2022-07-03T12:06:20-07:00
updated_utc: 2022-07-03T19:06:20+00:00
shot_hash: 9c98adfca12d1ef7193ce37f5f6f36a4
slug: multiple-hooks-same-file
body:
I found myself wanting to register more than one instance of a [Pluggy](https://pluggy.readthedocs.io/) plugin hook inside a single module.

Hooks are usually registered like this:

```python
@hookimpl
def filters_from_request(request, database, datasette):
    # ...
```
Where `filters_from_request` matches the name of a registered plugin hook.

It turns out you can do this instead:

```python
@hookimpl(specname="filters_from_request")
def filters_from_request_1(request, database, datasette):
    # ...

    
@hookimpl(specname="filters_from_request")
def filters_from_request_2(request, database, datasette):
    # ...
```
Which allows you to write more than one plugin implementation function in the same Python module file.

Note that the `specname` feature requires [Pluggy 1.0.0](https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst#pluggy-100-2021-08-25) or higher.

These can be combined with `tryfirst=` and `trylast=`. This example adds one link at the start of the Datasette application menu and one at the end, using the [menu_links hook](https://docs.datasette.io/en/stable/plugin_hooks.html#menu-links-datasette-actor-request).

```python
from datasette import hookimpl


@hookimpl(specname="menu_links", tryfirst=True)
def menu_links_1(datasette):
    return [
        {"href": datasette.urls.path("/"), "label": "Home"},
    ]


@hookimpl(specname="menu_links", trylast=True)
def menu_links_2():
    return [
        {
            "href": "http://www.example.com/",
            "label": "Link at the end",
        },
    ]
```

html:
<p>I found myself wanting to register more than one instance of a <a href="https://pluggy.readthedocs.io/" rel="nofollow">Pluggy</a> plugin hook inside a single module.</p>
<p>Hooks are usually registered like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">hookimpl</span></span>
<span class="pl-k">def</span> <span class="pl-en">filters_from_request</span>(<span class="pl-s1">request</span>, <span class="pl-s1">database</span>, <span class="pl-s1">datasette</span>):
    <span class="pl-c"># ...</span></pre></div>
<p>Where <code>filters_from_request</code> matches the name of a registered plugin hook.</p>
<p>It turns out you can do this instead:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-en">hookimpl</span>(<span class="pl-s1">specname</span><span class="pl-c1">=</span><span class="pl-s">"filters_from_request"</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">filters_from_request_1</span>(<span class="pl-s1">request</span>, <span class="pl-s1">database</span>, <span class="pl-s1">datasette</span>):
    <span class="pl-c"># ...</span>

    
<span class="pl-en">@<span class="pl-en">hookimpl</span>(<span class="pl-s1">specname</span><span class="pl-c1">=</span><span class="pl-s">"filters_from_request"</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">filters_from_request_2</span>(<span class="pl-s1">request</span>, <span class="pl-s1">database</span>, <span class="pl-s1">datasette</span>):
    <span class="pl-c"># ...</span></pre></div>
<p>Which allows you to write more than one plugin implementation function in the same Python module file.</p>
<p>Note that the <code>specname</code> feature requires <a href="https://github.com/pytest-dev/pluggy/blob/main/CHANGELOG.rst#pluggy-100-2021-08-25">Pluggy 1.0.0</a> or higher.</p>
<p>These can be combined with <code>tryfirst=</code> and <code>trylast=</code>. This example adds one link at the start of the Datasette application menu and one at the end, using the <a href="https://docs.datasette.io/en/stable/plugin_hooks.html#menu-links-datasette-actor-request" rel="nofollow">menu_links hook</a>.</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">datasette</span> <span class="pl-k">import</span> <span class="pl-s1">hookimpl</span>


<span class="pl-en">@<span class="pl-en">hookimpl</span>(<span class="pl-s1">specname</span><span class="pl-c1">=</span><span class="pl-s">"menu_links"</span>, <span class="pl-s1">tryfirst</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">menu_links_1</span>(<span class="pl-s1">datasette</span>):
    <span class="pl-k">return</span> [
        {<span class="pl-s">"href"</span>: <span class="pl-s1">datasette</span>.<span class="pl-s1">urls</span>.<span class="pl-en">path</span>(<span class="pl-s">"/"</span>), <span class="pl-s">"label"</span>: <span class="pl-s">"Home"</span>},
    ]


<span class="pl-en">@<span class="pl-en">hookimpl</span>(<span class="pl-s1">specname</span><span class="pl-c1">=</span><span class="pl-s">"menu_links"</span>, <span class="pl-s1">trylast</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">menu_links_2</span>():
    <span class="pl-k">return</span> [
        {
            <span class="pl-s">"href"</span>: <span class="pl-s">"http://www.example.com/"</span>,
            <span class="pl-s">"label"</span>: <span class="pl-s">"Link at the end"</span>,
        },
    ]</pre></div>


path: spatialite_minimal-spatialite-database-in-python.md
topic: spatialite
title: Creating a minimal SpatiaLite database with Python
url: https://github.com/simonw/til/blob/main/spatialite/minimal-spatialite-database-in-python.md
created: 2021-12-17T15:57:29-08:00
created_utc: 2021-12-17T23:57:29+00:00
updated: 2021-12-17T15:57:29-08:00
updated_utc: 2021-12-17T23:57:29+00:00
shot_hash: 2ddc943a9eedd28c593ee870504536b9
slug: minimal-spatialite-database-in-python
body:
When writing a test for [datasette-leaflet-freedraw](https://github.com/simonw/datasette-leaflet-freedraw) I realized I didn't have a simple tiny recipe for creating an in-memory SpatiaLite database in Python. I came up with this:

```python
import sqlite3

SPATIALITE = "/usr/local/lib/mod_spatialite.dylib"

db = sqlite3.connect(":memory:")
db.enable_load_extension(True)
db.execute("SELECT load_extension(?)", [SPATIALITE])
db.execute("SELECT InitSpatialMetadata(1)")
db.execute("CREATE TABLE places_spatialite (id integer primary key, name text)")
db.execute(
    "SELECT AddGeometryColumn('places_spatialite', 'geometry', 4326, 'POINT', 'XY');"
)
# Then to add a spatial index:
db.execute(
    "SELECT CreateSpatialIndex('places_spatialite', 'geometry');"
)
```
Datasette and `sqlite-utils` both have `find_spatialite()` utility functions. Here's how to call the Datasette one as a one-liner:
```
% python -c 'import datasette.utils; print(datasette.utils.find_spatialite())'
/usr/local/lib/mod_spatialite.dylib
```
I also remembered I have this script: [build_small_spatialite_db.py](https://github.com/simonw/datasette/blob/main/tests/build_small_spatialite_db.py)

html:
<p>When writing a test for <a href="https://github.com/simonw/datasette-leaflet-freedraw">datasette-leaflet-freedraw</a> I realized I didn't have a simple tiny recipe for creating an in-memory SpatiaLite database in Python. I came up with this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">sqlite3</span>

<span class="pl-v">SPATIALITE</span> <span class="pl-c1">=</span> <span class="pl-s">"/usr/local/lib/mod_spatialite.dylib"</span>

<span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite3</span>.<span class="pl-en">connect</span>(<span class="pl-s">":memory:"</span>)
<span class="pl-s1">db</span>.<span class="pl-en">enable_load_extension</span>(<span class="pl-c1">True</span>)
<span class="pl-s1">db</span>.<span class="pl-en">execute</span>(<span class="pl-s">"SELECT load_extension(?)"</span>, [<span class="pl-v">SPATIALITE</span>])
<span class="pl-s1">db</span>.<span class="pl-en">execute</span>(<span class="pl-s">"SELECT InitSpatialMetadata(1)"</span>)
<span class="pl-s1">db</span>.<span class="pl-en">execute</span>(<span class="pl-s">"CREATE TABLE places_spatialite (id integer primary key, name text)"</span>)
<span class="pl-s1">db</span>.<span class="pl-en">execute</span>(
    <span class="pl-s">"SELECT AddGeometryColumn('places_spatialite', 'geometry', 4326, 'POINT', 'XY');"</span>
)
<span class="pl-c"># Then to add a spatial index:</span>
<span class="pl-s1">db</span>.<span class="pl-en">execute</span>(
    <span class="pl-s">"SELECT CreateSpatialIndex('places_spatialite', 'geometry');"</span>
)</pre></div>
<p>Datasette and <code>sqlite-utils</code> both have <code>find_spatialite()</code> utility functions. Here's how to call the Datasette one as a one-liner:</p>
<pre><code>% python -c 'import datasette.utils; print(datasette.utils.find_spatialite())'
/usr/local/lib/mod_spatialite.dylib
</code></pre>
<p>I also remembered I have this script: <a href="https://github.com/simonw/datasette/blob/main/tests/build_small_spatialite_db.py">build_small_spatialite_db.py</a></p>


path: python_safe-output-json.md
topic: python
title: Safely outputting JSON
url: https://github.com/simonw/til/blob/main/python/safe-output-json.md
created: 2021-12-17T17:06:10-08:00
created_utc: 2021-12-18T01:06:10+00:00
updated: 2021-12-17T17:06:10-08:00
updated_utc: 2021-12-18T01:06:10+00:00
shot_hash: d6e49a22d73ecf8d886252552021cccd
slug: safe-output-json
body:
Carelessly including the output of `json.dumps()` in an HTML page can lead to an XSS hole, thanks to the following:

```pycon
>>> import json
>>> s = "</script><script>alert(document.location)</script>"
>>> print(json.dumps({"bad": s}))
{"bad": "</script><script>alert(document.location)</script>"}
```
Jinja has a function that avoids this in `jinja2.utils.htmlsafe_json_dumps()` - the (simplified) implementation [looks like this](https://github.com/pallets/jinja/blob/3.0.3/src/jinja2/utils.py#L704-L741):

```python
def htmlsafe_json_dumps(obj):
    return (
        json.dumps(obj)
        .replace("<", "\\u003c")
        .replace(">", "\\u003e")
        .replace("&", "\\u0026")
        .replace("'", "\\u0027")
    )
```
Which outputs like this:
```pycon
>>> print(htmlsafe_json_dumps({"bad": s}))
{"bad": "\u003c/script\u003e\u003cscript\u003ealert(document.location)\u003c/script\u003e"}
```

html:
<p>Carelessly including the output of <code>json.dumps()</code> in an HTML page can lead to an XSS hole, thanks to the following:</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; <span class="pl-k">import</span> json
&gt;&gt;&gt; s <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">"</span>&lt;/script&gt;&lt;script&gt;alert(document.location)&lt;/script&gt;<span class="pl-pds">"</span></span>
&gt;&gt;&gt; <span class="pl-c1">print</span>(json.dumps({<span class="pl-s"><span class="pl-pds">"</span>bad<span class="pl-pds">"</span></span>: s}))
{"bad": "&lt;/script&gt;&lt;script&gt;alert(document.location)&lt;/script&gt;"}</pre></div>
<p>Jinja has a function that avoids this in <code>jinja2.utils.htmlsafe_json_dumps()</code> - the (simplified) implementation <a href="https://github.com/pallets/jinja/blob/3.0.3/src/jinja2/utils.py#L704-L741">looks like this</a>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">htmlsafe_json_dumps</span>(<span class="pl-s1">obj</span>):
    <span class="pl-k">return</span> (
        <span class="pl-s1">json</span>.<span class="pl-en">dumps</span>(<span class="pl-s1">obj</span>)
        .<span class="pl-en">replace</span>(<span class="pl-s">"&lt;"</span>, <span class="pl-s">"<span class="pl-cce">\\</span>u003c"</span>)
        .<span class="pl-en">replace</span>(<span class="pl-s">"&gt;"</span>, <span class="pl-s">"<span class="pl-cce">\\</span>u003e"</span>)
        .<span class="pl-en">replace</span>(<span class="pl-s">"&amp;"</span>, <span class="pl-s">"<span class="pl-cce">\\</span>u0026"</span>)
        .<span class="pl-en">replace</span>(<span class="pl-s">"'"</span>, <span class="pl-s">"<span class="pl-cce">\\</span>u0027"</span>)
    )</pre></div>
<p>Which outputs like this:</p>
<div class="highlight highlight-text-python-console"><pre>&gt;&gt;&gt; <span class="pl-c1">print</span>(htmlsafe_json_dumps({<span class="pl-s"><span class="pl-pds">"</span>bad<span class="pl-pds">"</span></span>: s}))
{"bad": "\u003c/script\u003e\u003cscript\u003ealert(document.location)\u003c/script\u003e"}</pre></div>


path: python_annotated-dataklasses.md
topic: python
title: Annotated explanation of David Beazley's dataklasses
url: https://github.com/simonw/til/blob/main/python/annotated-dataklasses.md
created: 2021-12-19T20:30:57-08:00
created_utc: 2021-12-20T04:30:57+00:00
updated: 2021-12-21T08:39:26-08:00
updated_utc: 2021-12-21T16:39:26+00:00
shot_hash: 68e1aa62c7eb9c8184648319f5aef7b4
slug: annotated-dataklasses
body:
David Beazley [on Twitter](https://twitter.com/dabeaz/status/1472742536649351173):

> So, I just published this deliciously evil spin on dataclasses. It's tiny and the resulting classes import about 15-20 faster than dataclasses.  https://github.com/dabeaz/dataklasses

I decided to create a heavily annotated version of [his original code](https://github.com/dabeaz/dataklasses/blob/78d29d40091eb55d2e0196a9eee9d842b5b82835/dataklasses.py) to figure out for myself how it worked.

## What this code does

It takes this:
```python
@dataklass
class Coordinates:
    x: int
    y: int
```
And turns it into this:
```python
class Coordinates:
    __match_args__ = {'x': <class 'int'>, 'y': <class 'int'>}

    def __init__(self, x, y):
        self.x = x
        self.y = y
    
    def __repr__(self):
        return f"Coordinates({self.x!r}, {self.y!r})"

    def __eq__(self, other):
        if self.__class__ is other.__class__:
            return (self,x, self.y,) == (other.x, other.y,)
        else:
            return NotImplemented
```
## The key trick: code generation

The key to understanding how the code works is to understand that it is using code generation. David introspects out the annotated class properties, from the class itself and any superclasses, and then counts how many they are. He then generates methods that look like this:

```python
def __init__(self, _0, _1):
    self._0 = _0
    self._1 = _1
```
Then uses the `func.__code__.replace(co_names=repl_co_names, co_varnames=repl_co_varnames)` method to rename those `_0` and `_1` variables back to `x` and `y`.

I don't yet understand why he does this as opposed to generating the code using `x` and `y` directly. I'll update this with an explanation once I find one!

Update: Jacob Kaplan-Moss [explains it here](https://twitter.com/jacobian/status/1472789373552762884)!

> That's the absolutely _wild_ trick at the heart of this, and what makes it fast:
>
> the bytecode for `__init__(self, x, y)` is exactly the same as the bytecode for `__init__(self, foo, bar)`!
>
> So that means Dave can cache the generated code for for "any `__init__` function with two args" and then _re-use the exact same code_ for any later 2-arity `__init__`s!

## My annotated copy

First, including the copyright message as required by the copyright message:
```python
# dataklasses.py
#
#     https://github.com/dabeaz/dataklasses
#
# Author: David Beazley (@dabeaz).
#         http://www.dabeaz.com
#
# Copyright (C) 2021-2022.
#
# Permission is granted to use, copy, and modify this code in any
# manner as long as this copyright message and disclaimer remain in
# the source code.  There is no warranty.  Try to use the code for the
# greater good.
```
My annotations start here (I also ran it through [Black](https://black.vercel.app/)).
```python
__all__ = ["dataklass"]
# This ensures "from dataklasses import *" will only import the dataklass symbol

from functools import lru_cache, reduce

# This decorator takes a function that returns Python source code and compiles
# that into a Python function. So if func(fields) returns the following string:
#
# def __init__(self, a):
#     self.a = a
#
# The return value will be that compiled function. With some enhancements.
def codegen(func):
    # This caches the result so if you pass the same numfields it won't have to
    # carry out the same work twice.
    #
    # This is the key performance optimization for this code - it means that
    # the same generated code can be reused for any class that has the same
    # number of arguments!
    @lru_cache
    def make_func_code(numfields):
        # numfields in len(fields)
        names = [f"_{n}" for n in range(numfields)]
        # names is now ["_0", "_1", "_2"] depending on numfields
        #
        # We call func() with that list of names and exec() the
        # returned source code. The third argument to exec() is
        # locals() - it's a dictionary that the exec() function
        # will populate with any new symbols.
        #
        # We use the d:={} walrus operator here so that we can
        # refer to that dictionary by the name d in a moment.
        exec(func(names), globals(), d := {})
        # d.popitem() returns the first (key, value) pair in
        # the dictionary. We know that the locals() dictionary
        # is only going to have a single symbol in it, because
        # we now that the code returned by func(names) only
        # defines a single function. So this returns the value
        # from the first item in locals(), the function object
        return d.popitem()[1]

    # This decorate() is the return value from the decorator,
    # which means that the following code:
    #
    # @codegen
    # def make__init__(fields):
    #     # ...
    # Turns make__init__ into the function defined by decorate()
    def decorate(fields):
        # As shown above, this uses exec() to compile and return
        # a function body created using the generated source code
        func = make_func_code(len(fields))

        # But remember: because we cache and reuse method bodies, this has
        # ugly _0, _1 parameters and variables that we want to make nicer.

        # co_names: tuple of names other than arguments and function locals
        co_names = func.__code__.co_names
        # For the Coordinates example, co_names = ("_0", "_1")

        # co_varnames: tuple of names of arguments and local variables
        co_varnames = func.__code__.co_varnames
        # For the Coordinates example, co_varnames = ("self", "_0", "_1")

        # We are about to replace co_names and co_varnames with modified
        # versions - so we need to create two replacement tuples for them.
        #
        # start := co_names.index("_0") uses the walrus operator to both
        # figure out the 0-based index of the _0 symbol and assign it to s
        #
        # We use (*a, *b, *c) to create a new tuple that is the result of
        # combining those three input tuples.
        repl_co_names = (
            # Slice everything in co_names up to that first _0 symbol
            *co_names[: (start := co_names.index("_0"))],
            # Then insert the fields, which are the annotated class properties
            *fields,
            # Now everything in co_names following 
            *co_names[start + len(fields) :],
        )
        # For Coordinates this is now ("x", "y")

        repl_co_varnames = (
            # We only modify co_varnames if _0 is one of them, otherwise
            # we leave them unchanged.
            (
                *co_varnames[: (start := co_varnames.index("_0"))],
                *fields,
                *co_varnames[start + len(fields) :],
            )
            if "_0" in co_varnames
            else co_varnames
        )
        # For Coordinates this is now ("self", "x", "y")

        # type(func) returns a Python internal object called "function"
        # which is callable and has this function signature:
        #
        # function(code, globals, name=None, argdefs=None, closure=None)
        #
        # It creates a brand new function object
        return type(func)(
            # func.__code__.replace docstring says:
            # "Return a copy of the code object with new values for the specified fields"
            # So we are rewriting the co_names and co_varnames here
            func.__code__.replace(co_names=repl_co_names, co_varnames=repl_co_varnames),
            func.__globals__,
        )

    return decorate


def all_hints(cls):
    # This introspects the class to find all of the annotated class members
    # In Coordinates case this returns {'x': <class 'int'>, 'y': <class 'int'>}
    #
    # The "x" to that lambda starts out as that {} empty dictionary and each time
    # is the dictionary with new stuff added to it.
    #
    # reversed(Coordinates.__mro__) loops through every superclass of the current
    # class, starting at "object".
    #
    # So the lambda is called against each superclass in turn, and each time it
    # reads the __annotations__ field, if one is available.
    #
    # Coordinates.__annotations__ returns {'x': int, 'y': int}
    #
    # dict1 | dict2 in Python returns a new dict that combines the previous two
    #
    # So this function returns a combined dictionary of the __annotations__ from
    # every class in the superclass hierarchy.
    return reduce(
        lambda x, y: x | getattr(y, "__annotations__", {}), reversed(cls.__mro__), {}
    )


# Next are the functions that generate the different methods. Remember they get passed
# fields which is {'x': <class 'int'>, 'y': <class 'int'>}
@codegen
def make__init__(fields):
    # Calling ",".join(dict) joins just the keys of that dictionary
    code = "def __init__(self, " + ",".join(fields) + "):\n"
    # So here we have:
    #    def __init__(self, x, y):
    return code + "\n".join(f" self.{name} = {name}\n" for name in fields)
    # This adds on:
    #   self.x = x
    #   self.y = y


@codegen
def make__repr__(fields):
    return (
        "def __repr__(self):\n"
        # type(self).__name__ gives us the class name: "Coordinates"
        ' return f"{type(self).__name__}('
        # This gives us {self.x!r}, {self.y!r} which in an f-string
        # gives us the __repr__() version of those object properties
        + ", ".join("{self." + name + "!r}" for name in fields)
        + ')"\n'
    )
    # So this generates:
    # def __repr__(self):
    #  return f"Coordinates({self.x!r}, {self.y!r})" 


@codegen
def make__eq__(fields):
    selfvals = ",".join(f"self.{name}" for name in fields)
    othervals = ",".join(f"other.{name}" for name in fields)
    return (
        "def __eq__(self, other):\n"
        "  if self.__class__ is other.__class__:\n"
        f"    return ({selfvals},) == ({othervals},)\n"
        "  else:\n"
        "    return NotImplemented\n"
    )
    # This generates:
    # def __eq__(self, other):
    #   if self.__class__ is other.__class__:
    #     return (self,x, self.y,) == (other.x, other.y)
    #   else:
    #     return NotImplemented


@codegen
def make__iter__(fields):
    return "def __iter__(self):\n" + "\n".join(
        f"   yield self.{name}" for name in fields
    )


@codegen
def make__hash__(fields):
    self_tuple = "(" + ",".join(f"self.{name}" for name in fields) + ",)"
    return "def __hash__(self):\n" f"    return hash({self_tuple})\n"


def dataklass(cls):
    fields = all_hints(cls)
    # fields is now {'x': <class 'int'>, 'y': <class 'int'>}
    clsdict = vars(cls)
    # clsdict looks like this:
    # {'__module__': '__main__',
    #  '__annotations__': {'x': <class 'int'>, 'y': <class 'int'>},
    #  '__dict__': <attribute '__dict__' of 'Coordinates' objects>,
    #  '__weakref__': <attribute '__weakref__' of 'Coordinates' objects>,
    #  '__doc__': None}
    #
    # The purpose of this function is mainly to add __init__ and
    # __repr__ and __eq__ methods, but only if they are not yet defined
    if not "__init__" in clsdict:
        cls.__init__ = make__init__(fields)
    if not "__repr__" in clsdict:
        cls.__repr__ = make__repr__(fields)
    if not "__eq__" in clsdict:
        cls.__eq__ = make__eq__(fields)
    # Not sure why these are commented out right now:
    # if not '__iter__' in clsdict:  cls.__iter__ = make__iter__(fields)
    # if not '__hash__' in clsdict:  cls.__hash__ = make__hash__(fields)
    cls.__match_args__ = fields
    # This sets __match_args__ to {'x': <class 'int'>, 'y': <class 'int'>}
    # Usually __match_args__ is expected to be a tuple like ("x", "y")
    # but presumably this still works with a dictionary because it is used
    # for "name in __match_args__" style lookups.
    #
    # __match_args__ is a property used by Python structural typing:
    # https://www.python.org/dev/peps/pep-0622/#special-attribute-match-args
    return cls


# Example use
if __name__ == "__main__":

    @dataklass
    class Coordinates:
        x: int
        y: int
```

html:
<p>David Beazley <a href="https://twitter.com/dabeaz/status/1472742536649351173" rel="nofollow">on Twitter</a>:</p>
<blockquote>
<p>So, I just published this deliciously evil spin on dataclasses. It's tiny and the resulting classes import about 15-20 faster than dataclasses.  <a href="https://github.com/dabeaz/dataklasses">https://github.com/dabeaz/dataklasses</a></p>
</blockquote>
<p>I decided to create a heavily annotated version of <a href="https://github.com/dabeaz/dataklasses/blob/78d29d40091eb55d2e0196a9eee9d842b5b82835/dataklasses.py">his original code</a> to figure out for myself how it worked.</p>
<h2>
<a id="user-content-what-this-code-does" class="anchor" href="#what-this-code-does" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>What this code does</h2>
<p>It takes this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">dataklass</span></span>
<span class="pl-k">class</span> <span class="pl-v">Coordinates</span>:
    <span class="pl-s1">x</span>: <span class="pl-s1">int</span>
    <span class="pl-s1">y</span>: <span class="pl-s1">int</span></pre></div>
<p>And turns it into this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">class</span> <span class="pl-v">Coordinates</span>:
    <span class="pl-s1">__match_args__</span> <span class="pl-c1">=</span> {<span class="pl-s">'x'</span>:<span class="pl-s1"></span> <span class="pl-c1">&lt;</span><span class="pl-s1">class</span> <span class="pl-s">'int'</span><span class="pl-c1">&gt;</span>, <span class="pl-s">'y'</span>:<span class="pl-s1"></span> <span class="pl-c1">&lt;</span><span class="pl-s1">class</span> <span class="pl-s">'int'</span><span class="pl-c1">&gt;</span>}

    <span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">x</span>, <span class="pl-s1">y</span>):
        <span class="pl-s1">self</span>.<span class="pl-s1">x</span> <span class="pl-c1">=</span> <span class="pl-s1">x</span>
        <span class="pl-s1">self</span>.<span class="pl-s1">y</span> <span class="pl-c1">=</span> <span class="pl-s1">y</span>
    
    <span class="pl-k">def</span> <span class="pl-en">__repr__</span>(<span class="pl-s1">self</span>):
        <span class="pl-k">return</span> <span class="pl-s">f"Coordinates(<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">self</span>.<span class="pl-s1">x</span>!r<span class="pl-kos">}</span></span>, <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">self</span>.<span class="pl-s1">y</span>!r<span class="pl-kos">}</span></span>)"</span>

    <span class="pl-k">def</span> <span class="pl-en">__eq__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">other</span>):
        <span class="pl-k">if</span> <span class="pl-s1">self</span>.<span class="pl-s1">__class__</span> <span class="pl-c1">is</span> <span class="pl-s1">other</span>.<span class="pl-s1">__class__</span>:
            <span class="pl-k">return</span> (<span class="pl-s1">self</span>,<span class="pl-s1">x</span>, <span class="pl-s1">self</span>.<span class="pl-s1">y</span>,) <span class="pl-c1">==</span> (<span class="pl-s1">other</span>.<span class="pl-s1">x</span>, <span class="pl-s1">other</span>.<span class="pl-s1">y</span>,)
        <span class="pl-k">else</span>:
            <span class="pl-k">return</span> <span class="pl-v">NotImplemented</span></pre></div>
<h2>
<a id="user-content-the-key-trick-code-generation" class="anchor" href="#the-key-trick-code-generation" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>The key trick: code generation</h2>
<p>The key to understanding how the code works is to understand that it is using code generation. David introspects out the annotated class properties, from the class itself and any superclasses, and then counts how many they are. He then generates methods that look like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">__init__</span>(<span class="pl-s1">self</span>, <span class="pl-s1">_0</span>, <span class="pl-s1">_1</span>):
    <span class="pl-s1">self</span>.<span class="pl-s1">_0</span> <span class="pl-c1">=</span> <span class="pl-s1">_0</span>
    <span class="pl-s1">self</span>.<span class="pl-s1">_1</span> <span class="pl-c1">=</span> <span class="pl-s1">_1</span></pre></div>
<p>Then uses the <code>func.__code__.replace(co_names=repl_co_names, co_varnames=repl_co_varnames)</code> method to rename those <code>_0</code> and <code>_1</code> variables back to <code>x</code> and <code>y</code>.</p>
<p>I don't yet understand why he does this as opposed to generating the code using <code>x</code> and <code>y</code> directly. I'll update this with an explanation once I find one!</p>
<p>Update: Jacob Kaplan-Moss <a href="https://twitter.com/jacobian/status/1472789373552762884" rel="nofollow">explains it here</a>!</p>
<blockquote>
<p>That's the absolutely <em>wild</em> trick at the heart of this, and what makes it fast:</p>
<p>the bytecode for <code>__init__(self, x, y)</code> is exactly the same as the bytecode for <code>__init__(self, foo, bar)</code>!</p>
<p>So that means Dave can cache the generated code for for "any <code>__init__</code> function with two args" and then <em>re-use the exact same code</em> for any later 2-arity <code>__init__</code>s!</p>
</blockquote>
<h2>
<a id="user-content-my-annotated-copy" class="anchor" href="#my-annotated-copy" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>My annotated copy</h2>
<p>First, including the copyright message as required by the copyright message:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-c"># dataklasses.py</span>
<span class="pl-c">#</span>
<span class="pl-c">#     https://github.com/dabeaz/dataklasses</span>
<span class="pl-c">#</span>
<span class="pl-c"># Author: David Beazley (@dabeaz).</span>
<span class="pl-c">#         http://www.dabeaz.com</span>
<span class="pl-c">#</span>
<span class="pl-c"># Copyright (C) 2021-2022.</span>
<span class="pl-c">#</span>
<span class="pl-c"># Permission is granted to use, copy, and modify this code in any</span>
<span class="pl-c"># manner as long as this copyright message and disclaimer remain in</span>
<span class="pl-c"># the source code.  There is no warranty.  Try to use the code for the</span>
<span class="pl-c"># greater good.</span></pre></div>
<p>My annotations start here (I also ran it through <a href="https://black.vercel.app/" rel="nofollow">Black</a>).</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">__all__</span> <span class="pl-c1">=</span> [<span class="pl-s">"dataklass"</span>]
<span class="pl-c"># This ensures "from dataklasses import *" will only import the dataklass symbol</span>

<span class="pl-k">from</span> <span class="pl-s1">functools</span> <span class="pl-k">import</span> <span class="pl-s1">lru_cache</span>, <span class="pl-s1">reduce</span>

<span class="pl-c"># This decorator takes a function that returns Python source code and compiles</span>
<span class="pl-c"># that into a Python function. So if func(fields) returns the following string:</span>
<span class="pl-c">#</span>
<span class="pl-c"># def __init__(self, a):</span>
<span class="pl-c">#     self.a = a</span>
<span class="pl-c">#</span>
<span class="pl-c"># The return value will be that compiled function. With some enhancements.</span>
<span class="pl-k">def</span> <span class="pl-en">codegen</span>(<span class="pl-s1">func</span>):
    <span class="pl-c"># This caches the result so if you pass the same numfields it won't have to</span>
    <span class="pl-c"># carry out the same work twice.</span>
    <span class="pl-c">#</span>
    <span class="pl-c"># This is the key performance optimization for this code - it means that</span>
    <span class="pl-c"># the same generated code can be reused for any class that has the same</span>
    <span class="pl-c"># number of arguments!</span>
    <span class="pl-en">@<span class="pl-s1">lru_cache</span></span>
    <span class="pl-k">def</span> <span class="pl-en">make_func_code</span>(<span class="pl-s1">numfields</span>):
        <span class="pl-c"># numfields in len(fields)</span>
        <span class="pl-s1">names</span> <span class="pl-c1">=</span> [<span class="pl-s">f"_<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">n</span><span class="pl-kos">}</span></span>"</span> <span class="pl-k">for</span> <span class="pl-s1">n</span> <span class="pl-c1">in</span> <span class="pl-en">range</span>(<span class="pl-s1">numfields</span>)]
        <span class="pl-c"># names is now ["_0", "_1", "_2"] depending on numfields</span>
        <span class="pl-c">#</span>
        <span class="pl-c"># We call func() with that list of names and exec() the</span>
        <span class="pl-c"># returned source code. The third argument to exec() is</span>
        <span class="pl-c"># locals() - it's a dictionary that the exec() function</span>
        <span class="pl-c"># will populate with any new symbols.</span>
        <span class="pl-c">#</span>
        <span class="pl-c"># We use the d:={} walrus operator here so that we can</span>
        <span class="pl-c"># refer to that dictionary by the name d in a moment.</span>
        <span class="pl-en">exec</span>(<span class="pl-en">func</span>(<span class="pl-s1">names</span>), <span class="pl-en">globals</span>(), <span class="pl-s1">d</span> <span class="pl-c1">:=</span> {})
        <span class="pl-c"># d.popitem() returns the first (key, value) pair in</span>
        <span class="pl-c"># the dictionary. We know that the locals() dictionary</span>
        <span class="pl-c"># is only going to have a single symbol in it, because</span>
        <span class="pl-c"># we now that the code returned by func(names) only</span>
        <span class="pl-c"># defines a single function. So this returns the value</span>
        <span class="pl-c"># from the first item in locals(), the function object</span>
        <span class="pl-k">return</span> <span class="pl-s1">d</span>.<span class="pl-en">popitem</span>()[<span class="pl-c1">1</span>]

    <span class="pl-c"># This decorate() is the return value from the decorator,</span>
    <span class="pl-c"># which means that the following code:</span>
    <span class="pl-c">#</span>
    <span class="pl-c"># @codegen</span>
    <span class="pl-c"># def make__init__(fields):</span>
    <span class="pl-c">#     # ...</span>
    <span class="pl-c"># Turns make__init__ into the function defined by decorate()</span>
    <span class="pl-k">def</span> <span class="pl-en">decorate</span>(<span class="pl-s1">fields</span>):
        <span class="pl-c"># As shown above, this uses exec() to compile and return</span>
        <span class="pl-c"># a function body created using the generated source code</span>
        <span class="pl-s1">func</span> <span class="pl-c1">=</span> <span class="pl-en">make_func_code</span>(<span class="pl-en">len</span>(<span class="pl-s1">fields</span>))

        <span class="pl-c"># But remember: because we cache and reuse method bodies, this has</span>
        <span class="pl-c"># ugly _0, _1 parameters and variables that we want to make nicer.</span>

        <span class="pl-c"># co_names: tuple of names other than arguments and function locals</span>
        <span class="pl-s1">co_names</span> <span class="pl-c1">=</span> <span class="pl-s1">func</span>.<span class="pl-s1">__code__</span>.<span class="pl-s1">co_names</span>
        <span class="pl-c"># For the Coordinates example, co_names = ("_0", "_1")</span>

        <span class="pl-c"># co_varnames: tuple of names of arguments and local variables</span>
        <span class="pl-s1">co_varnames</span> <span class="pl-c1">=</span> <span class="pl-s1">func</span>.<span class="pl-s1">__code__</span>.<span class="pl-s1">co_varnames</span>
        <span class="pl-c"># For the Coordinates example, co_varnames = ("self", "_0", "_1")</span>

        <span class="pl-c"># We are about to replace co_names and co_varnames with modified</span>
        <span class="pl-c"># versions - so we need to create two replacement tuples for them.</span>
        <span class="pl-c">#</span>
        <span class="pl-c"># start := co_names.index("_0") uses the walrus operator to both</span>
        <span class="pl-c"># figure out the 0-based index of the _0 symbol and assign it to s</span>
        <span class="pl-c">#</span>
        <span class="pl-c"># We use (*a, *b, *c) to create a new tuple that is the result of</span>
        <span class="pl-c"># combining those three input tuples.</span>
        <span class="pl-s1">repl_co_names</span> <span class="pl-c1">=</span> (
            <span class="pl-c"># Slice everything in co_names up to that first _0 symbol</span>
            <span class="pl-c1">*</span><span class="pl-s1">co_names</span>[: (<span class="pl-s1">start</span> <span class="pl-c1">:=</span> <span class="pl-s1">co_names</span>.<span class="pl-en">index</span>(<span class="pl-s">"_0"</span>))],
            <span class="pl-c"># Then insert the fields, which are the annotated class properties</span>
            <span class="pl-c1">*</span><span class="pl-s1">fields</span>,
            <span class="pl-c"># Now everything in co_names following </span>
            <span class="pl-c1">*</span><span class="pl-s1">co_names</span>[<span class="pl-s1">start</span> <span class="pl-c1">+</span> <span class="pl-en">len</span>(<span class="pl-s1">fields</span>) :],
        )
        <span class="pl-c"># For Coordinates this is now ("x", "y")</span>

        <span class="pl-s1">repl_co_varnames</span> <span class="pl-c1">=</span> (
            <span class="pl-c"># We only modify co_varnames if _0 is one of them, otherwise</span>
            <span class="pl-c"># we leave them unchanged.</span>
            (
                <span class="pl-c1">*</span><span class="pl-s1">co_varnames</span>[: (<span class="pl-s1">start</span> <span class="pl-c1">:=</span> <span class="pl-s1">co_varnames</span>.<span class="pl-en">index</span>(<span class="pl-s">"_0"</span>))],
                <span class="pl-c1">*</span><span class="pl-s1">fields</span>,
                <span class="pl-c1">*</span><span class="pl-s1">co_varnames</span>[<span class="pl-s1">start</span> <span class="pl-c1">+</span> <span class="pl-en">len</span>(<span class="pl-s1">fields</span>) :],
            )
            <span class="pl-k">if</span> <span class="pl-s">"_0"</span> <span class="pl-c1">in</span> <span class="pl-s1">co_varnames</span>
            <span class="pl-k">else</span> <span class="pl-s1">co_varnames</span>
        )
        <span class="pl-c"># For Coordinates this is now ("self", "x", "y")</span>

        <span class="pl-c"># type(func) returns a Python internal object called "function"</span>
        <span class="pl-c"># which is callable and has this function signature:</span>
        <span class="pl-c">#</span>
        <span class="pl-c"># function(code, globals, name=None, argdefs=None, closure=None)</span>
        <span class="pl-c">#</span>
        <span class="pl-c"># It creates a brand new function object</span>
        <span class="pl-k">return</span> <span class="pl-en">type</span>(<span class="pl-s1">func</span>)(
            <span class="pl-c"># func.__code__.replace docstring says:</span>
            <span class="pl-c"># "Return a copy of the code object with new values for the specified fields"</span>
            <span class="pl-c"># So we are rewriting the co_names and co_varnames here</span>
            <span class="pl-s1">func</span>.<span class="pl-s1">__code__</span>.<span class="pl-en">replace</span>(<span class="pl-s1">co_names</span><span class="pl-c1">=</span><span class="pl-s1">repl_co_names</span>, <span class="pl-s1">co_varnames</span><span class="pl-c1">=</span><span class="pl-s1">repl_co_varnames</span>),
            <span class="pl-s1">func</span>.<span class="pl-s1">__globals__</span>,
        )

    <span class="pl-k">return</span> <span class="pl-s1">decorate</span>


<span class="pl-k">def</span> <span class="pl-en">all_hints</span>(<span class="pl-s1">cls</span>):
    <span class="pl-c"># This introspects the class to find all of the annotated class members</span>
    <span class="pl-c"># In Coordinates case this returns {'x': &lt;class 'int'&gt;, 'y': &lt;class 'int'&gt;}</span>
    <span class="pl-c">#</span>
    <span class="pl-c"># The "x" to that lambda starts out as that {} empty dictionary and each time</span>
    <span class="pl-c"># is the dictionary with new stuff added to it.</span>
    <span class="pl-c">#</span>
    <span class="pl-c"># reversed(Coordinates.__mro__) loops through every superclass of the current</span>
    <span class="pl-c"># class, starting at "object".</span>
    <span class="pl-c">#</span>
    <span class="pl-c"># So the lambda is called against each superclass in turn, and each time it</span>
    <span class="pl-c"># reads the __annotations__ field, if one is available.</span>
    <span class="pl-c">#</span>
    <span class="pl-c"># Coordinates.__annotations__ returns {'x': int, 'y': int}</span>
    <span class="pl-c">#</span>
    <span class="pl-c"># dict1 | dict2 in Python returns a new dict that combines the previous two</span>
    <span class="pl-c">#</span>
    <span class="pl-c"># So this function returns a combined dictionary of the __annotations__ from</span>
    <span class="pl-c"># every class in the superclass hierarchy.</span>
    <span class="pl-k">return</span> <span class="pl-en">reduce</span>(
        <span class="pl-k">lambda</span> <span class="pl-s1">x</span>, <span class="pl-s1">y</span>: <span class="pl-s1">x</span> <span class="pl-c1">|</span> <span class="pl-en">getattr</span>(<span class="pl-s1">y</span>, <span class="pl-s">"__annotations__"</span>, {}), <span class="pl-en">reversed</span>(<span class="pl-s1">cls</span>.<span class="pl-s1">__mro__</span>), {}
    )


<span class="pl-c"># Next are the functions that generate the different methods. Remember they get passed</span>
<span class="pl-c"># fields which is {'x': &lt;class 'int'&gt;, 'y': &lt;class 'int'&gt;}</span>
<span class="pl-en">@<span class="pl-s1">codegen</span></span>
<span class="pl-k">def</span> <span class="pl-en">make__init__</span>(<span class="pl-s1">fields</span>):
    <span class="pl-c"># Calling ",".join(dict) joins just the keys of that dictionary</span>
    <span class="pl-s1">code</span> <span class="pl-c1">=</span> <span class="pl-s">"def __init__(self, "</span> <span class="pl-c1">+</span> <span class="pl-s">","</span>.<span class="pl-en">join</span>(<span class="pl-s1">fields</span>) <span class="pl-c1">+</span> <span class="pl-s">"):<span class="pl-cce">\n</span>"</span>
    <span class="pl-c"># So here we have:</span>
    <span class="pl-c">#    def __init__(self, x, y):</span>
    <span class="pl-k">return</span> <span class="pl-s1">code</span> <span class="pl-c1">+</span> <span class="pl-s">"<span class="pl-cce">\n</span>"</span>.<span class="pl-en">join</span>(<span class="pl-s">f" self.<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">name</span><span class="pl-kos">}</span></span> = <span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">name</span><span class="pl-kos">}</span></span><span class="pl-cce">\n</span>"</span> <span class="pl-k">for</span> <span class="pl-s1">name</span> <span class="pl-c1">in</span> <span class="pl-s1">fields</span>)
    <span class="pl-c"># This adds on:</span>
    <span class="pl-c">#   self.x = x</span>
    <span class="pl-c">#   self.y = y</span>


<span class="pl-en">@<span class="pl-s1">codegen</span></span>
<span class="pl-k">def</span> <span class="pl-en">make__repr__</span>(<span class="pl-s1">fields</span>):
    <span class="pl-k">return</span> (
        <span class="pl-s">"def __repr__(self):<span class="pl-cce">\n</span>"</span>
        <span class="pl-c"># type(self).__name__ gives us the class name: "Coordinates"</span>
        <span class="pl-s">' return f"{type(self).__name__}('</span>
        <span class="pl-c"># This gives us {self.x!r}, {self.y!r} which in an f-string</span>
        <span class="pl-c"># gives us the __repr__() version of those object properties</span>
        <span class="pl-c1">+</span> <span class="pl-s">", "</span>.<span class="pl-en">join</span>(<span class="pl-s">"{self."</span> <span class="pl-c1">+</span> <span class="pl-s1">name</span> <span class="pl-c1">+</span> <span class="pl-s">"!r}"</span> <span class="pl-k">for</span> <span class="pl-s1">name</span> <span class="pl-c1">in</span> <span class="pl-s1">fields</span>)
        <span class="pl-c1">+</span> <span class="pl-s">')"<span class="pl-cce">\n</span>'</span>
    )
    <span class="pl-c"># So this generates:</span>
    <span class="pl-c"># def __repr__(self):</span>
    <span class="pl-c">#  return f"Coordinates({self.x!r}, {self.y!r})" </span>


<span class="pl-en">@<span class="pl-s1">codegen</span></span>
<span class="pl-k">def</span> <span class="pl-en">make__eq__</span>(<span class="pl-s1">fields</span>):
    <span class="pl-s1">selfvals</span> <span class="pl-c1">=</span> <span class="pl-s">","</span>.<span class="pl-en">join</span>(<span class="pl-s">f"self.<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">name</span><span class="pl-kos">}</span></span>"</span> <span class="pl-k">for</span> <span class="pl-s1">name</span> <span class="pl-c1">in</span> <span class="pl-s1">fields</span>)
    <span class="pl-s1">othervals</span> <span class="pl-c1">=</span> <span class="pl-s">","</span>.<span class="pl-en">join</span>(<span class="pl-s">f"other.<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">name</span><span class="pl-kos">}</span></span>"</span> <span class="pl-k">for</span> <span class="pl-s1">name</span> <span class="pl-c1">in</span> <span class="pl-s1">fields</span>)
    <span class="pl-k">return</span> (
        <span class="pl-s">"def __eq__(self, other):<span class="pl-cce">\n</span>"</span>
        <span class="pl-s">"  if self.__class__ is other.__class__:<span class="pl-cce">\n</span>"</span>
        <span class="pl-s">f"    return (<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">selfvals</span><span class="pl-kos">}</span></span>,) == (<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">othervals</span><span class="pl-kos">}</span></span>,)<span class="pl-cce">\n</span>"</span>
        <span class="pl-s">"  else:<span class="pl-cce">\n</span>"</span>
        <span class="pl-s">"    return NotImplemented<span class="pl-cce">\n</span>"</span>
    )
    <span class="pl-c"># This generates:</span>
    <span class="pl-c"># def __eq__(self, other):</span>
    <span class="pl-c">#   if self.__class__ is other.__class__:</span>
    <span class="pl-c">#     return (self,x, self.y,) == (other.x, other.y)</span>
    <span class="pl-c">#   else:</span>
    <span class="pl-c">#     return NotImplemented</span>


<span class="pl-en">@<span class="pl-s1">codegen</span></span>
<span class="pl-k">def</span> <span class="pl-en">make__iter__</span>(<span class="pl-s1">fields</span>):
    <span class="pl-k">return</span> <span class="pl-s">"def __iter__(self):<span class="pl-cce">\n</span>"</span> <span class="pl-c1">+</span> <span class="pl-s">"<span class="pl-cce">\n</span>"</span>.<span class="pl-en">join</span>(
        <span class="pl-s">f"   yield self.<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">name</span><span class="pl-kos">}</span></span>"</span> <span class="pl-k">for</span> <span class="pl-s1">name</span> <span class="pl-c1">in</span> <span class="pl-s1">fields</span>
    )


<span class="pl-en">@<span class="pl-s1">codegen</span></span>
<span class="pl-k">def</span> <span class="pl-en">make__hash__</span>(<span class="pl-s1">fields</span>):
    <span class="pl-s1">self_tuple</span> <span class="pl-c1">=</span> <span class="pl-s">"("</span> <span class="pl-c1">+</span> <span class="pl-s">","</span>.<span class="pl-en">join</span>(<span class="pl-s">f"self.<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">name</span><span class="pl-kos">}</span></span>"</span> <span class="pl-k">for</span> <span class="pl-s1">name</span> <span class="pl-c1">in</span> <span class="pl-s1">fields</span>) <span class="pl-c1">+</span> <span class="pl-s">",)"</span>
    <span class="pl-k">return</span> <span class="pl-s">"def __hash__(self):<span class="pl-cce">\n</span>"</span> <span class="pl-s">f"    return hash(<span class="pl-s1"><span class="pl-kos">{</span><span class="pl-s1">self_tuple</span><span class="pl-kos">}</span></span>)<span class="pl-cce">\n</span>"</span>


<span class="pl-k">def</span> <span class="pl-en">dataklass</span>(<span class="pl-s1">cls</span>):
    <span class="pl-s1">fields</span> <span class="pl-c1">=</span> <span class="pl-en">all_hints</span>(<span class="pl-s1">cls</span>)
    <span class="pl-c"># fields is now {'x': &lt;class 'int'&gt;, 'y': &lt;class 'int'&gt;}</span>
    <span class="pl-s1">clsdict</span> <span class="pl-c1">=</span> <span class="pl-en">vars</span>(<span class="pl-s1">cls</span>)
    <span class="pl-c"># clsdict looks like this:</span>
    <span class="pl-c"># {'__module__': '__main__',</span>
    <span class="pl-c">#  '__annotations__': {'x': &lt;class 'int'&gt;, 'y': &lt;class 'int'&gt;},</span>
    <span class="pl-c">#  '__dict__': &lt;attribute '__dict__' of 'Coordinates' objects&gt;,</span>
    <span class="pl-c">#  '__weakref__': &lt;attribute '__weakref__' of 'Coordinates' objects&gt;,</span>
    <span class="pl-c">#  '__doc__': None}</span>
    <span class="pl-c">#</span>
    <span class="pl-c"># The purpose of this function is mainly to add __init__ and</span>
    <span class="pl-c"># __repr__ and __eq__ methods, but only if they are not yet defined</span>
    <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s">"__init__"</span> <span class="pl-c1">in</span> <span class="pl-s1">clsdict</span>:
        <span class="pl-s1">cls</span>.<span class="pl-s1">__init__</span> <span class="pl-c1">=</span> <span class="pl-en">make__init__</span>(<span class="pl-s1">fields</span>)
    <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s">"__repr__"</span> <span class="pl-c1">in</span> <span class="pl-s1">clsdict</span>:
        <span class="pl-s1">cls</span>.<span class="pl-s1">__repr__</span> <span class="pl-c1">=</span> <span class="pl-en">make__repr__</span>(<span class="pl-s1">fields</span>)
    <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s">"__eq__"</span> <span class="pl-c1">in</span> <span class="pl-s1">clsdict</span>:
        <span class="pl-s1">cls</span>.<span class="pl-s1">__eq__</span> <span class="pl-c1">=</span> <span class="pl-en">make__eq__</span>(<span class="pl-s1">fields</span>)
    <span class="pl-c"># Not sure why these are commented out right now:</span>
    <span class="pl-c"># if not '__iter__' in clsdict:  cls.__iter__ = make__iter__(fields)</span>
    <span class="pl-c"># if not '__hash__' in clsdict:  cls.__hash__ = make__hash__(fields)</span>
    <span class="pl-s1">cls</span>.<span class="pl-s1">__match_args__</span> <span class="pl-c1">=</span> <span class="pl-s1">fields</span>
    <span class="pl-c"># This sets __match_args__ to {'x': &lt;class 'int'&gt;, 'y': &lt;class 'int'&gt;}</span>
    <span class="pl-c"># Usually __match_args__ is expected to be a tuple like ("x", "y")</span>
    <span class="pl-c"># but presumably this still works with a dictionary because it is used</span>
    <span class="pl-c"># for "name in __match_args__" style lookups.</span>
    <span class="pl-c">#</span>
    <span class="pl-c"># __match_args__ is a property used by Python structural typing:</span>
    <span class="pl-c"># https://www.python.org/dev/peps/pep-0622/#special-attribute-match-args</span>
    <span class="pl-k">return</span> <span class="pl-s1">cls</span>


<span class="pl-c"># Example use</span>
<span class="pl-k">if</span> <span class="pl-s1">__name__</span> <span class="pl-c1">==</span> <span class="pl-s">"__main__"</span>:

    <span class="pl-en">@<span class="pl-s1">dataklass</span></span>
    <span class="pl-k">class</span> <span class="pl-v">Coordinates</span>:
        <span class="pl-s1">x</span>: <span class="pl-s1">int</span>
        <span class="pl-s1">y</span>: <span class="pl-s1">int</span></pre></div>


path: cloudflare_robots-txt-cloudflare-workers.md
topic: cloudflare
title: Adding a robots.txt using Cloudflare workers
url: https://github.com/simonw/til/blob/main/cloudflare/robots-txt-cloudflare-workers.md
created: 2021-12-21T15:07:51-08:00
created_utc: 2021-12-21T23:07:51+00:00
updated: 2021-12-21T15:07:51-08:00
updated_utc: 2021-12-21T23:07:51+00:00
shot_hash: 36dbb3210d6769fb6c768cd1b12f367f
slug: robots-txt-cloudflare-workers
body:
I got an unexpected traffic spike to https://russian-ira-facebook-ads.datasettes.com/ - which runs on Cloud Run - and decided to use `robots.txt` to block crawlers.

Re-deploying that instance was a little hard because I didn't have a clean repeatable deployment script in place for it (it's an older project) - so I decided to try using Cloudflare workers for this instead.

DNS was already running through Cloudflare, so switching it to "proxy" mode to enable Cloudflare caching and workers could be done in the Cloudflare control panel. 

![Having turned on the Proxied toggle in the Cloudlfare control panel](https://user-images.githubusercontent.com/9599/147008621-6f87de32-4f6d-4d6b-a685-542fd21da7aa.png)

I navigated to the "Workers" section of the Cloudflare dashboard and clicked "Create a Service", then used their "Introduction (HTTP handler)" starting template. I modified it to look like this and saved it as `block-all-robots`:

```javascript
addEventListener("fetch", (event) => {
  event.respondWith(
    handleRequest(event.request).catch(
      (err) => new Response(err.stack, { status: 500 })
    )
  );
});

async function handleRequest(request) {
  const { pathname } = new URL(request.url);
  if (pathname == "/robots.txt") {
    return new Response("User-agent: *\nDisallow: /", {
      headers: { "Content-Type": "text/plain" },
    });
  }
}
```
After deploying it, https://block-all-robots.simonw.workers.dev/robots.txt started serving my new `robots.txt` file:
```
User-agent: *
Disallow: /
```
Then in the Cloudflare dashboard for `datasettes.com` I found the "Workers" section (not to be confused with the "Workers" section where you create and edit workers) I clicked "Add route" and used the following settings:

![Screenshot of the Add Route dialog](https://user-images.githubusercontent.com/9599/147009015-222346ab-aa0f-403f-acdf-ca9788f525e6.png)

Route: `russian-ira-facebook-ads.datasettes.com/robots.txt`

Service: `block-all-robots`

Environment: `production`

I clicked "Save" and https://russian-ira-facebook-ads.datasettes.com/robots.txt instantly started serving the new file.

html:
<p>I got an unexpected traffic spike to <a href="https://russian-ira-facebook-ads.datasettes.com/" rel="nofollow">https://russian-ira-facebook-ads.datasettes.com/</a> - which runs on Cloud Run - and decided to use <code>robots.txt</code> to block crawlers.</p>
<p>Re-deploying that instance was a little hard because I didn't have a clean repeatable deployment script in place for it (it's an older project) - so I decided to try using Cloudflare workers for this instead.</p>
<p>DNS was already running through Cloudflare, so switching it to "proxy" mode to enable Cloudflare caching and workers could be done in the Cloudflare control panel.</p>
<p><a href="https://user-images.githubusercontent.com/9599/147008621-6f87de32-4f6d-4d6b-a685-542fd21da7aa.png" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/147008621-6f87de32-4f6d-4d6b-a685-542fd21da7aa.png" alt="Having turned on the Proxied toggle in the Cloudlfare control panel" style="max-width:100%;"></a></p>
<p>I navigated to the "Workers" section of the Cloudflare dashboard and clicked "Create a Service", then used their "Introduction (HTTP handler)" starting template. I modified it to look like this and saved it as <code>block-all-robots</code>:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-en">addEventListener</span><span class="pl-kos">(</span><span class="pl-s">"fetch"</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-s1">event</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
  <span class="pl-s1">event</span><span class="pl-kos">.</span><span class="pl-en">respondWith</span><span class="pl-kos">(</span>
    <span class="pl-en">handleRequest</span><span class="pl-kos">(</span><span class="pl-s1">event</span><span class="pl-kos">.</span><span class="pl-c1">request</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">catch</span><span class="pl-kos">(</span>
      <span class="pl-kos">(</span><span class="pl-s1">err</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-k">new</span> <span class="pl-v">Response</span><span class="pl-kos">(</span><span class="pl-s1">err</span><span class="pl-kos">.</span><span class="pl-c1">stack</span><span class="pl-kos">,</span> <span class="pl-kos">{</span> <span class="pl-c1">status</span>: <span class="pl-c1">500</span> <span class="pl-kos">}</span><span class="pl-kos">)</span>
    <span class="pl-kos">)</span>
  <span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

<span class="pl-k">async</span> <span class="pl-k">function</span> <span class="pl-en">handleRequest</span><span class="pl-kos">(</span><span class="pl-s1">request</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
  <span class="pl-k">const</span> <span class="pl-kos">{</span> pathname <span class="pl-kos">}</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-c1">URL</span><span class="pl-kos">(</span><span class="pl-s1">request</span><span class="pl-kos">.</span><span class="pl-c1">url</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">pathname</span> <span class="pl-c1">==</span> <span class="pl-s">"/robots.txt"</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-k">return</span> <span class="pl-k">new</span> <span class="pl-v">Response</span><span class="pl-kos">(</span><span class="pl-s">"User-agent: *\nDisallow: /"</span><span class="pl-kos">,</span> <span class="pl-kos">{</span>
      <span class="pl-c1">headers</span>: <span class="pl-kos">{</span> <span class="pl-s">"Content-Type"</span>: <span class="pl-s">"text/plain"</span> <span class="pl-kos">}</span><span class="pl-kos">,</span>
    <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span>
<span class="pl-kos">}</span></pre></div>
<p>After deploying it, <a href="https://block-all-robots.simonw.workers.dev/robots.txt" rel="nofollow">https://block-all-robots.simonw.workers.dev/robots.txt</a> started serving my new <code>robots.txt</code> file:</p>
<pre><code>User-agent: *
Disallow: /
</code></pre>
<p>Then in the Cloudflare dashboard for <code>datasettes.com</code> I found the "Workers" section (not to be confused with the "Workers" section where you create and edit workers) I clicked "Add route" and used the following settings:</p>
<p><a href="https://user-images.githubusercontent.com/9599/147009015-222346ab-aa0f-403f-acdf-ca9788f525e6.png" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/147009015-222346ab-aa0f-403f-acdf-ca9788f525e6.png" alt="Screenshot of the Add Route dialog" style="max-width:100%;"></a></p>
<p>Route: <code>russian-ira-facebook-ads.datasettes.com/robots.txt</code></p>
<p>Service: <code>block-all-robots</code></p>
<p>Environment: <code>production</code></p>
<p>I clicked "Save" and <a href="https://russian-ira-facebook-ads.datasettes.com/robots.txt" rel="nofollow">https://russian-ira-facebook-ads.datasettes.com/robots.txt</a> instantly started serving the new file.</p>


path: github_transfer-issue-private-to-public.md
topic: github
title: Transferring a GitHub issue from a private to a public repository
url: https://github.com/simonw/til/blob/main/github/transfer-issue-private-to-public.md
created: 2021-12-22T17:46:27-08:00
created_utc: 2021-12-23T01:46:27+00:00
updated: 2021-12-22T17:46:27-08:00
updated_utc: 2021-12-23T01:46:27+00:00
shot_hash: a43d82a535b6b523e9bac2ea57076745
slug: transfer-issue-private-to-public
body:
I have my own private `notes` repository where I sometimes create research threads. Occasionally I want to transfer these to a public repository to publish their contents.

https://docs.github.com/en/issues/tracking-your-work-with-issues/transferring-an-issue-to-another-repository says:

> You can't transfer an issue from a private repository to a public repository.

I found this workaround:

1. Create a new private repository. I called mine `simonw/temp`
2. Transfer the issue from your original repository to this new temporary repository
3. Use the "Settings" tab in the temporary repository to change the entire repository's visibility from private to public
4. Transfer the issue from the temporary repository to the public repository that you want it to live in

## Using the gh tool

You can perform transfers using the web interface, but I also learned how to do it using the `gh` tool.

Install that with `brew install gh`

Then you can run this:

    gh issue transfer https://github.com/simonw/temp/issues/1 simonw/datasette-tiddlywiki

I used this trick today to transfer https://github.com/simonw/datasette-tiddlywiki/issues/2 out of my private `notes` repo.

html:
<p>I have my own private <code>notes</code> repository where I sometimes create research threads. Occasionally I want to transfer these to a public repository to publish their contents.</p>
<p><a href="https://docs.github.com/en/issues/tracking-your-work-with-issues/transferring-an-issue-to-another-repository">https://docs.github.com/en/issues/tracking-your-work-with-issues/transferring-an-issue-to-another-repository</a> says:</p>
<blockquote>
<p>You can't transfer an issue from a private repository to a public repository.</p>
</blockquote>
<p>I found this workaround:</p>
<ol>
<li>Create a new private repository. I called mine <code>simonw/temp</code>
</li>
<li>Transfer the issue from your original repository to this new temporary repository</li>
<li>Use the "Settings" tab in the temporary repository to change the entire repository's visibility from private to public</li>
<li>Transfer the issue from the temporary repository to the public repository that you want it to live in</li>
</ol>
<h2>
<a id="user-content-using-the-gh-tool" class="anchor" href="#using-the-gh-tool" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using the gh tool</h2>
<p>You can perform transfers using the web interface, but I also learned how to do it using the <code>gh</code> tool.</p>
<p>Install that with <code>brew install gh</code></p>
<p>Then you can run this:</p>
<pre><code>gh issue transfer https://github.com/simonw/temp/issues/1 simonw/datasette-tiddlywiki
</code></pre>
<p>I used this trick today to transfer <a href="https://github.com/simonw/datasette-tiddlywiki/issues/2">https://github.com/simonw/datasette-tiddlywiki/issues/2</a> out of my private <code>notes</code> repo.</p>


path: kubernetes_kubectl-proxy.md
topic: kubernetes
title: kubectl proxy
url: https://github.com/simonw/til/blob/main/kubernetes/kubectl-proxy.md
created: 2021-12-28T17:06:34-08:00
created_utc: 2021-12-29T01:06:34+00:00
updated: 2021-12-29T08:59:18-08:00
updated_utc: 2021-12-29T16:59:18+00:00
shot_hash: 27d4e85cf37aea43f8d2efbe44205111
slug: kubectl-proxy
body:
Learned about this today as a way of accessing the Kubernetes REST API.

Assuming you have `kubectl` setup and authorized against a cluster (I'm using DigitalOcean K8S) you can start a `localhost` proxy for talking to the API server in the cluster like this:

    kubectl proxy --port 9000

This starts a proxy running on `localhost` port 9000 which can be used to make authenticated API calls to the cluster. The authentication wrapper (which I think defaults to client certificates) is added automatically, so you can just hit `http://localhost:9000/` using `curl`.
```
curl localhost:9000
{
  "paths": [
    "/.well-known/openid-configuration",
    "/api",
    "/api/v1",
    "/apis",
    "/apis/",
    "/apis/admissionregistration.k8s.io",
    "/apis/admissionregistration.k8s.io/v1",
    "/apis/admissionregistration.k8s.io/v1beta1",
    "/apis/apiextensions.k8s.io",
...
```
Absolutely everything in Kubernetes is exposed via the API. Hitting the homepage, as above, shows a list of API paths. Then you can do things like this:
```
% # List nodes in the cluster
% curl localhost:9000/api/v1/nodes
{
  "kind": "NodeList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "18429049"
  },
  "items": [
    {
      "metadata": {
        "name": "..."

% # List pods (effectively containers) in the cluster:
% curl localhost:9000/api/v1/pods 
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "18429226"
  },
  "items": [
    {
      "metadata": {
        "name": "alpaca-prod",
        "namespace": "default",
        "uid": "50b03bf7-c46d-4ebb-ab93-df089940fa9c",
        "resourceVersion": "1207774",
        "creationTimestamp": "2021-10-31T21:18:08Z",
        "labels": {

% # Show Kubernetes version
% curl localhost:9000/version    
{
  "major": "1",
  "minor": "21",
  "gitVersion": "v1.21.5",
  "gitCommit": "aea7bbadd2fc0cd689de94a54e5b7b758869d691",
  "gitTreeState": "clean",
  "buildDate": "2021-09-15T21:04:16Z",
  "goVersion": "go1.16.8",
  "compiler": "gc",
  "platform": "linux/amd64"
}
```

html:
<p>Learned about this today as a way of accessing the Kubernetes REST API.</p>
<p>Assuming you have <code>kubectl</code> setup and authorized against a cluster (I'm using DigitalOcean K8S) you can start a <code>localhost</code> proxy for talking to the API server in the cluster like this:</p>
<pre><code>kubectl proxy --port 9000
</code></pre>
<p>This starts a proxy running on <code>localhost</code> port 9000 which can be used to make authenticated API calls to the cluster. The authentication wrapper (which I think defaults to client certificates) is added automatically, so you can just hit <code>http://localhost:9000/</code> using <code>curl</code>.</p>
<pre><code>curl localhost:9000
{
  "paths": [
    "/.well-known/openid-configuration",
    "/api",
    "/api/v1",
    "/apis",
    "/apis/",
    "/apis/admissionregistration.k8s.io",
    "/apis/admissionregistration.k8s.io/v1",
    "/apis/admissionregistration.k8s.io/v1beta1",
    "/apis/apiextensions.k8s.io",
...
</code></pre>
<p>Absolutely everything in Kubernetes is exposed via the API. Hitting the homepage, as above, shows a list of API paths. Then you can do things like this:</p>
<pre><code>% # List nodes in the cluster
% curl localhost:9000/api/v1/nodes
{
  "kind": "NodeList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "18429049"
  },
  "items": [
    {
      "metadata": {
        "name": "..."

% # List pods (effectively containers) in the cluster:
% curl localhost:9000/api/v1/pods 
{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {
    "resourceVersion": "18429226"
  },
  "items": [
    {
      "metadata": {
        "name": "alpaca-prod",
        "namespace": "default",
        "uid": "50b03bf7-c46d-4ebb-ab93-df089940fa9c",
        "resourceVersion": "1207774",
        "creationTimestamp": "2021-10-31T21:18:08Z",
        "labels": {

% # Show Kubernetes version
% curl localhost:9000/version    
{
  "major": "1",
  "minor": "21",
  "gitVersion": "v1.21.5",
  "gitCommit": "aea7bbadd2fc0cd689de94a54e5b7b758869d691",
  "gitTreeState": "clean",
  "buildDate": "2021-09-15T21:04:16Z",
  "goVersion": "go1.16.8",
  "compiler": "gc",
  "platform": "linux/amd64"
}
</code></pre>


path: webauthn_webauthn-browser-support.md
topic: webauthn
title: WebAuthn browser support
url: https://github.com/simonw/til/blob/main/webauthn/webauthn-browser-support.md
created: 2021-12-29T14:15:11-08:00
created_utc: 2021-12-29T22:15:11+00:00
updated: 2021-12-29T14:15:11-08:00
updated_utc: 2021-12-29T22:15:11+00:00
shot_hash: c04076a8d5b85c1042a49d76f9ac1267
slug: webauthn-browser-support
body:
I [started exploring](https://twitter.com/simonw/status/1476249939516616704) **[WebAuthn](https://webauthn.guide/)** today - a set of browser standards that adds support for both Yubikey 2FA hardware devices and "platform" authentication using things like Touch ID and Face ID.

## Trying it out

I have an iPhone with Face ID and a MacBook Pro (16", 2019) running macOS Catalina. I could get it to work on my iPhone but not on my Mac - at least not without a separate Yubikey (I have one lost in a bag somewhere, I should dig that out and try it).

The easiest way to try it out is using the demo on https://webauthn.io/

## Browser support

https://caniuse.com/?search=webauthn has a support matrix.

https://webauthn.me/browser-support has more browser support information, including code that detects the level of support for your current browser.

## Other places that support it

- PyPI let you use it for 2FA at https://pypi.org/manage/account/webauthn-provision - [source code here](https://github.com/pypa/warehouse/blob/eb241f9061633752672a07851cd553ad4c5cd24a/warehouse/manage/views.py#L568)
- GitHub offer it for 2FA - they call it "security keys" and the flow starts at https://github.com/settings/two_factor_authentication/configure
- Best Buy, apparently: https://www.bestbuy.com/identity/global/signin - but I think you need to configure it in an existing account before you use it to sign in.

html:
<p>I <a href="https://twitter.com/simonw/status/1476249939516616704" rel="nofollow">started exploring</a> <strong><a href="https://webauthn.guide/" rel="nofollow">WebAuthn</a></strong> today - a set of browser standards that adds support for both Yubikey 2FA hardware devices and "platform" authentication using things like Touch ID and Face ID.</p>
<h2>
<a id="user-content-trying-it-out" class="anchor" href="#trying-it-out" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Trying it out</h2>
<p>I have an iPhone with Face ID and a MacBook Pro (16", 2019) running macOS Catalina. I could get it to work on my iPhone but not on my Mac - at least not without a separate Yubikey (I have one lost in a bag somewhere, I should dig that out and try it).</p>
<p>The easiest way to try it out is using the demo on <a href="https://webauthn.io/" rel="nofollow">https://webauthn.io/</a></p>
<h2>
<a id="user-content-browser-support" class="anchor" href="#browser-support" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Browser support</h2>
<p><a href="https://caniuse.com/?search=webauthn" rel="nofollow">https://caniuse.com/?search=webauthn</a> has a support matrix.</p>
<p><a href="https://webauthn.me/browser-support" rel="nofollow">https://webauthn.me/browser-support</a> has more browser support information, including code that detects the level of support for your current browser.</p>
<h2>
<a id="user-content-other-places-that-support-it" class="anchor" href="#other-places-that-support-it" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Other places that support it</h2>
<ul>
<li>PyPI let you use it for 2FA at <a href="https://pypi.org/manage/account/webauthn-provision" rel="nofollow">https://pypi.org/manage/account/webauthn-provision</a> - <a href="https://github.com/pypa/warehouse/blob/eb241f9061633752672a07851cd553ad4c5cd24a/warehouse/manage/views.py#L568">source code here</a>
</li>
<li>GitHub offer it for 2FA - they call it "security keys" and the flow starts at <a href="https://github.com/settings/two_factor_authentication/configure">https://github.com/settings/two_factor_authentication/configure</a>
</li>
<li>Best Buy, apparently: <a href="https://www.bestbuy.com/identity/global/signin" rel="nofollow">https://www.bestbuy.com/identity/global/signin</a> - but I think you need to configure it in an existing account before you use it to sign in.</li>
</ul>


path: aws_s3-cors.md
topic: aws
title: Adding a CORS policy to an S3 bucket
url: https://github.com/simonw/til/blob/main/aws/s3-cors.md
created: 2022-01-04T15:42:13-08:00
created_utc: 2022-01-04T23:42:13+00:00
updated: 2022-01-04T20:33:18-08:00
updated_utc: 2022-01-05T04:33:18+00:00
shot_hash: 5e0c5c7fe12cdcc87bde2f7bc050a915
slug: s3-cors
body:
Amazon S3 buckets that are configured to work as public websites can support CORS, allowing assets such as JavaScript modules to be loaded by JavaScript running on other domains.

This configuration happens at the bucket level - it's not something that can be applied to individual items.

[Here's their documentation](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html). As with so many AWS things it involves hand-crafting a JSON document: the documentation for that format, with useful examples, [is here](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ManageCorsUsing.html).

I opted to use the S3 web console option - find the bucket in the console interface, click the "Security" tab and you can paste in a JSON configuration.

The configuration I tried first was this one:

```json
[
    {
        "AllowedHeaders": [
            "*"
        ],
        "AllowedMethods": [
            "GET"
        ],
        "AllowedOrigins": [
            "https://simonwillison.net/"
        ],
        "ExposeHeaders": []
    }
]
```
This should enable CORS access for GET requests from code running on my https://simonwillison.net/ site.

The `AllowedOrigins` key is interesting: it works by inspecting the `Origin` header on the incoming request, and returning CORS headers based on if that origin matches one of the values in the list.

I used `curl -i ... -H "Origin: value"` to confirm that this worked:

```
~ % curl -i 'http://static.simonwillison.net.s3-website-us-west-1.amazonaws.com/static/2022/photoswipe/photoswipe-lightbox.esm.js' \
  -H "Origin: https://simonwillison.net" | head -n 20
-x-amz-request-id: 4YY7ZBCVJ167XCR9
 Date: Tue, 04 Jan 2022 21:02:44 GMT
-Access-Control-Allow-Origin: *
-Access-Control-Allow-Methods: GET
:Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
-Last-Modified: Tue, 04 Jan 2022 20:10:26 GMT
-ETag: "8e26fa2b966ca8bac30678cdd6af765c"
:Content-Type: text/javascript
-Server: AmazonS3

~ % curl -i 'http://static.simonwillison.net.s3-website-us-west-1.amazonaws.com/static/2022/photoswipe/photoswipe-lightbox.esm.js' | head -n 20
x-amz-request-id: MPD20P9P3X45BR1Q
Date: Tue, 04 Jan 2022 21:02:48 GMT
Last-Modified: Tue, 04 Jan 2022 20:10:26 GMT
ETag: "8e26fa2b966ca8bac30678cdd6af765c"
Content-Type: text/javascript
Server: AmazonS3
```

With the `Origin` header on the request it returns the `Access-Control-Allow-Origin` headers. Without it does not.

I'm running my S3 bucket behind a Cloudflare cache. As you can see above, S3 returns a `Vary: Origin` header so caches know that they should respect that header when returning cached content.

But... while Cloudflare [added support for Vary](https://blog.cloudflare.com/vary-for-images-serve-the-correct-images-to-the-correct-browsers/) in September 2021 they only support it for images, not for other file formats! So sadly I don't think you can use CORS for JavaScript modules in this way if you are using Cloudflare.

I also tried using `"AllowedOrigins": ["*"]` in my S3 configuration, but I found that if you make a request without an `Origin` header S3 still doesn't return `Access-Control-Allow-Origin` - so under a cache that does not support Vary you run the risk of caching an asset without those headers.

html:
<p>Amazon S3 buckets that are configured to work as public websites can support CORS, allowing assets such as JavaScript modules to be loaded by JavaScript running on other domains.</p>
<p>This configuration happens at the bucket level - it's not something that can be applied to individual items.</p>
<p><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html" rel="nofollow">Here's their documentation</a>. As with so many AWS things it involves hand-crafting a JSON document: the documentation for that format, with useful examples, <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/ManageCorsUsing.html" rel="nofollow">is here</a>.</p>
<p>I opted to use the S3 web console option - find the bucket in the console interface, click the "Security" tab and you can paste in a JSON configuration.</p>
<p>The configuration I tried first was this one:</p>
<div class="highlight highlight-source-json"><pre>[
    {
        <span class="pl-ent">"AllowedHeaders"</span>: [
            <span class="pl-s"><span class="pl-pds">"</span>*<span class="pl-pds">"</span></span>
        ],
        <span class="pl-ent">"AllowedMethods"</span>: [
            <span class="pl-s"><span class="pl-pds">"</span>GET<span class="pl-pds">"</span></span>
        ],
        <span class="pl-ent">"AllowedOrigins"</span>: [
            <span class="pl-s"><span class="pl-pds">"</span>https://simonwillison.net/<span class="pl-pds">"</span></span>
        ],
        <span class="pl-ent">"ExposeHeaders"</span>: []
    }
]</pre></div>
<p>This should enable CORS access for GET requests from code running on my <a href="https://simonwillison.net/" rel="nofollow">https://simonwillison.net/</a> site.</p>
<p>The <code>AllowedOrigins</code> key is interesting: it works by inspecting the <code>Origin</code> header on the incoming request, and returning CORS headers based on if that origin matches one of the values in the list.</p>
<p>I used <code>curl -i ... -H "Origin: value"</code> to confirm that this worked:</p>
<pre><code>~ % curl -i 'http://static.simonwillison.net.s3-website-us-west-1.amazonaws.com/static/2022/photoswipe/photoswipe-lightbox.esm.js' \
  -H "Origin: https://simonwillison.net" | head -n 20
-x-amz-request-id: 4YY7ZBCVJ167XCR9
 Date: Tue, 04 Jan 2022 21:02:44 GMT
-Access-Control-Allow-Origin: *
-Access-Control-Allow-Methods: GET
:Vary: Origin, Access-Control-Request-Headers, Access-Control-Request-Method
-Last-Modified: Tue, 04 Jan 2022 20:10:26 GMT
-ETag: "8e26fa2b966ca8bac30678cdd6af765c"
:Content-Type: text/javascript
-Server: AmazonS3

~ % curl -i 'http://static.simonwillison.net.s3-website-us-west-1.amazonaws.com/static/2022/photoswipe/photoswipe-lightbox.esm.js' | head -n 20
x-amz-request-id: MPD20P9P3X45BR1Q
Date: Tue, 04 Jan 2022 21:02:48 GMT
Last-Modified: Tue, 04 Jan 2022 20:10:26 GMT
ETag: "8e26fa2b966ca8bac30678cdd6af765c"
Content-Type: text/javascript
Server: AmazonS3
</code></pre>
<p>With the <code>Origin</code> header on the request it returns the <code>Access-Control-Allow-Origin</code> headers. Without it does not.</p>
<p>I'm running my S3 bucket behind a Cloudflare cache. As you can see above, S3 returns a <code>Vary: Origin</code> header so caches know that they should respect that header when returning cached content.</p>
<p>But... while Cloudflare <a href="https://blog.cloudflare.com/vary-for-images-serve-the-correct-images-to-the-correct-browsers/" rel="nofollow">added support for Vary</a> in September 2021 they only support it for images, not for other file formats! So sadly I don't think you can use CORS for JavaScript modules in this way if you are using Cloudflare.</p>
<p>I also tried using <code>"AllowedOrigins": ["*"]</code> in my S3 configuration, but I found that if you make a request without an <code>Origin</code> header S3 still doesn't return <code>Access-Control-Allow-Origin</code> - so under a cache that does not support Vary you run the risk of caching an asset without those headers.</p>


path: pytest_pytest-argparse.md
topic: pytest
title: Writing pytest tests against tools written with argparse
url: https://github.com/simonw/til/blob/main/pytest/pytest-argparse.md
created: 2022-01-08T12:42:34-08:00
created_utc: 2022-01-08T20:42:34+00:00
updated: 2022-01-08T12:42:34-08:00
updated_utc: 2022-01-08T20:42:34+00:00
shot_hash: d7356f1cdf3c7024ef5a7f88c3a2a94f
slug: pytest-argparse
body:
I usually build command-line tools using [Click](https://click.palletsprojects.com/) (and my [click-app](https://github.com/simonw/click-app) cookiecutter template), which includes a really nice [set of tools](https://click.palletsprojects.com/en/8.0.x/testing/) for writing tests.

Today I decided to try building a tool called [stream-delay](https://github.com/simonw/stream-delay) using [argparse]() from the Python standard library, since it didn't need any other dependencies.

The one challenge I had was how to write the tests. I used [pytest](https://pytest.org/) as a test-only dependency.

Here's the pattern I came up with, using the [capsys pytest fixture](https://docs.pytest.org/en/6.2.x/capture.html) to capture standard output from my tool.

```python
from stream_delay import main
import pytest

@pytest.mark.parametrize("option", ("-h", "--help"))
def test_help(capsys, option):
    try:
        main([option])
    except SystemExit:
        pass
    output = capsys.readouterr().out
    assert "Stream one or more files with a delay" in output
```
My `main()` function starts like this:
```python
import argparse, sys

parser = argparse.ArgumentParser(
    description="Stream one or more files with a delay between each line"
)
parser.add_argument("files", type=argparse.FileType("r"), nargs="*", default=["-"])
parser.add_argument("-d", "--delay-in-ms", type=int, default=100)


def main(args=None):
    parsed_args = parser.parse_args(args)
    delay_in_s = float(parsed_args.delay_in_ms) / 1000
    # ...
```
As you can see, `main()` takes an optional list of arguments. The default for that is `None` which will cause `argparse` to read `sys.argv` - but I can inject arguments to the function from my tests if I need to.

I'm catching the `SystemExit` exception because this will be raised by default if you use `-h` or `--help` - but I still want to finish my test execution so I can inspect the captured output.

Complete code:

- [stream_delay.py](https://github.com/simonw/stream-delay/blob/0.1/stream_delay.py)
- [tests/test_stream_delay.py](https://github.com/simonw/stream-delay/blob/0.1/tests/test_stream_delay.py)

html:
<p>I usually build command-line tools using <a href="https://click.palletsprojects.com/" rel="nofollow">Click</a> (and my <a href="https://github.com/simonw/click-app">click-app</a> cookiecutter template), which includes a really nice <a href="https://click.palletsprojects.com/en/8.0.x/testing/" rel="nofollow">set of tools</a> for writing tests.</p>
<p>Today I decided to try building a tool called <a href="https://github.com/simonw/stream-delay">stream-delay</a> using <a href="">argparse</a> from the Python standard library, since it didn't need any other dependencies.</p>
<p>The one challenge I had was how to write the tests. I used <a href="https://pytest.org/" rel="nofollow">pytest</a> as a test-only dependency.</p>
<p>Here's the pattern I came up with, using the <a href="https://docs.pytest.org/en/6.2.x/capture.html" rel="nofollow">capsys pytest fixture</a> to capture standard output from my tool.</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">stream_delay</span> <span class="pl-k">import</span> <span class="pl-s1">main</span>
<span class="pl-k">import</span> <span class="pl-s1">pytest</span>

<span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-s1">mark</span>.<span class="pl-en">parametrize</span>(<span class="pl-s">"option"</span>, (<span class="pl-s">"-h"</span>, <span class="pl-s">"--help"</span>))</span>
<span class="pl-k">def</span> <span class="pl-en">test_help</span>(<span class="pl-s1">capsys</span>, <span class="pl-s1">option</span>):
    <span class="pl-k">try</span>:
        <span class="pl-en">main</span>([<span class="pl-s1">option</span>])
    <span class="pl-k">except</span> <span class="pl-v">SystemExit</span>:
        <span class="pl-k">pass</span>
    <span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-s1">capsys</span>.<span class="pl-en">readouterr</span>().<span class="pl-s1">out</span>
    <span class="pl-k">assert</span> <span class="pl-s">"Stream one or more files with a delay"</span> <span class="pl-c1">in</span> <span class="pl-s1">output</span></pre></div>
<p>My <code>main()</code> function starts like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">argparse</span>, <span class="pl-s1">sys</span>

<span class="pl-s1">parser</span> <span class="pl-c1">=</span> <span class="pl-s1">argparse</span>.<span class="pl-v">ArgumentParser</span>(
    <span class="pl-s1">description</span><span class="pl-c1">=</span><span class="pl-s">"Stream one or more files with a delay between each line"</span>
)
<span class="pl-s1">parser</span>.<span class="pl-en">add_argument</span>(<span class="pl-s">"files"</span>, <span class="pl-s1">type</span><span class="pl-c1">=</span><span class="pl-s1">argparse</span>.<span class="pl-v">FileType</span>(<span class="pl-s">"r"</span>), <span class="pl-s1">nargs</span><span class="pl-c1">=</span><span class="pl-s">"*"</span>, <span class="pl-s1">default</span><span class="pl-c1">=</span>[<span class="pl-s">"-"</span>])
<span class="pl-s1">parser</span>.<span class="pl-en">add_argument</span>(<span class="pl-s">"-d"</span>, <span class="pl-s">"--delay-in-ms"</span>, <span class="pl-s1">type</span><span class="pl-c1">=</span><span class="pl-s1">int</span>, <span class="pl-s1">default</span><span class="pl-c1">=</span><span class="pl-c1">100</span>)


<span class="pl-k">def</span> <span class="pl-en">main</span>(<span class="pl-s1">args</span><span class="pl-c1">=</span><span class="pl-c1">None</span>):
    <span class="pl-s1">parsed_args</span> <span class="pl-c1">=</span> <span class="pl-s1">parser</span>.<span class="pl-en">parse_args</span>(<span class="pl-s1">args</span>)
    <span class="pl-s1">delay_in_s</span> <span class="pl-c1">=</span> <span class="pl-en">float</span>(<span class="pl-s1">parsed_args</span>.<span class="pl-s1">delay_in_ms</span>) <span class="pl-c1">/</span> <span class="pl-c1">1000</span>
    <span class="pl-c"># ...</span></pre></div>
<p>As you can see, <code>main()</code> takes an optional list of arguments. The default for that is <code>None</code> which will cause <code>argparse</code> to read <code>sys.argv</code> - but I can inject arguments to the function from my tests if I need to.</p>
<p>I'm catching the <code>SystemExit</code> exception because this will be raised by default if you use <code>-h</code> or <code>--help</code> - but I still want to finish my test execution so I can inspect the captured output.</p>
<p>Complete code:</p>
<ul>
<li><a href="https://github.com/simonw/stream-delay/blob/0.1/stream_delay.py">stream_delay.py</a></li>
<li><a href="https://github.com/simonw/stream-delay/blob/0.1/tests/test_stream_delay.py">tests/test_stream_delay.py</a></li>
</ul>


path: pytest_test-click-app-with-streaming-input.md
topic: pytest
title: Testing a Click app with streaming input
url: https://github.com/simonw/til/blob/main/pytest/test-click-app-with-streaming-input.md
created: 2022-01-09T20:47:37-08:00
created_utc: 2022-01-10T04:47:37+00:00
updated: 2022-01-09T20:47:37-08:00
updated_utc: 2022-01-10T04:47:37+00:00
shot_hash: 66fbfa1c2530bb546d53b8524f087813
slug: test-click-app-with-streaming-input
body:
For [sqlite-utils#364](https://github.com/simonw/sqlite-utils/issues/364) I needed to write a test for a [Click](https://click.palletsprojects.com/) app which dealt with input streamed to standard input. I needed to run some assertions during that process, which ruled out the usual [CliRunner.invoke()](https://click.palletsprojects.com/en/8.0.x/testing/) testing tool since that works by running the command until completion.

I decided to use `subprocess` to run the application. Here's the pattern I came up with for the test:
```python
def test_insert_streaming_batch_size_1(db_path):
    # https://github.com/simonw/sqlite-utils/issues/364
    # Streaming with --batch-size 1 should commit on each record
    # Can't use CliRunner().invoke() here bacuse we need to
    # run assertions in between writing to process stdin
    proc = subprocess.Popen(
        [
            sys.executable,
            "-m",
            "sqlite_utils",
            "insert",
            db_path,
            "rows",
            "-",
            "--nl",
            "--batch-size",
            "1",
        ],
        stdin=subprocess.PIPE
    )
    proc.stdin.write(b'{"name": "Azi"}\n')
    proc.stdin.flush()
    # Without this delay the data wasn't yet visible
    time.sleep(0.2)
    assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}]
    proc.stdin.write(b'{"name": "Suna"}\n')
    proc.stdin.flush()
    time.sleep(0.2)
    assert list(Database(db_path)["rows"].rows) == [{"name": "Azi"}, {"name": "Suna"}]
    proc.stdin.close()
    proc.wait()
    assert proc.returncode == 0
```
The first trick I'm using here is running `sys.executable` to start a Python process. This ensures I run the Python that is available in the current virtual environment.

I modified my `sqlite-utils` command such that it could also be run using `python -m sqlite_utils` - see [sqlite-utils#368](https://github.com/simonw/sqlite-utils/issues/368) for details.

Setting `stdin=subprocess.PIPE` allows me to write data to the process's standard input using `proc.stdin.write()`.

I realized I needed to call `proc.stdin.flush()` after each write to ensure the write was pushed to the process in a predictable manner.

At the end of the test, running `proc.stdin.close()` is equivalent to sending an end-of-file, then `proc.wait()` ensures the process has finished and terminated.

html:
<p>For <a href="https://github.com/simonw/sqlite-utils/issues/364">sqlite-utils#364</a> I needed to write a test for a <a href="https://click.palletsprojects.com/" rel="nofollow">Click</a> app which dealt with input streamed to standard input. I needed to run some assertions during that process, which ruled out the usual <a href="https://click.palletsprojects.com/en/8.0.x/testing/" rel="nofollow">CliRunner.invoke()</a> testing tool since that works by running the command until completion.</p>
<p>I decided to use <code>subprocess</code> to run the application. Here's the pattern I came up with for the test:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">test_insert_streaming_batch_size_1</span>(<span class="pl-s1">db_path</span>):
    <span class="pl-c"># https://github.com/simonw/sqlite-utils/issues/364</span>
    <span class="pl-c"># Streaming with --batch-size 1 should commit on each record</span>
    <span class="pl-c"># Can't use CliRunner().invoke() here bacuse we need to</span>
    <span class="pl-c"># run assertions in between writing to process stdin</span>
    <span class="pl-s1">proc</span> <span class="pl-c1">=</span> <span class="pl-s1">subprocess</span>.<span class="pl-v">Popen</span>(
        [
            <span class="pl-s1">sys</span>.<span class="pl-s1">executable</span>,
            <span class="pl-s">"-m"</span>,
            <span class="pl-s">"sqlite_utils"</span>,
            <span class="pl-s">"insert"</span>,
            <span class="pl-s1">db_path</span>,
            <span class="pl-s">"rows"</span>,
            <span class="pl-s">"-"</span>,
            <span class="pl-s">"--nl"</span>,
            <span class="pl-s">"--batch-size"</span>,
            <span class="pl-s">"1"</span>,
        ],
        <span class="pl-s1">stdin</span><span class="pl-c1">=</span><span class="pl-s1">subprocess</span>.<span class="pl-v">PIPE</span>
    )
    <span class="pl-s1">proc</span>.<span class="pl-s1">stdin</span>.<span class="pl-en">write</span>(<span class="pl-s">b'{"name": "Azi"}<span class="pl-cce">\n</span>'</span>)
    <span class="pl-s1">proc</span>.<span class="pl-s1">stdin</span>.<span class="pl-en">flush</span>()
    <span class="pl-c"># Without this delay the data wasn't yet visible</span>
    <span class="pl-s1">time</span>.<span class="pl-en">sleep</span>(<span class="pl-c1">0.2</span>)
    <span class="pl-k">assert</span> <span class="pl-en">list</span>(<span class="pl-v">Database</span>(<span class="pl-s1">db_path</span>)[<span class="pl-s">"rows"</span>].<span class="pl-s1">rows</span>) <span class="pl-c1">==</span> [{<span class="pl-s">"name"</span>: <span class="pl-s">"Azi"</span>}]
    <span class="pl-s1">proc</span>.<span class="pl-s1">stdin</span>.<span class="pl-en">write</span>(<span class="pl-s">b'{"name": "Suna"}<span class="pl-cce">\n</span>'</span>)
    <span class="pl-s1">proc</span>.<span class="pl-s1">stdin</span>.<span class="pl-en">flush</span>()
    <span class="pl-s1">time</span>.<span class="pl-en">sleep</span>(<span class="pl-c1">0.2</span>)
    <span class="pl-k">assert</span> <span class="pl-en">list</span>(<span class="pl-v">Database</span>(<span class="pl-s1">db_path</span>)[<span class="pl-s">"rows"</span>].<span class="pl-s1">rows</span>) <span class="pl-c1">==</span> [{<span class="pl-s">"name"</span>: <span class="pl-s">"Azi"</span>}, {<span class="pl-s">"name"</span>: <span class="pl-s">"Suna"</span>}]
    <span class="pl-s1">proc</span>.<span class="pl-s1">stdin</span>.<span class="pl-en">close</span>()
    <span class="pl-s1">proc</span>.<span class="pl-en">wait</span>()
    <span class="pl-k">assert</span> <span class="pl-s1">proc</span>.<span class="pl-s1">returncode</span> <span class="pl-c1">==</span> <span class="pl-c1">0</span></pre></div>
<p>The first trick I'm using here is running <code>sys.executable</code> to start a Python process. This ensures I run the Python that is available in the current virtual environment.</p>
<p>I modified my <code>sqlite-utils</code> command such that it could also be run using <code>python -m sqlite_utils</code> - see <a href="https://github.com/simonw/sqlite-utils/issues/368">sqlite-utils#368</a> for details.</p>
<p>Setting <code>stdin=subprocess.PIPE</code> allows me to write data to the process's standard input using <code>proc.stdin.write()</code>.</p>
<p>I realized I needed to call <code>proc.stdin.flush()</code> after each write to ensure the write was pushed to the process in a predictable manner.</p>
<p>At the end of the test, running <code>proc.stdin.close()</code> is equivalent to sending an end-of-file, then <code>proc.wait()</code> ensures the process has finished and terminated.</p>


path: github_dependabot-python-setup.md
topic: github
title: Configuring Dependabot for a Python project with dependencies in setup.py
url: https://github.com/simonw/til/blob/main/github/dependabot-python-setup.md
created: 2022-01-14T16:33:03-08:00
created_utc: 2022-01-15T00:33:03+00:00
updated: 2022-01-14T16:33:03-08:00
updated_utc: 2022-01-15T00:33:03+00:00
shot_hash: bb8939fd1f0403a79fc2be0cf5774e1d
slug: dependabot-python-setup
body:
The [Dependabot setup instructions](https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/configuration-options-for-dependency-updates) don't explicitly mention projects which keep all of their dependency information in `setup.py`.

It works just fine with those kinds of projects. To start it working, create a file in `.github/dependabot.yml` with the following contents:

```yaml
version: 2
updates:
  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      interval: "daily"
```
Then navigate to https://github.com/simonw/s3-credentials/network/updates (but for your project) - that's Insights -> Dependency graph -> Dependabot - to confirm that it worked.

html:
<p>The <a href="https://docs.github.com/en/code-security/supply-chain-security/keeping-your-dependencies-updated-automatically/configuration-options-for-dependency-updates">Dependabot setup instructions</a> don't explicitly mention projects which keep all of their dependency information in <code>setup.py</code>.</p>
<p>It works just fine with those kinds of projects. To start it working, create a file in <code>.github/dependabot.yml</code> with the following contents:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">version</span>: <span class="pl-c1">2</span>
<span class="pl-ent">updates</span>:
  - <span class="pl-ent">package-ecosystem</span>: <span class="pl-s"><span class="pl-pds">"</span>pip<span class="pl-pds">"</span></span>
    <span class="pl-ent">directory</span>: <span class="pl-s"><span class="pl-pds">"</span>/<span class="pl-pds">"</span></span>
    <span class="pl-ent">schedule</span>:
      <span class="pl-ent">interval</span>: <span class="pl-s"><span class="pl-pds">"</span>daily<span class="pl-pds">"</span></span></pre></div>
<p>Then navigate to <a href="https://github.com/simonw/s3-credentials/network/updates">https://github.com/simonw/s3-credentials/network/updates</a> (but for your project) - that's Insights -&gt; Dependency graph -&gt; Dependabot - to confirm that it worked.</p>


path: javascript_javascript-date-objects.md
topic: javascript
title: JavaScript date objects
url: https://github.com/simonw/til/blob/main/javascript/javascript-date-objects.md
created: 2022-01-16T12:16:03-08:00
created_utc: 2022-01-16T20:16:03+00:00
updated: 2022-01-16T13:20:28-08:00
updated_utc: 2022-01-16T21:20:28+00:00
shot_hash: 2cd70fdccffd368e6b3bd465455cbe67
slug: javascript-date-objects
body:
A few notes on JavaScript `Date` object, based on trying to do some basic things with them in Observable notebooks.

## Parsing ISO format strings into a Date object

I had date times that looked like this:

`2022-03-05 14:35`

Passing this string to the `Date()` constructor breaks in interesting ways: it assumes they are in the browser's current timezone, and it breaks in Safari producing an `Invalid Date` error.

This worked instead:

```javascript
new Date(o.datetime.replace(" ", "T") + ":00Z"))
```
Safari requires dates to look like this - without the `T` you get `Invalid Date`:

`2022-03-05T14:35:00Z`

Adding the trailing `Z` causes the date to be treated as if it was UTC. This can be helpful if you plan to work with them in a mostly timezone-unaware fashion, though see the note about using `{timeZone: "UTC"}` with the formatting methods later in this document.

## Timezones

JavaScript date objects REALLY want to be helpful with timezones. Consider the following examples, here using `.toString()`:

```javascript
(new Date("2021-01-15T14:31:00")).toString()
"Fri Jan 15 2021 14:31:00 GMT-0800 (Pacific Standard Time)"

(new Date("2021-01-15T14:31:00Z")).toString()
"Fri Jan 15 2021 06:31:00 GMT-0800 (Pacific Standard Time)"
```
In both cases the `.toString()` method converts to my current timezone, Pacific Standard Time. Note how the one with the `Z` gets displayed as `06:31:00` while the one without the `Z` is shown as `14:31:00`.

## Differences between dates

I wanted to calculate the number of minutes difference between a start date and an end date. Subtracting the two gives you a difference between them as in integer number of milliseconds! So this gives you minutes:

```javascript
let diff = end.getTime() - start.getTime();
let minutes = Math.round(diff / 60 * 1000);
```

## Displaying dates

The closest JavaScript gets to Python's handy [strftime() method](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes) is the much more unwieldly `toLocaleDateString()` method. Some examples:

```javascript
(new Date).toLocaleDateString("en-US", {
  weekday: "long",
  year: "numeric",
  month: "long",
  day: "numeric",
})
"Sunday, January 16, 2022"
(new Date).toLocaleTimeString("en-US")
"12:12:29 PM"
```
These are also timezone aware. Passing `timeZone: "UTC"` to them can be useful, for example:
```javascript
// This outputs in my timezone, PST:
(new Date("2021-01-22T15:03:00Z")).toLocaleTimeString("en-US")
"7:03:00 AM"
// This keeps the output in UTC:
(new Date("2021-01-22T15:03:00Z")).toLocaleTimeString(
    "en-US", {timeZone: "UTC"}
)
"3:03:00 PM"
```
MDN documentation:

- [toLocaleDateString()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toLocaleDateString)
- [toLocaleTimeString()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toLocaleTimeString)

## Discussion

Lots of useful comments on this in [replies on Twitter](https://twitter.com/simonw/status/1482810703123607552), including recommendations for these libraries:

- [Intl.DateTimeFormat()](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/DateTimeFormat/DateTimeFormat) for even more advanced formatting options in the browser standards
- [Luxon](https://moment.github.io/luxon/) - the more modern alternative to the [no longer recommended](https://momentjs.com/docs/#/-project-status/) Moment.js
- [date-fns](https://date-fns.org/) as a more modular modern alternative
- GitHub's [time-elements](https://github.com/github/time-elements) web components
- There's a new browser standard for JavaScript date operations coming in the future called Temporal, [described in detail here](https://2ality.com/2021/06/temporal-api.html)

html:
<p>A few notes on JavaScript <code>Date</code> object, based on trying to do some basic things with them in Observable notebooks.</p>
<h2>
<a id="user-content-parsing-iso-format-strings-into-a-date-object" class="anchor" href="#parsing-iso-format-strings-into-a-date-object" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Parsing ISO format strings into a Date object</h2>
<p>I had date times that looked like this:</p>
<p><code>2022-03-05 14:35</code></p>
<p>Passing this string to the <code>Date()</code> constructor breaks in interesting ways: it assumes they are in the browser's current timezone, and it breaks in Safari producing an <code>Invalid Date</code> error.</p>
<p>This worked instead:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">new</span> <span class="pl-v">Date</span><span class="pl-kos">(</span><span class="pl-s1">o</span><span class="pl-kos">.</span><span class="pl-c1">datetime</span><span class="pl-kos">.</span><span class="pl-en">replace</span><span class="pl-kos">(</span><span class="pl-s">" "</span><span class="pl-kos">,</span> <span class="pl-s">"T"</span><span class="pl-kos">)</span> <span class="pl-c1">+</span> <span class="pl-s">":00Z"</span><span class="pl-kos">)</span><span class="pl-kos">)</span></pre></div>
<p>Safari requires dates to look like this - without the <code>T</code> you get <code>Invalid Date</code>:</p>
<p><code>2022-03-05T14:35:00Z</code></p>
<p>Adding the trailing <code>Z</code> causes the date to be treated as if it was UTC. This can be helpful if you plan to work with them in a mostly timezone-unaware fashion, though see the note about using <code>{timeZone: "UTC"}</code> with the formatting methods later in this document.</p>
<h2>
<a id="user-content-timezones" class="anchor" href="#timezones" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Timezones</h2>
<p>JavaScript date objects REALLY want to be helpful with timezones. Consider the following examples, here using <code>.toString()</code>:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-kos">(</span><span class="pl-k">new</span> <span class="pl-v">Date</span><span class="pl-kos">(</span><span class="pl-s">"2021-01-15T14:31:00"</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">toString</span><span class="pl-kos">(</span><span class="pl-kos">)</span>
<span class="pl-s">"Fri Jan 15 2021 14:31:00 GMT-0800 (Pacific Standard Time)"</span>

<span class="pl-kos">(</span><span class="pl-k">new</span> <span class="pl-v">Date</span><span class="pl-kos">(</span><span class="pl-s">"2021-01-15T14:31:00Z"</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">toString</span><span class="pl-kos">(</span><span class="pl-kos">)</span>
<span class="pl-s">"Fri Jan 15 2021 06:31:00 GMT-0800 (Pacific Standard Time)"</span></pre></div>
<p>In both cases the <code>.toString()</code> method converts to my current timezone, Pacific Standard Time. Note how the one with the <code>Z</code> gets displayed as <code>06:31:00</code> while the one without the <code>Z</code> is shown as <code>14:31:00</code>.</p>
<h2>
<a id="user-content-differences-between-dates" class="anchor" href="#differences-between-dates" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Differences between dates</h2>
<p>I wanted to calculate the number of minutes difference between a start date and an end date. Subtracting the two gives you a difference between them as in integer number of milliseconds! So this gives you minutes:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">let</span> <span class="pl-s1">diff</span> <span class="pl-c1">=</span> <span class="pl-s1">end</span><span class="pl-kos">.</span><span class="pl-en">getTime</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">-</span> <span class="pl-s1">start</span><span class="pl-kos">.</span><span class="pl-en">getTime</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-k">let</span> <span class="pl-s1">minutes</span> <span class="pl-c1">=</span> <span class="pl-v">Math</span><span class="pl-kos">.</span><span class="pl-en">round</span><span class="pl-kos">(</span><span class="pl-s1">diff</span> <span class="pl-c1">/</span> <span class="pl-c1">60</span> <span class="pl-c1">*</span> <span class="pl-c1">1000</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<h2>
<a id="user-content-displaying-dates" class="anchor" href="#displaying-dates" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Displaying dates</h2>
<p>The closest JavaScript gets to Python's handy <a href="https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes" rel="nofollow">strftime() method</a> is the much more unwieldly <code>toLocaleDateString()</code> method. Some examples:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-kos">(</span><span class="pl-k">new</span> <span class="pl-v">Date</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">toLocaleDateString</span><span class="pl-kos">(</span><span class="pl-s">"en-US"</span><span class="pl-kos">,</span> <span class="pl-kos">{</span>
  <span class="pl-c1">weekday</span>: <span class="pl-s">"long"</span><span class="pl-kos">,</span>
  <span class="pl-c1">year</span>: <span class="pl-s">"numeric"</span><span class="pl-kos">,</span>
  <span class="pl-c1">month</span>: <span class="pl-s">"long"</span><span class="pl-kos">,</span>
  <span class="pl-c1">day</span>: <span class="pl-s">"numeric"</span><span class="pl-kos">,</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span>
<span class="pl-s">"Sunday, January 16, 2022"</span>
<span class="pl-kos">(</span><span class="pl-k">new</span> <span class="pl-v">Date</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">toLocaleTimeString</span><span class="pl-kos">(</span><span class="pl-s">"en-US"</span><span class="pl-kos">)</span>
<span class="pl-s">"12:12:29 PM"</span></pre></div>
<p>These are also timezone aware. Passing <code>timeZone: "UTC"</code> to them can be useful, for example:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-c">// This outputs in my timezone, PST:</span>
<span class="pl-kos">(</span><span class="pl-k">new</span> <span class="pl-v">Date</span><span class="pl-kos">(</span><span class="pl-s">"2021-01-22T15:03:00Z"</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">toLocaleTimeString</span><span class="pl-kos">(</span><span class="pl-s">"en-US"</span><span class="pl-kos">)</span>
<span class="pl-s">"7:03:00 AM"</span>
<span class="pl-c">// This keeps the output in UTC:</span>
<span class="pl-kos">(</span><span class="pl-k">new</span> <span class="pl-v">Date</span><span class="pl-kos">(</span><span class="pl-s">"2021-01-22T15:03:00Z"</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">toLocaleTimeString</span><span class="pl-kos">(</span>
    <span class="pl-s">"en-US"</span><span class="pl-kos">,</span> <span class="pl-kos">{</span><span class="pl-c1">timeZone</span>: <span class="pl-s">"UTC"</span><span class="pl-kos">}</span>
<span class="pl-kos">)</span>
<span class="pl-s">"3:03:00 PM"</span></pre></div>
<p>MDN documentation:</p>
<ul>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toLocaleDateString" rel="nofollow">toLocaleDateString()</a></li>
<li><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/toLocaleTimeString" rel="nofollow">toLocaleTimeString()</a></li>
</ul>
<h2>
<a id="user-content-discussion" class="anchor" href="#discussion" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Discussion</h2>
<p>Lots of useful comments on this in <a href="https://twitter.com/simonw/status/1482810703123607552" rel="nofollow">replies on Twitter</a>, including recommendations for these libraries:</p>
<ul>
<li>
<a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/DateTimeFormat/DateTimeFormat" rel="nofollow">Intl.DateTimeFormat()</a> for even more advanced formatting options in the browser standards</li>
<li>
<a href="https://moment.github.io/luxon/" rel="nofollow">Luxon</a> - the more modern alternative to the <a href="https://momentjs.com/docs/#/-project-status/" rel="nofollow">no longer recommended</a> Moment.js</li>
<li>
<a href="https://date-fns.org/" rel="nofollow">date-fns</a> as a more modular modern alternative</li>
<li>GitHub's <a href="https://github.com/github/time-elements">time-elements</a> web components</li>
<li>There's a new browser standard for JavaScript date operations coming in the future called Temporal, <a href="https://2ality.com/2021/06/temporal-api.html" rel="nofollow">described in detail here</a>
</li>
</ul>


path: python_output-json-array-streaming.md
topic: python
title: Streaming indented output of a JSON array
url: https://github.com/simonw/til/blob/main/python/output-json-array-streaming.md
created: 2022-01-17T15:52:18-08:00
created_utc: 2022-01-17T23:52:18+00:00
updated: 2022-01-17T16:41:20-08:00
updated_utc: 2022-01-18T00:41:20+00:00
shot_hash: 4f7202a6e84a2055150d311e82ab0004
slug: output-json-array-streaming
body:
I wanted to produce the following output from a command-line tool:

```json
[
  {
    "id": 1,
    "name": "Simon"
  },
  {
    "id": 2,
    "name": "Cleo"
  },
  {
    "id": 3,
    "name": "Azi"
  }
]
```
But I wanted to do it from a streaming iterator - without first buffering the entire output in an in-memory Python list and calling `json.dumps()` on it.

Here's the pattern I came up with:

```python
import itertools, json, textwrap

def stream_indented_json(iterator, indent=2):
    # We have to iterate two-at-a-time so we can know if we
    # should output a trailing comma or if we have reached
    # the last item.
    current_iter, next_iter = itertools.tee(iterator, 2)
    next(next_iter, None)
    first = True
    for item, next_item in itertools.zip_longest(current_iter, next_iter):
        is_last = next_item is None
        data = item
        line = "{first}{serialized}{separator}{last}".format(
            first="[\n" if first else "",
            serialized=textwrap.indent(
                json.dumps(data, indent=indent, default=repr), " " * indent
            ),
            separator="," if not is_last else "",
            last="\n]" if is_last else "",
        )
        yield line
        first = False
    if first:
        # We didn't output anything, so yield the empty list
        yield "[]"
```
Example usage:
```python
for line in stream_indented_json(item_iterator):
    click.echo(line)
```
I built this for [s3-credentials issue #48](https://github.com/simonw/s3-credentials/issues/48).

html:
<p>I wanted to produce the following output from a command-line tool:</p>
<div class="highlight highlight-source-json"><pre>[
  {
    <span class="pl-ent">"id"</span>: <span class="pl-c1">1</span>,
    <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Simon<span class="pl-pds">"</span></span>
  },
  {
    <span class="pl-ent">"id"</span>: <span class="pl-c1">2</span>,
    <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Cleo<span class="pl-pds">"</span></span>
  },
  {
    <span class="pl-ent">"id"</span>: <span class="pl-c1">3</span>,
    <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Azi<span class="pl-pds">"</span></span>
  }
]</pre></div>
<p>But I wanted to do it from a streaming iterator - without first buffering the entire output in an in-memory Python list and calling <code>json.dumps()</code> on it.</p>
<p>Here's the pattern I came up with:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">itertools</span>, <span class="pl-s1">json</span>, <span class="pl-s1">textwrap</span>

<span class="pl-k">def</span> <span class="pl-en">stream_indented_json</span>(<span class="pl-s1">iterator</span>, <span class="pl-s1">indent</span><span class="pl-c1">=</span><span class="pl-c1">2</span>):
    <span class="pl-c"># We have to iterate two-at-a-time so we can know if we</span>
    <span class="pl-c"># should output a trailing comma or if we have reached</span>
    <span class="pl-c"># the last item.</span>
    <span class="pl-s1">current_iter</span>, <span class="pl-s1">next_iter</span> <span class="pl-c1">=</span> <span class="pl-s1">itertools</span>.<span class="pl-en">tee</span>(<span class="pl-s1">iterator</span>, <span class="pl-c1">2</span>)
    <span class="pl-en">next</span>(<span class="pl-s1">next_iter</span>, <span class="pl-c1">None</span>)
    <span class="pl-s1">first</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
    <span class="pl-k">for</span> <span class="pl-s1">item</span>, <span class="pl-s1">next_item</span> <span class="pl-c1">in</span> <span class="pl-s1">itertools</span>.<span class="pl-en">zip_longest</span>(<span class="pl-s1">current_iter</span>, <span class="pl-s1">next_iter</span>):
        <span class="pl-s1">is_last</span> <span class="pl-c1">=</span> <span class="pl-s1">next_item</span> <span class="pl-c1">is</span> <span class="pl-c1">None</span>
        <span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">item</span>
        <span class="pl-s1">line</span> <span class="pl-c1">=</span> <span class="pl-s">"{first}{serialized}{separator}{last}"</span>.<span class="pl-en">format</span>(
            <span class="pl-s1">first</span><span class="pl-c1">=</span><span class="pl-s">"[<span class="pl-cce">\n</span>"</span> <span class="pl-k">if</span> <span class="pl-s1">first</span> <span class="pl-k">else</span> <span class="pl-s">""</span>,
            <span class="pl-s1">serialized</span><span class="pl-c1">=</span><span class="pl-s1">textwrap</span>.<span class="pl-en">indent</span>(
                <span class="pl-s1">json</span>.<span class="pl-en">dumps</span>(<span class="pl-s1">data</span>, <span class="pl-s1">indent</span><span class="pl-c1">=</span><span class="pl-s1">indent</span>, <span class="pl-s1">default</span><span class="pl-c1">=</span><span class="pl-s1">repr</span>), <span class="pl-s">" "</span> <span class="pl-c1">*</span> <span class="pl-s1">indent</span>
            ),
            <span class="pl-s1">separator</span><span class="pl-c1">=</span><span class="pl-s">","</span> <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">is_last</span> <span class="pl-k">else</span> <span class="pl-s">""</span>,
            <span class="pl-s1">last</span><span class="pl-c1">=</span><span class="pl-s">"<span class="pl-cce">\n</span>]"</span> <span class="pl-k">if</span> <span class="pl-s1">is_last</span> <span class="pl-k">else</span> <span class="pl-s">""</span>,
        )
        <span class="pl-k">yield</span> <span class="pl-s1">line</span>
        <span class="pl-s1">first</span> <span class="pl-c1">=</span> <span class="pl-c1">False</span>
    <span class="pl-k">if</span> <span class="pl-s1">first</span>:
        <span class="pl-c"># We didn't output anything, so yield the empty list</span>
        <span class="pl-k">yield</span> <span class="pl-s">"[]"</span></pre></div>
<p>Example usage:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">for</span> <span class="pl-s1">line</span> <span class="pl-c1">in</span> <span class="pl-en">stream_indented_json</span>(<span class="pl-s1">item_iterator</span>):
    <span class="pl-s1">click</span>.<span class="pl-en">echo</span>(<span class="pl-s1">line</span>)</pre></div>
<p>I built this for <a href="https://github.com/simonw/s3-credentials/issues/48">s3-credentials issue #48</a>.</p>


path: sqlite_json-extract-path.md
topic: sqlite
title: json_extract() path syntax in SQLite
url: https://github.com/simonw/til/blob/main/sqlite/json-extract-path.md
created: 2022-01-18T15:00:56-08:00
created_utc: 2022-01-18T23:00:56+00:00
updated: 2022-09-20T13:52:31-07:00
updated_utc: 2022-09-20T20:52:31+00:00
shot_hash: 9d81713e62dae42d599dce9f4533a569
slug: json-extract-path
body:
Several of the [SQLite JSON functions](), such as `json_extract()` and `json_array_length()`, take a path argument. This uses custom syntax along the lines of `$.a[2].b`, as described [in the documentation here](https://sqlite.org/json1.html#path_arguments).

The syntax is similar to that used by MySQL, which is [documented here](https://dev.mysql.com/doc/refman/8.0/en/json.html#json-path-syntax).

Today I figured out the full rules for the path syntax, based on [this forum thread](https://sqlite.org/forum/forumpost/e1d3b6a054) and some dives into the SQLite source code.

## Basic syntax for objects and arrays

Paths must always start with a `$`, which represents the root of the JSON value.

This can be followed by `.key` or `."key"` to navigate into object keys, and `[0]` to navigate into arrays.

The double quote syntax is useful if your key includes `.` characters.

Given this example document:

```json
{
  "creatures": [
    {
      "name": "Cleo",
      "species": "dog"
    },
    {
      "name": "Azi",
      "species": "chicken",
      "weight.lb": 1.6
    },
  ]
}
```
- `$.creatures` returns the JSON array ([demo](https://latest.datasette.io/_memory?sql=select+json_extract%28%27%7B%0D%0A++++%22creatures%22%3A+%5B%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Cleo%22%2C%0D%0A++++++++++++%22species%22%3A+%22dog%22%0D%0A++++++++%7D%2C%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Azi%22%2C%0D%0A++++++++++++%22species%22%3A+%22chicken%22%2C%0D%0A++++++++++++%22weight.lb%22%3A+1.6%0D%0A++++++++%7D%0D%0A++++%5D%0D%0A%7D%27%2C+%3Apath%29&path=%24.creatures))
- `$.creatures[0].name` returns `Cleo` ([demo](https://latest.datasette.io/_memory?sql=select+json_extract%28%27%7B%0D%0A++++%22creatures%22%3A+%5B%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Cleo%22%2C%0D%0A++++++++++++%22species%22%3A+%22dog%22%0D%0A++++++++%7D%2C%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Azi%22%2C%0D%0A++++++++++++%22species%22%3A+%22chicken%22%2C%0D%0A++++++++++++%22weight.lb%22%3A+1.6%0D%0A++++++++%7D%0D%0A++++%5D%0D%0A%7D%27%2C+%3Apath%29&path=%24.creatures%5B0%5D.name))
- `$.creatures[1]."weight.lb"` returns `1.6` ([demo](https://latest.datasette.io/_memory?sql=select+json_extract%28%27%7B%0D%0A++++%22creatures%22%3A+%5B%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Cleo%22%2C%0D%0A++++++++++++%22species%22%3A+%22dog%22%0D%0A++++++++%7D%2C%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Azi%22%2C%0D%0A++++++++++++%22species%22%3A+%22chicken%22%2C%0D%0A++++++++++++%22weight.lb%22%3A+1.6%0D%0A++++++++%7D%0D%0A++++%5D%0D%0A%7D%27%2C+%3Apath%29&path=%24.creatures%5B1%5D.%22weight.lb%22))

## \#-1 to access arrays by index from the end

You can also use `#` inside the `[]` array syntax to refer to the length of the array.

This means `$.creatures[#]` (demo) will return `null` - because array indexing is from 0 so using the length as an index returns the item that's just past the end.

But... you can apply a single integer subtraction operation to that `#` - so you can return the name of the last creature in the array using this:

- `$.creatures[#-1].name` returns `Azi` ([demo](https://latest.datasette.io/_memory?sql=select+json_extract%28%27%7B%0D%0A++++%22creatures%22%3A+%5B%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Cleo%22%2C%0D%0A++++++++++++%22species%22%3A+%22dog%22%0D%0A++++++++%7D%2C%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Azi%22%2C%0D%0A++++++++++++%22species%22%3A+%22chicken%22%2C%0D%0A++++++++++++%22weight.lb%22%3A+1.6%0D%0A++++++++%7D%0D%0A++++%5D%0D%0A%7D%27%2C+%3Apath%29&path=%24.creatures%5B%23-1%5D.name))

Here's [the commit](https://sqlite.org/src/info/35ed68a651f) that added that custom SQLite extension in 2019.

## Keys containing a double quote

If your object key contains a `"` character you can't use the `$."..."` syntax to access it - but provided it does not also contain a `.` character you can escape it like this:

`$.has\" quotes in it`

For example ([demo](https://latest.datasette.io/_memory?sql=select+json_extract(%27%7B%0D%0A++++%22has%5C%22+quotes+in+it%22:+%22hello%22%0D%0A%7D%27,+%27$.has%5C%22+quotes+in+it%27)&path=$.has%5C%22+quotes+in+it)):

```sql
select json_extract('{
    "has\" quotes in it": "hello"
}', '$.has\" quotes in it')
```
Outputs `hello`.

## Source code

The latest source code for the JSON module can be found in [ext/misc/json.c](https://www3.sqlite.org/src/file?name=ext/misc/json.c) - in particular the `static JsonNode *jsonLookup(...)
` function.

The unit tests are really useful - those are spread across these six files:

- [test/json1.test](https://www3.sqlite.org/src/file?name=test/json1.test)
- [test/json101.test](https://www3.sqlite.org/src/file?name=test/json101.test)
- [test/json102.test](https://www3.sqlite.org/src/file?name=test/json102.test)
- [test/json103.test](https://www3.sqlite.org/src/file?name=test/json103.test)
- [test/json104.test](https://www3.sqlite.org/src/file?name=test/json104.test)
- [test/json105.test](https://www3.sqlite.org/src/file?name=test/json105.test) - this one has the tests for `[#]` syntax.

html:
<p>Several of the <a href="">SQLite JSON functions</a>, such as <code>json_extract()</code> and <code>json_array_length()</code>, take a path argument. This uses custom syntax along the lines of <code>$.a[2].b</code>, as described <a href="https://sqlite.org/json1.html#path_arguments" rel="nofollow">in the documentation here</a>.</p>
<p>The syntax is similar to that used by MySQL, which is <a href="https://dev.mysql.com/doc/refman/8.0/en/json.html#json-path-syntax" rel="nofollow">documented here</a>.</p>
<p>Today I figured out the full rules for the path syntax, based on <a href="https://sqlite.org/forum/forumpost/e1d3b6a054" rel="nofollow">this forum thread</a> and some dives into the SQLite source code.</p>
<h2><a id="user-content-basic-syntax-for-objects-and-arrays" class="anchor" aria-hidden="true" href="#basic-syntax-for-objects-and-arrays"><span aria-hidden="true" class="octicon octicon-link"></span></a>Basic syntax for objects and arrays</h2>
<p>Paths must always start with a <code>$</code>, which represents the root of the JSON value.</p>
<p>This can be followed by <code>.key</code> or <code>."key"</code> to navigate into object keys, and <code>[0]</code> to navigate into arrays.</p>
<p>The double quote syntax is useful if your key includes <code>.</code> characters.</p>
<p>Given this example document:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-ent">"creatures"</span>: [
    {
      <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Cleo<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"species"</span>: <span class="pl-s"><span class="pl-pds">"</span>dog<span class="pl-pds">"</span></span>
    },
    {
      <span class="pl-ent">"name"</span>: <span class="pl-s"><span class="pl-pds">"</span>Azi<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"species"</span>: <span class="pl-s"><span class="pl-pds">"</span>chicken<span class="pl-pds">"</span></span>,
      <span class="pl-ent">"weight.lb"</span>: <span class="pl-c1">1.6</span>
    },
  ]
}</pre></div>
<ul>
<li>
<code>$.creatures</code> returns the JSON array (<a href="https://latest.datasette.io/_memory?sql=select+json_extract%28%27%7B%0D%0A++++%22creatures%22%3A+%5B%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Cleo%22%2C%0D%0A++++++++++++%22species%22%3A+%22dog%22%0D%0A++++++++%7D%2C%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Azi%22%2C%0D%0A++++++++++++%22species%22%3A+%22chicken%22%2C%0D%0A++++++++++++%22weight.lb%22%3A+1.6%0D%0A++++++++%7D%0D%0A++++%5D%0D%0A%7D%27%2C+%3Apath%29&amp;path=%24.creatures" rel="nofollow">demo</a>)</li>
<li>
<code>$.creatures[0].name</code> returns <code>Cleo</code> (<a href="https://latest.datasette.io/_memory?sql=select+json_extract%28%27%7B%0D%0A++++%22creatures%22%3A+%5B%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Cleo%22%2C%0D%0A++++++++++++%22species%22%3A+%22dog%22%0D%0A++++++++%7D%2C%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Azi%22%2C%0D%0A++++++++++++%22species%22%3A+%22chicken%22%2C%0D%0A++++++++++++%22weight.lb%22%3A+1.6%0D%0A++++++++%7D%0D%0A++++%5D%0D%0A%7D%27%2C+%3Apath%29&amp;path=%24.creatures%5B0%5D.name" rel="nofollow">demo</a>)</li>
<li>
<code>$.creatures[1]."weight.lb"</code> returns <code>1.6</code> (<a href="https://latest.datasette.io/_memory?sql=select+json_extract%28%27%7B%0D%0A++++%22creatures%22%3A+%5B%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Cleo%22%2C%0D%0A++++++++++++%22species%22%3A+%22dog%22%0D%0A++++++++%7D%2C%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Azi%22%2C%0D%0A++++++++++++%22species%22%3A+%22chicken%22%2C%0D%0A++++++++++++%22weight.lb%22%3A+1.6%0D%0A++++++++%7D%0D%0A++++%5D%0D%0A%7D%27%2C+%3Apath%29&amp;path=%24.creatures%5B1%5D.%22weight.lb%22" rel="nofollow">demo</a>)</li>
</ul>
<h2><a id="user-content--1-to-access-arrays-by-index-from-the-end" class="anchor" aria-hidden="true" href="#-1-to-access-arrays-by-index-from-the-end"><span aria-hidden="true" class="octicon octicon-link"></span></a>#-1 to access arrays by index from the end</h2>
<p>You can also use <code>#</code> inside the <code>[]</code> array syntax to refer to the length of the array.</p>
<p>This means <code>$.creatures[#]</code> (demo) will return <code>null</code> - because array indexing is from 0 so using the length as an index returns the item that's just past the end.</p>
<p>But... you can apply a single integer subtraction operation to that <code>#</code> - so you can return the name of the last creature in the array using this:</p>
<ul>
<li>
<code>$.creatures[#-1].name</code> returns <code>Azi</code> (<a href="https://latest.datasette.io/_memory?sql=select+json_extract%28%27%7B%0D%0A++++%22creatures%22%3A+%5B%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Cleo%22%2C%0D%0A++++++++++++%22species%22%3A+%22dog%22%0D%0A++++++++%7D%2C%0D%0A++++++++%7B%0D%0A++++++++++++%22name%22%3A+%22Azi%22%2C%0D%0A++++++++++++%22species%22%3A+%22chicken%22%2C%0D%0A++++++++++++%22weight.lb%22%3A+1.6%0D%0A++++++++%7D%0D%0A++++%5D%0D%0A%7D%27%2C+%3Apath%29&amp;path=%24.creatures%5B%23-1%5D.name" rel="nofollow">demo</a>)</li>
</ul>
<p>Here's <a href="https://sqlite.org/src/info/35ed68a651f" rel="nofollow">the commit</a> that added that custom SQLite extension in 2019.</p>
<h2><a id="user-content-keys-containing-a-double-quote" class="anchor" aria-hidden="true" href="#keys-containing-a-double-quote"><span aria-hidden="true" class="octicon octicon-link"></span></a>Keys containing a double quote</h2>
<p>If your object key contains a <code>"</code> character you can't use the <code>$."..."</code> syntax to access it - but provided it does not also contain a <code>.</code> character you can escape it like this:</p>
<p><code>$.has\" quotes in it</code></p>
<p>For example (<a href="https://latest.datasette.io/_memory?sql=select+json_extract(%27%7B%0D%0A++++%22has%5C%22+quotes+in+it%22:+%22hello%22%0D%0A%7D%27,+%27%24.has%5C%22+quotes+in+it%27)&amp;path=%24.has%5C%22+quotes+in+it" rel="nofollow">demo</a>):</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> json_extract(<span class="pl-s"><span class="pl-pds">'</span>{</span>
<span class="pl-s">    "has<span class="pl-cce">\"</span> quotes in it": "hello"</span>
<span class="pl-s">}<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>$.has<span class="pl-cce">\"</span> quotes in it<span class="pl-pds">'</span></span>)</pre></div>
<p>Outputs <code>hello</code>.</p>
<h2><a id="user-content-source-code" class="anchor" aria-hidden="true" href="#source-code"><span aria-hidden="true" class="octicon octicon-link"></span></a>Source code</h2>
<p>The latest source code for the JSON module can be found in <a href="https://www3.sqlite.org/src/file?name=ext/misc/json.c" rel="nofollow">ext/misc/json.c</a> - in particular the <code>static JsonNode *jsonLookup(...) </code> function.</p>
<p>The unit tests are really useful - those are spread across these six files:</p>
<ul>
<li><a href="https://www3.sqlite.org/src/file?name=test/json1.test" rel="nofollow">test/json1.test</a></li>
<li><a href="https://www3.sqlite.org/src/file?name=test/json101.test" rel="nofollow">test/json101.test</a></li>
<li><a href="https://www3.sqlite.org/src/file?name=test/json102.test" rel="nofollow">test/json102.test</a></li>
<li><a href="https://www3.sqlite.org/src/file?name=test/json103.test" rel="nofollow">test/json103.test</a></li>
<li><a href="https://www3.sqlite.org/src/file?name=test/json104.test" rel="nofollow">test/json104.test</a></li>
<li>
<a href="https://www3.sqlite.org/src/file?name=test/json105.test" rel="nofollow">test/json105.test</a> - this one has the tests for <code>[#]</code> syntax.</li>
</ul>


path: aws_helper-for-boto-aws-pagination.md
topic: aws
title: Helper function for pagination using AWS boto3
url: https://github.com/simonw/til/blob/main/aws/helper-for-boto-aws-pagination.md
created: 2022-01-19T11:49:58-08:00
created_utc: 2022-01-19T19:49:58+00:00
updated: 2022-01-19T11:49:58-08:00
updated_utc: 2022-01-19T19:49:58+00:00
shot_hash: c6247a16c6d08af6de0042edcc3e518d
slug: helper-for-boto-aws-pagination
body:
I noticed that a lot of my boto3 code in [s3-credentials](https://github.com/simonw/s3-credentials) looked like this:

```python
paginator = iam.get_paginator("list_user_policies")
for response in paginator.paginate(UserName=username):
    for policy_name in response["PolicyNames"]:
        print(policy_name)
```
This was enough verbosity that I was hesitating on implementing pagination properly for some method calls.

I came up with this helper function to use instead:

```python
def paginate(service, method, list_key, **kwargs):
    paginator = service.get_paginator(method)
    for response in paginator.paginate(**kwargs):
        yield from response[list_key]
```
Now the above becomes:
```python
for policy_name in paginate(iam, "list_user_policies", "PolicyNames", UserName=username):
    print(policy_name)
```
Here's [the issue](https://github.com/simonw/s3-credentials/issues/63) and the [refactoring commit](https://github.com/simonw/s3-credentials/commit/fc1e06ca3ffa2c73e196cffe741ef4e950204240).

html:
<p>I noticed that a lot of my boto3 code in <a href="https://github.com/simonw/s3-credentials">s3-credentials</a> looked like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">paginator</span> <span class="pl-c1">=</span> <span class="pl-s1">iam</span>.<span class="pl-en">get_paginator</span>(<span class="pl-s">"list_user_policies"</span>)
<span class="pl-k">for</span> <span class="pl-s1">response</span> <span class="pl-c1">in</span> <span class="pl-s1">paginator</span>.<span class="pl-en">paginate</span>(<span class="pl-v">UserName</span><span class="pl-c1">=</span><span class="pl-s1">username</span>):
    <span class="pl-k">for</span> <span class="pl-s1">policy_name</span> <span class="pl-c1">in</span> <span class="pl-s1">response</span>[<span class="pl-s">"PolicyNames"</span>]:
        <span class="pl-en">print</span>(<span class="pl-s1">policy_name</span>)</pre></div>
<p>This was enough verbosity that I was hesitating on implementing pagination properly for some method calls.</p>
<p>I came up with this helper function to use instead:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">paginate</span>(<span class="pl-s1">service</span>, <span class="pl-s1">method</span>, <span class="pl-s1">list_key</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
    <span class="pl-s1">paginator</span> <span class="pl-c1">=</span> <span class="pl-s1">service</span>.<span class="pl-en">get_paginator</span>(<span class="pl-s1">method</span>)
    <span class="pl-k">for</span> <span class="pl-s1">response</span> <span class="pl-c1">in</span> <span class="pl-s1">paginator</span>.<span class="pl-en">paginate</span>(<span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
        <span class="pl-k">yield</span> <span class="pl-k">from</span> <span class="pl-s1">response</span>[<span class="pl-s1">list_key</span>]</pre></div>
<p>Now the above becomes:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">for</span> <span class="pl-s1">policy_name</span> <span class="pl-c1">in</span> <span class="pl-en">paginate</span>(<span class="pl-s1">iam</span>, <span class="pl-s">"list_user_policies"</span>, <span class="pl-s">"PolicyNames"</span>, <span class="pl-v">UserName</span><span class="pl-c1">=</span><span class="pl-s1">username</span>):
    <span class="pl-en">print</span>(<span class="pl-s1">policy_name</span>)</pre></div>
<p>Here's <a href="https://github.com/simonw/s3-credentials/issues/63">the issue</a> and the <a href="https://github.com/simonw/s3-credentials/commit/fc1e06ca3ffa2c73e196cffe741ef4e950204240">refactoring commit</a>.</p>


path: pixelmator_pixel-editing-favicon.md
topic: pixelmator
title: Pixel editing a favicon with Pixelmator
url: https://github.com/simonw/til/blob/main/pixelmator/pixel-editing-favicon.md
created: 2022-01-20T11:15:41-08:00
created_utc: 2022-01-20T19:15:41+00:00
updated: 2022-01-20T11:20:59-08:00
updated_utc: 2022-01-20T19:20:59+00:00
shot_hash: 787dde7273ed859699cbd484188bdc1a
slug: pixel-editing-favicon
body:
I wanted to [add a favicon](https://github.com/simonw/datasette/issues/1603) to Datasette, using a PNG image served from `/favicon.ico` as suggested in [this article by Adam Johnson](https://adamj.eu/tech/2022/01/18/how-to-add-a-favicon-to-your-django-site/).

Initially I created a 128x128 icon image (using Figma and exporting as PNG) and allowed the browsers to resize it down to 32x32 - but I wasn't satisfied with the result:

![Screenshot showing my resized icon in Firefox, Chrome and Safari](https://user-images.githubusercontent.com/9599/150405946-46d4aadc-deda-47f7-a562-0db5232af36c.png)

I decided to use [Pixelmator](https://www.pixelmator.com/pro/) on my Mac (since I already have a paid license) to hand-edit the icon at 32x32 to see if I could get better results.

Pixelmator's default interface doesn't include a tool for setting individual pixels - even at 1px size the default brushes affect nearby pixels too.

Thanks to [Enabling a Pixel Brush in Pixelmator to Draw Pixel Art on Mac](https://osxdaily.com/2016/11/17/enable-pixel-brush-pixelmator-mac/) on OSXDaily I found the preference pane (in Preferences -> Tools -> Paintig) that lets you drag a pixel editing tool onto the tool palette.

With that added, I could bump up the zoom level on a brand new 32x32 image and start editing pixels.

I used the `I` keyboard shortcut to switch to the eyedropper to select colours, and the `P` keyboard shortcut to switch back to the pixel tool to edit the pixels.

I couldn't figure out how to delete individual pixels (in order to achieve a transparent PNG background) so I used the eraser tool to fuzzy-erase areas, then redrew the pixels by hand with the pixel tool.

![Screenshot of the Pixelmator interface](https://user-images.githubusercontent.com/9599/150406510-b563af5b-21e0-48b2-b73f-bbf4031f3050.png)

I exported the finished image as a PNG, then used https://squoosh.app/ to shrink it down to just 208 bytes.

![Screenshot of the Squoosh interface - I used effort=3 and colors=8](https://user-images.githubusercontent.com/9599/150406568-4081e9aa-2161-4f6c-bc97-5e99780d6ea0.png)

The end result, comparing the browser-resized icon to my hand-edited one:

![The one I let the browser resize has some fuzzy edges. The hand-edited one does not.](https://user-images.githubusercontent.com/9599/150407521-a7b0c823-d9f3-4c31-b927-6ac9d93e9d85.png)

html:
<p>I wanted to <a href="https://github.com/simonw/datasette/issues/1603">add a favicon</a> to Datasette, using a PNG image served from <code>/favicon.ico</code> as suggested in <a href="https://adamj.eu/tech/2022/01/18/how-to-add-a-favicon-to-your-django-site/" rel="nofollow">this article by Adam Johnson</a>.</p>
<p>Initially I created a 128x128 icon image (using Figma and exporting as PNG) and allowed the browsers to resize it down to 32x32 - but I wasn't satisfied with the result:</p>
<p><a href="https://user-images.githubusercontent.com/9599/150405946-46d4aadc-deda-47f7-a562-0db5232af36c.png" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/150405946-46d4aadc-deda-47f7-a562-0db5232af36c.png" alt="Screenshot showing my resized icon in Firefox, Chrome and Safari" style="max-width:100%;"></a></p>
<p>I decided to use <a href="https://www.pixelmator.com/pro/" rel="nofollow">Pixelmator</a> on my Mac (since I already have a paid license) to hand-edit the icon at 32x32 to see if I could get better results.</p>
<p>Pixelmator's default interface doesn't include a tool for setting individual pixels - even at 1px size the default brushes affect nearby pixels too.</p>
<p>Thanks to <a href="https://osxdaily.com/2016/11/17/enable-pixel-brush-pixelmator-mac/" rel="nofollow">Enabling a Pixel Brush in Pixelmator to Draw Pixel Art on Mac</a> on OSXDaily I found the preference pane (in Preferences -&gt; Tools -&gt; Paintig) that lets you drag a pixel editing tool onto the tool palette.</p>
<p>With that added, I could bump up the zoom level on a brand new 32x32 image and start editing pixels.</p>
<p>I used the <code>I</code> keyboard shortcut to switch to the eyedropper to select colours, and the <code>P</code> keyboard shortcut to switch back to the pixel tool to edit the pixels.</p>
<p>I couldn't figure out how to delete individual pixels (in order to achieve a transparent PNG background) so I used the eraser tool to fuzzy-erase areas, then redrew the pixels by hand with the pixel tool.</p>
<p><a href="https://user-images.githubusercontent.com/9599/150406510-b563af5b-21e0-48b2-b73f-bbf4031f3050.png" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/150406510-b563af5b-21e0-48b2-b73f-bbf4031f3050.png" alt="Screenshot of the Pixelmator interface" style="max-width:100%;"></a></p>
<p>I exported the finished image as a PNG, then used <a href="https://squoosh.app/" rel="nofollow">https://squoosh.app/</a> to shrink it down to just 208 bytes.</p>
<p><a href="https://user-images.githubusercontent.com/9599/150406568-4081e9aa-2161-4f6c-bc97-5e99780d6ea0.png" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/150406568-4081e9aa-2161-4f6c-bc97-5e99780d6ea0.png" alt="Screenshot of the Squoosh interface - I used effort=3 and colors=8" style="max-width:100%;"></a></p>
<p>The end result, comparing the browser-resized icon to my hand-edited one:</p>
<p><a href="https://user-images.githubusercontent.com/9599/150407521-a7b0c823-d9f3-4c31-b927-6ac9d93e9d85.png" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/150407521-a7b0c823-d9f3-4c31-b927-6ac9d93e9d85.png" alt="The one I let the browser resize has some fuzzy edges. The hand-edited one does not." style="max-width:100%;"></a></p>


path: readthedocs_documentation-seo-canonical.md
topic: readthedocs
title: Promoting the stable version of the documentation using rel=canonical
url: https://github.com/simonw/til/blob/main/readthedocs/documentation-seo-canonical.md
created: 2022-01-20T15:15:38-08:00
created_utc: 2022-01-20T23:15:38+00:00
updated: 2022-01-20T15:15:38-08:00
updated_utc: 2022-01-20T23:15:38+00:00
shot_hash: 58f05ec048b933566e0470fa4d175331
slug: documentation-seo-canonical
body:
I was thinking about documentation SEO today. Like many projects, Datasette offers multiple versions of the documentation:

- https://docs.datasette.io/en/latest/ is the latest `main` branch on GitHub
- https://docs.datasette.io/en/stable/ is the most recent stable (non alpha or beta) release - currently 0.60
- https://docs.datasette.io/en/0.59.4/ is the documentation for that specific version - I have more than 70 of those now

For other projects that do this I've often found myself running a Google search and landing on on older version of their documentation. How best to avoid that?

I started by looking at how Django addresses this. Django uses `rel=canonical` - so, on this page: https://docs.djangoproject.com/en/2.2/topics/db/ they have the following HTML:

```html
<link rel="canonical" href="https://docs.djangoproject.com/en/4.0/topics/db/">
```

I tend to default to imitating Django, so I decided to see if I could do that for the Datasette documentation...

... and found out it's already solved for me! [Read The Docs](https://readthedocs.org/), the platform I use for the Datasette documentation, already implements exactly this pattern.

So https://docs.datasette.io/en/0.50/introspection.html includes the following HTML:
```html
<link rel="canonical" href="https://docs.datasette.io/en/stable/introspection.html" />
```
Here's [their documentation](https://docs.readthedocs.io/en/latest/custom_domains.html#canonical-urls) covering this feature. I think you need to have configured a "default version" (though they may set a sensible default for that already) - for my project the page for doing that is the Advanced settings page at https://readthedocs.org/dashboard/datasette/advanced/

This TIL started life as [a Twitter thread](https://twitter.com/simonw/status/1484287724773203971).

html:
<p>I was thinking about documentation SEO today. Like many projects, Datasette offers multiple versions of the documentation:</p>
<ul>
<li>
<a href="https://docs.datasette.io/en/latest/" rel="nofollow">https://docs.datasette.io/en/latest/</a> is the latest <code>main</code> branch on GitHub</li>
<li>
<a href="https://docs.datasette.io/en/stable/" rel="nofollow">https://docs.datasette.io/en/stable/</a> is the most recent stable (non alpha or beta) release - currently 0.60</li>
<li>
<a href="https://docs.datasette.io/en/0.59.4/" rel="nofollow">https://docs.datasette.io/en/0.59.4/</a> is the documentation for that specific version - I have more than 70 of those now</li>
</ul>
<p>For other projects that do this I've often found myself running a Google search and landing on on older version of their documentation. How best to avoid that?</p>
<p>I started by looking at how Django addresses this. Django uses <code>rel=canonical</code> - so, on this page: <a href="https://docs.djangoproject.com/en/2.2/topics/db/" rel="nofollow">https://docs.djangoproject.com/en/2.2/topics/db/</a> they have the following HTML:</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">link</span> <span class="pl-c1">rel</span>="<span class="pl-s">canonical</span>" <span class="pl-c1">href</span>="<span class="pl-s">https://docs.djangoproject.com/en/4.0/topics/db/</span>"<span class="pl-kos">&gt;</span></pre></div>
<p>I tend to default to imitating Django, so I decided to see if I could do that for the Datasette documentation...</p>
<p>... and found out it's already solved for me! <a href="https://readthedocs.org/" rel="nofollow">Read The Docs</a>, the platform I use for the Datasette documentation, already implements exactly this pattern.</p>
<p>So <a href="https://docs.datasette.io/en/0.50/introspection.html" rel="nofollow">https://docs.datasette.io/en/0.50/introspection.html</a> includes the following HTML:</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">link</span> <span class="pl-c1">rel</span>="<span class="pl-s">canonical</span>" <span class="pl-c1">href</span>="<span class="pl-s">https://docs.datasette.io/en/stable/introspection.html</span>" /&gt;</pre></div>
<p>Here's <a href="https://docs.readthedocs.io/en/latest/custom_domains.html#canonical-urls" rel="nofollow">their documentation</a> covering this feature. I think you need to have configured a "default version" (though they may set a sensible default for that already) - for my project the page for doing that is the Advanced settings page at <a href="https://readthedocs.org/dashboard/datasette/advanced/" rel="nofollow">https://readthedocs.org/dashboard/datasette/advanced/</a></p>
<p>This TIL started life as <a href="https://twitter.com/simonw/status/1484287724773203971" rel="nofollow">a Twitter thread</a>.</p>


path: readthedocs_link-from-latest-to-stable.md
topic: readthedocs
title: Linking from /latest/ to /stable/ on Read The Docs
url: https://github.com/simonw/til/blob/main/readthedocs/link-from-latest-to-stable.md
created: 2022-01-20T15:26:56-08:00
created_utc: 2022-01-20T23:26:56+00:00
updated: 2022-01-20T15:26:56-08:00
updated_utc: 2022-01-20T23:26:56+00:00
shot_hash: 30a3568e724075567ea4bee74b73dce8
slug: link-from-latest-to-stable
body:
[Read The Docs](https://readthedocs.org/) has a handy feature where documentation for older versions will automatically link to the latest release, for example [on this page](https://docs.datasette.io/en/0.56/spatialite.html):

<img width="978" alt="A documentation page with a note that says: You are not reading the most recent version of this documentation. 0.60 is the latest version available." src="https://user-images.githubusercontent.com/9599/150437341-14554fe7-1c47-4462-a1d9-9b8d822aaea8.png">

That feature is enabled by a "Show version warning" check box in their Advanced Settings preference pane.

It's implemented by [this JavaScript](https://github.com/readthedocs/readthedocs.org/blob/0852d7c10d725d954d3e9a93513171baa1116d9f/readthedocs/core/static-src/core/js/doc-embed/version-compare.js#L13-L21) in their default theme, called [from here](https://github.com/readthedocs/readthedocs.org/blob/bc3e147770e5740314a8e8c33fec5d111c850498/readthedocs/core/static-src/core/js/doc-embed/footer.js#L66-L86).

I had an extra requirement: I wanted pages on my `/en/latest/` documentation (which shows documentation for the in-development `main` branch on GitHub) to link back to the `/en/stable/` equivalent - but only if that page also existed in the stable documentation.

I ended up [adding this snippet](https://github.com/simonw/datasette/commit/ffca55dfd7cc9b53522c2e5a2fa1ff67c9beadf2) of jQuery JavaScript to my custom ` docs/_templates/layout.html` template:

```html+jinja
{% block footer %}
{{ super() }}
<script>
jQuery(function ($) {
  // Show banner linking to /stable/ if this is a /latest/ page
  if (!/\/latest\//.test(location.pathname)) {
    return;
  }
  var stableUrl = location.pathname.replace("/latest/", "/stable/");
  // Check it's not a 404
  fetch(stableUrl, { method: "HEAD" }).then((response) => {
    if (response.status == 200) {
      var warning = $(
        `<div class="admonition warning">
           <p class="first admonition-title">Note</p>
           <p class="last">
             This documentation covers the <strong>development version</strong> of Datasette.</p>
             <p>See <a href="${stableUrl}">this page</a> for the current stable release.
           </p>
        </div>`
      );
      warning.find("a").attr("href", stableUrl);
      var body = $("div.body");
      if (!body.length) {
        body = $("div.document");
      }
      body.prepend(warning);
    }
  });
});
</script>
{% endblock %}
```
The neatest piece of this solution is the way it uses an HTTP `HEAD` request via `fetch()` to confirm that the equivalent stable page exists before adding a link to it:
```javascript
  var stableUrl = location.pathname.replace("/latest/", "/stable/");
  // Check it's not a 404
  fetch(stableUrl, { method: "HEAD" }).then((response) => {
    if (response.status == 200) {
      // Add the link
```

Here's what my fix looks like, running on https://docs.datasette.io/en/latest/csv_export.html

<img width="978" alt="This page has a banner that says:  This documentation covers the development version of Datasette. See this page for the current stable release." src="https://user-images.githubusercontent.com/9599/150438021-0ab3db8f-7f65-4846-b2d4-880e10dce79d.png">

## Alternative solution: sphinx-version-warning

Just minutes after I committed my fix I was informed of the existence of [sphinx-version-warning](https://sphinx-version-warning.readthedocs.io/en/latest/), a Sphinx plugin that can solve this problem too. There's an example of using that to add a message to the `/latest/` page in [its own documentation configuration here](https://github.com/humitos/sphinx-version-warning/blob/a82156c2ea08e5feab406514d0ccd9d48a345f48/docs/conf.py#L32-L38).

```python
# -- Version Warning Banner configuration ------------------------------------
versionwarning_messages = {
    'latest': 'This is a custom message only for version "latest" of this documentation.',
}
versionwarning_admonition_type = 'tip'
versionwarning_banner_title = 'Tip'
versionwarning_body_selector = 'div[itemprop="articleBody"]'
```
I decided to stick with my version, mainly because I like the `fetch()` solution I used.

GitHub issue: [ Documentation should clarify /stable/ vs /latest/ #1608](https://github.com/simonw/datasette/issues/1608)

html:
<p><a href="https://readthedocs.org/" rel="nofollow">Read The Docs</a> has a handy feature where documentation for older versions will automatically link to the latest release, for example <a href="https://docs.datasette.io/en/0.56/spatialite.html" rel="nofollow">on this page</a>:</p>
<p><a href="https://user-images.githubusercontent.com/9599/150437341-14554fe7-1c47-4462-a1d9-9b8d822aaea8.png" target="_blank" rel="nofollow"><img width="978" alt="A documentation page with a note that says: You are not reading the most recent version of this documentation. 0.60 is the latest version available." src="https://user-images.githubusercontent.com/9599/150437341-14554fe7-1c47-4462-a1d9-9b8d822aaea8.png" style="max-width:100%;"></a></p>
<p>That feature is enabled by a "Show version warning" check box in their Advanced Settings preference pane.</p>
<p>It's implemented by <a href="https://github.com/readthedocs/readthedocs.org/blob/0852d7c10d725d954d3e9a93513171baa1116d9f/readthedocs/core/static-src/core/js/doc-embed/version-compare.js#L13-L21">this JavaScript</a> in their default theme, called <a href="https://github.com/readthedocs/readthedocs.org/blob/bc3e147770e5740314a8e8c33fec5d111c850498/readthedocs/core/static-src/core/js/doc-embed/footer.js#L66-L86">from here</a>.</p>
<p>I had an extra requirement: I wanted pages on my <code>/en/latest/</code> documentation (which shows documentation for the in-development <code>main</code> branch on GitHub) to link back to the <code>/en/stable/</code> equivalent - but only if that page also existed in the stable documentation.</p>
<p>I ended up <a href="https://github.com/simonw/datasette/commit/ffca55dfd7cc9b53522c2e5a2fa1ff67c9beadf2">adding this snippet</a> of jQuery JavaScript to my custom <code> docs/_templates/layout.html</code> template:</p>
<div class="highlight highlight-text-html-django"><pre><span class="pl-e">{%</span> <span class="pl-k">block</span> <span class="pl-s">footer</span> <span class="pl-e">%}</span>
{{ super() }}
&lt;<span class="pl-ent">script</span>&gt;<span class="pl-s1"></span>
<span class="pl-s1"><span class="pl-en">jQuery</span>(<span class="pl-k">function</span> (<span class="pl-smi">$</span>) {</span>
<span class="pl-s1">  <span class="pl-c"><span class="pl-c">//</span> Show banner linking to /stable/ if this is a /latest/ page</span></span>
<span class="pl-s1">  <span class="pl-k">if</span> (<span class="pl-k">!</span><span class="pl-sr"><span class="pl-pds">/</span><span class="pl-cce">\/</span>latest<span class="pl-cce">\/</span><span class="pl-pds">/</span></span>.<span class="pl-c1">test</span>(<span class="pl-smi">location</span>.<span class="pl-c1">pathname</span>)) {</span>
<span class="pl-s1">    <span class="pl-k">return</span>;</span>
<span class="pl-s1">  }</span>
<span class="pl-s1">  <span class="pl-k">var</span> stableUrl <span class="pl-k">=</span> <span class="pl-smi">location</span>.<span class="pl-c1">pathname</span>.<span class="pl-c1">replace</span>(<span class="pl-s"><span class="pl-pds">"</span>/latest/<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>/stable/<span class="pl-pds">"</span></span>);</span>
<span class="pl-s1">  <span class="pl-c"><span class="pl-c">//</span> Check it's not a 404</span></span>
<span class="pl-s1">  <span class="pl-en">fetch</span>(stableUrl, { method<span class="pl-k">:</span> <span class="pl-s"><span class="pl-pds">"</span>HEAD<span class="pl-pds">"</span></span> }).<span class="pl-c1">then</span>((<span class="pl-smi">response</span>) <span class="pl-k">=&gt;</span> {</span>
<span class="pl-s1">    <span class="pl-k">if</span> (<span class="pl-smi">response</span>.<span class="pl-c1">status</span> <span class="pl-k">==</span> <span class="pl-c1">200</span>) {</span>
<span class="pl-s1">      <span class="pl-k">var</span> warning <span class="pl-k">=</span> <span class="pl-en">$</span>(</span>
<span class="pl-s1">        <span class="pl-s"><span class="pl-pds">`</span>&lt;div class="admonition warning"&gt;</span></span>
<span class="pl-s1"><span class="pl-s">           &lt;p class="first admonition-title"&gt;Note&lt;/p&gt;</span></span>
<span class="pl-s1"><span class="pl-s">           &lt;p class="last"&gt;</span></span>
<span class="pl-s1"><span class="pl-s">             This documentation covers the &lt;strong&gt;development version&lt;/strong&gt; of Datasette.&lt;/p&gt;</span></span>
<span class="pl-s1"><span class="pl-s">             &lt;p&gt;See &lt;a href="<span class="pl-s1"><span class="pl-pse">${</span>stableUrl<span class="pl-pse">}</span></span>"&gt;this page&lt;/a&gt; for the current stable release.</span></span>
<span class="pl-s1"><span class="pl-s">           &lt;/p&gt;</span></span>
<span class="pl-s1"><span class="pl-s">        &lt;/div&gt;<span class="pl-pds">`</span></span></span>
<span class="pl-s1">      );</span>
<span class="pl-s1">      <span class="pl-smi">warning</span>.<span class="pl-c1">find</span>(<span class="pl-s"><span class="pl-pds">"</span>a<span class="pl-pds">"</span></span>).<span class="pl-en">attr</span>(<span class="pl-s"><span class="pl-pds">"</span>href<span class="pl-pds">"</span></span>, stableUrl);</span>
<span class="pl-s1">      <span class="pl-k">var</span> body <span class="pl-k">=</span> <span class="pl-en">$</span>(<span class="pl-s"><span class="pl-pds">"</span>div.body<span class="pl-pds">"</span></span>);</span>
<span class="pl-s1">      <span class="pl-k">if</span> (<span class="pl-k">!</span><span class="pl-smi">body</span>.<span class="pl-c1">length</span>) {</span>
<span class="pl-s1">        body <span class="pl-k">=</span> <span class="pl-en">$</span>(<span class="pl-s"><span class="pl-pds">"</span>div.document<span class="pl-pds">"</span></span>);</span>
<span class="pl-s1">      }</span>
<span class="pl-s1">      <span class="pl-smi">body</span>.<span class="pl-c1">prepend</span>(warning);</span>
<span class="pl-s1">    }</span>
<span class="pl-s1">  });</span>
<span class="pl-s1">});</span>
<span class="pl-s1"></span>&lt;/<span class="pl-ent">script</span>&gt;
<span class="pl-e">{%</span> <span class="pl-k">endblock</span> <span class="pl-e">%}</span></pre></div>
<p>The neatest piece of this solution is the way it uses an HTTP <code>HEAD</code> request via <code>fetch()</code> to confirm that the equivalent stable page exists before adding a link to it:</p>
<div class="highlight highlight-source-js"><pre>  <span class="pl-k">var</span> <span class="pl-s1">stableUrl</span> <span class="pl-c1">=</span> <span class="pl-s1">location</span><span class="pl-kos">.</span><span class="pl-c1">pathname</span><span class="pl-kos">.</span><span class="pl-en">replace</span><span class="pl-kos">(</span><span class="pl-s">"/latest/"</span><span class="pl-kos">,</span> <span class="pl-s">"/stable/"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-c">// Check it's not a 404</span>
  <span class="pl-en">fetch</span><span class="pl-kos">(</span><span class="pl-s1">stableUrl</span><span class="pl-kos">,</span> <span class="pl-kos">{</span> <span class="pl-c1">method</span>: <span class="pl-s">"HEAD"</span> <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-c1">then</span><span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-s1">response</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">response</span><span class="pl-kos">.</span><span class="pl-c1">status</span> <span class="pl-c1">==</span> <span class="pl-c1">200</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
      <span class="pl-c">// Add the link</span></pre></div>
<p>Here's what my fix looks like, running on <a href="https://docs.datasette.io/en/latest/csv_export.html" rel="nofollow">https://docs.datasette.io/en/latest/csv_export.html</a></p>
<p><a href="https://user-images.githubusercontent.com/9599/150438021-0ab3db8f-7f65-4846-b2d4-880e10dce79d.png" target="_blank" rel="nofollow"><img width="978" alt="This page has a banner that says:  This documentation covers the development version of Datasette. See this page for the current stable release." src="https://user-images.githubusercontent.com/9599/150438021-0ab3db8f-7f65-4846-b2d4-880e10dce79d.png" style="max-width:100%;"></a></p>
<h2>
<a id="user-content-alternative-solution-sphinx-version-warning" class="anchor" href="#alternative-solution-sphinx-version-warning" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Alternative solution: sphinx-version-warning</h2>
<p>Just minutes after I committed my fix I was informed of the existence of <a href="https://sphinx-version-warning.readthedocs.io/en/latest/" rel="nofollow">sphinx-version-warning</a>, a Sphinx plugin that can solve this problem too. There's an example of using that to add a message to the <code>/latest/</code> page in <a href="https://github.com/humitos/sphinx-version-warning/blob/a82156c2ea08e5feab406514d0ccd9d48a345f48/docs/conf.py#L32-L38">its own documentation configuration here</a>.</p>
<div class="highlight highlight-source-python"><pre><span class="pl-c"># -- Version Warning Banner configuration ------------------------------------</span>
<span class="pl-s1">versionwarning_messages</span> <span class="pl-c1">=</span> {
    <span class="pl-s">'latest'</span>: <span class="pl-s">'This is a custom message only for version "latest" of this documentation.'</span>,
}
<span class="pl-s1">versionwarning_admonition_type</span> <span class="pl-c1">=</span> <span class="pl-s">'tip'</span>
<span class="pl-s1">versionwarning_banner_title</span> <span class="pl-c1">=</span> <span class="pl-s">'Tip'</span>
<span class="pl-s1">versionwarning_body_selector</span> <span class="pl-c1">=</span> <span class="pl-s">'div[itemprop="articleBody"]'</span></pre></div>
<p>I decided to stick with my version, mainly because I like the <code>fetch()</code> solution I used.</p>
<p>GitHub issue: <a href="https://github.com/simonw/datasette/issues/1608"> Documentation should clarify /stable/ vs /latest/ #1608</a></p>


path: fly_undocumented-graphql-api.md
topic: fly
title: Using the undocumented Fly GraphQL API
url: https://github.com/simonw/til/blob/main/fly/undocumented-graphql-api.md
created: 2022-01-21T14:59:35-08:00
created_utc: 2022-01-21T22:59:35+00:00
updated: 2022-02-01T15:07:06-08:00
updated_utc: 2022-02-01T23:07:06+00:00
shot_hash: 74a5e0005a6e44d69b773fef0a2b0928
slug: undocumented-graphql-api
body:
[Fly](https://fly.io/) has a GraphQL API which is used by some of their own tools - I found it while [browsing around their code](https://github.com/superfly/flyctl/blob/603b0adccf5416188eabaa7dc73f9c0ec88fa6ca/api/resource_volumes.go#L5-L40) on GitHub.

It's very much undocumented, which means you would be very foolish to write any software against it and expect it to continue to work as Fly make changes.

Only it is *kind of* documented, because GraphQL introspection provides decent documentation.

(Also it's used [by example code](https://github.com/fly-apps/hostnamesapi) published by Fly, so maybe it's more supported than I initially thought.)

The endpoint is `https://api.fly.io/graphql` - you need a `Authorization: Bearer xxx` HTTP header to access it, where you can get the `xxx` token by running `flyctl auth token`.

Or, you can point your browser directly at https://api.fly.io/graphql - they are running a copy of [GraphiQL](https://github.com/graphql/graphiql) there which provides an interactive explorer plus documentation and schema tabs.

And if you're signed in to the Fly web interface it will use your `.fly.io` cookies to authenticate your GraphQL requests - so no need to worry about that `Authorization` header.

Here's a query I used to answer the question "what volumes do I have attached, across all of my instances?"

```graphql
{
  apps {
    nodes {
      name
      volumes {
        nodes {
          name
        }
      }
    }
  }
}
```
Here's a much more fun query:
```graphql
{
  # Your user account:
  viewer {
    avatarUrl
    createdAt
    email
    # This returned the following for me:
    # ["backend_wordpress", "response_headers_middleware", "firecracker", "dashboard_logs"]
    featureFlags
  }
  nearestRegion {
    # This returned "sjc"
    code
  }
  personalOrganization {
    name
    creditBalance
    creditBalanceFormatted
    # Not sure what these are but they look interesting - I have 7
    loggedCertificates {
      totalCount
      nodes {
        cert
        id
        root
      }
    }
    isCreditCardSaved
    wireGuardPeers {
      # Returned one entry for me, with name:
      # interactive-Simons-MacBook-Pro-swillison-gmail-com-26
      # Presumably the flyctl CLI command set this up
      totalCount
      nodes {
        name
        network
        peerip
        pubkey
        region
      }
    }
  }
}
```
This one returns recent logs (only for the past hour / max of 50 values - those are the highest numbers that can be used for those parameters):
```graphql
{
  apps {
    nodes {
      name
      vms {
        totalCount
        nodes {
          recentLogs(limit: 50, range: 3600) {
            id
            region
            message
            timestamp
          }
        }
      }
    }
  }
}
```
And another one which digs into the details of attached volumes:
```graphql
{
  apps {
    nodes {
      name
      services {
        checks {
          httpPath
          httpMethod
          name
        }
        description
      }
      volumes {
        nodes {
          id
          name
          createdAt
          host {
            id
          }
          sizeGb
          status
          usedBytes
          region
          app {
            name
          }
          attachedAllocation {
            privateIP
            # Not sure why attachedAllocation on a volume gives app HTTP traffic logs:
            recentLogs {
              id
              message
              timestamp
            }
            canary
            events {
              message
              timestamp
            }
          }
        }
      }
    }
  }
}
```

html:
<p><a href="https://fly.io/" rel="nofollow">Fly</a> has a GraphQL API which is used by some of their own tools - I found it while <a href="https://github.com/superfly/flyctl/blob/603b0adccf5416188eabaa7dc73f9c0ec88fa6ca/api/resource_volumes.go#L5-L40">browsing around their code</a> on GitHub.</p>
<p>It's very much undocumented, which means you would be very foolish to write any software against it and expect it to continue to work as Fly make changes.</p>
<p>Only it is <em>kind of</em> documented, because GraphQL introspection provides decent documentation.</p>
<p>(Also it's used <a href="https://github.com/fly-apps/hostnamesapi">by example code</a> published by Fly, so maybe it's more supported than I initially thought.)</p>
<p>The endpoint is <code>https://api.fly.io/graphql</code> - you need a <code>Authorization: Bearer xxx</code> HTTP header to access it, where you can get the <code>xxx</code> token by running <code>flyctl auth token</code>.</p>
<p>Or, you can point your browser directly at <a href="https://api.fly.io/graphql" rel="nofollow">https://api.fly.io/graphql</a> - they are running a copy of <a href="https://github.com/graphql/graphiql">GraphiQL</a> there which provides an interactive explorer plus documentation and schema tabs.</p>
<p>And if you're signed in to the Fly web interface it will use your <code>.fly.io</code> cookies to authenticate your GraphQL requests - so no need to worry about that <code>Authorization</code> header.</p>
<p>Here's a query I used to answer the question "what volumes do I have attached, across all of my instances?"</p>
<div class="highlight highlight-source-graphql"><pre>{
  <span class="pl-v">apps</span> {
    <span class="pl-v">nodes</span> {
      <span class="pl-v">name</span>
      <span class="pl-v">volumes</span> {
        <span class="pl-v">nodes</span> {
          <span class="pl-v">name</span>
        }
      }
    }
  }
}</pre></div>
<p>Here's a much more fun query:</p>
<div class="highlight highlight-source-graphql"><pre>{
<span class="pl-c">  # Your user account:</span>
  <span class="pl-v">viewer</span> {
    <span class="pl-v">avatarUrl</span>
    <span class="pl-v">createdAt</span>
    <span class="pl-v">email</span>
<span class="pl-c">    # This returned the following for me:</span>
<span class="pl-c">    # ["backend_wordpress", "response_headers_middleware", "firecracker", "dashboard_logs"]</span>
    <span class="pl-v">featureFlags</span>
  }
  <span class="pl-v">nearestRegion</span> {
<span class="pl-c">    # This returned "sjc"</span>
    <span class="pl-v">code</span>
  }
  <span class="pl-v">personalOrganization</span> {
    <span class="pl-v">name</span>
    <span class="pl-v">creditBalance</span>
    <span class="pl-v">creditBalanceFormatted</span>
<span class="pl-c">    # Not sure what these are but they look interesting - I have 7</span>
    <span class="pl-v">loggedCertificates</span> {
      <span class="pl-v">totalCount</span>
      <span class="pl-v">nodes</span> {
        <span class="pl-v">cert</span>
        <span class="pl-v">id</span>
        <span class="pl-v">root</span>
      }
    }
    <span class="pl-v">isCreditCardSaved</span>
    <span class="pl-v">wireGuardPeers</span> {
<span class="pl-c">      # Returned one entry for me, with name:</span>
<span class="pl-c">      # interactive-Simons-MacBook-Pro-swillison-gmail-com-26</span>
<span class="pl-c">      # Presumably the flyctl CLI command set this up</span>
      <span class="pl-v">totalCount</span>
      <span class="pl-v">nodes</span> {
        <span class="pl-v">name</span>
        <span class="pl-v">network</span>
        <span class="pl-v">peerip</span>
        <span class="pl-v">pubkey</span>
        <span class="pl-v">region</span>
      }
    }
  }
}</pre></div>
<p>This one returns recent logs (only for the past hour / max of 50 values - those are the highest numbers that can be used for those parameters):</p>
<div class="highlight highlight-source-graphql"><pre>{
  <span class="pl-v">apps</span> {
    <span class="pl-v">nodes</span> {
      <span class="pl-v">name</span>
      <span class="pl-v">vms</span> {
        <span class="pl-v">totalCount</span>
        <span class="pl-v">nodes</span> {
          <span class="pl-v">recentLogs</span>(<span class="pl-v">limit</span>: <span class="pl-c1">50</span>, <span class="pl-v">range</span>: <span class="pl-c1">3600</span>) {
            <span class="pl-v">id</span>
            <span class="pl-v">region</span>
            <span class="pl-v">message</span>
            <span class="pl-v">timestamp</span>
          }
        }
      }
    }
  }
}</pre></div>
<p>And another one which digs into the details of attached volumes:</p>
<div class="highlight highlight-source-graphql"><pre>{
  <span class="pl-v">apps</span> {
    <span class="pl-v">nodes</span> {
      <span class="pl-v">name</span>
      <span class="pl-v">services</span> {
        <span class="pl-v">checks</span> {
          <span class="pl-v">httpPath</span>
          <span class="pl-v">httpMethod</span>
          <span class="pl-v">name</span>
        }
        <span class="pl-v">description</span>
      }
      <span class="pl-v">volumes</span> {
        <span class="pl-v">nodes</span> {
          <span class="pl-v">id</span>
          <span class="pl-v">name</span>
          <span class="pl-v">createdAt</span>
          <span class="pl-v">host</span> {
            <span class="pl-v">id</span>
          }
          <span class="pl-v">sizeGb</span>
          <span class="pl-v">status</span>
          <span class="pl-v">usedBytes</span>
          <span class="pl-v">region</span>
          <span class="pl-v">app</span> {
            <span class="pl-v">name</span>
          }
          <span class="pl-v">attachedAllocation</span> {
            <span class="pl-v">privateIP</span>
<span class="pl-c">            # Not sure why attachedAllocation on a volume gives app HTTP traffic logs:</span>
            <span class="pl-v">recentLogs</span> {
              <span class="pl-v">id</span>
              <span class="pl-v">message</span>
              <span class="pl-v">timestamp</span>
            }
            <span class="pl-v">canary</span>
            <span class="pl-v">events</span> {
              <span class="pl-v">message</span>
              <span class="pl-v">timestamp</span>
            }
          }
        }
      }
    }
  }
}</pre></div>


path: pytest_only-run-integration.md
topic: pytest
title: Opt-in integration tests with pytest --integration
url: https://github.com/simonw/til/blob/main/pytest/only-run-integration.md
created: 2022-01-26T13:59:39-08:00
created_utc: 2022-01-26T21:59:39+00:00
updated: 2022-01-26T13:59:39-08:00
updated_utc: 2022-01-26T21:59:39+00:00
shot_hash: 38b2312e41da5d19f64d432e91988d16
slug: only-run-integration
body:
For both [s3-credentials](https://github.com/simonw/s3-credentials) and [datasette-publish-fly](https://github.com/simonw/datasette-publish-fly) I have a need for real-world integration tests that actually interact with the underlying APIs (AWS or Fly) to create and destroy resources on those platforms.

Most of the time I want my tests to run without doing these. I want the option to run `pytest --integration` to opt-in to running those extra integration tests.

Here's the pattern I'm using. First, in `tests/conftest.py`:

```python
import pytest


def pytest_addoption(parser):
    parser.addoption(
        "--integration",
        action="store_true",
        default=False,
        help="run integration tests",
    )


def pytest_configure(config):
    config.addinivalue_line(
        "markers",
        "integration: mark test as integration test, only run with --integration",
    )


def pytest_collection_modifyitems(config, items):
    if config.getoption("--integration"):
        # Also run integration tests
        return
    skip_integration = pytest.mark.skip(reason="use --integration option to run")
    for item in items:
        if "integration" in item.keywords:
            item.add_marker(skip_integration)
```
This implements a `@pytest.mark.integration` marker which I can use to mark any test that should be considered part of the integration test suite. These will be skipped by default... but will not be skipped if the `--integration` option is passed to `pytest`.

Then in the tests I can either do this:

```python
@pytest.mark.integration
def test_integration_s3():
    # ...
```
Or if I have a module that only contains integration tests - `tests/test_integration.py` - I can use the following line to apply that decorator to every test in the module:
```python
import pytest

pytestmark = pytest.mark.integration

def test_integration_s3():
    # ...
```

html:
<p>For both <a href="https://github.com/simonw/s3-credentials">s3-credentials</a> and <a href="https://github.com/simonw/datasette-publish-fly">datasette-publish-fly</a> I have a need for real-world integration tests that actually interact with the underlying APIs (AWS or Fly) to create and destroy resources on those platforms.</p>
<p>Most of the time I want my tests to run without doing these. I want the option to run <code>pytest --integration</code> to opt-in to running those extra integration tests.</p>
<p>Here's the pattern I'm using. First, in <code>tests/conftest.py</code>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">pytest</span>


<span class="pl-k">def</span> <span class="pl-en">pytest_addoption</span>(<span class="pl-s1">parser</span>):
    <span class="pl-s1">parser</span>.<span class="pl-en">addoption</span>(
        <span class="pl-s">"--integration"</span>,
        <span class="pl-s1">action</span><span class="pl-c1">=</span><span class="pl-s">"store_true"</span>,
        <span class="pl-s1">default</span><span class="pl-c1">=</span><span class="pl-c1">False</span>,
        <span class="pl-s1">help</span><span class="pl-c1">=</span><span class="pl-s">"run integration tests"</span>,
    )


<span class="pl-k">def</span> <span class="pl-en">pytest_configure</span>(<span class="pl-s1">config</span>):
    <span class="pl-s1">config</span>.<span class="pl-en">addinivalue_line</span>(
        <span class="pl-s">"markers"</span>,
        <span class="pl-s">"integration: mark test as integration test, only run with --integration"</span>,
    )


<span class="pl-k">def</span> <span class="pl-en">pytest_collection_modifyitems</span>(<span class="pl-s1">config</span>, <span class="pl-s1">items</span>):
    <span class="pl-k">if</span> <span class="pl-s1">config</span>.<span class="pl-en">getoption</span>(<span class="pl-s">"--integration"</span>):
        <span class="pl-c"># Also run integration tests</span>
        <span class="pl-k">return</span>
    <span class="pl-s1">skip_integration</span> <span class="pl-c1">=</span> <span class="pl-s1">pytest</span>.<span class="pl-s1">mark</span>.<span class="pl-en">skip</span>(<span class="pl-s1">reason</span><span class="pl-c1">=</span><span class="pl-s">"use --integration option to run"</span>)
    <span class="pl-k">for</span> <span class="pl-s1">item</span> <span class="pl-c1">in</span> <span class="pl-s1">items</span>:
        <span class="pl-k">if</span> <span class="pl-s">"integration"</span> <span class="pl-c1">in</span> <span class="pl-s1">item</span>.<span class="pl-s1">keywords</span>:
            <span class="pl-s1">item</span>.<span class="pl-en">add_marker</span>(<span class="pl-s1">skip_integration</span>)</pre></div>
<p>This implements a <code>@pytest.mark.integration</code> marker which I can use to mark any test that should be considered part of the integration test suite. These will be skipped by default... but will not be skipped if the <code>--integration</code> option is passed to <code>pytest</code>.</p>
<p>Then in the tests I can either do this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-s1">mark</span>.<span class="pl-s1">integration</span></span>
<span class="pl-k">def</span> <span class="pl-en">test_integration_s3</span>():
    <span class="pl-c"># ...</span></pre></div>
<p>Or if I have a module that only contains integration tests - <code>tests/test_integration.py</code> - I can use the following line to apply that decorator to every test in the module:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">pytest</span>

<span class="pl-s1">pytestmark</span> <span class="pl-c1">=</span> <span class="pl-s1">pytest</span>.<span class="pl-s1">mark</span>.<span class="pl-s1">integration</span>

<span class="pl-k">def</span> <span class="pl-en">test_integration_s3</span>():
    <span class="pl-c"># ...</span></pre></div>


path: graphql_get-graphql-schema.md
topic: graphql
title: get-graphql-schema
url: https://github.com/simonw/til/blob/main/graphql/get-graphql-schema.md
created: 2022-02-01T14:55:44-08:00
created_utc: 2022-02-01T22:55:44+00:00
updated: 2022-02-05T11:49:00-08:00
updated_utc: 2022-02-05T19:49:00+00:00
shot_hash: 6a589407167a7697f85d231c55d3964a
slug: get-graphql-schema
body:
The GraphQL schema language is a concise way to represent the available schema provided by a GraphQL endpoint. It looks something like this:

```graphql
type assets {
  url: String
  node_id: String
  name: String
  label: String
  content_type: String
  state: String
  created_at: String
  updated_at: String
  browser_download_url: String
  id: Int
  size: Int
  download_count: Int
  uploader: users
  release: releases
}

type assetsCollection {
  totalCount: Int
  pageInfo: PageInfo
  nodes: [assets]
  edges: [assetsEdge]
}
```
You can retrieve a JSON version of this from the GraphQL server itself using the query at the bottom of this document.

But... if you want it back in readable text as shown above, you can use the [get-graphql-schema](https://www.npmjs.com/package/get-graphql-schema) tool. Run that like this:

    npx get-graphql-schema https://github-to-sqlite.dogsheep.net/graphql

If the API requires authentication, use the `--header` option like this:

    npx get-graphql-schema https://api.github.com/graphql -h 'Authorization=Bearer XXX'

## Retrieving the schema as JSON using GraphQL

I found this query using the network inspector against a GraphiQL instance (https://api.fly.io/graphql) that offered a "Schema" tab:

```graphql
query IntrospectionQuery {
  __schema {
    queryType {
      name
    }
    mutationType {
      name
    }
    subscriptionType {
      name
    }
    types {
      ...FullType
    }
    directives {
      name
      description
      locations
      args {
        ...InputValue
      }
    }
  }
}
fragment FullType on __Type {
  kind
  name
  description
  fields(includeDeprecated: true) {
    name
    description
    args {
      ...InputValue
    }
    type {
      ...TypeRef
    }
    isDeprecated
    deprecationReason
  }
  inputFields {
    ...InputValue
  }
  interfaces {
    ...TypeRef
  }
  enumValues(includeDeprecated: true) {
    name
    description
    isDeprecated
    deprecationReason
  }
  possibleTypes {
    ...TypeRef
  }
}
fragment InputValue on __InputValue {
  name
  description
  type {
    ...TypeRef
  }
  defaultValue
}
fragment TypeRef on __Type {
  kind
  name
  ofType {
    kind
    name
    ofType {
      kind
      name
      ofType {
        kind
        name
        ofType {
          kind
          name
          ofType {
            kind
            name
            ofType {
              kind
              name
              ofType {
                kind
                name
              }
            }
          }
        }
      }
    }
  }
}
```

html:
<p>The GraphQL schema language is a concise way to represent the available schema provided by a GraphQL endpoint. It looks something like this:</p>
<div class="highlight highlight-source-graphql"><pre><span class="pl-k">type</span> <span class="pl-c1">assets</span> {
  <span class="pl-v">url</span>: <span class="pl-c1">String</span>
  <span class="pl-v">node_id</span>: <span class="pl-c1">String</span>
  <span class="pl-v">name</span>: <span class="pl-c1">String</span>
  <span class="pl-v">label</span>: <span class="pl-c1">String</span>
  <span class="pl-v">content_type</span>: <span class="pl-c1">String</span>
  <span class="pl-v">state</span>: <span class="pl-c1">String</span>
  <span class="pl-v">created_at</span>: <span class="pl-c1">String</span>
  <span class="pl-v">updated_at</span>: <span class="pl-c1">String</span>
  <span class="pl-v">browser_download_url</span>: <span class="pl-c1">String</span>
  <span class="pl-v">id</span>: <span class="pl-c1">Int</span>
  <span class="pl-v">size</span>: <span class="pl-c1">Int</span>
  <span class="pl-v">download_count</span>: <span class="pl-c1">Int</span>
  <span class="pl-v">uploader</span>: <span class="pl-c1">users</span>
  <span class="pl-v">release</span>: <span class="pl-c1">releases</span>
}

<span class="pl-k">type</span> <span class="pl-c1">assetsCollection</span> {
  <span class="pl-v">totalCount</span>: <span class="pl-c1">Int</span>
  <span class="pl-v">pageInfo</span>: <span class="pl-c1">PageInfo</span>
  <span class="pl-v">nodes</span>: [<span class="pl-c1">assets</span>]
  <span class="pl-v">edges</span>: [<span class="pl-c1">assetsEdge</span>]
}</pre></div>
<p>You can retrieve a JSON version of this from the GraphQL server itself using the query at the bottom of this document.</p>
<p>But... if you want it back in readable text as shown above, you can use the <a href="https://www.npmjs.com/package/get-graphql-schema" rel="nofollow">get-graphql-schema</a> tool. Run that like this:</p>
<pre><code>npx get-graphql-schema https://github-to-sqlite.dogsheep.net/graphql
</code></pre>
<p>If the API requires authentication, use the <code>--header</code> option like this:</p>
<pre><code>npx get-graphql-schema https://api.github.com/graphql -h 'Authorization=Bearer XXX'
</code></pre>
<h2>
<a id="user-content-retrieving-the-schema-as-json-using-graphql" class="anchor" href="#retrieving-the-schema-as-json-using-graphql" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Retrieving the schema as JSON using GraphQL</h2>
<p>I found this query using the network inspector against a GraphiQL instance (<a href="https://api.fly.io/graphql" rel="nofollow">https://api.fly.io/graphql</a>) that offered a "Schema" tab:</p>
<div class="highlight highlight-source-graphql"><pre><span class="pl-k">query</span> <span class="pl-en">IntrospectionQuery</span> {
  <span class="pl-v">__schema</span> {
    <span class="pl-v">queryType</span> {
      <span class="pl-v">name</span>
    }
    <span class="pl-v">mutationType</span> {
      <span class="pl-v">name</span>
    }
    <span class="pl-v">subscriptionType</span> {
      <span class="pl-v">name</span>
    }
    <span class="pl-v">types</span> {
      <span class="pl-k">...</span><span class="pl-v">FullType</span>
    }
    <span class="pl-v">directives</span> {
      <span class="pl-v">name</span>
      <span class="pl-v">description</span>
      <span class="pl-v">locations</span>
      <span class="pl-v">args</span> {
        <span class="pl-k">...</span><span class="pl-v">InputValue</span>
      }
    }
  }
}
<span class="pl-k">fragment</span> <span class="pl-en">FullType</span> <span class="pl-k">on</span> <span class="pl-c1">__Type</span> {
  <span class="pl-v">kind</span>
  <span class="pl-v">name</span>
  <span class="pl-v">description</span>
  <span class="pl-v">fields</span>(<span class="pl-v">includeDeprecated</span>: <span class="pl-c1">true</span>) {
    <span class="pl-v">name</span>
    <span class="pl-v">description</span>
    <span class="pl-v">args</span> {
      <span class="pl-k">...</span><span class="pl-v">InputValue</span>
    }
    <span class="pl-v">type</span> {
      <span class="pl-k">...</span><span class="pl-v">TypeRef</span>
    }
    <span class="pl-v">isDeprecated</span>
    <span class="pl-v">deprecationReason</span>
  }
  <span class="pl-v">inputFields</span> {
    <span class="pl-k">...</span><span class="pl-v">InputValue</span>
  }
  <span class="pl-v">interfaces</span> {
    <span class="pl-k">...</span><span class="pl-v">TypeRef</span>
  }
  <span class="pl-v">enumValues</span>(<span class="pl-v">includeDeprecated</span>: <span class="pl-c1">true</span>) {
    <span class="pl-v">name</span>
    <span class="pl-v">description</span>
    <span class="pl-v">isDeprecated</span>
    <span class="pl-v">deprecationReason</span>
  }
  <span class="pl-v">possibleTypes</span> {
    <span class="pl-k">...</span><span class="pl-v">TypeRef</span>
  }
}
<span class="pl-k">fragment</span> <span class="pl-en">InputValue</span> <span class="pl-k">on</span> <span class="pl-c1">__InputValue</span> {
  <span class="pl-v">name</span>
  <span class="pl-v">description</span>
  <span class="pl-v">type</span> {
    <span class="pl-k">...</span><span class="pl-v">TypeRef</span>
  }
  <span class="pl-v">defaultValue</span>
}
<span class="pl-k">fragment</span> <span class="pl-en">TypeRef</span> <span class="pl-k">on</span> <span class="pl-c1">__Type</span> {
  <span class="pl-v">kind</span>
  <span class="pl-v">name</span>
  <span class="pl-v">ofType</span> {
    <span class="pl-v">kind</span>
    <span class="pl-v">name</span>
    <span class="pl-v">ofType</span> {
      <span class="pl-v">kind</span>
      <span class="pl-v">name</span>
      <span class="pl-v">ofType</span> {
        <span class="pl-v">kind</span>
        <span class="pl-v">name</span>
        <span class="pl-v">ofType</span> {
          <span class="pl-v">kind</span>
          <span class="pl-v">name</span>
          <span class="pl-v">ofType</span> {
            <span class="pl-v">kind</span>
            <span class="pl-v">name</span>
            <span class="pl-v">ofType</span> {
              <span class="pl-v">kind</span>
              <span class="pl-v">name</span>
              <span class="pl-v">ofType</span> {
                <span class="pl-v">kind</span>
                <span class="pl-v">name</span>
              }
            }
          }
        }
      }
    }
  }
}</pre></div>


path: github-actions_python-3-11.md
topic: github-actions
title: Testing against Python 3.11 preview using GitHub Actions
url: https://github.com/simonw/til/blob/main/github-actions/python-3-11.md
created: 2022-02-02T14:20:31-08:00
created_utc: 2022-02-02T22:20:31+00:00
updated: 2022-08-08T11:57:55-07:00
updated_utc: 2022-08-08T18:57:55+00:00
shot_hash: 488231a258c473bfbbe641647d4eb951
slug: python-3-11
body:
I decided to run my CI tests against the Python 3.11 preview, to avoid the problem I had when Python 3.10 came out with [a bug that affected Datasette](https://simonwillison.net/2021/Oct/9/finding-and-reporting-a-bug/).

I used the new [GitHub Code Search](https://cs.github.com/) to figure out how to do this. I searched for:

    3.11 path:workflows/*.yml

And found [this example](https://github.com/urllib3/urllib3/blob/7bec77e81aa0a194c98381053225813f5347c9d2/.github/workflows/ci.yml#L60) from `urllib3` which showed that the version tag to use is:

    3.11-dev

I added that to my test matrix like so:

```yaml
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11-dev"]
    steps:
    - uses: actions/checkout@v2
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v2
      with:
        python-version: ${{ matrix.python-version }}
    # ...
```
Here's the [full workflow](https://github.com/simonw/datasette/blob/a9d8824617268c4d214dd3be2174ac452044f737/.github/workflows/test.yml).

html:
<p>I decided to run my CI tests against the Python 3.11 preview, to avoid the problem I had when Python 3.10 came out with <a href="https://simonwillison.net/2021/Oct/9/finding-and-reporting-a-bug/" rel="nofollow">a bug that affected Datasette</a>.</p>
<p>I used the new <a href="https://cs.github.com/">GitHub Code Search</a> to figure out how to do this. I searched for:</p>
<pre><code>3.11 path:workflows/*.yml
</code></pre>
<p>And found <a href="https://github.com/urllib3/urllib3/blob/7bec77e81aa0a194c98381053225813f5347c9d2/.github/workflows/ci.yml#L60">this example</a> from <code>urllib3</code> which showed that the version tag to use is:</p>
<pre><code>3.11-dev
</code></pre>
<p>I added that to my test matrix like so:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">jobs</span>:
  <span class="pl-ent">test</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">strategy</span>:
      <span class="pl-ent">matrix</span>:
        <span class="pl-ent">python-version</span>: <span class="pl-s">["3.7", "3.8", "3.9", "3.10", "3.11-dev"]</span>
    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v2</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Set up Python ${{ matrix.python-version }}</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">actions/setup-python@v2</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">python-version</span>: <span class="pl-s">${{ matrix.python-version }}</span>
    <span class="pl-c"><span class="pl-c">#</span> ...</span></pre></div>
<p>Here's the <a href="https://github.com/simonw/datasette/blob/a9d8824617268c4d214dd3be2174ac452044f737/.github/workflows/test.yml">full workflow</a>.</p>


path: sqlite_ordered-group-concat.md
topic: sqlite
title: Ordered group_concat() in SQLite
url: https://github.com/simonw/til/blob/main/sqlite/ordered-group-concat.md
created: 2022-02-06T15:08:39-08:00
created_utc: 2022-02-06T23:08:39+00:00
updated: 2022-06-21T08:55:38-07:00
updated_utc: 2022-06-21T15:55:38+00:00
shot_hash: 4c1a2856d5b6c825187923f7054e44da
slug: ordered-group-concat
body:
I was trying to use `group_concat()` to glue together some column values into a stiched together Markdown document. My first attempt looked like this:

```sql
select group_concat('## ' || chapter || '
                    
> ' || quote, '
                    
') from highlights order by timestamp
```

This attempt didn't work, because the order of the elements combined by a `group_concat()` [is undefined](https://www.sqlite.org/lang_aggfunc.html#group_concat):

> The group_concat() function returns a string which is the concatenation of all non-NULL values of X. If parameter Y is present then it is used as the separator between instances of X. A comma (",") is used as the separator if Y is omitted. **The order of the concatenated elements is arbitrary.**

It turns out you can fix this using a subselect:

```sql
select group_concat('## ' || chapter || '
                    
> ' || quote, '
                    
') from (select chapter, quote from highlights order by timestamp)
```
See [this explanation](https://sqlite.org/forum/forumpost/228bb96e12a746ce) by Keith Medcalf on the SQLite forum.

I think it may also be possible to solve this using Window functions. I tried doing this:
```sql
select group_concat('## ' || chapter || '
                    
> ' || quote, '
                    
') OVER (ORDER BY timestamp) from highlights
```
Which almost worked... but it returned one row for each row in `highlights`, each one with a growing combined result - the result I wanted was in the last returned row.

html:
<p>I was trying to use <code>group_concat()</code> to glue together some column values into a stiched together Markdown document. My first attempt looked like this:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> group_concat(<span class="pl-s"><span class="pl-pds">'</span>## <span class="pl-pds">'</span></span> <span class="pl-k">||</span> chapter <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s">                    </span>
<span class="pl-s">&gt; <span class="pl-pds">'</span></span> <span class="pl-k">||</span> quote, <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s">                    </span>
<span class="pl-s"><span class="pl-pds">'</span></span>) <span class="pl-k">from</span> highlights <span class="pl-k">order by</span> <span class="pl-k">timestamp</span></pre></div>
<p>This attempt didn't work, because the order of the elements combined by a <code>group_concat()</code> <a href="https://www.sqlite.org/lang_aggfunc.html#group_concat" rel="nofollow">is undefined</a>:</p>
<blockquote>
<p>The group_concat() function returns a string which is the concatenation of all non-NULL values of X. If parameter Y is present then it is used as the separator between instances of X. A comma (",") is used as the separator if Y is omitted. <strong>The order of the concatenated elements is arbitrary.</strong></p>
</blockquote>
<p>It turns out you can fix this using a subselect:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> group_concat(<span class="pl-s"><span class="pl-pds">'</span>## <span class="pl-pds">'</span></span> <span class="pl-k">||</span> chapter <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s">                    </span>
<span class="pl-s">&gt; <span class="pl-pds">'</span></span> <span class="pl-k">||</span> quote, <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s">                    </span>
<span class="pl-s"><span class="pl-pds">'</span></span>) <span class="pl-k">from</span> (<span class="pl-k">select</span> chapter, quote <span class="pl-k">from</span> highlights <span class="pl-k">order by</span> <span class="pl-k">timestamp</span>)</pre></div>
<p>See <a href="https://sqlite.org/forum/forumpost/228bb96e12a746ce" rel="nofollow">this explanation</a> by Keith Medcalf on the SQLite forum.</p>
<p>I think it may also be possible to solve this using Window functions. I tried doing this:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> group_concat(<span class="pl-s"><span class="pl-pds">'</span>## <span class="pl-pds">'</span></span> <span class="pl-k">||</span> chapter <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s">                    </span>
<span class="pl-s">&gt; <span class="pl-pds">'</span></span> <span class="pl-k">||</span> quote, <span class="pl-s"><span class="pl-pds">'</span></span>
<span class="pl-s">                    </span>
<span class="pl-s"><span class="pl-pds">'</span></span>) OVER (<span class="pl-k">ORDER BY</span> <span class="pl-k">timestamp</span>) <span class="pl-k">from</span> highlights</pre></div>
<p>Which almost worked... but it returned one row for each row in <code>highlights</code>, each one with a growing combined result - the result I wanted was in the last returned row.</p>


path: npm_annotated-package-json.md
topic: npm
title: Annotated package.json for idb-keyval
url: https://github.com/simonw/til/blob/main/npm/annotated-package-json.md
created: 2022-02-10T17:32:05-08:00
created_utc: 2022-02-11T01:32:05+00:00
updated: 2022-02-10T17:32:05-08:00
updated_utc: 2022-02-11T01:32:05+00:00
shot_hash: 91c9af8f971234f9d31572bcd8138e3f
slug: annotated-package-json
body:
Jake Archibald [pointed to](https://twitter.com/jaffathecake/status/1491771402294370306) his `package.json` for `idb-keyval` as a "modernish example" of NPM packaging on Twitter this morning.

I've been frustrated at my lack of understanding of `package.json` for too long, so I decided to do some research and put together an annotated copy of [that full package.json file](https://github.com/jakearchibald/idb-keyval/blob/fdf795b9b55b417eb6ed22db9721c53877a19569/package.json).

## Relevant links

- The [idb-keyval README](https://github.com/jakearchibald/idb-keyval/blob/main/README.md) has a bunch of extra details about the packaging
- The [NPM package.json docs](https://docs.npmjs.com/cli/v7/configuring-npm/package-json) are a great starting point
- [idb-keyval npm listing](https://www.npmjs.com/package/idb-keyval) (which displays the metadata) 
- [Node.js Packages](https://nodejs.org/api/packages.html) explains the Node.js extensions to `package.json`
- [UNPGK CDN docs](https://unpkg.com/) explaning the `"unpgk"` extension
- [webpack Tree Shaking docs](https://webpack.js.org/guides/tree-shaking/) explaining `"sideEffects"`
- Documentation for [Husky](https://typicode.github.io/husky/) and [lint-staged](https://github.com/okonet/lint-staged)

## Annotated package.json

```javascript
{
  // Name and version are required if you want to publish to npm
  "name": "idb-keyval",
  "version": "6.1.0",

  // Description is listed by npm search and used on websites
  "description": "A super-simple-small keyval store built on top of IndexedDB",

  // The “primary entry point” to your program - if the user installs the package and
  // runs require("package-name") the exports object from this module will be returned
  "main": "./dist/compat.cjs",
```
This is complicated, [see StackOverflow](https://stackoverflow.com/questions/42708484/what-is-the-module-package-json-field-for) - it’s a not-fully-accepted proposal which
is something of a de facto standard. It indicates the version of the package that
can be used as an ES Module. I don’t yet understand the full implications of this.
```javascript
  "module": "./dist/compat.js",

  // The unpkg CDN says “If you omit the file path (i.e. use a “bare” URL), unpkg will
  // serve the file specified by the unpkg field in package.json, or fall back to main.”
  // - this should be what happens when you hit https://unpkg.com/idb-keyval - but I
  // tried that just now and got an error saying “Cannot find "/dist/iife-compat.js"
  // in idb-keyval@6.1.0” so something isn’t quite working right here.
  //
  // The iife-compat.js filename here suggests that this file will use the Immediately
  // invoked function expression pattern, which looks like this:
  //
  // ​​(function () { /* ... */ })();
  //
  // It’s a pattern to avoid polluting the global namespace with variables and functions.
  "unpkg": "./dist/iife-compat.js",

  // This is a Node.js extension. The Node docs say: “The "exports" field provides an
  // alternative to "main" where the package main entry point can be defined while also
  // encapsulating the package, preventing any other entry points besides those defined
  // in "exports". This encapsulation allows module authors to define a public interface
  // for their package.”
  "exports": {
    // I don’t yet understand why this is a nested object
    ".": {
      "module": "./dist/index.js",
      "import": "./dist/index.js",
      "default": "./dist/index.cjs"
    },
    "./dist/*": "./dist/*",
    "./package.json": "./package.json"
  },

  // These files are included when the package is installed as a dependency
  "files": [
    "dist/**"
  ],

  // Another node.js extension: https://nodejs.org/api/packages.html#type says: Files
  // ending with .js are loaded as ES modules when the nearest parent package.json file
  // contains a top-level field "type" with a value of "module".
  "type": "module",

  // TypeScript extension indicating the location of type information for this module
  "types": "./dist/index.d.ts",

  // The webpack Tree Shaking docs say “The new webpack 4 release expands on this capability
  // with a way to provide hints to the compiler via the "sideEffects" package.json property
  // to denote which files in your project are "pure" and therefore safe to prune if unused”
  "sideEffects": false,

  // Running “npm run build” or “npm run dev” or “npm run prepack” runs these:
  "scripts": {
    "build": "rollup -c && node lib/size-report.js",
    "dev": "rollup -cw & serve",
    "prepack": "npm run build"
  },

  // A bunch more metadata used when the package is shown on npm
  "repository": {
    "type": "git",
    "url": "git+https://github.com/jakearchibald/idb-keyval.git"
  },
  "keywords": [
    "idb",
    "indexeddb",
    "store",
    "keyval",
    "localstorage",
    "storage",
    "promise"
  ],
  "author": "Jake Archibald",
  "license": "Apache-2.0",
  "bugs": {
    "url": "https://github.com/jakearchibald/idb-keyval/issues"
  },
  "homepage": "https://github.com/jakearchibald/idb-keyval#readme",

  // These are the dependencies that get installed when you run “npm install” from the
  // root of the package - but they are not installed when an end-user uses
  // “npm install name-of-package”
  "devDependencies": {
    "@babel/core": "^7.16.0",
    "@babel/plugin-external-helpers": "^7.16.0",
    "@babel/plugin-transform-runtime": "^7.16.4",
    "@babel/preset-env": "^7.16.4",
    "@babel/runtime": "^7.16.3",
    "@rollup/plugin-babel": "^5.3.0",
    "@rollup/plugin-commonjs": "^21.0.1",
    "@rollup/plugin-node-resolve": "^13.0.6",
    "@types/chai": "^4.2.22",
    "@types/mocha": "^9.0.0",
    "chai": "^4.3.4",
    "conditional-type-checks": "^1.0.5",
    "del": "^6.0.0",
    "filesize": "^8.0.6",
    "glob": "^7.2.0",
    "husky": "^7.0.4",
    "lint-staged": "^12.1.2",
    "mocha": "^9.1.3",
    "prettier": "^2.5.0",
    "rollup": "^2.60.2",
    "rollup-plugin-terser": "^7.0.2",
    "serve": "^13.0.2",
    "typescript": "^4.3.5"
  },

  // Husky is a tool for managing git pre-commit hooks
  "husky": {
    "hooks": {
      "pre-commit": "lint-staged"
    }
  },

  // lint-staged runs linters against staged files - here it is called by Husky
  "lint-staged": {
    "*.{js,css,md,ts,html}": "prettier --write"
  },

  // This package has only one non-dev dependency
  "dependencies": {
    // safari-14-idb-fix provides a idbReady().then(() => … function for waiting
    // until IDB is ready to be accessed in Safari, as a workaround for a bug that
    // existed prior to Safari 14.7 
    "safari-14-idb-fix": "^3.0.0"
  }
}
```

html:
<p>Jake Archibald <a href="https://twitter.com/jaffathecake/status/1491771402294370306" rel="nofollow">pointed to</a> his <code>package.json</code> for <code>idb-keyval</code> as a "modernish example" of NPM packaging on Twitter this morning.</p>
<p>I've been frustrated at my lack of understanding of <code>package.json</code> for too long, so I decided to do some research and put together an annotated copy of <a href="https://github.com/jakearchibald/idb-keyval/blob/fdf795b9b55b417eb6ed22db9721c53877a19569/package.json">that full package.json file</a>.</p>
<h2>
<a id="user-content-relevant-links" class="anchor" href="#relevant-links" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Relevant links</h2>
<ul>
<li>The <a href="https://github.com/jakearchibald/idb-keyval/blob/main/README.md">idb-keyval README</a> has a bunch of extra details about the packaging</li>
<li>The <a href="https://docs.npmjs.com/cli/v7/configuring-npm/package-json" rel="nofollow">NPM package.json docs</a> are a great starting point</li>
<li>
<a href="https://www.npmjs.com/package/idb-keyval" rel="nofollow">idb-keyval npm listing</a> (which displays the metadata)</li>
<li>
<a href="https://nodejs.org/api/packages.html" rel="nofollow">Node.js Packages</a> explains the Node.js extensions to <code>package.json</code>
</li>
<li>
<a href="https://unpkg.com/" rel="nofollow">UNPGK CDN docs</a> explaning the <code>"unpgk"</code> extension</li>
<li>
<a href="https://webpack.js.org/guides/tree-shaking/" rel="nofollow">webpack Tree Shaking docs</a> explaining <code>"sideEffects"</code>
</li>
<li>Documentation for <a href="https://typicode.github.io/husky/" rel="nofollow">Husky</a> and <a href="https://github.com/okonet/lint-staged">lint-staged</a>
</li>
</ul>
<h2>
<a id="user-content-annotated-packagejson" class="anchor" href="#annotated-packagejson" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Annotated package.json</h2>
<div class="highlight highlight-source-js"><pre><span class="pl-kos">{</span>
  <span class="pl-c">// Name and version are required if you want to publish to npm</span>
  <span class="pl-s">"name"</span>: <span class="pl-s">"idb-keyval"</span><span class="pl-kos">,</span>
  <span class="pl-s">"version"</span>: <span class="pl-s">"6.1.0"</span><span class="pl-kos">,</span>

  <span class="pl-c">// Description is listed by npm search and used on websites</span>
  <span class="pl-s">"description"</span>: <span class="pl-s">"A super-simple-small keyval store built on top of IndexedDB"</span><span class="pl-kos">,</span>

  <span class="pl-c">// The “primary entry point” to your program - if the user installs the package and</span>
  <span class="pl-c">// runs require("package-name") the exports object from this module will be returned</span>
  <span class="pl-s">"main"</span>: <span class="pl-s">"./dist/compat.cjs"</span><span class="pl-kos">,</span></pre></div>
<p>This is complicated, <a href="https://stackoverflow.com/questions/42708484/what-is-the-module-package-json-field-for" rel="nofollow">see StackOverflow</a> - it’s a not-fully-accepted proposal which
is something of a de facto standard. It indicates the version of the package that
can be used as an ES Module. I don’t yet understand the full implications of this.</p>
<div class="highlight highlight-source-js"><pre>  <span class="pl-s">"module"</span>: <span class="pl-s">"./dist/compat.js"</span><span class="pl-kos">,</span>

  <span class="pl-c">// The unpkg CDN says “If you omit the file path (i.e. use a “bare” URL), unpkg will</span>
  <span class="pl-c">// serve the file specified by the unpkg field in package.json, or fall back to main.”</span>
  <span class="pl-c">// - this should be what happens when you hit https://unpkg.com/idb-keyval - but I</span>
  <span class="pl-c">// tried that just now and got an error saying “Cannot find "/dist/iife-compat.js"</span>
  <span class="pl-c">// in idb-keyval@6.1.0” so something isn’t quite working right here.</span>
  <span class="pl-c">//</span>
  <span class="pl-c">// The iife-compat.js filename here suggests that this file will use the Immediately</span>
  <span class="pl-c">// invoked function expression pattern, which looks like this:</span>
  <span class="pl-c">//</span>
  <span class="pl-c">// ​​(function () { /* ... */ })();</span>
  <span class="pl-c">//</span>
  <span class="pl-c">// It’s a pattern to avoid polluting the global namespace with variables and functions.</span>
  <span class="pl-s">"unpkg"</span>: <span class="pl-s">"./dist/iife-compat.js"</span><span class="pl-kos">,</span>

  <span class="pl-c">// This is a Node.js extension. The Node docs say: “The "exports" field provides an</span>
  <span class="pl-c">// alternative to "main" where the package main entry point can be defined while also</span>
  <span class="pl-c">// encapsulating the package, preventing any other entry points besides those defined</span>
  <span class="pl-c">// in "exports". This encapsulation allows module authors to define a public interface</span>
  <span class="pl-c">// for their package.”</span>
  <span class="pl-s">"exports"</span>: <span class="pl-kos">{</span>
    <span class="pl-c">// I don’t yet understand why this is a nested object</span>
    <span class="pl-s">"."</span>: <span class="pl-kos">{</span>
      <span class="pl-s">"module"</span>: <span class="pl-s">"./dist/index.js"</span><span class="pl-kos">,</span>
      <span class="pl-s">"import"</span>: <span class="pl-s">"./dist/index.js"</span><span class="pl-kos">,</span>
      <span class="pl-s">"default"</span>: <span class="pl-s">"./dist/index.cjs"</span>
    <span class="pl-kos">}</span><span class="pl-kos">,</span>
    <span class="pl-s">"./dist/*"</span>: <span class="pl-s">"./dist/*"</span><span class="pl-kos">,</span>
    <span class="pl-s">"./package.json"</span>: <span class="pl-s">"./package.json"</span>
  <span class="pl-kos">}</span><span class="pl-kos">,</span>

  <span class="pl-c">// These files are included when the package is installed as a dependency</span>
  <span class="pl-s">"files"</span>: <span class="pl-kos">[</span>
    <span class="pl-s">"dist/**"</span>
  <span class="pl-kos">]</span><span class="pl-kos">,</span>

  <span class="pl-c">// Another node.js extension: https://nodejs.org/api/packages.html#type says: Files</span>
  <span class="pl-c">// ending with .js are loaded as ES modules when the nearest parent package.json file</span>
  <span class="pl-c">// contains a top-level field "type" with a value of "module".</span>
  <span class="pl-s">"type"</span>: <span class="pl-s">"module"</span><span class="pl-kos">,</span>

  <span class="pl-c">// TypeScript extension indicating the location of type information for this module</span>
  <span class="pl-s">"types"</span>: <span class="pl-s">"./dist/index.d.ts"</span><span class="pl-kos">,</span>

  <span class="pl-c">// The webpack Tree Shaking docs say “The new webpack 4 release expands on this capability</span>
  <span class="pl-c">// with a way to provide hints to the compiler via the "sideEffects" package.json property</span>
  <span class="pl-c">// to denote which files in your project are "pure" and therefore safe to prune if unused”</span>
  <span class="pl-s">"sideEffects"</span>: <span class="pl-c1">false</span><span class="pl-kos">,</span>

  <span class="pl-c">// Running “npm run build” or “npm run dev” or “npm run prepack” runs these:</span>
  <span class="pl-s">"scripts"</span>: <span class="pl-kos">{</span>
    <span class="pl-s">"build"</span>: <span class="pl-s">"rollup -c &amp;&amp; node lib/size-report.js"</span><span class="pl-kos">,</span>
    <span class="pl-s">"dev"</span>: <span class="pl-s">"rollup -cw &amp; serve"</span><span class="pl-kos">,</span>
    <span class="pl-s">"prepack"</span>: <span class="pl-s">"npm run build"</span>
  <span class="pl-kos">}</span><span class="pl-kos">,</span>

  <span class="pl-c">// A bunch more metadata used when the package is shown on npm</span>
  <span class="pl-s">"repository"</span>: <span class="pl-kos">{</span>
    <span class="pl-s">"type"</span>: <span class="pl-s">"git"</span><span class="pl-kos">,</span>
    <span class="pl-s">"url"</span>: <span class="pl-s">"git+https://github.com/jakearchibald/idb-keyval.git"</span>
  <span class="pl-kos">}</span><span class="pl-kos">,</span>
  <span class="pl-s">"keywords"</span>: <span class="pl-kos">[</span>
    <span class="pl-s">"idb"</span><span class="pl-kos">,</span>
    <span class="pl-s">"indexeddb"</span><span class="pl-kos">,</span>
    <span class="pl-s">"store"</span><span class="pl-kos">,</span>
    <span class="pl-s">"keyval"</span><span class="pl-kos">,</span>
    <span class="pl-s">"localstorage"</span><span class="pl-kos">,</span>
    <span class="pl-s">"storage"</span><span class="pl-kos">,</span>
    <span class="pl-s">"promise"</span>
  <span class="pl-kos">]</span><span class="pl-kos">,</span>
  <span class="pl-s">"author"</span>: <span class="pl-s">"Jake Archibald"</span><span class="pl-kos">,</span>
  <span class="pl-s">"license"</span>: <span class="pl-s">"Apache-2.0"</span><span class="pl-kos">,</span>
  <span class="pl-s">"bugs"</span>: <span class="pl-kos">{</span>
    <span class="pl-s">"url"</span>: <span class="pl-s">"https://github.com/jakearchibald/idb-keyval/issues"</span>
  <span class="pl-kos">}</span><span class="pl-kos">,</span>
  <span class="pl-s">"homepage"</span>: <span class="pl-s">"https://github.com/jakearchibald/idb-keyval#readme"</span><span class="pl-kos">,</span>

  <span class="pl-c">// These are the dependencies that get installed when you run “npm install” from the</span>
  <span class="pl-c">// root of the package - but they are not installed when an end-user uses</span>
  <span class="pl-c">// “npm install name-of-package”</span>
  <span class="pl-s">"devDependencies"</span>: <span class="pl-kos">{</span>
    <span class="pl-s">"@babel/core"</span>: <span class="pl-s">"^7.16.0"</span><span class="pl-kos">,</span>
    <span class="pl-s">"@babel/plugin-external-helpers"</span>: <span class="pl-s">"^7.16.0"</span><span class="pl-kos">,</span>
    <span class="pl-s">"@babel/plugin-transform-runtime"</span>: <span class="pl-s">"^7.16.4"</span><span class="pl-kos">,</span>
    <span class="pl-s">"@babel/preset-env"</span>: <span class="pl-s">"^7.16.4"</span><span class="pl-kos">,</span>
    <span class="pl-s">"@babel/runtime"</span>: <span class="pl-s">"^7.16.3"</span><span class="pl-kos">,</span>
    <span class="pl-s">"@rollup/plugin-babel"</span>: <span class="pl-s">"^5.3.0"</span><span class="pl-kos">,</span>
    <span class="pl-s">"@rollup/plugin-commonjs"</span>: <span class="pl-s">"^21.0.1"</span><span class="pl-kos">,</span>
    <span class="pl-s">"@rollup/plugin-node-resolve"</span>: <span class="pl-s">"^13.0.6"</span><span class="pl-kos">,</span>
    <span class="pl-s">"@types/chai"</span>: <span class="pl-s">"^4.2.22"</span><span class="pl-kos">,</span>
    <span class="pl-s">"@types/mocha"</span>: <span class="pl-s">"^9.0.0"</span><span class="pl-kos">,</span>
    <span class="pl-s">"chai"</span>: <span class="pl-s">"^4.3.4"</span><span class="pl-kos">,</span>
    <span class="pl-s">"conditional-type-checks"</span>: <span class="pl-s">"^1.0.5"</span><span class="pl-kos">,</span>
    <span class="pl-s">"del"</span>: <span class="pl-s">"^6.0.0"</span><span class="pl-kos">,</span>
    <span class="pl-s">"filesize"</span>: <span class="pl-s">"^8.0.6"</span><span class="pl-kos">,</span>
    <span class="pl-s">"glob"</span>: <span class="pl-s">"^7.2.0"</span><span class="pl-kos">,</span>
    <span class="pl-s">"husky"</span>: <span class="pl-s">"^7.0.4"</span><span class="pl-kos">,</span>
    <span class="pl-s">"lint-staged"</span>: <span class="pl-s">"^12.1.2"</span><span class="pl-kos">,</span>
    <span class="pl-s">"mocha"</span>: <span class="pl-s">"^9.1.3"</span><span class="pl-kos">,</span>
    <span class="pl-s">"prettier"</span>: <span class="pl-s">"^2.5.0"</span><span class="pl-kos">,</span>
    <span class="pl-s">"rollup"</span>: <span class="pl-s">"^2.60.2"</span><span class="pl-kos">,</span>
    <span class="pl-s">"rollup-plugin-terser"</span>: <span class="pl-s">"^7.0.2"</span><span class="pl-kos">,</span>
    <span class="pl-s">"serve"</span>: <span class="pl-s">"^13.0.2"</span><span class="pl-kos">,</span>
    <span class="pl-s">"typescript"</span>: <span class="pl-s">"^4.3.5"</span>
  <span class="pl-kos">}</span><span class="pl-kos">,</span>

  <span class="pl-c">// Husky is a tool for managing git pre-commit hooks</span>
  <span class="pl-s">"husky"</span>: <span class="pl-kos">{</span>
    <span class="pl-s">"hooks"</span>: <span class="pl-kos">{</span>
      <span class="pl-s">"pre-commit"</span>: <span class="pl-s">"lint-staged"</span>
    <span class="pl-kos">}</span>
  <span class="pl-kos">}</span><span class="pl-kos">,</span>

  <span class="pl-c">// lint-staged runs linters against staged files - here it is called by Husky</span>
  <span class="pl-s">"lint-staged"</span>: <span class="pl-kos">{</span>
    <span class="pl-s">"*.{js,css,md,ts,html}"</span>: <span class="pl-s">"prettier --write"</span>
  <span class="pl-kos">}</span><span class="pl-kos">,</span>

  <span class="pl-c">// This package has only one non-dev dependency</span>
  <span class="pl-s">"dependencies"</span>: <span class="pl-kos">{</span>
    <span class="pl-c">// safari-14-idb-fix provides a idbReady().then(() =&gt; … function for waiting</span>
    <span class="pl-c">// until IDB is ready to be accessed in Safari, as a workaround for a bug that</span>
    <span class="pl-c">// existed prior to Safari 14.7 </span>
    <span class="pl-s">"safari-14-idb-fix"</span>: <span class="pl-s">"^3.0.0"</span>
  <span class="pl-kos">}</span>
<span class="pl-kos">}</span></pre></div>


