# til

path: sqlite_null-case.md
topic: sqlite
title: Null case comparisons in SQLite
url: https://github.com/simonw/til/blob/main/sqlite/null-case.md
created: 2020-04-21T09:43:38-07:00
created_utc: 2020-04-21T16:43:38+00:00
updated: 2020-04-22T06:36:01-07:00
updated_utc: 2020-04-22T13:36:01+00:00
shot_hash: fa2da46f9c7d31336020cbe23bc41f78
slug: null-case
body:
I wanted to say "output this transformed value if it's not null, otherwise nothing". The recipe I figured out was:

```sql
  case
    when (media_url_https is not null) then json_object('img_src', media_url_https, 'width', 300)
  end as photo
```

Full query example:

```sql
select
  created_at,
  regexp_match('.*?(\d+(\.\d+))lb.*', full_text, 1) as lbs,
  full_text,
  case
    when (media_url_https is not null) then json_object('img_src', media_url_https, 'width', 300)
  end as photo
from
  tweets
  left join media_tweets on tweets.id = media_tweets.tweets_id
  left join media on media.id = media_tweets.media_id
where
  full_text like '%lb%'
  and user = 3166449535
  and lbs is not null
group by
  tweets.id
order by
  created_at
```
This uses [datasette-rure](https://github.com/simonw/datasette-rure) for the `regexp_match()` function. Example output here: https://twitter.com/simonw/status/1249400425138155523

html:
<p>I wanted to say "output this transformed value if it's not null, otherwise nothing". The recipe I figured out was:</p>
<div class="highlight highlight-source-sql"><pre>  case
    when (media_url_https <span class="pl-k">is not null</span>) then json_object(<span class="pl-s"><span class="pl-pds">'</span>img_src<span class="pl-pds">'</span></span>, media_url_https, <span class="pl-s"><span class="pl-pds">'</span>width<span class="pl-pds">'</span></span>, <span class="pl-c1">300</span>)
  end <span class="pl-k">as</span> photo</pre></div>
<p>Full query example:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  created_at,
  regexp_match(<span class="pl-s"><span class="pl-pds">'</span>.*?(<span class="pl-cce">\d</span>+(<span class="pl-cce">\.\d</span>+))lb.*<span class="pl-pds">'</span></span>, full_text, <span class="pl-c1">1</span>) <span class="pl-k">as</span> lbs,
  full_text,
  case
    when (media_url_https <span class="pl-k">is not null</span>) then json_object(<span class="pl-s"><span class="pl-pds">'</span>img_src<span class="pl-pds">'</span></span>, media_url_https, <span class="pl-s"><span class="pl-pds">'</span>width<span class="pl-pds">'</span></span>, <span class="pl-c1">300</span>)
  end <span class="pl-k">as</span> photo
<span class="pl-k">from</span>
  tweets
  <span class="pl-k">left join</span> media_tweets <span class="pl-k">on</span> <span class="pl-c1">tweets</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">media_tweets</span>.<span class="pl-c1">tweets_id</span>
  <span class="pl-k">left join</span> media <span class="pl-k">on</span> <span class="pl-c1">media</span>.<span class="pl-c1">id</span> <span class="pl-k">=</span> <span class="pl-c1">media_tweets</span>.<span class="pl-c1">media_id</span>
<span class="pl-k">where</span>
  full_text <span class="pl-k">like</span> <span class="pl-s"><span class="pl-pds">'</span>%lb%<span class="pl-pds">'</span></span>
  <span class="pl-k">and</span> user <span class="pl-k">=</span> <span class="pl-c1">3166449535</span>
  <span class="pl-k">and</span> lbs <span class="pl-k">is not null</span>
<span class="pl-k">group by</span>
  <span class="pl-c1">tweets</span>.<span class="pl-c1">id</span>
<span class="pl-k">order by</span>
  created_at</pre></div>
<p>This uses <a href="https://github.com/simonw/datasette-rure">datasette-rure</a> for the <code>regexp_match()</code> function. Example output here: <a href="https://twitter.com/simonw/status/1249400425138155523" rel="nofollow">https://twitter.com/simonw/status/1249400425138155523</a></p>


path: sqlite_compile-sqlite3-ubuntu.md
topic: sqlite
title: Compile a new sqlite3 binary on Ubuntu
url: https://github.com/simonw/til/blob/main/sqlite/compile-sqlite3-ubuntu.md
created: 2020-04-30T10:40:20-07:00
created_utc: 2020-04-30T17:40:20+00:00
updated: 2020-09-02T13:33:03-07:00
updated_utc: 2020-09-02T20:33:03+00:00
shot_hash: f5f5f7c0a60ce37490c31bba23bcb603
slug: compile-sqlite3-ubuntu
body:
I wanted to try the `vacuum into` backup command that was released in SQLite3 3.27.0 [on 2019-02-07](https://www.sqlite.org/changes.html#version_3_27_0).

Ubuntu 18.04.4 LTS has SQLite 3.22.0 from 2018-01-22.

Thankfully it's really easy to compile a new `sqlite3` binary on Ubuntu, using the amalgamation release.

```
cd /tmp
wget https://www.sqlite.org/2020/sqlite-amalgamation-3310100.zip
unzip sqlite-amalgamation-3310100.zip
cd sqlite-amalgamation-3310100
gcc shell.c sqlite3.c -lpthread -ldl
./a.out --version
3.31.1 2020-01-27 19:55:54 3bfa9cc97da10598521b342961df8f5f68c7388fa117345eeb516eaa837bb4d6
```
Now you can move the `a.out` file somewhere else:
```
mkdir ~/bin
mv ./a.out ~/bin/sqlite3
```
I used the backup command like this:
```
/home/simon/bin/sqlite3 data.db "vacuum into '/tmp/backup.db'"
```

html:
<p>I wanted to try the <code>vacuum into</code> backup command that was released in SQLite3 3.27.0 <a href="https://www.sqlite.org/changes.html#version_3_27_0" rel="nofollow">on 2019-02-07</a>.</p>
<p>Ubuntu 18.04.4 LTS has SQLite 3.22.0 from 2018-01-22.</p>
<p>Thankfully it's really easy to compile a new <code>sqlite3</code> binary on Ubuntu, using the amalgamation release.</p>
<pre><code>cd /tmp
wget https://www.sqlite.org/2020/sqlite-amalgamation-3310100.zip
unzip sqlite-amalgamation-3310100.zip
cd sqlite-amalgamation-3310100
gcc shell.c sqlite3.c -lpthread -ldl
./a.out --version
3.31.1 2020-01-27 19:55:54 3bfa9cc97da10598521b342961df8f5f68c7388fa117345eeb516eaa837bb4d6
</code></pre>
<p>Now you can move the <code>a.out</code> file somewhere else:</p>
<pre><code>mkdir ~/bin
mv ./a.out ~/bin/sqlite3
</code></pre>
<p>I used the backup command like this:</p>
<pre><code>/home/simon/bin/sqlite3 data.db "vacuum into '/tmp/backup.db'"
</code></pre>


path: sqlite_enabling-wal-mode.md
topic: sqlite
title: Enabling WAL mode for SQLite database files
url: https://github.com/simonw/til/blob/main/sqlite/enabling-wal-mode.md
created: 2020-08-09T20:23:23-07:00
created_utc: 2020-08-10T03:23:23+00:00
updated: 2022-08-24T09:19:52-07:00
updated_utc: 2022-08-24T16:19:52+00:00
shot_hash: 327bc55f18a5c4fc0d8fcf59c7ad3987
slug: enabling-wal-mode
body:
I was getting occasional `Error: database is locked` messages from a Datasette instance that was running against a bunch of different SQLite files that were updated by cron scripts (my personal [Dogsheep](https://dogsheep.github.io/)).

I had read about SQLite's WAL mode but never fully understood how it works. I asked [some clarifying questions](https://sqlite.org/forum/forumpost/e6c238e854) on the SQLite forum and learned that WAL is actually a property of the database file itself, not of the connection to that database.

This means that turning on WAL is a thing you can do directly to a database file!

Here's the incantation:

    sqlite3 github.db 'PRAGMA journal_mode=WAL;'

I ran this against all of the `*.db` files in a directory like this:

    ls *.db | while read filename;
      do sqlite3 $filename 'PRAGMA journal_mode=WAL;';
    done;

The first time I ran this it worked on all but one file, which showed the `Error: database is locked` message - so I kept trying against that file until it worked.

After running this each `.db` file has an accompanying `.db-shm` and `.db-wal` file. So far I've not seen the "database is locked" message, so I think it had the desired effect.

## Turning WAL mode off again

If you want to turn WAL mode off and go back to the SQLite default, the unintuitive way to do that is:

    PRAGMA journal_mode=delete;

## Using sqlite-utils

I added a command to [sqlite-utils 2.15](https://sqlite-utils.datasette.io/en/stable/changelog.html#v2-15) that does this:

    sqlite-utils enable-wal *.db

The `disable-Wal` command disables it again.

## Futher notes

Ben Johnson wrote about how WAL mode internals work in great detail in [How SQLite Scales Read Concurrency ](https://fly.io/blog/sqlite-internals-wal/).

In a Hacker News comment, Ben [points out](https://news.ycombinator.com/item?id=32581486):

>  The other odd thing is that the journal_mode is only persistent for WAL, I believe. The DELETE, TRUNCATE, & PERSIST modes are per-connection. It makes sense though since those 3 modes deal with the rollback journal and are compatible with each other while the WAL is totally separate. https://www.sqlite.org/pragma.html#pragma_journal_mode

html:
<p>I was getting occasional <code>Error: database is locked</code> messages from a Datasette instance that was running against a bunch of different SQLite files that were updated by cron scripts (my personal <a href="https://dogsheep.github.io/" rel="nofollow">Dogsheep</a>).</p>
<p>I had read about SQLite's WAL mode but never fully understood how it works. I asked <a href="https://sqlite.org/forum/forumpost/e6c238e854" rel="nofollow">some clarifying questions</a> on the SQLite forum and learned that WAL is actually a property of the database file itself, not of the connection to that database.</p>
<p>This means that turning on WAL is a thing you can do directly to a database file!</p>
<p>Here's the incantation:</p>
<pre><code>sqlite3 github.db 'PRAGMA journal_mode=WAL;'
</code></pre>
<p>I ran this against all of the <code>*.db</code> files in a directory like this:</p>
<pre><code>ls *.db | while read filename;
  do sqlite3 $filename 'PRAGMA journal_mode=WAL;';
done;
</code></pre>
<p>The first time I ran this it worked on all but one file, which showed the <code>Error: database is locked</code> message - so I kept trying against that file until it worked.</p>
<p>After running this each <code>.db</code> file has an accompanying <code>.db-shm</code> and <code>.db-wal</code> file. So far I've not seen the "database is locked" message, so I think it had the desired effect.</p>
<h2><a id="user-content-turning-wal-mode-off-again" class="anchor" aria-hidden="true" href="#turning-wal-mode-off-again"><span aria-hidden="true" class="octicon octicon-link"></span></a>Turning WAL mode off again</h2>
<p>If you want to turn WAL mode off and go back to the SQLite default, the unintuitive way to do that is:</p>
<pre><code>PRAGMA journal_mode=delete;
</code></pre>
<h2><a id="user-content-using-sqlite-utils" class="anchor" aria-hidden="true" href="#using-sqlite-utils"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using sqlite-utils</h2>
<p>I added a command to <a href="https://sqlite-utils.datasette.io/en/stable/changelog.html#v2-15" rel="nofollow">sqlite-utils 2.15</a> that does this:</p>
<pre><code>sqlite-utils enable-wal *.db
</code></pre>
<p>The <code>disable-Wal</code> command disables it again.</p>
<h2><a id="user-content-futher-notes" class="anchor" aria-hidden="true" href="#futher-notes"><span aria-hidden="true" class="octicon octicon-link"></span></a>Futher notes</h2>
<p>Ben Johnson wrote about how WAL mode internals work in great detail in <a href="https://fly.io/blog/sqlite-internals-wal/" rel="nofollow">How SQLite Scales Read Concurrency </a>.</p>
<p>In a Hacker News comment, Ben <a href="https://news.ycombinator.com/item?id=32581486" rel="nofollow">points out</a>:</p>
<blockquote>
<p>The other odd thing is that the journal_mode is only persistent for WAL, I believe. The DELETE, TRUNCATE, &amp; PERSIST modes are per-connection. It makes sense though since those 3 modes deal with the rollback journal and are compatible with each other while the WAL is totally separate. <a href="https://www.sqlite.org/pragma.html#pragma_journal_mode" rel="nofollow">https://www.sqlite.org/pragma.html#pragma_journal_mode</a></p>
</blockquote>


path: sqlite_list-all-columns-in-a-database.md
topic: sqlite
title: List all columns in a SQLite database
url: https://github.com/simonw/til/blob/main/sqlite/list-all-columns-in-a-database.md
created: 2020-05-06T08:58:14-07:00
created_utc: 2020-05-06T15:58:14+00:00
updated: 2020-12-17T17:45:24-08:00
updated_utc: 2020-12-18T01:45:24+00:00
shot_hash: 564b94b476390aad13273b6d07ebb3cc
slug: list-all-columns-in-a-database
body:
Here's a devious trick for listing ALL columns in a SQLite database, using a SQL query that generates another SQL query.

The first query ([demo](https://latest.datasette.io/fixtures?sql=select+group_concat%28%0D%0A++%22select+%27%22+%7C%7C+name+%7C%7C+%22%27+as+table_name%2C+*+from+pragma_table_info%28%27%22+%7C%7C+name+%7C%7C+%22%27%29%22%0D%0A%2C+%27+union+%27%29+%7C%7C+%27+order+by+table_name%2C+cid%27%0D%0A++from+sqlite_master+where+type+%3D+%27table%27%3B)):

```sql
select group_concat(
  "select '" || name || "' as table_name, * from pragma_table_info('" || name || "')"
, ' union ') || ' order by table_name, cid'
  from sqlite_master where type = 'table';
```
This outputs the second query, which will look something like this ([demo](https://latest.datasette.io/fixtures?sql=select+%27simple_primary_key%27+as+table_name%2C+*+from+pragma_table_info%28%27simple_primary_key%27%29+union+select+%27primary_key_multiple_columns%27+as+table_name%2C+*+from+pragma_table_info%28%27primary_key_multiple_columns%27%29+union+select+%27primary_key_multiple_columns_explicit_label%27+as+table_name%2C+*+from+pragma_table_info%28%27primary_key_multiple_columns_explicit_label%27%29+union+select+%27compound_primary_key%27+as+table_name%2C+*+from+pragma_table_info%28%27compound_primary_key%27%29+union+select+%27compound_three_primary_keys%27+as+table_name%2C+*+from+pragma_table_info%28%27compound_three_primary_keys%27%29+union+select+%27foreign_key_references%27+as+table_name%2C+*+from+pragma_table_info%28%27foreign_key_references%27%29+union+select+%27sortable%27+as+table_name%2C+*+from+pragma_table_info%28%27sortable%27%29+union+select+%27no_primary_key%27+as+table_name%2C+*+from+pragma_table_info%28%27no_primary_key%27%29+union+select+%27123_starts_with_digits%27+as+table_name%2C+*+from+pragma_table_info%28%27123_starts_with_digits%27%29+union+select+%27Table+With+Space+In+Name%27+as+table_name%2C+*+from+pragma_table_info%28%27Table+With+Space+In+Name%27%29+union+select+%27table%2Fwith%2Fslashes.csv%27+as+table_name%2C+*+from+pragma_table_info%28%27table%2Fwith%2Fslashes.csv%27%29+union+select+%27complex_foreign_keys%27+as+table_name%2C+*+from+pragma_table_info%28%27complex_foreign_keys%27%29+union+select+%27custom_foreign_key_label%27+as+table_name%2C+*+from+pragma_table_info%28%27custom_foreign_key_label%27%29+union+select+%27units%27+as+table_name%2C+*+from+pragma_table_info%28%27units%27%29+union+select+%27tags%27+as+table_name%2C+*+from+pragma_table_info%28%27tags%27%29+union+select+%27searchable%27+as+table_name%2C+*+from+pragma_table_info%28%27searchable%27%29+union+select+%27searchable_tags%27+as+table_name%2C+*+from+pragma_table_info%28%27searchable_tags%27%29+union+select+%27searchable_fts%27+as+table_name%2C+*+from+pragma_table_info%28%27searchable_fts%27%29+union+select+%27searchable_fts_content%27+as+table_name%2C+*+from+pragma_table_info%28%27searchable_fts_content%27%29+union+select+%27searchable_fts_segments%27+as+table_name%2C+*+from+pragma_table_info%28%27searchable_fts_segments%27%29+union+select+%27searchable_fts_segdir%27+as+table_name%2C+*+from+pragma_table_info%28%27searchable_fts_segdir%27%29+union+select+%27select%27+as+table_name%2C+*+from+pragma_table_info%28%27select%27%29+union+select+%27infinity%27+as+table_name%2C+*+from+pragma_table_info%28%27infinity%27%29+union+select+%27facet_cities%27+as+table_name%2C+*+from+pragma_table_info%28%27facet_cities%27%29+union+select+%27facetable%27+as+table_name%2C+*+from+pragma_table_info%28%27facetable%27%29+union+select+%27binary_data%27+as+table_name%2C+*+from+pragma_table_info%28%27binary_data%27%29+union+select+%27roadside_attractions%27+as+table_name%2C+*+from+pragma_table_info%28%27roadside_attractions%27%29+union+select+%27attraction_characteristic%27+as+table_name%2C+*+from+pragma_table_info%28%27attraction_characteristic%27%29+union+select+%27roadside_attraction_characteristics%27+as+table_name%2C+*+from+pragma_table_info%28%27roadside_attraction_characteristics%27%29+order+by+table_name%2C+cid)):
```sql
select 'simple_primary_key' as table_name, * from pragma_table_info('simple_primary_key') union
select 'primary_key_multiple_columns' as table_name, * from pragma_table_info('primary_key_multiple_columns') union
select 'primary_key_multiple_columns_explicit_label' as table_name, * from pragma_table_info('primary_key_multiple_columns_explicit_label') union
select 'compound_primary_key' as table_name, * from pragma_table_info('compound_primary_key') union
select 'compound_three_primary_keys' as table_name, * from pragma_table_info('compound_three_primary_keys') union
select 'foreign_key_references' as table_name, * from pragma_table_info('foreign_key_references') union
select 'sortable' as table_name, * from pragma_table_info('sortable') union
select 'no_primary_key' as table_name, * from pragma_table_info('no_primary_key') union
select '123_starts_with_digits' as table_name, * from pragma_table_info('123_starts_with_digits') union
select 'Table With Space In Name' as table_name, * from pragma_table_info('Table With Space In Name') union
select 'table/with/slashes.csv' as table_name, * from pragma_table_info('table/with/slashes.csv') union
select 'complex_foreign_keys' as table_name, * from pragma_table_info('complex_foreign_keys') union
select 'custom_foreign_key_label' as table_name, * from pragma_table_info('custom_foreign_key_label') union
select 'units' as table_name, * from pragma_table_info('units') union
select 'tags' as table_name, * from pragma_table_info('tags') union
select 'searchable' as table_name, * from pragma_table_info('searchable') union
select 'searchable_tags' as table_name, * from pragma_table_info('searchable_tags') union
select 'searchable_fts' as table_name, * from pragma_table_info('searchable_fts') union
select 'searchable_fts_content' as table_name, * from pragma_table_info('searchable_fts_content') union
select 'searchable_fts_segments' as table_name, * from pragma_table_info('searchable_fts_segments') union
select 'searchable_fts_segdir' as table_name, * from pragma_table_info('searchable_fts_segdir') union
select 'select' as table_name, * from pragma_table_info('select') union
select 'infinity' as table_name, * from pragma_table_info('infinity') union
select 'facet_cities' as table_name, * from pragma_table_info('facet_cities') union
select 'facetable' as table_name, * from pragma_table_info('facetable') union
select 'binary_data' as table_name, * from pragma_table_info('binary_data') union
select 'roadside_attractions' as table_name, * from pragma_table_info('roadside_attractions') union
select 'attraction_characteristic' as table_name, * from pragma_table_info('attraction_characteristic') union
select 'roadside_attraction_characteristics' as table_name, * from pragma_table_info('roadside_attraction_characteristics')
order by table_name, cid
```
Executing that second query will return results like this:
|table_name|cid|name|type|notnull|dflt_value|pk|
|--- |--- |--- |--- |--- |--- |--- |
|123_starts_with_digits|0|content|text|0||0|
|Table With Space In Name|0|pk|varchar(30)|0||1|
|Table With Space In Name|1|content|text|0||0|
|attraction_characteristic|0|pk|integer|0||1|
|attraction_characteristic|1|name|text|0||0|
|binary_data|0|data|BLOB|0||0|
|complex_foreign_keys|0|pk|varchar(30)|0||1|
|complex_foreign_keys|1|f1|text|0||0|
|complex_foreign_keys|2|f2|text|0||0|
|complex_foreign_keys|3|f3|text|0||0|
|compound_primary_key|0|pk1|varchar(30)|0||1|
|compound_primary_key|1|pk2|varchar(30)|0||2|
|compound_primary_key|2|content|text|0||0|
|compound_three_primary_keys|0|pk1|varchar(30)|0||1|
|compound_three_primary_keys|1|pk2|varchar(30)|0||2|
|compound_three_primary_keys|2|pk3|varchar(30)|0||3|
|compound_three_primary_keys|3|content|text|0||0|
|custom_foreign_key_label|0|pk|varchar(30)|0||1|
|custom_foreign_key_label|1|foreign_key_with_custom_label|text|0||0|
|facet_cities|0|id|integer|0||1|
|facet_cities|1|name|text|0||0|
|facetable|0|pk|integer|0||1|
|facetable|1|created|text|0||0|
|facetable|2|planet_int|integer|0||0|
|facetable|3|on_earth|integer|0||0|
|facetable|4|state|text|0||0|
|facetable|5|city_id|integer|0||0|
|facetable|6|neighborhood|text|0||0|
|facetable|7|tags|text|0||0|
|facetable|8|complex_array|text|0||0|
|facetable|9|distinct_some_null||0||0|
|foreign_key_references|0|pk|varchar(30)|0||1|
|foreign_key_references|1|foreign_key_with_label|varchar(30)|0||0|
|foreign_key_references|2|foreign_key_with_no_label|varchar(30)|0||0|
|infinity|0|value|REAL|0||0|
|no_primary_key|0|content|text|0||0|
|no_primary_key|1|a|text|0||0|
|no_primary_key|2|b|text|0||0|
|no_primary_key|3|c|text|0||0|
|primary_key_multiple_columns|0|id|varchar(30)|0||1|
|primary_key_multiple_columns|1|content|text|0||0|
|primary_key_multiple_columns|2|content2|text|0||0|
|primary_key_multiple_columns_explicit_label|0|id|varchar(30)|0||1|
|primary_key_multiple_columns_explicit_label|1|content|text|0||0|
|primary_key_multiple_columns_explicit_label|2|content2|text|0||0|
|roadside_attraction_characteristics|0|attraction_id|INTEGER|0||0|
|roadside_attraction_characteristics|1|characteristic_id|INTEGER|0||0|
|roadside_attractions|0|pk|integer|0||1|
|roadside_attractions|1|name|text|0||0|
|roadside_attractions|2|address|text|0||0|
|roadside_attractions|3|latitude|real|0||0|
|roadside_attractions|4|longitude|real|0||0|
|searchable|0|pk|integer|0||1|
|searchable|1|text1|text|0||0|
|searchable|2|text2|text|0||0|
|searchable|3|name with . and spaces|text|0||0|
|searchable_fts|0|text1||0||0|
|searchable_fts|1|text2||0||0|
|searchable_fts|2|name with . and spaces||0||0|
|searchable_fts|3|content||0||0|
|searchable_fts_content|0|docid|INTEGER|0||1|
|searchable_fts_content|1|c0text1||0||0|
|searchable_fts_content|2|c1text2||0||0|
|searchable_fts_content|3|c2name with . and spaces||0||0|
|searchable_fts_content|4|c3content||0||0|
|searchable_fts_segdir|0|level|INTEGER|0||1|
|searchable_fts_segdir|1|idx|INTEGER|0||2|
|searchable_fts_segdir|2|start_block|INTEGER|0||0|
|searchable_fts_segdir|3|leaves_end_block|INTEGER|0||0|
|searchable_fts_segdir|4|end_block|INTEGER|0||0|
|searchable_fts_segdir|5|root|BLOB|0||0|
|searchable_fts_segments|0|blockid|INTEGER|0||1|
|searchable_fts_segments|1|block|BLOB|0||0|
|searchable_tags|0|searchable_id|integer|0||1|
|searchable_tags|1|tag|text|0||2|
|select|0|group|text|0||0|
|select|1|having|text|0||0|
|select|2|and|text|0||0|
|select|3|json|text|0||0|
|simple_primary_key|0|id|varchar(30)|0||1|
|simple_primary_key|1|content|text|0||0|
|sortable|0|pk1|varchar(30)|0||1|
|sortable|1|pk2|varchar(30)|0||2|
|sortable|2|content|text|0||0|
|sortable|3|sortable|integer|0||0|
|sortable|4|sortable_with_nulls|real|0||0|
|sortable|5|sortable_with_nulls_2|real|0||0|
|sortable|6|text|text|0||0|
|table/with/slashes.csv|0|pk|varchar(30)|0||1|
|table/with/slashes.csv|1|content|text|0||0|
|tags|0|tag|TEXT|0||1|
|units|0|pk|integer|0||1|
|units|1|distance|int|0||0|
|units|2|frequency|int|0||0|

(I generated that Markdown table by pasting the HTML from Datasette into this tool: https://jmalarcon.github.io/markdowntables/)

## Better alternative using a join

```sql
select
  sqlite_master.name as table_name,
  table_info.*
from
  sqlite_master
  join pragma_table_info(sqlite_master.name) as table_info
order by
  sqlite_master.name
```
[Demo](https://latest.datasette.io/fixtures?sql=select%0D%0A++sqlite_master.name+as+table_name%2C%0D%0A++table_info.*%0D%0Afrom%0D%0A++sqlite_master%0D%0A++join+pragma_table_info%28sqlite_master.name%29+as+table_info%0D%0Aorder+by%0D%0A++sqlite_master.name%2C%0D%0A++table_info.cid).

This works with the `pragma_table_info` and `pragma_index_list` and `pragma_foreign_key_list` functions too.

## Another recipe

[Amjith pointed](https://twitter.com/amjithr/status/1258576704164909056) to [this query](https://github.com/dbcli/litecli/blob/829220b1e2c3fea84d7c3f0ea8f791f3c28e6230/litecli/sqlexecute.py#L33-L39) used in litecli:

```sql
SELECT
  m.name as tableName,
  p.name as columnName
FROM
  sqlite_master m
  LEFT OUTER JOIN pragma_table_info((m.name)) p ON m.name <> p.name
WHERE
  m.type IN ('table', 'view')
  AND m.name NOT LIKE 'sqlite_%'
ORDER BY
  tableName,
  columnName
```
[Demo](https://latest.datasette.io/fixtures?sql=SELECT+m.name+as+tableName%2C+p.name+as+columnName%0D%0A++++++++FROM+sqlite_master+m%0D%0A++++++++LEFT+OUTER+JOIN+pragma_table_info%28%28m.name%29%29+p+ON+m.name+%3C%3E+p.name%0D%0A++++++++WHERE+m.type+IN+%28%27table%27%2C%27view%27%29+AND+m.name+NOT+LIKE+%27sqlite_%25%27%0D%0A++++++++ORDER+BY+tableName%2C+columnName).

html:
<p>Here's a devious trick for listing ALL columns in a SQLite database, using a SQL query that generates another SQL query.</p>
<p>The first query (<a href="https://latest.datasette.io/fixtures?sql=select+group_concat%28%0D%0A++%22select+%27%22+%7C%7C+name+%7C%7C+%22%27+as+table_name%2C+*+from+pragma_table_info%28%27%22+%7C%7C+name+%7C%7C+%22%27%29%22%0D%0A%2C+%27+union+%27%29+%7C%7C+%27+order+by+table_name%2C+cid%27%0D%0A++from+sqlite_master+where+type+%3D+%27table%27%3B" rel="nofollow">demo</a>):</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> group_concat(
  <span class="pl-s"><span class="pl-pds">"</span>select '<span class="pl-pds">"</span></span> <span class="pl-k">||</span> name <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">"</span>' as table_name, * from pragma_table_info('<span class="pl-pds">"</span></span> <span class="pl-k">||</span> name <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">"</span>')<span class="pl-pds">"</span></span>
, <span class="pl-s"><span class="pl-pds">'</span> union <span class="pl-pds">'</span></span>) <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span> order by table_name, cid<span class="pl-pds">'</span></span>
  <span class="pl-k">from</span> sqlite_master <span class="pl-k">where</span> type <span class="pl-k">=</span> <span class="pl-s"><span class="pl-pds">'</span>table<span class="pl-pds">'</span></span>;</pre></div>
<p>This outputs the second query, which will look something like this (<a href="https://latest.datasette.io/fixtures?sql=select+%27simple_primary_key%27+as+table_name%2C+*+from+pragma_table_info%28%27simple_primary_key%27%29+union+select+%27primary_key_multiple_columns%27+as+table_name%2C+*+from+pragma_table_info%28%27primary_key_multiple_columns%27%29+union+select+%27primary_key_multiple_columns_explicit_label%27+as+table_name%2C+*+from+pragma_table_info%28%27primary_key_multiple_columns_explicit_label%27%29+union+select+%27compound_primary_key%27+as+table_name%2C+*+from+pragma_table_info%28%27compound_primary_key%27%29+union+select+%27compound_three_primary_keys%27+as+table_name%2C+*+from+pragma_table_info%28%27compound_three_primary_keys%27%29+union+select+%27foreign_key_references%27+as+table_name%2C+*+from+pragma_table_info%28%27foreign_key_references%27%29+union+select+%27sortable%27+as+table_name%2C+*+from+pragma_table_info%28%27sortable%27%29+union+select+%27no_primary_key%27+as+table_name%2C+*+from+pragma_table_info%28%27no_primary_key%27%29+union+select+%27123_starts_with_digits%27+as+table_name%2C+*+from+pragma_table_info%28%27123_starts_with_digits%27%29+union+select+%27Table+With+Space+In+Name%27+as+table_name%2C+*+from+pragma_table_info%28%27Table+With+Space+In+Name%27%29+union+select+%27table%2Fwith%2Fslashes.csv%27+as+table_name%2C+*+from+pragma_table_info%28%27table%2Fwith%2Fslashes.csv%27%29+union+select+%27complex_foreign_keys%27+as+table_name%2C+*+from+pragma_table_info%28%27complex_foreign_keys%27%29+union+select+%27custom_foreign_key_label%27+as+table_name%2C+*+from+pragma_table_info%28%27custom_foreign_key_label%27%29+union+select+%27units%27+as+table_name%2C+*+from+pragma_table_info%28%27units%27%29+union+select+%27tags%27+as+table_name%2C+*+from+pragma_table_info%28%27tags%27%29+union+select+%27searchable%27+as+table_name%2C+*+from+pragma_table_info%28%27searchable%27%29+union+select+%27searchable_tags%27+as+table_name%2C+*+from+pragma_table_info%28%27searchable_tags%27%29+union+select+%27searchable_fts%27+as+table_name%2C+*+from+pragma_table_info%28%27searchable_fts%27%29+union+select+%27searchable_fts_content%27+as+table_name%2C+*+from+pragma_table_info%28%27searchable_fts_content%27%29+union+select+%27searchable_fts_segments%27+as+table_name%2C+*+from+pragma_table_info%28%27searchable_fts_segments%27%29+union+select+%27searchable_fts_segdir%27+as+table_name%2C+*+from+pragma_table_info%28%27searchable_fts_segdir%27%29+union+select+%27select%27+as+table_name%2C+*+from+pragma_table_info%28%27select%27%29+union+select+%27infinity%27+as+table_name%2C+*+from+pragma_table_info%28%27infinity%27%29+union+select+%27facet_cities%27+as+table_name%2C+*+from+pragma_table_info%28%27facet_cities%27%29+union+select+%27facetable%27+as+table_name%2C+*+from+pragma_table_info%28%27facetable%27%29+union+select+%27binary_data%27+as+table_name%2C+*+from+pragma_table_info%28%27binary_data%27%29+union+select+%27roadside_attractions%27+as+table_name%2C+*+from+pragma_table_info%28%27roadside_attractions%27%29+union+select+%27attraction_characteristic%27+as+table_name%2C+*+from+pragma_table_info%28%27attraction_characteristic%27%29+union+select+%27roadside_attraction_characteristics%27+as+table_name%2C+*+from+pragma_table_info%28%27roadside_attraction_characteristics%27%29+order+by+table_name%2C+cid" rel="nofollow">demo</a>):</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>simple_primary_key<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>simple_primary_key<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>primary_key_multiple_columns<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>primary_key_multiple_columns<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>primary_key_multiple_columns_explicit_label<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>primary_key_multiple_columns_explicit_label<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>compound_primary_key<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>compound_primary_key<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>compound_three_primary_keys<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>compound_three_primary_keys<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>foreign_key_references<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>foreign_key_references<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>sortable<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>sortable<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>no_primary_key<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>no_primary_key<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>123_starts_with_digits<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>123_starts_with_digits<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>Table With Space In Name<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>Table With Space In Name<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>table/with/slashes.csv<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>table/with/slashes.csv<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>complex_foreign_keys<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>complex_foreign_keys<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>custom_foreign_key_label<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>custom_foreign_key_label<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>units<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>units<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>tags<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>tags<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>searchable<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>searchable<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>searchable_tags<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>searchable_tags<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>searchable_fts<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>searchable_fts<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>searchable_fts_content<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>searchable_fts_content<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>searchable_fts_segments<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>searchable_fts_segments<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>searchable_fts_segdir<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>searchable_fts_segdir<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>select<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>select<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>infinity<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>infinity<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>facet_cities<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>facet_cities<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>facetable<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>facetable<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>binary_data<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>binary_data<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>roadside_attractions<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>roadside_attractions<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>attraction_characteristic<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>attraction_characteristic<span class="pl-pds">'</span></span>) <span class="pl-k">union</span>
<span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>roadside_attraction_characteristics<span class="pl-pds">'</span></span> <span class="pl-k">as</span> table_name, <span class="pl-k">*</span> <span class="pl-k">from</span> pragma_table_info(<span class="pl-s"><span class="pl-pds">'</span>roadside_attraction_characteristics<span class="pl-pds">'</span></span>)
<span class="pl-k">order by</span> table_name, cid</pre></div>
<p>Executing that second query will return results like this:</p>
<table>
<thead>
<tr>
<th>table_name</th>
<th>cid</th>
<th>name</th>
<th>type</th>
<th>notnull</th>
<th>dflt_value</th>
<th>pk</th>
</tr>
</thead>
<tbody>
<tr>
<td>123_starts_with_digits</td>
<td>0</td>
<td>content</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>Table With Space In Name</td>
<td>0</td>
<td>pk</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>Table With Space In Name</td>
<td>1</td>
<td>content</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>attraction_characteristic</td>
<td>0</td>
<td>pk</td>
<td>integer</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>attraction_characteristic</td>
<td>1</td>
<td>name</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>binary_data</td>
<td>0</td>
<td>data</td>
<td>BLOB</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>complex_foreign_keys</td>
<td>0</td>
<td>pk</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>complex_foreign_keys</td>
<td>1</td>
<td>f1</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>complex_foreign_keys</td>
<td>2</td>
<td>f2</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>complex_foreign_keys</td>
<td>3</td>
<td>f3</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>compound_primary_key</td>
<td>0</td>
<td>pk1</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>compound_primary_key</td>
<td>1</td>
<td>pk2</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>2</td>
</tr>
<tr>
<td>compound_primary_key</td>
<td>2</td>
<td>content</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>compound_three_primary_keys</td>
<td>0</td>
<td>pk1</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>compound_three_primary_keys</td>
<td>1</td>
<td>pk2</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>2</td>
</tr>
<tr>
<td>compound_three_primary_keys</td>
<td>2</td>
<td>pk3</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>3</td>
</tr>
<tr>
<td>compound_three_primary_keys</td>
<td>3</td>
<td>content</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>custom_foreign_key_label</td>
<td>0</td>
<td>pk</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>custom_foreign_key_label</td>
<td>1</td>
<td>foreign_key_with_custom_label</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>facet_cities</td>
<td>0</td>
<td>id</td>
<td>integer</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>facet_cities</td>
<td>1</td>
<td>name</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>facetable</td>
<td>0</td>
<td>pk</td>
<td>integer</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>facetable</td>
<td>1</td>
<td>created</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>facetable</td>
<td>2</td>
<td>planet_int</td>
<td>integer</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>facetable</td>
<td>3</td>
<td>on_earth</td>
<td>integer</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>facetable</td>
<td>4</td>
<td>state</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>facetable</td>
<td>5</td>
<td>city_id</td>
<td>integer</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>facetable</td>
<td>6</td>
<td>neighborhood</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>facetable</td>
<td>7</td>
<td>tags</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>facetable</td>
<td>8</td>
<td>complex_array</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>facetable</td>
<td>9</td>
<td>distinct_some_null</td>
<td></td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>foreign_key_references</td>
<td>0</td>
<td>pk</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>foreign_key_references</td>
<td>1</td>
<td>foreign_key_with_label</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>foreign_key_references</td>
<td>2</td>
<td>foreign_key_with_no_label</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>infinity</td>
<td>0</td>
<td>value</td>
<td>REAL</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>no_primary_key</td>
<td>0</td>
<td>content</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>no_primary_key</td>
<td>1</td>
<td>a</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>no_primary_key</td>
<td>2</td>
<td>b</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>no_primary_key</td>
<td>3</td>
<td>c</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>primary_key_multiple_columns</td>
<td>0</td>
<td>id</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>primary_key_multiple_columns</td>
<td>1</td>
<td>content</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>primary_key_multiple_columns</td>
<td>2</td>
<td>content2</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>primary_key_multiple_columns_explicit_label</td>
<td>0</td>
<td>id</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>primary_key_multiple_columns_explicit_label</td>
<td>1</td>
<td>content</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>primary_key_multiple_columns_explicit_label</td>
<td>2</td>
<td>content2</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>roadside_attraction_characteristics</td>
<td>0</td>
<td>attraction_id</td>
<td>INTEGER</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>roadside_attraction_characteristics</td>
<td>1</td>
<td>characteristic_id</td>
<td>INTEGER</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>roadside_attractions</td>
<td>0</td>
<td>pk</td>
<td>integer</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>roadside_attractions</td>
<td>1</td>
<td>name</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>roadside_attractions</td>
<td>2</td>
<td>address</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>roadside_attractions</td>
<td>3</td>
<td>latitude</td>
<td>real</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>roadside_attractions</td>
<td>4</td>
<td>longitude</td>
<td>real</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable</td>
<td>0</td>
<td>pk</td>
<td>integer</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>searchable</td>
<td>1</td>
<td>text1</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable</td>
<td>2</td>
<td>text2</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable</td>
<td>3</td>
<td>name with . and spaces</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_fts</td>
<td>0</td>
<td>text1</td>
<td></td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_fts</td>
<td>1</td>
<td>text2</td>
<td></td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_fts</td>
<td>2</td>
<td>name with . and spaces</td>
<td></td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_fts</td>
<td>3</td>
<td>content</td>
<td></td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_fts_content</td>
<td>0</td>
<td>docid</td>
<td>INTEGER</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>searchable_fts_content</td>
<td>1</td>
<td>c0text1</td>
<td></td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_fts_content</td>
<td>2</td>
<td>c1text2</td>
<td></td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_fts_content</td>
<td>3</td>
<td>c2name with . and spaces</td>
<td></td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_fts_content</td>
<td>4</td>
<td>c3content</td>
<td></td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_fts_segdir</td>
<td>0</td>
<td>level</td>
<td>INTEGER</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>searchable_fts_segdir</td>
<td>1</td>
<td>idx</td>
<td>INTEGER</td>
<td>0</td>
<td></td>
<td>2</td>
</tr>
<tr>
<td>searchable_fts_segdir</td>
<td>2</td>
<td>start_block</td>
<td>INTEGER</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_fts_segdir</td>
<td>3</td>
<td>leaves_end_block</td>
<td>INTEGER</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_fts_segdir</td>
<td>4</td>
<td>end_block</td>
<td>INTEGER</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_fts_segdir</td>
<td>5</td>
<td>root</td>
<td>BLOB</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_fts_segments</td>
<td>0</td>
<td>blockid</td>
<td>INTEGER</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>searchable_fts_segments</td>
<td>1</td>
<td>block</td>
<td>BLOB</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>searchable_tags</td>
<td>0</td>
<td>searchable_id</td>
<td>integer</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>searchable_tags</td>
<td>1</td>
<td>tag</td>
<td>text</td>
<td>0</td>
<td></td>
<td>2</td>
</tr>
<tr>
<td>select</td>
<td>0</td>
<td>group</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>select</td>
<td>1</td>
<td>having</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>select</td>
<td>2</td>
<td>and</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>select</td>
<td>3</td>
<td>json</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>simple_primary_key</td>
<td>0</td>
<td>id</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>simple_primary_key</td>
<td>1</td>
<td>content</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>sortable</td>
<td>0</td>
<td>pk1</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>sortable</td>
<td>1</td>
<td>pk2</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>2</td>
</tr>
<tr>
<td>sortable</td>
<td>2</td>
<td>content</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>sortable</td>
<td>3</td>
<td>sortable</td>
<td>integer</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>sortable</td>
<td>4</td>
<td>sortable_with_nulls</td>
<td>real</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>sortable</td>
<td>5</td>
<td>sortable_with_nulls_2</td>
<td>real</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>sortable</td>
<td>6</td>
<td>text</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>table/with/slashes.csv</td>
<td>0</td>
<td>pk</td>
<td>varchar(30)</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>table/with/slashes.csv</td>
<td>1</td>
<td>content</td>
<td>text</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>tags</td>
<td>0</td>
<td>tag</td>
<td>TEXT</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>units</td>
<td>0</td>
<td>pk</td>
<td>integer</td>
<td>0</td>
<td></td>
<td>1</td>
</tr>
<tr>
<td>units</td>
<td>1</td>
<td>distance</td>
<td>int</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
<tr>
<td>units</td>
<td>2</td>
<td>frequency</td>
<td>int</td>
<td>0</td>
<td></td>
<td>0</td>
</tr>
</tbody>
</table>
<p>(I generated that Markdown table by pasting the HTML from Datasette into this tool: <a href="https://jmalarcon.github.io/markdowntables/" rel="nofollow">https://jmalarcon.github.io/markdowntables/</a>)</p>
<h2>
<a id="user-content-better-alternative-using-a-join" class="anchor" href="#better-alternative-using-a-join" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Better alternative using a join</h2>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  <span class="pl-c1">sqlite_master</span>.<span class="pl-c1">name</span> <span class="pl-k">as</span> table_name,
  table_info.<span class="pl-k">*</span>
<span class="pl-k">from</span>
  sqlite_master
  <span class="pl-k">join</span> pragma_table_info(<span class="pl-c1">sqlite_master</span>.<span class="pl-c1">name</span>) <span class="pl-k">as</span> table_info
<span class="pl-k">order by</span>
  <span class="pl-c1">sqlite_master</span>.<span class="pl-c1">name</span></pre></div>
<p><a href="https://latest.datasette.io/fixtures?sql=select%0D%0A++sqlite_master.name+as+table_name%2C%0D%0A++table_info.*%0D%0Afrom%0D%0A++sqlite_master%0D%0A++join+pragma_table_info%28sqlite_master.name%29+as+table_info%0D%0Aorder+by%0D%0A++sqlite_master.name%2C%0D%0A++table_info.cid" rel="nofollow">Demo</a>.</p>
<p>This works with the <code>pragma_table_info</code> and <code>pragma_index_list</code> and <code>pragma_foreign_key_list</code> functions too.</p>
<h2>
<a id="user-content-another-recipe" class="anchor" href="#another-recipe" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Another recipe</h2>
<p><a href="https://twitter.com/amjithr/status/1258576704164909056" rel="nofollow">Amjith pointed</a> to <a href="https://github.com/dbcli/litecli/blob/829220b1e2c3fea84d7c3f0ea8f791f3c28e6230/litecli/sqlexecute.py#L33-L39">this query</a> used in litecli:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">SELECT</span>
  <span class="pl-c1">m</span>.<span class="pl-c1">name</span> <span class="pl-k">as</span> tableName,
  <span class="pl-c1">p</span>.<span class="pl-c1">name</span> <span class="pl-k">as</span> columnName
<span class="pl-k">FROM</span>
  sqlite_master m
  <span class="pl-k">LEFT OUTER JOIN</span> pragma_table_info((<span class="pl-c1">m</span>.<span class="pl-c1">name</span>)) p <span class="pl-k">ON</span> <span class="pl-c1">m</span>.<span class="pl-c1">name</span> <span class="pl-k">&lt;&gt;</span> <span class="pl-c1">p</span>.<span class="pl-c1">name</span>
<span class="pl-k">WHERE</span>
  <span class="pl-c1">m</span>.<span class="pl-c1">type</span> <span class="pl-k">IN</span> (<span class="pl-s"><span class="pl-pds">'</span>table<span class="pl-pds">'</span></span>, <span class="pl-s"><span class="pl-pds">'</span>view<span class="pl-pds">'</span></span>)
  <span class="pl-k">AND</span> <span class="pl-c1">m</span>.<span class="pl-c1">name</span> NOT <span class="pl-k">LIKE</span> <span class="pl-s"><span class="pl-pds">'</span>sqlite_%<span class="pl-pds">'</span></span>
<span class="pl-k">ORDER BY</span>
  tableName,
  columnName</pre></div>
<p><a href="https://latest.datasette.io/fixtures?sql=SELECT+m.name+as+tableName%2C+p.name+as+columnName%0D%0A++++++++FROM+sqlite_master+m%0D%0A++++++++LEFT+OUTER+JOIN+pragma_table_info%28%28m.name%29%29+p+ON+m.name+%3C%3E+p.name%0D%0A++++++++WHERE+m.type+IN+%28%27table%27%2C%27view%27%29+AND+m.name+NOT+LIKE+%27sqlite_%25%27%0D%0A++++++++ORDER+BY+tableName%2C+columnName" rel="nofollow">Demo</a>.</p>


path: cloudrun_use-labels-for-billing-breakdown.md
topic: cloudrun
title: Use labels on Cloud Run services for a billing breakdown
url: https://github.com/simonw/til/blob/main/cloudrun/use-labels-for-billing-breakdown.md
created: 2020-04-21T17:52:57-07:00
created_utc: 2020-04-22T00:52:57+00:00
updated: 2021-12-21T13:02:50-08:00
updated_utc: 2021-12-21T21:02:50+00:00
shot_hash: 9157d6cd2112e335ce93afbece19a833
slug: use-labels-for-billing-breakdown
body:
Thanks to [@glasnt](https://github.com/glasnt) for the tip on this one. If you want a per-service breakdown of pricing on your Google Cloud Run services within a project (each service is a different deployed application) the easiest way to do it is to apply labels to those services, then request a by-label pricing breakdown.

This command will update a service (restarting it) with a new label:

```bash
gcloud run services update csvconf --region=us-central1 --platform=managed --update-labels service=csvconf
```

I found it needed the `--platform=managed` and `--region=X` options to avoid it asking interactive questions.

Here's a bash script which loops through all of the services that do NOT have a `service` label and applies one:

```bash
#!/bin/bash
for line in $(
  gcloud run services list --platform=managed \
    --format="csv(SERVICE,REGION)" \
    --filter "NOT metadata.labels.service:*" \
  | tail -n +2)
do
  IFS=$','; service_and_region=($line); unset IFS;
  service=${service_and_region[0]}
  region=${service_and_region[1]}
  echo "service: $service    region: $region"
  gcloud run services update $service \
    --region=$region --platform=managed \
    --update-labels service=$service
  echo
done
```

It runs the equivalent of this for each service:
```
gcloud run services update asgi-log-demo --region=us-central1 --platform=managed --update-labels service=asgi-log-demo
```

I saved that as a `runme.sh` script, run `chmod 755 runme.sh` and then  `./runme.sh` to run it.

The output of the script looked like this (one entry for each service) - each one took ~30s to run.
```
Service [covid-19] revision [covid-19-00122-zod] has been deployed and is serving 100 percent of traffic at https://covid-19-j7hipcg4aq-uc.a.run.app
✓ Deploying... Done.                                                                                                                                                                                                 
  ✓ Creating Revision...                                                                                                                                                                                             
  ✓ Routing traffic...                                                                                                                                                                                               
Done.                                                                                                                                                                                                                
```
I had to wait a couple of days for this to take effect, but once it did I could get results by visiting Billing -> Reports, then selecting `service` from the group by menu here:

<img src="https://raw.githubusercontent.com/simonw/til/main/cloudrun/use-labels-for-billing-breakdown-1.png" width="300">

The graph (I picked bar chart over line chart) looked like this:

![Graph](https://raw.githubusercontent.com/simonw/til/main/cloudrun/use-labels-for-billing-breakdown-2.png)

## Using the metrics explorer

Even without setting up these extra `service` labels the [Metrics explorer](https://console.cloud.google.com/monitoring/metrics-explorer) can show you breakdowns of billing time against different services.

Here's how to configure that:

<img src="https://raw.githubusercontent.com/simonw/til/main/cloudrun/billing-metrics-explorer.png" width="300">

[Bookmarked query](https://console.cloud.google.com/monitoring/metrics-explorer?project=datasette-222320&pageState=%7B%22xyChart%22:%7B%22dataSets%22:%5B%7B%22timeSeriesFilter%22:%7B%22filter%22:%22metric.type%3D%5C%22run.googleapis.com%2Fcontainer%2Fbillable_instance_time%5C%22%20resource.type%3D%5C%22cloud_run_revision%5C%22%22,%22minAlignmentPeriod%22:%2260s%22,%22aggregations%22:%5B%7B%22perSeriesAligner%22:%22ALIGN_RATE%22,%22crossSeriesReducer%22:%22REDUCE_NONE%22,%22alignmentPeriod%22:%2260s%22,%22groupByFields%22:%5B%5D%7D,%7B%22crossSeriesReducer%22:%22REDUCE_NONE%22,%22alignmentPeriod%22:%2260s%22,%22groupByFields%22:%5B%5D%7D%5D%7D,%22targetAxis%22:%22Y1%22,%22plotType%22:%22LINE%22%7D%5D,%22options%22:%7B%22mode%22:%22COLOR%22%7D,%22constantLines%22:%5B%5D,%22timeshiftDuration%22:%220s%22,%22y1Axis%22:%7B%22label%22:%22y1Axis%22,%22scale%22:%22LINEAR%22%7D%7D,%22isAutoRefresh%22:true,%22timeSelection%22:%7B%22timeRange%22:%221h%22%7D%7D) (only works for me)

html:
<p>Thanks to <a href="https://github.com/glasnt">@glasnt</a> for the tip on this one. If you want a per-service breakdown of pricing on your Google Cloud Run services within a project (each service is a different deployed application) the easiest way to do it is to apply labels to those services, then request a by-label pricing breakdown.</p>
<p>This command will update a service (restarting it) with a new label:</p>
<div class="highlight highlight-source-shell"><pre>gcloud run services update csvconf --region=us-central1 --platform=managed --update-labels service=csvconf</pre></div>
<p>I found it needed the <code>--platform=managed</code> and <code>--region=X</code> options to avoid it asking interactive questions.</p>
<p>Here's a bash script which loops through all of the services that do NOT have a <code>service</code> label and applies one:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c"><span class="pl-c">#!</span>/bin/bash</span>
<span class="pl-k">for</span> <span class="pl-smi">line</span> <span class="pl-k">in</span> <span class="pl-s"><span class="pl-pds">$(</span></span>
<span class="pl-s">  gcloud run services list --platform=managed \</span>
<span class="pl-s">    --format=<span class="pl-s"><span class="pl-pds">"</span>csv(SERVICE,REGION)<span class="pl-pds">"</span></span> \</span>
<span class="pl-s">    --filter <span class="pl-s"><span class="pl-pds">"</span>NOT metadata.labels.service:*<span class="pl-pds">"</span></span> \</span>
<span class="pl-s">  <span class="pl-k">|</span> tail -n +2<span class="pl-pds">)</span></span>
<span class="pl-k">do</span>
  IFS=<span class="pl-s"><span class="pl-pds">$'</span>,<span class="pl-pds">'</span></span><span class="pl-k">;</span> service_and_region=(<span class="pl-smi">$line</span>)<span class="pl-k">;</span> <span class="pl-c1">unset</span> IFS<span class="pl-k">;</span>
  service=<span class="pl-smi">${service_and_region[0]}</span>
  region=<span class="pl-smi">${service_and_region[1]}</span>
  <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>service: <span class="pl-smi">$service</span>    region: <span class="pl-smi">$region</span><span class="pl-pds">"</span></span>
  gcloud run services update <span class="pl-smi">$service</span> \
    --region=<span class="pl-smi">$region</span> --platform=managed \
    --update-labels service=<span class="pl-smi">$service</span>
  <span class="pl-c1">echo</span>
<span class="pl-k">done</span></pre></div>
<p>It runs the equivalent of this for each service:</p>
<pre><code>gcloud run services update asgi-log-demo --region=us-central1 --platform=managed --update-labels service=asgi-log-demo
</code></pre>
<p>I saved that as a <code>runme.sh</code> script, run <code>chmod 755 runme.sh</code> and then  <code>./runme.sh</code> to run it.</p>
<p>The output of the script looked like this (one entry for each service) - each one took ~30s to run.</p>
<pre><code>Service [covid-19] revision [covid-19-00122-zod] has been deployed and is serving 100 percent of traffic at https://covid-19-j7hipcg4aq-uc.a.run.app
✓ Deploying... Done.                                                                                                                                                                                                 
  ✓ Creating Revision...                                                                                                                                                                                             
  ✓ Routing traffic...                                                                                                                                                                                               
Done.                                                                                                                                                                                                                
</code></pre>
<p>I had to wait a couple of days for this to take effect, but once it did I could get results by visiting Billing -&gt; Reports, then selecting <code>service</code> from the group by menu here:</p>
<p><a href="https://raw.githubusercontent.com/simonw/til/main/cloudrun/use-labels-for-billing-breakdown-1.png" target="_blank" rel="nofollow"><img src="https://raw.githubusercontent.com/simonw/til/main/cloudrun/use-labels-for-billing-breakdown-1.png" width="300" style="max-width:100%;"></a></p>
<p>The graph (I picked bar chart over line chart) looked like this:</p>
<p><a href="https://raw.githubusercontent.com/simonw/til/main/cloudrun/use-labels-for-billing-breakdown-2.png" target="_blank" rel="nofollow"><img src="https://raw.githubusercontent.com/simonw/til/main/cloudrun/use-labels-for-billing-breakdown-2.png" alt="Graph" style="max-width:100%;"></a></p>
<h2>
<a id="user-content-using-the-metrics-explorer" class="anchor" href="#using-the-metrics-explorer" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using the metrics explorer</h2>
<p>Even without setting up these extra <code>service</code> labels the <a href="https://console.cloud.google.com/monitoring/metrics-explorer" rel="nofollow">Metrics explorer</a> can show you breakdowns of billing time against different services.</p>
<p>Here's how to configure that:</p>
<p><a href="https://raw.githubusercontent.com/simonw/til/main/cloudrun/billing-metrics-explorer.png" target="_blank" rel="nofollow"><img src="https://raw.githubusercontent.com/simonw/til/main/cloudrun/billing-metrics-explorer.png" width="300" style="max-width:100%;"></a></p>
<p><a href="https://console.cloud.google.com/monitoring/metrics-explorer?project=datasette-222320&amp;pageState=%7B%22xyChart%22:%7B%22dataSets%22:%5B%7B%22timeSeriesFilter%22:%7B%22filter%22:%22metric.type%3D%5C%22run.googleapis.com%2Fcontainer%2Fbillable_instance_time%5C%22%20resource.type%3D%5C%22cloud_run_revision%5C%22%22,%22minAlignmentPeriod%22:%2260s%22,%22aggregations%22:%5B%7B%22perSeriesAligner%22:%22ALIGN_RATE%22,%22crossSeriesReducer%22:%22REDUCE_NONE%22,%22alignmentPeriod%22:%2260s%22,%22groupByFields%22:%5B%5D%7D,%7B%22crossSeriesReducer%22:%22REDUCE_NONE%22,%22alignmentPeriod%22:%2260s%22,%22groupByFields%22:%5B%5D%7D%5D%7D,%22targetAxis%22:%22Y1%22,%22plotType%22:%22LINE%22%7D%5D,%22options%22:%7B%22mode%22:%22COLOR%22%7D,%22constantLines%22:%5B%5D,%22timeshiftDuration%22:%220s%22,%22y1Axis%22:%7B%22label%22:%22y1Axis%22,%22scale%22:%22LINEAR%22%7D%7D,%22isAutoRefresh%22:true,%22timeSelection%22:%7B%22timeRange%22:%221h%22%7D%7D" rel="nofollow">Bookmarked query</a> (only works for me)</p>


path: cloudrun_ship-dockerfile-to-cloud-run.md
topic: cloudrun
title: How to deploy a folder with a Dockerfile to Cloud Run
url: https://github.com/simonw/til/blob/main/cloudrun/ship-dockerfile-to-cloud-run.md
created: 2020-08-04T20:36:31-07:00
created_utc: 2020-08-05T03:36:31+00:00
updated: 2020-12-29T13:55:23-08:00
updated_utc: 2020-12-29T21:55:23+00:00
shot_hash: b086f0e3bf24398095e41516db57e0cc
slug: ship-dockerfile-to-cloud-run
body:
I deployed https://metmusem.datasettes.com/ by creating a folder on my computer containing a Dockerfile and then shipping that folder up to Google Cloud Run.

Normally I use [datasette publish cloudrun](https://docs.datasette.io/en/stable/publish.html#publishing-to-google-cloud-run) to deploy to Cloud Run, but in this case I decided to do it by hand.

I created a folder and dropped two files into it: a `Dockerfile` and a `metadata.json`. BUT... this trick would work with more files in the same directory - it uploads the entire directory contents to be built by Google's cloud builder.

`Dockerfile`
```dockerfile
FROM python:3.6-slim-stretch
RUN apt update
RUN apt install -y python3-dev gcc wget
ADD metadata.json metadata.json
RUN wget -q "https://static.simonwillison.net/static/2018/MetObjects.db"
RUN pip install datasette
RUN datasette inspect MetObjects.db --inspect-file inspect-data.json

EXPOSE $PORT

CMD datasette serve MetObjects.db --host 0.0.0.0 --cors --port $PORT --inspect-file inspect-data.json -m metadata.json
```
The `PORT` is provided by Cloud Run. It's 8080 but they may change that in the future, so it's best to use an environment variable.

Here's the `metadata.json`:
```json
{
    "title": "The Metropolitan Museum of Art Open Access",
    "source": "metmuseum/openaccess",
    "source_url": "https://github.com/metmuseum/openaccess",
    "license": "CC0",
    "license_url": "https://creativecommons.org/publicdomain/zero/1.0/"
}
```
Finally here's my `deploy.sh` script which I used to run the deploy. This needs to be run from within that directory:
```bash
#!/bin/bash
NAME="metmuseum"
PROJECT=$(gcloud config get-value project)
IMAGE="gcr.io/$PROJECT/$NAME"

gcloud builds submit --tag $IMAGE
gcloud run deploy --allow-unauthenticated --platform=managed --image $IMAGE $NAME --memory 2Gi
```
Before running the script I had installed the Cloud Run SDK and run `gcloud init` to login.

The `NAME` variable ends up being used as the name of both my built image and my service. This needs to be unique in your Cloud Run account, or your deploy will over-write an existing service.

Cloud Run deployed my site to https://metmuseum-j7hipcg4aq-uc.a.run.app/

I then used the "Domain mappings" feature in the Cloud Run web console to point a better web address at it.

html:
<p>I deployed <a href="https://metmusem.datasettes.com/" rel="nofollow">https://metmusem.datasettes.com/</a> by creating a folder on my computer containing a Dockerfile and then shipping that folder up to Google Cloud Run.</p>
<p>Normally I use <a href="https://docs.datasette.io/en/stable/publish.html#publishing-to-google-cloud-run" rel="nofollow">datasette publish cloudrun</a> to deploy to Cloud Run, but in this case I decided to do it by hand.</p>
<p>I created a folder and dropped two files into it: a <code>Dockerfile</code> and a <code>metadata.json</code>. BUT... this trick would work with more files in the same directory - it uploads the entire directory contents to be built by Google's cloud builder.</p>
<p><code>Dockerfile</code></p>
<div class="highlight highlight-source-dockerfile"><pre><span class="pl-k">FROM</span> python:3.6-slim-stretch
<span class="pl-k">RUN</span> apt update
<span class="pl-k">RUN</span> apt install -y python3-dev gcc wget
<span class="pl-k">ADD</span> metadata.json metadata.json
<span class="pl-k">RUN</span> wget -q <span class="pl-s">"https://static.simonwillison.net/static/2018/MetObjects.db"</span>
<span class="pl-k">RUN</span> pip install datasette
<span class="pl-k">RUN</span> datasette inspect MetObjects.db --inspect-file inspect-data.json

<span class="pl-k">EXPOSE</span> $PORT

<span class="pl-k">CMD</span> datasette serve MetObjects.db --host 0.0.0.0 --cors --port $PORT --inspect-file inspect-data.json -m metadata.json</pre></div>
<p>The <code>PORT</code> is provided by Cloud Run. It's 8080 but they may change that in the future, so it's best to use an environment variable.</p>
<p>Here's the <code>metadata.json</code>:</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-s"><span class="pl-pds">"</span>title<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>The Metropolitan Museum of Art Open Access<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>source<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>metmuseum/openaccess<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>source_url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://github.com/metmuseum/openaccess<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>license<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>CC0<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>license_url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>https://creativecommons.org/publicdomain/zero/1.0/<span class="pl-pds">"</span></span>
}</pre></div>
<p>Finally here's my <code>deploy.sh</code> script which I used to run the deploy. This needs to be run from within that directory:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c"><span class="pl-c">#!</span>/bin/bash</span>
NAME=<span class="pl-s"><span class="pl-pds">"</span>metmuseum<span class="pl-pds">"</span></span>
PROJECT=<span class="pl-s"><span class="pl-pds">$(</span>gcloud config get-value project<span class="pl-pds">)</span></span>
IMAGE=<span class="pl-s"><span class="pl-pds">"</span>gcr.io/<span class="pl-smi">$PROJECT</span>/<span class="pl-smi">$NAME</span><span class="pl-pds">"</span></span>

gcloud builds submit --tag <span class="pl-smi">$IMAGE</span>
gcloud run deploy --allow-unauthenticated --platform=managed --image <span class="pl-smi">$IMAGE</span> <span class="pl-smi">$NAME</span> --memory 2Gi</pre></div>
<p>Before running the script I had installed the Cloud Run SDK and run <code>gcloud init</code> to login.</p>
<p>The <code>NAME</code> variable ends up being used as the name of both my built image and my service. This needs to be unique in your Cloud Run account, or your deploy will over-write an existing service.</p>
<p>Cloud Run deployed my site to <a href="https://metmuseum-j7hipcg4aq-uc.a.run.app/" rel="nofollow">https://metmuseum-j7hipcg4aq-uc.a.run.app/</a></p>
<p>I then used the "Domain mappings" feature in the Cloud Run web console to point a better web address at it.</p>


path: readthedocs_readthedocs-search-api.md
topic: readthedocs
title: Read the Docs Search API
url: https://github.com/simonw/til/blob/main/readthedocs/readthedocs-search-api.md
created: 2020-08-16T14:08:23-07:00
created_utc: 2020-08-16T21:08:23+00:00
updated: 2020-08-16T14:08:23-07:00
updated_utc: 2020-08-16T21:08:23+00:00
shot_hash: e03a99ee8544e1742f32d5b52f06fea8
slug: readthedocs-search-api
body:
I stumbled across this API today: https://docs.datasette.io/_/api/v2/docsearch/?q=startup&project=datasette&version=stable&language=en

It's used by the search feature at https://docs.datasette.io/en/stable/search.html?q=startup&check_keywords=yes&area=default - I had assumed that feature was implemented in JavaScript (as is common in Sphinx world), but Read the Docs have upgraded it with their own backend search index.

It's built on Elasticsearch - they have developer docs for it here: https://github.com/readthedocs/readthedocs.org/blob/master/docs/development/search.rst

It looks like the key bits of the server-side implementation live here:

- https://github.com/readthedocs/readthedocs.org/blob/master/readthedocs/search/api.py
- https://github.com/readthedocs/readthedocs.org/blob/master/readthedocs/search/faceted_search.py

html:
<p>I stumbled across this API today: <a href="https://docs.datasette.io/_/api/v2/docsearch/?q=startup&amp;project=datasette&amp;version=stable&amp;language=en" rel="nofollow">https://docs.datasette.io/_/api/v2/docsearch/?q=startup&amp;project=datasette&amp;version=stable&amp;language=en</a></p>
<p>It's used by the search feature at <a href="https://docs.datasette.io/en/stable/search.html?q=startup&amp;check_keywords=yes&amp;area=default" rel="nofollow">https://docs.datasette.io/en/stable/search.html?q=startup&amp;check_keywords=yes&amp;area=default</a> - I had assumed that feature was implemented in JavaScript (as is common in Sphinx world), but Read the Docs have upgraded it with their own backend search index.</p>
<p>It's built on Elasticsearch - they have developer docs for it here: <a href="https://github.com/readthedocs/readthedocs.org/blob/master/docs/development/search.rst">https://github.com/readthedocs/readthedocs.org/blob/master/docs/development/search.rst</a></p>
<p>It looks like the key bits of the server-side implementation live here:</p>
<ul>
<li><a href="https://github.com/readthedocs/readthedocs.org/blob/master/readthedocs/search/api.py">https://github.com/readthedocs/readthedocs.org/blob/master/readthedocs/search/api.py</a></li>
<li><a href="https://github.com/readthedocs/readthedocs.org/blob/master/readthedocs/search/faceted_search.py">https://github.com/readthedocs/readthedocs.org/blob/master/readthedocs/search/faceted_search.py</a></li>
</ul>


path: readthedocs_custom-subdomain.md
topic: readthedocs
title: Pointing a custom subdomain at Read the Docs
url: https://github.com/simonw/til/blob/main/readthedocs/custom-subdomain.md
created: 2020-08-14T14:12:43-07:00
created_utc: 2020-08-14T21:12:43+00:00
updated: 2020-08-14T14:12:43-07:00
updated_utc: 2020-08-14T21:12:43+00:00
shot_hash: 7b693e6d7eaffb5027c6b67c7c7adc0d
slug: custom-subdomain
body:
I host the documentation for Datasette on [Read the Docs](https://readthedocs.org/). Until today it lived at https://datasette.readthedocs.io/ but today I moved it to a custom subdomain, https://docs.datasette.io/

Most of this is handled by the https://readthedocs.org/dashboard/datasette/domains/ interface, but there were a couple of extra details.

## CNAME record

First, I needed to add a `CNAME` record for `docs.datasette.io` pointing to `readthedocs.org.`

It's good to do that in advance of adding the domain to Read the Docs, because when you add the domain they instantly start redirecting traffic from the old `datasette.readthedocs.io` domain to the new custom `docs.datasette.io` domain - even if that domain hasn't finished updating DNS and issuing certificates yet!

When I first tried this I got the following error in the *Read the Docs* interface:

>  SSL certificate status: pending_issuance: caa_error: docs.datasette.io

It turns out this was because my DNS hosting provider, Vercel, had added a `CAA` record to the root `datasette.io` domain restricting which certificate authorities could issue certificates. It was locked down to `letsencrypt.org`, but for the Read the Docs certificate mechanism to work I needed to add two more CAA records.

With the help of Read the Docs support I added these:

- `CAA` for `datasette.io`: `0 issue "comodoca.com"`
- `CAA` for `datasette.io`: `0 issue "digicert.com"`

I confirmed that they were working using `dig CAA +short datasette.io`:

```
~ % dig CAA +short datasette.io
0 issue "letsencrypt.org"
0 issue "digicert.com"
0 issue "comodoca.com"
```

Having done this, I added the domain in the Read the Docs interface once more and this time it worked!

html:
<p>I host the documentation for Datasette on <a href="https://readthedocs.org/" rel="nofollow">Read the Docs</a>. Until today it lived at <a href="https://datasette.readthedocs.io/" rel="nofollow">https://datasette.readthedocs.io/</a> but today I moved it to a custom subdomain, <a href="https://docs.datasette.io/" rel="nofollow">https://docs.datasette.io/</a></p>
<p>Most of this is handled by the <a href="https://readthedocs.org/dashboard/datasette/domains/" rel="nofollow">https://readthedocs.org/dashboard/datasette/domains/</a> interface, but there were a couple of extra details.</p>
<h2>
<a id="user-content-cname-record" class="anchor" href="#cname-record" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>CNAME record</h2>
<p>First, I needed to add a <code>CNAME</code> record for <code>docs.datasette.io</code> pointing to <code>readthedocs.org.</code></p>
<p>It's good to do that in advance of adding the domain to Read the Docs, because when you add the domain they instantly start redirecting traffic from the old <code>datasette.readthedocs.io</code> domain to the new custom <code>docs.datasette.io</code> domain - even if that domain hasn't finished updating DNS and issuing certificates yet!</p>
<p>When I first tried this I got the following error in the <em>Read the Docs</em> interface:</p>
<blockquote>
<p>SSL certificate status: pending_issuance: caa_error: docs.datasette.io</p>
</blockquote>
<p>It turns out this was because my DNS hosting provider, Vercel, had added a <code>CAA</code> record to the root <code>datasette.io</code> domain restricting which certificate authorities could issue certificates. It was locked down to <code>letsencrypt.org</code>, but for the Read the Docs certificate mechanism to work I needed to add two more CAA records.</p>
<p>With the help of Read the Docs support I added these:</p>
<ul>
<li>
<code>CAA</code> for <code>datasette.io</code>: <code>0 issue "comodoca.com"</code>
</li>
<li>
<code>CAA</code> for <code>datasette.io</code>: <code>0 issue "digicert.com"</code>
</li>
</ul>
<p>I confirmed that they were working using <code>dig CAA +short datasette.io</code>:</p>
<pre><code>~ % dig CAA +short datasette.io
0 issue "letsencrypt.org"
0 issue "digicert.com"
0 issue "comodoca.com"
</code></pre>
<p>Having done this, I added the domain in the Read the Docs interface once more and this time it worked!</p>


path: github_graphql-pagination-python.md
topic: github
title: Paginating through the GitHub GraphQL API with Python
url: https://github.com/simonw/til/blob/main/github/graphql-pagination-python.md
created: 2020-07-09T21:00:20-07:00
created_utc: 2020-07-10T04:00:20+00:00
updated: 2020-07-10T10:56:36-07:00
updated_utc: 2020-07-10T17:56:36+00:00
shot_hash: 30a44c5998d5ecdb16ee91a55d5fae77
slug: graphql-pagination-python
body:
(See also [Building a self-updating profile README for GitHub](https://simonwillison.net/2020/Jul/10/self-updating-profile-readme/) on my blog)

For my [auto-updating personal README](https://twitter.com/simonw/status/1281435464474324993) I needed to fetch the latest release for every repository I have on GitHub. Since I have 316 public repos I wanted the most efficent way possible to do this. I decided to use the [GitHub GraphQL API](https://developer.github.com/v4/).

Their API allows you to fetch up to 100 repositories at once, and each one can return up to 100 releases. Since I only wanted the most recent release my query ended up looking like this:

```graphql
query {
  viewer {
    repositories(first: 100, privacy: PUBLIC) {
      pageInfo {
        hasNextPage
        endCursor
      }
      nodes {
        name
        releases(last:1) {
          totalCount
          nodes {
            name
            publishedAt
            url
          }
        }
      }
    }
  }
}
```
This gives me back my 100 first repos, and for each one returns the most recent release (if a release exists).

Just one problem: I needed to paginate through all 316. The way you do this with the GitHub GraphQL API is using the `after:` argument and the `endcursor` returned from `pageInfo`. You can send `after:null` to get the first page, then `after:TOKEN` where TOKEN is the `endCursor` from the previous results.

My Python code ended up looking like this (using [python-graphql-client](https://pypi.org/project/python-graphql-client/)):

```python
from python_graphql_client import GraphqlClient

client = GraphqlClient(endpoint="https://api.github.com/graphql")

def make_query(after_cursor=None):
    return """
query {
  viewer {
    repositories(first: 100, privacy: PUBLIC, after:AFTER) {
      pageInfo {
        hasNextPage
        endCursor
      }
      nodes {
        name
        releases(last:1) {
          totalCount
          nodes {
            name
            publishedAt
            url
          }
        }
      }
    }
  }
}
""".replace(
        "AFTER", '"{}"'.format(after_cursor) if after_cursor else "null"
    )


def fetch_releases(oauth_token):
    repos = []
    releases = []
    repo_names = set()
    has_next_page = True
    after_cursor = None

    while has_next_page:
        data = client.execute(
            query=make_query(after_cursor),
            headers={"Authorization": "Bearer {}".format(oauth_token)},
        )
        print()
        print(json.dumps(data, indent=4))
        print()
        for repo in data["data"]["viewer"]["repositories"]["nodes"]:
            if repo["releases"]["totalCount"] and repo["name"] not in repo_names:
                repos.append(repo)
                repo_names.add(repo["name"])
                releases.append(
                    {
                        "repo": repo["name"],
                        "release": repo["releases"]["nodes"][0]["name"]
                        .replace(repo["name"], "")
                        .strip(),
                        "published_at": repo["releases"]["nodes"][0][
                            "publishedAt"
                        ].split("T")[0],
                        "url": repo["releases"]["nodes"][0]["url"],
                    }
                )
        has_next_page = data["data"]["viewer"]["repositories"]["pageInfo"][
            "hasNextPage"
        ]
        after_cursor = data["data"]["viewer"]["repositories"]["pageInfo"]["endCursor"]
    return releases
```
Full code here: https://github.com/simonw/simonw/blob/50d4188f9f067b68b2203540f1983750d51800db/build_readme.py

html:
<p>(See also <a href="https://simonwillison.net/2020/Jul/10/self-updating-profile-readme/" rel="nofollow">Building a self-updating profile README for GitHub</a> on my blog)</p>
<p>For my <a href="https://twitter.com/simonw/status/1281435464474324993" rel="nofollow">auto-updating personal README</a> I needed to fetch the latest release for every repository I have on GitHub. Since I have 316 public repos I wanted the most efficent way possible to do this. I decided to use the <a href="https://developer.github.com/v4/">GitHub GraphQL API</a>.</p>
<p>Their API allows you to fetch up to 100 repositories at once, and each one can return up to 100 releases. Since I only wanted the most recent release my query ended up looking like this:</p>
<div class="highlight highlight-source-graphql"><pre><span class="pl-k">query</span> {
  <span class="pl-v">viewer</span> {
    <span class="pl-v">repositories</span>(<span class="pl-v">first</span>: <span class="pl-c1">100</span>, <span class="pl-v">privacy</span>:<span class="pl-c1"> PUBLIC</span>) {
      <span class="pl-v">pageInfo</span> {
        <span class="pl-v">hasNextPage</span>
        <span class="pl-v">endCursor</span>
      }
      <span class="pl-v">nodes</span> {
        <span class="pl-v">name</span>
        <span class="pl-v">releases</span>(<span class="pl-v">last</span>:<span class="pl-c1">1</span>) {
          <span class="pl-v">totalCount</span>
          <span class="pl-v">nodes</span> {
            <span class="pl-v">name</span>
            <span class="pl-v">publishedAt</span>
            <span class="pl-v">url</span>
          }
        }
      }
    }
  }
}</pre></div>
<p>This gives me back my 100 first repos, and for each one returns the most recent release (if a release exists).</p>
<p>Just one problem: I needed to paginate through all 316. The way you do this with the GitHub GraphQL API is using the <code>after:</code> argument and the <code>endcursor</code> returned from <code>pageInfo</code>. You can send <code>after:null</code> to get the first page, then <code>after:TOKEN</code> where TOKEN is the <code>endCursor</code> from the previous results.</p>
<p>My Python code ended up looking like this (using <a href="https://pypi.org/project/python-graphql-client/" rel="nofollow">python-graphql-client</a>):</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">python_graphql_client</span> <span class="pl-k">import</span> <span class="pl-v">GraphqlClient</span>

<span class="pl-s1">client</span> <span class="pl-c1">=</span> <span class="pl-v">GraphqlClient</span>(<span class="pl-s1">endpoint</span><span class="pl-c1">=</span><span class="pl-s">"https://api.github.com/graphql"</span>)

<span class="pl-k">def</span> <span class="pl-en">make_query</span>(<span class="pl-s1">after_cursor</span><span class="pl-c1">=</span><span class="pl-c1">None</span>):
    <span class="pl-k">return</span> <span class="pl-s">"""</span>
<span class="pl-s">query {</span>
<span class="pl-s">  viewer {</span>
<span class="pl-s">    repositories(first: 100, privacy: PUBLIC, after:AFTER) {</span>
<span class="pl-s">      pageInfo {</span>
<span class="pl-s">        hasNextPage</span>
<span class="pl-s">        endCursor</span>
<span class="pl-s">      }</span>
<span class="pl-s">      nodes {</span>
<span class="pl-s">        name</span>
<span class="pl-s">        releases(last:1) {</span>
<span class="pl-s">          totalCount</span>
<span class="pl-s">          nodes {</span>
<span class="pl-s">            name</span>
<span class="pl-s">            publishedAt</span>
<span class="pl-s">            url</span>
<span class="pl-s">          }</span>
<span class="pl-s">        }</span>
<span class="pl-s">      }</span>
<span class="pl-s">    }</span>
<span class="pl-s">  }</span>
<span class="pl-s">}</span>
<span class="pl-s">"""</span>.<span class="pl-en">replace</span>(
        <span class="pl-s">"AFTER"</span>, <span class="pl-s">'"{}"'</span>.<span class="pl-en">format</span>(<span class="pl-s1">after_cursor</span>) <span class="pl-k">if</span> <span class="pl-s1">after_cursor</span> <span class="pl-k">else</span> <span class="pl-s">"null"</span>
    )


<span class="pl-k">def</span> <span class="pl-en">fetch_releases</span>(<span class="pl-s1">oauth_token</span>):
    <span class="pl-s1">repos</span> <span class="pl-c1">=</span> []
    <span class="pl-s1">releases</span> <span class="pl-c1">=</span> []
    <span class="pl-s1">repo_names</span> <span class="pl-c1">=</span> <span class="pl-en">set</span>()
    <span class="pl-s1">has_next_page</span> <span class="pl-c1">=</span> <span class="pl-c1">True</span>
    <span class="pl-s1">after_cursor</span> <span class="pl-c1">=</span> <span class="pl-c1">None</span>

    <span class="pl-k">while</span> <span class="pl-s1">has_next_page</span>:
        <span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">client</span>.<span class="pl-en">execute</span>(
            <span class="pl-s1">query</span><span class="pl-c1">=</span><span class="pl-en">make_query</span>(<span class="pl-s1">after_cursor</span>),
            <span class="pl-s1">headers</span><span class="pl-c1">=</span>{<span class="pl-s">"Authorization"</span>: <span class="pl-s">"Bearer {}"</span>.<span class="pl-en">format</span>(<span class="pl-s1">oauth_token</span>)},
        )
        <span class="pl-en">print</span>()
        <span class="pl-en">print</span>(<span class="pl-s1">json</span>.<span class="pl-en">dumps</span>(<span class="pl-s1">data</span>, <span class="pl-s1">indent</span><span class="pl-c1">=</span><span class="pl-c1">4</span>))
        <span class="pl-en">print</span>()
        <span class="pl-k">for</span> <span class="pl-s1">repo</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>[<span class="pl-s">"data"</span>][<span class="pl-s">"viewer"</span>][<span class="pl-s">"repositories"</span>][<span class="pl-s">"nodes"</span>]:
            <span class="pl-k">if</span> <span class="pl-s1">repo</span>[<span class="pl-s">"releases"</span>][<span class="pl-s">"totalCount"</span>] <span class="pl-c1">and</span> <span class="pl-s1">repo</span>[<span class="pl-s">"name"</span>] <span class="pl-c1">not</span> <span class="pl-c1">in</span> <span class="pl-s1">repo_names</span>:
                <span class="pl-s1">repos</span>.<span class="pl-en">append</span>(<span class="pl-s1">repo</span>)
                <span class="pl-s1">repo_names</span>.<span class="pl-en">add</span>(<span class="pl-s1">repo</span>[<span class="pl-s">"name"</span>])
                <span class="pl-s1">releases</span>.<span class="pl-en">append</span>(
                    {
                        <span class="pl-s">"repo"</span>: <span class="pl-s1">repo</span>[<span class="pl-s">"name"</span>],
                        <span class="pl-s">"release"</span>: <span class="pl-s1">repo</span>[<span class="pl-s">"releases"</span>][<span class="pl-s">"nodes"</span>][<span class="pl-c1">0</span>][<span class="pl-s">"name"</span>]
                        .<span class="pl-en">replace</span>(<span class="pl-s1">repo</span>[<span class="pl-s">"name"</span>], <span class="pl-s">""</span>)
                        .<span class="pl-en">strip</span>(),
                        <span class="pl-s">"published_at"</span>: <span class="pl-s1">repo</span>[<span class="pl-s">"releases"</span>][<span class="pl-s">"nodes"</span>][<span class="pl-c1">0</span>][
                            <span class="pl-s">"publishedAt"</span>
                        ].<span class="pl-en">split</span>(<span class="pl-s">"T"</span>)[<span class="pl-c1">0</span>],
                        <span class="pl-s">"url"</span>: <span class="pl-s1">repo</span>[<span class="pl-s">"releases"</span>][<span class="pl-s">"nodes"</span>][<span class="pl-c1">0</span>][<span class="pl-s">"url"</span>],
                    }
                )
        <span class="pl-s1">has_next_page</span> <span class="pl-c1">=</span> <span class="pl-s1">data</span>[<span class="pl-s">"data"</span>][<span class="pl-s">"viewer"</span>][<span class="pl-s">"repositories"</span>][<span class="pl-s">"pageInfo"</span>][
            <span class="pl-s">"hasNextPage"</span>
        ]
        <span class="pl-s1">after_cursor</span> <span class="pl-c1">=</span> <span class="pl-s1">data</span>[<span class="pl-s">"data"</span>][<span class="pl-s">"viewer"</span>][<span class="pl-s">"repositories"</span>][<span class="pl-s">"pageInfo"</span>][<span class="pl-s">"endCursor"</span>]
    <span class="pl-k">return</span> <span class="pl-s1">releases</span></pre></div>
<p>Full code here: <a href="https://github.com/simonw/simonw/blob/50d4188f9f067b68b2203540f1983750d51800db/build_readme.py">https://github.com/simonw/simonw/blob/50d4188f9f067b68b2203540f1983750d51800db/build_readme.py</a></p>


path: github_dependencies-graphql-api.md
topic: github
title: Accessing repository dependencies in the GitHub GraphQL API
url: https://github.com/simonw/til/blob/main/github/dependencies-graphql-api.md
created: 2020-04-30T14:46:40-07:00
created_utc: 2020-04-30T21:46:40+00:00
updated: 2020-04-30T16:18:03-07:00
updated_utc: 2020-04-30T23:18:03+00:00
shot_hash: 485fde079e65449bd7ede45172bede7e
slug: dependencies-graphql-api
body:
[Access to a Repositories Dependency Graph](https://developer.github.com/v4/previews/#access-to-a-repositories-dependency-graph) describes a preview API for accessing GitHub repository dependencies.

To access the preview you need to send this `Accept:` header: `application/vnd.github.hawkgirl-preview+json`

But... GitHub's own GraphQL explorer at https://developer.github.com/v4/explorer/ doesn't currently allow you to set custom HTTP headers.

## Installing GraphiQL

So I decided to try https://github.com/skevy/graphiql-app - since it's mentioned in the GitHub docs [here](https://developer.github.com/v4/guides/using-the-explorer/#using-graphiql).

    brew cask install graphiql

Then you can launch the app from `/Applications/GraphiQL.app`.

I created a personal access token at https://github.com/settings/tokens called graphiql and gave it all of the `read:` permissions.

I launched the graphiql app and added the following headers:

    Authorization: Bearer MY-PERSONAl-ACCESS-TOKEN
    Accept: application/vnd.github.hawkgirl-preview+json

I added `https://api.github.com/graphql` as the endpoint.

I tested it by running these queries:
```graphql
{
  user(login:"simonw") {
    id
    name
    url
    websiteUrl
  }
}
```
```graphql
{
  repository(owner:"simonw", name:"datasette") {
    id
    name
    issues {
      totalCount
    }
  }
}
```
```graphql
{
  search(type:REPOSITORY query:"datasette" first:10) {
    repositoryCount
    edges {
      node {
        __typename
        ... on Repository {
          nameWithOwner
          description
        }
      }
    }
  }
}
```
Then I figured out this query (using autocomplete) which returns dependencies detected for a repo:
```graphql
{
  repository(owner:"simonw", name:"datasette") {
    dependencyGraphManifests {
      totalCount
      nodes {
        filename
      }
      edges {
        node {
          blobPath
          dependencies {
            totalCount
            nodes {
              packageName
              requirements
              hasDependencies
              packageManager
            }
          }
        }
      }
    }
  }
}
```
This gave me back:
```json
{
  "data": {
    "repository": {
      "dependencyGraphManifests": {
        "totalCount": 1,
        "nodes": [
          {
            "filename": "setup.py"
          }
        ],
        "edges": [
          {
            "node": {
              "blobPath": "/simonw/datasette/blob/master/setup.py",
              "dependencies": {
                "totalCount": 12,
                "nodes": [
                  {
                    "packageName": "aiofiles",
                    "requirements": "~> 0.4.0",
                    "hasDependencies": false,
                    "packageManager": "PIP"
                  },
                  {
                    "packageName": "click",
                    "requirements": "~> 7.1.1",
                    "hasDependencies": false,
                    "packageManager": "PIP"
                  },
                  {
                    "packageName": "click-default-group",
                    "requirements": "~> 1.2.2",
                    "hasDependencies": true,
                    "packageManager": "PIP"
                  },
                  {
                    "packageName": "datasette",
                    "requirements": "",
                    "hasDependencies": true,
                    "packageManager": "PIP"
                  },
                  {
                    "packageName": "hupper",
                    "requirements": "~> 1.9",
                    "hasDependencies": true,
                    "packageManager": "PIP"
                  },
                  {
                    "packageName": "janus",
                    "requirements": "~> 0.4.0",
                    "hasDependencies": false,
                    "packageManager": "PIP"
                  },
                  {
                    "packageName": "jinja2",
                    "requirements": "~> 2.10.3",
                    "hasDependencies": true,
                    "packageManager": "PIP"
                  },
                  {
                    "packageName": "mergedeep",
                    "requirements": "~> 1.1.1",
                    "hasDependencies": false,
                    "packageManager": "PIP"
                  },
                  {
                    "packageName": "pint",
                    "requirements": "~> 0.9",
                    "hasDependencies": false,
                    "packageManager": "PIP"
                  },
                  {
                    "packageName": "pluggy",
                    "requirements": "~> 0.13.0",
                    "hasDependencies": true,
                    "packageManager": "PIP"
                  },
                  {
                    "packageName": "pyyaml",
                    "requirements": "~> 5.3",
                    "hasDependencies": false,
                    "packageManager": "PIP"
                  },
                  {
                    "packageName": "uvicorn",
                    "requirements": "~> 0.11",
                    "hasDependencies": true,
                    "packageManager": "PIP"
                  }
                ]
              }
            }
          }
        ]
      }
    }
  }
}
```

Sadly I really wanted to get the dependents, not the dependencies - but that's [still not available](https://stackoverflow.com/questions/58734176/how-to-use-github-api-to-get-a-repositorys-dependents-information-in-github) even as a preview API. So I [scraped them](https://github.com/dogsheep/github-to-sqlite/issues/34) instead.

html:
<p><a href="https://developer.github.com/v4/previews/#access-to-a-repositories-dependency-graph">Access to a Repositories Dependency Graph</a> describes a preview API for accessing GitHub repository dependencies.</p>
<p>To access the preview you need to send this <code>Accept:</code> header: <code>application/vnd.github.hawkgirl-preview+json</code></p>
<p>But... GitHub's own GraphQL explorer at <a href="https://developer.github.com/v4/explorer/">https://developer.github.com/v4/explorer/</a> doesn't currently allow you to set custom HTTP headers.</p>
<h2>
<a id="user-content-installing-graphiql" class="anchor" href="#installing-graphiql" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing GraphiQL</h2>
<p>So I decided to try <a href="https://github.com/skevy/graphiql-app">https://github.com/skevy/graphiql-app</a> - since it's mentioned in the GitHub docs <a href="https://developer.github.com/v4/guides/using-the-explorer/#using-graphiql">here</a>.</p>
<pre><code>brew cask install graphiql
</code></pre>
<p>Then you can launch the app from <code>/Applications/GraphiQL.app</code>.</p>
<p>I created a personal access token at <a href="https://github.com/settings/tokens">https://github.com/settings/tokens</a> called graphiql and gave it all of the <code>read:</code> permissions.</p>
<p>I launched the graphiql app and added the following headers:</p>
<pre><code>Authorization: Bearer MY-PERSONAl-ACCESS-TOKEN
Accept: application/vnd.github.hawkgirl-preview+json
</code></pre>
<p>I added <code>https://api.github.com/graphql</code> as the endpoint.</p>
<p>I tested it by running these queries:</p>
<div class="highlight highlight-source-graphql"><pre>{
  <span class="pl-v">user</span>(<span class="pl-v">login</span>:<span class="pl-s"><span class="pl-pds">"</span>simonw<span class="pl-pds">"</span></span>) {
    <span class="pl-v">id</span>
    <span class="pl-v">name</span>
    <span class="pl-v">url</span>
    <span class="pl-v">websiteUrl</span>
  }
}</pre></div>
<div class="highlight highlight-source-graphql"><pre>{
  <span class="pl-v">repository</span>(<span class="pl-v">owner</span>:<span class="pl-s"><span class="pl-pds">"</span>simonw<span class="pl-pds">"</span></span>, <span class="pl-v">name</span>:<span class="pl-s"><span class="pl-pds">"</span>datasette<span class="pl-pds">"</span></span>) {
    <span class="pl-v">id</span>
    <span class="pl-v">name</span>
    <span class="pl-v">issues</span> {
      <span class="pl-v">totalCount</span>
    }
  }
}</pre></div>
<div class="highlight highlight-source-graphql"><pre>{
  <span class="pl-v">search</span>(<span class="pl-v">type</span>:<span class="pl-c1">REPOSITORY</span> <span class="pl-v">query</span>:<span class="pl-s"><span class="pl-pds">"</span>datasette<span class="pl-pds">"</span></span> <span class="pl-v">first</span>:<span class="pl-c1">10</span>) {
    <span class="pl-v">repositoryCount</span>
    <span class="pl-v">edges</span> {
      <span class="pl-v">node</span> {
        <span class="pl-v">__typename</span>
        <span class="pl-k">...</span> <span class="pl-k">on</span> <span class="pl-c1">Repository</span> {
          <span class="pl-v">nameWithOwner</span>
          <span class="pl-v">description</span>
        }
      }
    }
  }
}</pre></div>
<p>Then I figured out this query (using autocomplete) which returns dependencies detected for a repo:</p>
<div class="highlight highlight-source-graphql"><pre>{
  <span class="pl-v">repository</span>(<span class="pl-v">owner</span>:<span class="pl-s"><span class="pl-pds">"</span>simonw<span class="pl-pds">"</span></span>, <span class="pl-v">name</span>:<span class="pl-s"><span class="pl-pds">"</span>datasette<span class="pl-pds">"</span></span>) {
    <span class="pl-v">dependencyGraphManifests</span> {
      <span class="pl-v">totalCount</span>
      <span class="pl-v">nodes</span> {
        <span class="pl-v">filename</span>
      }
      <span class="pl-v">edges</span> {
        <span class="pl-v">node</span> {
          <span class="pl-v">blobPath</span>
          <span class="pl-v">dependencies</span> {
            <span class="pl-v">totalCount</span>
            <span class="pl-v">nodes</span> {
              <span class="pl-v">packageName</span>
              <span class="pl-v">requirements</span>
              <span class="pl-v">hasDependencies</span>
              <span class="pl-v">packageManager</span>
            }
          }
        }
      }
    }
  }
}</pre></div>
<p>This gave me back:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-s"><span class="pl-pds">"</span>data<span class="pl-pds">"</span></span>: {
    <span class="pl-s"><span class="pl-pds">"</span>repository<span class="pl-pds">"</span></span>: {
      <span class="pl-s"><span class="pl-pds">"</span>dependencyGraphManifests<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>totalCount<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>,
        <span class="pl-s"><span class="pl-pds">"</span>nodes<span class="pl-pds">"</span></span>: [
          {
            <span class="pl-s"><span class="pl-pds">"</span>filename<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>setup.py<span class="pl-pds">"</span></span>
          }
        ],
        <span class="pl-s"><span class="pl-pds">"</span>edges<span class="pl-pds">"</span></span>: [
          {
            <span class="pl-s"><span class="pl-pds">"</span>node<span class="pl-pds">"</span></span>: {
              <span class="pl-s"><span class="pl-pds">"</span>blobPath<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>/simonw/datasette/blob/master/setup.py<span class="pl-pds">"</span></span>,
              <span class="pl-s"><span class="pl-pds">"</span>dependencies<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>totalCount<span class="pl-pds">"</span></span>: <span class="pl-c1">12</span>,
                <span class="pl-s"><span class="pl-pds">"</span>nodes<span class="pl-pds">"</span></span>: [
                  {
                    <span class="pl-s"><span class="pl-pds">"</span>packageName<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>aiofiles<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>requirements<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>~&gt; 0.4.0<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>hasDependencies<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
                    <span class="pl-s"><span class="pl-pds">"</span>packageManager<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>PIP<span class="pl-pds">"</span></span>
                  },
                  {
                    <span class="pl-s"><span class="pl-pds">"</span>packageName<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>click<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>requirements<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>~&gt; 7.1.1<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>hasDependencies<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
                    <span class="pl-s"><span class="pl-pds">"</span>packageManager<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>PIP<span class="pl-pds">"</span></span>
                  },
                  {
                    <span class="pl-s"><span class="pl-pds">"</span>packageName<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>click-default-group<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>requirements<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>~&gt; 1.2.2<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>hasDependencies<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
                    <span class="pl-s"><span class="pl-pds">"</span>packageManager<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>PIP<span class="pl-pds">"</span></span>
                  },
                  {
                    <span class="pl-s"><span class="pl-pds">"</span>packageName<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>datasette<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>requirements<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span><span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>hasDependencies<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
                    <span class="pl-s"><span class="pl-pds">"</span>packageManager<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>PIP<span class="pl-pds">"</span></span>
                  },
                  {
                    <span class="pl-s"><span class="pl-pds">"</span>packageName<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>hupper<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>requirements<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>~&gt; 1.9<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>hasDependencies<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
                    <span class="pl-s"><span class="pl-pds">"</span>packageManager<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>PIP<span class="pl-pds">"</span></span>
                  },
                  {
                    <span class="pl-s"><span class="pl-pds">"</span>packageName<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>janus<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>requirements<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>~&gt; 0.4.0<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>hasDependencies<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
                    <span class="pl-s"><span class="pl-pds">"</span>packageManager<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>PIP<span class="pl-pds">"</span></span>
                  },
                  {
                    <span class="pl-s"><span class="pl-pds">"</span>packageName<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>jinja2<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>requirements<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>~&gt; 2.10.3<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>hasDependencies<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
                    <span class="pl-s"><span class="pl-pds">"</span>packageManager<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>PIP<span class="pl-pds">"</span></span>
                  },
                  {
                    <span class="pl-s"><span class="pl-pds">"</span>packageName<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>mergedeep<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>requirements<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>~&gt; 1.1.1<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>hasDependencies<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
                    <span class="pl-s"><span class="pl-pds">"</span>packageManager<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>PIP<span class="pl-pds">"</span></span>
                  },
                  {
                    <span class="pl-s"><span class="pl-pds">"</span>packageName<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>pint<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>requirements<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>~&gt; 0.9<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>hasDependencies<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
                    <span class="pl-s"><span class="pl-pds">"</span>packageManager<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>PIP<span class="pl-pds">"</span></span>
                  },
                  {
                    <span class="pl-s"><span class="pl-pds">"</span>packageName<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>pluggy<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>requirements<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>~&gt; 0.13.0<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>hasDependencies<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
                    <span class="pl-s"><span class="pl-pds">"</span>packageManager<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>PIP<span class="pl-pds">"</span></span>
                  },
                  {
                    <span class="pl-s"><span class="pl-pds">"</span>packageName<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>pyyaml<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>requirements<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>~&gt; 5.3<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>hasDependencies<span class="pl-pds">"</span></span>: <span class="pl-c1">false</span>,
                    <span class="pl-s"><span class="pl-pds">"</span>packageManager<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>PIP<span class="pl-pds">"</span></span>
                  },
                  {
                    <span class="pl-s"><span class="pl-pds">"</span>packageName<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>uvicorn<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>requirements<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>~&gt; 0.11<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>hasDependencies<span class="pl-pds">"</span></span>: <span class="pl-c1">true</span>,
                    <span class="pl-s"><span class="pl-pds">"</span>packageManager<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>PIP<span class="pl-pds">"</span></span>
                  }
                ]
              }
            }
          }
        ]
      }
    }
  }
}</pre></div>
<p>Sadly I really wanted to get the dependents, not the dependencies - but that's <a href="https://stackoverflow.com/questions/58734176/how-to-use-github-api-to-get-a-repositorys-dependents-information-in-github" rel="nofollow">still not available</a> even as a preview API. So I <a href="https://github.com/dogsheep/github-to-sqlite/issues/34">scraped them</a> instead.</p>


path: linux_echo-pipe-to-file-su.md
topic: linux
title: Piping echo to a file owned by root using sudo and tee
url: https://github.com/simonw/til/blob/main/linux/echo-pipe-to-file-su.md
created: 2020-08-24T17:30:21-07:00
created_utc: 2020-08-25T00:30:21+00:00
updated: 2020-08-24T17:30:21-07:00
updated_utc: 2020-08-25T00:30:21+00:00
shot_hash: e1aa050e08a51078cb2be6e2019edd35
slug: echo-pipe-to-file-su
body:
I wanted to create a file with a shell one-liner where the file lived somewhere owned by root.

I tried this but it didn't work:
```bash
sudo echo '#!/bin/sh
/usr/bin/tarsnap -c \
  -f "$(uname -n)-$(date +%Y-%m-%d_%H-%M-%S)" \
  /home/simon/team-storage' > /root/tarsnap-backup.sh
```
Running `echo` using `sudo` didn't pass through to the `> filename` bit.

Here's what did work:

```bash
echo '#!/bin/sh
/usr/bin/tarsnap -c \
  -f "$(uname -n)-$(date +%Y-%m-%d_%H-%M-%S)" \
  /home/simon/team-storage' | sudo tee /root/tarsnap-backup.sh > /dev/null
```

No need for `sudo` on the `echo` - but it pipes the output to `sudo tee` which can then write the file to disk.

The `> /dev/null` at the end supresses any output from the `tee` command. If you want to see the output you can do this instead:

```bash
echo '#!/bin/sh
/usr/bin/tarsnap -c \
  -f "$(uname -n)-$(date +%Y-%m-%d_%H-%M-%S)" \
  /home/simon/team-storage' | sudo tee /root/tarsnap-backup.sh
```

html:
<p>I wanted to create a file with a shell one-liner where the file lived somewhere owned by root.</p>
<p>I tried this but it didn't work:</p>
<div class="highlight highlight-source-shell"><pre>sudo <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">'</span>#!/bin/sh</span>
<span class="pl-s">/usr/bin/tarsnap -c \</span>
<span class="pl-s">  -f "$(uname -n)-$(date +%Y-%m-%d_%H-%M-%S)" \</span>
<span class="pl-s">  /home/simon/team-storage<span class="pl-pds">'</span></span> <span class="pl-k">&gt;</span> /root/tarsnap-backup.sh</pre></div>
<p>Running <code>echo</code> using <code>sudo</code> didn't pass through to the <code>&gt; filename</code> bit.</p>
<p>Here's what did work:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">'</span>#!/bin/sh</span>
<span class="pl-s">/usr/bin/tarsnap -c \</span>
<span class="pl-s">  -f "$(uname -n)-$(date +%Y-%m-%d_%H-%M-%S)" \</span>
<span class="pl-s">  /home/simon/team-storage<span class="pl-pds">'</span></span> <span class="pl-k">|</span> sudo tee /root/tarsnap-backup.sh <span class="pl-k">&gt;</span> /dev/null</pre></div>
<p>No need for <code>sudo</code> on the <code>echo</code> - but it pipes the output to <code>sudo tee</code> which can then write the file to disk.</p>
<p>The <code>&gt; /dev/null</code> at the end supresses any output from the <code>tee</code> command. If you want to see the output you can do this instead:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">'</span>#!/bin/sh</span>
<span class="pl-s">/usr/bin/tarsnap -c \</span>
<span class="pl-s">  -f "$(uname -n)-$(date +%Y-%m-%d_%H-%M-%S)" \</span>
<span class="pl-s">  /home/simon/team-storage<span class="pl-pds">'</span></span> <span class="pl-k">|</span> sudo tee /root/tarsnap-backup.sh</pre></div>


path: tailscale_lock-down-sshd.md
topic: tailscale
title: Restricting SSH connections to devices within a Tailscale network
url: https://github.com/simonw/til/blob/main/tailscale/lock-down-sshd.md
created: 2020-04-23T07:21:22-07:00
created_utc: 2020-04-23T14:21:22+00:00
updated: 2020-04-29T18:29:59-07:00
updated_utc: 2020-04-30T01:29:59+00:00
shot_hash: a78d2b07dda6a8773e59e92c364afebc
slug: lock-down-sshd
body:
I'm running an AWS Lightsail instance and I want to only be able to SSH to it from devices connected to my [Tailscale](https://tailscale.com/) network.

I installed Tailscale on the instance using their [Ubuntu installation instructions](https://tailscale.com/kb/1037/install-ubuntu-1804). I have it running on my laptop and phone as well.

I ran `ifconfig tailscale0` to find the Tailscale IP for instance:
```
$ ifconfig tailscale0
tailscale0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST>  mtu 1420
        inet 100.122.168.55  netmask 255.192.0.0  destination 100.122.168.55
        inet6 fe80::33a:342a:2733:186a  prefixlen 64  scopeid 0x20<link>
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 2147  bytes 95030 (95.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 990  bytes 66448 (66.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
```
Then I ran `sudo vi /etc/ssh/sshd_config` and added that address as the only `ListenAddress`:
```
#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
ListenAddress 100.122.168.55
```
Then restarted SSH:
```
sudo service ssh restart
```
I can now SSH to Tailscale from my laptop, but only if I use the Tailscale IP address for the server (I thought it was broken at first because I was still SSHing to the internet public IP):
```
ssh ubuntu@100.122.168.55 -i lightsail.pem
```
Bonus: point a real DNS subdomain at the Tailscale IP and you can `ssh ubuntu@realsubdomain.example.com` instead of remembering the IP address.

Handy debugging tip: `tail -f /var/log/auth.log` shows recent sign-in attempts.

Thanks to [@apenwarr for tips](https://twitter.com/apenwarr/status/1253318250131263489).

## Alternative pattern

[This conversation](https://twitter.com/moderat10n/status/1253407976330690561) questions if the above recipe will work correctly when a server reboots. It seems it's possible that sshd might start up before the `tailscale0` network has been created, resulting in problems.

[@bradfitz instead recommends](https://twitter.com/bradfitz/status/1253420075018645505) adding the following line:

    AllowUsers *@100.64.0.0/10

This will allow SSH access only from users within the Tailscale range of IPs. It shouldn't cause any problems during server startup.

## ⚠️ Warning!

This is complicated. See https://github.com/simonw/til/issues/7 for an explanation of a potential vulnerability in this pattern.

html:
<p>I'm running an AWS Lightsail instance and I want to only be able to SSH to it from devices connected to my <a href="https://tailscale.com/" rel="nofollow">Tailscale</a> network.</p>
<p>I installed Tailscale on the instance using their <a href="https://tailscale.com/kb/1037/install-ubuntu-1804" rel="nofollow">Ubuntu installation instructions</a>. I have it running on my laptop and phone as well.</p>
<p>I ran <code>ifconfig tailscale0</code> to find the Tailscale IP for instance:</p>
<pre><code>$ ifconfig tailscale0
tailscale0: flags=4305&lt;UP,POINTOPOINT,RUNNING,NOARP,MULTICAST&gt;  mtu 1420
        inet 100.122.168.55  netmask 255.192.0.0  destination 100.122.168.55
        inet6 fe80::33a:342a:2733:186a  prefixlen 64  scopeid 0x20&lt;link&gt;
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 500  (UNSPEC)
        RX packets 2147  bytes 95030 (95.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 990  bytes 66448 (66.4 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
</code></pre>
<p>Then I ran <code>sudo vi /etc/ssh/sshd_config</code> and added that address as the only <code>ListenAddress</code>:</p>
<pre><code>#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::
ListenAddress 100.122.168.55
</code></pre>
<p>Then restarted SSH:</p>
<pre><code>sudo service ssh restart
</code></pre>
<p>I can now SSH to Tailscale from my laptop, but only if I use the Tailscale IP address for the server (I thought it was broken at first because I was still SSHing to the internet public IP):</p>
<pre><code>ssh ubuntu@100.122.168.55 -i lightsail.pem
</code></pre>
<p>Bonus: point a real DNS subdomain at the Tailscale IP and you can <code>ssh ubuntu@realsubdomain.example.com</code> instead of remembering the IP address.</p>
<p>Handy debugging tip: <code>tail -f /var/log/auth.log</code> shows recent sign-in attempts.</p>
<p>Thanks to <a href="https://twitter.com/apenwarr/status/1253318250131263489" rel="nofollow">@apenwarr for tips</a>.</p>
<h2>
<a id="user-content-alternative-pattern" class="anchor" href="#alternative-pattern" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Alternative pattern</h2>
<p><a href="https://twitter.com/moderat10n/status/1253407976330690561" rel="nofollow">This conversation</a> questions if the above recipe will work correctly when a server reboots. It seems it's possible that sshd might start up before the <code>tailscale0</code> network has been created, resulting in problems.</p>
<p><a href="https://twitter.com/bradfitz/status/1253420075018645505" rel="nofollow">@bradfitz instead recommends</a> adding the following line:</p>
<pre><code>AllowUsers *@100.64.0.0/10
</code></pre>
<p>This will allow SSH access only from users within the Tailscale range of IPs. It shouldn't cause any problems during server startup.</p>
<h2>
<a id="user-content-️-warning" class="anchor" href="#%EF%B8%8F-warning" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a><g-emoji class="g-emoji" alias="warning" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/26a0.png">⚠️</g-emoji> Warning!</h2>
<p>This is complicated. See <a href="https://github.com/simonw/til/issues/7">https://github.com/simonw/til/issues/7</a> for an explanation of a potential vulnerability in this pattern.</p>


path: presenting_stickies-for-workshop-links.md
topic: presenting
title: Using macOS stickies to display a workshop link on the screen
url: https://github.com/simonw/til/blob/main/presenting/stickies-for-workshop-links.md
created: 2020-04-20T09:41:01-07:00
created_utc: 2020-04-20T16:41:01+00:00
updated: 2020-04-20T09:43:38-07:00
updated_utc: 2020-04-20T16:43:38+00:00
shot_hash: c21170e8479583cc9021ff04924def67
slug: stickies-for-workshop-links
body:
When giving a workshop it's often useful to have a URL to the workshop materials visible on screen at all times. I use a bit.ly link for these.

macOS stickies work great for this: create a sticky with the URL, locate it in a bottom corner of the screen and then turn on the "Float on Top" and "Translucent" options in the Window menu.

![Screenshot of stickies](https://raw.githubusercontent.com/simonw/til/master/presenting/Tipsheet__https___bit_ly_%E2%80%A6_and_New_File_and_Zoom.png)

html:
<p>When giving a workshop it's often useful to have a URL to the workshop materials visible on screen at all times. I use a bit.ly link for these.</p>
<p>macOS stickies work great for this: create a sticky with the URL, locate it in a bottom corner of the screen and then turn on the "Float on Top" and "Translucent" options in the Window menu.</p>
<p><a href="https://raw.githubusercontent.com/simonw/til/master/presenting/Tipsheet__https___bit_ly_%E2%80%A6_and_New_File_and_Zoom.png" target="_blank" rel="nofollow"><img src="https://raw.githubusercontent.com/simonw/til/master/presenting/Tipsheet__https___bit_ly_%E2%80%A6_and_New_File_and_Zoom.png" alt="Screenshot of stickies" style="max-width:100%;"></a></p>


path: javascript_copy-button.md
topic: javascript
title: Implementing a "copy to clipboard" button
url: https://github.com/simonw/til/blob/main/javascript/copy-button.md
created: 2020-07-23T14:21:09-07:00
created_utc: 2020-07-23T21:21:09+00:00
updated: 2020-07-23T14:21:09-07:00
updated_utc: 2020-07-23T21:21:09+00:00
shot_hash: 804ecafec2aa216dcb5e3b4dfe3c470b
slug: copy-button
body:
I had to figure this out while building [datasette-copyable](https://github.com/simonw/datasette-copyable/) - demo [here](https://covid-19.datasettes.com/covid/ny_times_us_counties.copyable?state=Texas&county=Harris).

The trick is to run `document.execCommand("copy")` to copy to the clipboard - but only after first ensuring that some selectable content (probably in a textarea) has been selected.

So you do this:
```javascript
var ta = document.querySelector("textarea");
var button = document.querySelector("button");
button.onclick = () => {
    ta.select();
    document.execCommand("copy");
};
```
But that's not quite enough: there's no visible indication that anything happened!

I decided to fix this by updating the text on the button to say "Copied!" and then reverting the button text back again 1.5 seconds later.

I also opted to add the "copy" button using JavaScript, since without JavaScript enabled it won't do anything.

Here's the full HTML and JavaScript snippet I used:
```html
<div>
<textarea class="copyable">{{ copyable }}</textarea>
<p class="raw-link"><a href="{{ raw_link }}">Raw data</a></p>
</div>

<script>
var ta = document.querySelector("textarea.copyable");
var p = document.querySelector("p.raw-link");
var button = document.createElement("button");
button.className = "copyable-copy-button";
button.innerHTML = "Copy to clipboard";
button.onclick = () => {
    ta.select();
    document.execCommand("copy");
    button.innerHTML = "Copied!";
    setTimeout(() => {
        button.innerHTML = "Copy to clipboard";
    }, 1500);
};
p.appendChild(button);
p.insertAdjacentElement("afterbegin", button);
</script>
```

html:
<p>I had to figure this out while building <a href="https://github.com/simonw/datasette-copyable/">datasette-copyable</a> - demo <a href="https://covid-19.datasettes.com/covid/ny_times_us_counties.copyable?state=Texas&amp;county=Harris" rel="nofollow">here</a>.</p>
<p>The trick is to run <code>document.execCommand("copy")</code> to copy to the clipboard - but only after first ensuring that some selectable content (probably in a textarea) has been selected.</p>
<p>So you do this:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">var</span> <span class="pl-s1">ta</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">querySelector</span><span class="pl-kos">(</span><span class="pl-s">"textarea"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-k">var</span> <span class="pl-s1">button</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">querySelector</span><span class="pl-kos">(</span><span class="pl-s">"button"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-s1">button</span><span class="pl-kos">.</span><span class="pl-en">onclick</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-s1">ta</span><span class="pl-kos">.</span><span class="pl-en">select</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">execCommand</span><span class="pl-kos">(</span><span class="pl-s">"copy"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">;</span></pre></div>
<p>But that's not quite enough: there's no visible indication that anything happened!</p>
<p>I decided to fix this by updating the text on the button to say "Copied!" and then reverting the button text back again 1.5 seconds later.</p>
<p>I also opted to add the "copy" button using JavaScript, since without JavaScript enabled it won't do anything.</p>
<p>Here's the full HTML and JavaScript snippet I used:</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">div</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;</span><span class="pl-ent">textarea</span> <span class="pl-c1">class</span>="<span class="pl-s">copyable</span>"<span class="pl-kos">&gt;</span>{{ copyable }}<span class="pl-kos">&lt;/</span><span class="pl-ent">textarea</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;</span><span class="pl-ent">p</span> <span class="pl-c1">class</span>="<span class="pl-s">raw-link</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;</span><span class="pl-ent">a</span> <span class="pl-c1">href</span>="<span class="pl-s">{{ raw_link }}</span>"<span class="pl-kos">&gt;</span>Raw data<span class="pl-kos">&lt;/</span><span class="pl-ent">a</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">p</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;/</span><span class="pl-ent">div</span><span class="pl-kos">&gt;</span>

<span class="pl-kos">&lt;</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span>
<span class="pl-k">var</span> <span class="pl-s1">ta</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">querySelector</span><span class="pl-kos">(</span><span class="pl-s">"textarea.copyable"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-k">var</span> <span class="pl-s1">p</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">querySelector</span><span class="pl-kos">(</span><span class="pl-s">"p.raw-link"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-k">var</span> <span class="pl-s1">button</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">createElement</span><span class="pl-kos">(</span><span class="pl-s">"button"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-s1">button</span><span class="pl-kos">.</span><span class="pl-c1">className</span> <span class="pl-c1">=</span> <span class="pl-s">"copyable-copy-button"</span><span class="pl-kos">;</span>
<span class="pl-s1">button</span><span class="pl-kos">.</span><span class="pl-c1">innerHTML</span> <span class="pl-c1">=</span> <span class="pl-s">"Copy to clipboard"</span><span class="pl-kos">;</span>
<span class="pl-s1">button</span><span class="pl-kos">.</span><span class="pl-en">onclick</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-s1">ta</span><span class="pl-kos">.</span><span class="pl-en">select</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">execCommand</span><span class="pl-kos">(</span><span class="pl-s">"copy"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">button</span><span class="pl-kos">.</span><span class="pl-c1">innerHTML</span> <span class="pl-c1">=</span> <span class="pl-s">"Copied!"</span><span class="pl-kos">;</span>
    <span class="pl-en">setTimeout</span><span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
        <span class="pl-s1">button</span><span class="pl-kos">.</span><span class="pl-c1">innerHTML</span> <span class="pl-c1">=</span> <span class="pl-s">"Copy to clipboard"</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span><span class="pl-kos">,</span> <span class="pl-c1">1500</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">;</span>
<span class="pl-s1">p</span><span class="pl-kos">.</span><span class="pl-en">appendChild</span><span class="pl-kos">(</span><span class="pl-s1">button</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-s1">p</span><span class="pl-kos">.</span><span class="pl-en">insertAdjacentElement</span><span class="pl-kos">(</span><span class="pl-s">"afterbegin"</span><span class="pl-kos">,</span> <span class="pl-s1">button</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">&lt;/</span><span class="pl-ent">script</span><span class="pl-kos">&gt;</span></pre></div>


path: javascript_working-around-nodevalue-size-limit.md
topic: javascript
title: Working around the size limit for nodeValue in the DOM
url: https://github.com/simonw/til/blob/main/javascript/working-around-nodevalue-size-limit.md
created: 2020-08-21T16:13:38-07:00
created_utc: 2020-08-21T23:13:38+00:00
updated: 2020-08-21T16:13:38-07:00
updated_utc: 2020-08-21T23:13:38+00:00
shot_hash: b6da0e444bc8f5790c31b8d4866203de
slug: working-around-nodevalue-size-limit
body:
TIL that `nodeValue` in the DOM has a size limit!

I had a table cell element containing HTML-escaped JSON and I was doing this:

```javascript
data = JSON.parse(td.firstChild.nodeValue); 
```

This was breaking on larger JSON strings. It turns out that beyond a certain length limit browsers break up large chunks of text into multiple DOM text nodes.

The solution, [via Stackoverflow](https://stackoverflow.com/questions/4411229/size-limit-to-javascript-node-nodevalue-field), was this:

```javascript
const getFullNodeText = (el) => {
    // https://stackoverflow.com/a/4412151
    if (!el) {
        return '';
    }
    if (typeof(el.textContent) != "undefined") {
        return el.textContent;
    }
    return el.firstChild.nodeValue;
};
```
More details in [this issue](https://github.com/simonw/datasette-leaflet-geojson/issues/12).

html:
<p>TIL that <code>nodeValue</code> in the DOM has a size limit!</p>
<p>I had a table cell element containing HTML-escaped JSON and I was doing this:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-c1">JSON</span><span class="pl-kos">.</span><span class="pl-en">parse</span><span class="pl-kos">(</span><span class="pl-s1">td</span><span class="pl-kos">.</span><span class="pl-c1">firstChild</span><span class="pl-kos">.</span><span class="pl-c1">nodeValue</span><span class="pl-kos">)</span><span class="pl-kos">;</span> </pre></div>
<p>This was breaking on larger JSON strings. It turns out that beyond a certain length limit browsers break up large chunks of text into multiple DOM text nodes.</p>
<p>The solution, <a href="https://stackoverflow.com/questions/4411229/size-limit-to-javascript-node-nodevalue-field" rel="nofollow">via Stackoverflow</a>, was this:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">const</span> <span class="pl-en">getFullNodeText</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-s1">el</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-c">// https://stackoverflow.com/a/4412151</span>
    <span class="pl-k">if</span> <span class="pl-kos">(</span>!<span class="pl-s1">el</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-k">return</span> <span class="pl-s">''</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span>
    <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-k">typeof</span><span class="pl-kos">(</span><span class="pl-s1">el</span><span class="pl-kos">.</span><span class="pl-c1">textContent</span><span class="pl-kos">)</span> != <span class="pl-s">"undefined"</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-k">return</span> <span class="pl-s1">el</span><span class="pl-kos">.</span><span class="pl-c1">textContent</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span>
    <span class="pl-k">return</span> <span class="pl-s1">el</span><span class="pl-kos">.</span><span class="pl-c1">firstChild</span><span class="pl-kos">.</span><span class="pl-c1">nodeValue</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">;</span></pre></div>
<p>More details in <a href="https://github.com/simonw/datasette-leaflet-geojson/issues/12">this issue</a>.</p>


path: javascript_minifying-uglify-npx.md
topic: javascript
title: Minifying JavaScript with npx uglify-js
url: https://github.com/simonw/til/blob/main/javascript/minifying-uglify-npx.md
created: 2020-08-30T11:20:43-07:00
created_utc: 2020-08-30T18:20:43+00:00
updated: 2021-01-24T11:58:15-08:00
updated_utc: 2021-01-24T19:58:15+00:00
shot_hash: dadd1449909a7b8b2c3b9b4588f513e1
slug: minifying-uglify-npx
body:
While [upgrading CodeMirror](https://github.com/simonw/datasette/issues/948) in Datasette I figured out how to minify JavaScript using `uglify-js` on the command line without first installing any teels, using [npx](https://www.npmjs.com/package/npx) (which downloads and executes a CLI tool while skipping the install step):

    npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js

One problem: this stripped out the LICENSE comment at the top of the file.

Turns out you can tell `uglify-js` not to strip comments that match a specific regular expression.

So I edited the CodeMirror file to use a single `/* ... */` comment at the top of the file (instead of multiple `//` lines) and ran Uglify like this:

    npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/'

For CSS I used `clean-css-cli`:

    npx clean-css-cli codemirror-5.57.0.css -o codemirror-5.57.0.min.css

## Using tercer instead

It turns out `uglify-js` doesn't support ES6 at all. You can use [tercer](https://github.com/terser/terser) instead:

    npx terser codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/'

Discovered in [datasette-edit-tables #16](https://github.com/simonw/datasette-edit-tables/issues/16).

## Trying out Uglify interactively

[UglifyJS 3: Online JavaScript minifier](https://skalman.github.io/UglifyJS-online/) is a useful way to try out Ugllify since it shows you the results as you type, which makes it easy to spot tiny changes you can make that result in a shorter minified output.

html:
<p>While <a href="https://github.com/simonw/datasette/issues/948">upgrading CodeMirror</a> in Datasette I figured out how to minify JavaScript using <code>uglify-js</code> on the command line without first installing any teels, using <a href="https://www.npmjs.com/package/npx" rel="nofollow">npx</a> (which downloads and executes a CLI tool while skipping the install step):</p>
<pre><code>npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js
</code></pre>
<p>One problem: this stripped out the LICENSE comment at the top of the file.</p>
<p>Turns out you can tell <code>uglify-js</code> not to strip comments that match a specific regular expression.</p>
<p>So I edited the CodeMirror file to use a single <code>/* ... */</code> comment at the top of the file (instead of multiple <code>//</code> lines) and ran Uglify like this:</p>
<pre><code>npx uglify-js codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/'
</code></pre>
<p>For CSS I used <code>clean-css-cli</code>:</p>
<pre><code>npx clean-css-cli codemirror-5.57.0.css -o codemirror-5.57.0.min.css
</code></pre>
<h2>
<a id="user-content-using-tercer-instead" class="anchor" href="#using-tercer-instead" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using tercer instead</h2>
<p>It turns out <code>uglify-js</code> doesn't support ES6 at all. You can use <a href="https://github.com/terser/terser">tercer</a> instead:</p>
<pre><code>npx terser codemirror-5.57.0.js -o codemirror-5.57.0.min.js --comments '/LICENSE/'
</code></pre>
<p>Discovered in <a href="https://github.com/simonw/datasette-edit-tables/issues/16">datasette-edit-tables #16</a>.</p>
<h2>
<a id="user-content-trying-out-uglify-interactively" class="anchor" href="#trying-out-uglify-interactively" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Trying out Uglify interactively</h2>
<p><a href="https://skalman.github.io/UglifyJS-online/" rel="nofollow">UglifyJS 3: Online JavaScript minifier</a> is a useful way to try out Ugllify since it shows you the results as you type, which makes it easy to spot tiny changes you can make that result in a shorter minified output.</p>


path: javascript_dynamically-loading-assets.md
topic: javascript
title: Dynamically loading multiple assets with a callback
url: https://github.com/simonw/til/blob/main/javascript/dynamically-loading-assets.md
created: 2020-08-21T21:23:56-07:00
created_utc: 2020-08-22T04:23:56+00:00
updated: 2020-08-22T20:27:22-07:00
updated_utc: 2020-08-23T03:27:22+00:00
shot_hash: 6f5cd73bcdd420961bc30e75cdb75ca6
slug: dynamically-loading-assets
body:
For [datasette-leaflet-geojson](https://github.com/simonw/datasette-leaflet-geojson) I wanted to dynamically load some external CSS and JavaScript and then execute some code once they had loaded ([issue 14](https://github.com/simonw/datasette-leaflet-geojson/issues/14)).

I'm not using any frameworks for this so I wanted to do it with vanilla JavaScript.

I wanted to load the CSS and JavaScript in parallel and fire a callback function when both of them had finished loading.

Here's the pattern I came up with:
```javascript
const loadDependencies = (callback) => {
    let loaded = [];
    function hasLoaded() {
        loaded.push(this);
        if (loaded.length == 2) {
            callback();
        }
    }
    let stylesheet = document.createElement('link');
    stylesheet.setAttribute('type', 'text/css');
    stylesheet.setAttribute('rel', 'stylesheet');
    stylesheet.setAttribute('href', 'https://unpkg.com/leaflet@1.5.1/dist/leaflet.css');
    stylesheet.setAttribute('integrity', 'sha512-xwE/Az9zrjBIphAcBb3F6JVqxf46+CDLwfLMHloNu6KEQCAWi6HcDUbeOfBIptF7tcCzusKFjFw2yuvEpDL9wQ==');
    stylesheet.setAttribute('crossorigin', 'anonymous');
    stylesheet.onload = hasLoaded;
    document.head.appendChild(stylesheet);
    let script = document.createElement('script');
    script.src = 'https://unpkg.com/leaflet@1.5.1/dist/leaflet.js';
    script.setAttribute('integrity', 'sha512-GffPMF3RvMeYyc1LWMHtK8EbPv0iNZ8/oTtHPx9/cc2ILxQ+u905qIwdpULaqDkyBKgOaB57QTMg7ztg8Jm2Og==');
    script.setAttribute('crossorigin', 'anonymous');
    script.onload = stylesheet.onload = hasLoaded;
    document.head.appendChild(script);
};
```
Then called like this:
```javascript
loadDependencies(() => {
    /* Code here can use Leaflet */
});
```
## Suggestions from Carl Johnson

[@carlmjohnson commented](https://github.com/simonw/til/commit/f40b5a6483f2979f4973d8af50ff9e9855f2b388#comments):

> Two suggestions. 
> 
> First, the Javascript is overly complicated with pushing `this` (?) to an array. I think the reason you're doing this is because you are subconsciously influenced by Python which has broken nonlocals by default. Even ES5 Javascript, however, respects `var` declarations, and you're using `let`, which is ES6. So, you can radically simplify this:
> 
> ```JavaScript
> const loadDependencies = (callback) => {
>     let loaded = 0;
>     function hasLoaded() {
>         loaded++;
>         if (loaded == 2) {
>             callback();
>         }
>     }
> ```
> 
> Two, you can radically complicate this. :-) The idiomatic way in JavaScript to deal with callbacks is to _convert them to promises as soon as possible_. Again, you're writing ES6 (arrow funcs, `const`, `let`) anyway, plus this is a hobby project, so don't let IE11 hold you back. Just use a promise. Again, I think you're influenced by Python, in which async is a big deal architecturally and you can't just drop it in anywhere. In JS, because it's single threaded by the spec, you can just drop in async any time you like and converting callbacks to Promises is trivial. There are two options for converting to promises, first you can keep your existing API:
> 
> ```JavaScript
> const resolveOnLoad = (el) =>
>   new Promise((resolve) => {
>     el.onload = resolve;
>   });
> 
> const loadDependencies = (callback) => {
>     let stylesheet = document.createElement('link');
>     stylesheet.setAttribute('type', 'text/css');
>     stylesheet.setAttribute('rel', 'stylesheet');
>     stylesheet.setAttribute('href', 'https://unpkg.com/leaflet@1.5.1/dist/leaflet.css');
>     stylesheet.setAttribute('integrity', 'sha512-xwE/Az9zrjBIphAcBb3F6JVqxf46+CDLwfLMHloNu6KEQCAWi6HcDUbeOfBIptF7tcCzusKFjFw2yuvEpDL9wQ==');
>     stylesheet.setAttribute('crossorigin', 'anonymous');
> 
>     let script = document.createElement('script');
>     script.src = 'https://unpkg.com/leaflet@1.5.1/dist/leaflet.js';
>     script.setAttribute('integrity', 'sha512-GffPMF3RvMeYyc1LWMHtK8EbPv0iNZ8/oTtHPx9/cc2ILxQ+u905qIwdpULaqDkyBKgOaB57QTMg7ztg8Jm2Og==');
>     script.setAttribute('crossorigin', 'anonymous');
>     Promise.all([
>         resolveOnLoad(stylesheet),
>         resolveOnLoad(script),
>     ]).then(cb)
>     document.head.appendChild(stylesheet);
>     document.head.appendChild(script);
> };
> ```
> 
> Or second, you just move to all promises/async:
> 
> ```JavaScript
> const resolveOnLoad = (el) =>
>   new Promise((resolve) => {
>     el.onload = resolve;
>   });
> 
> const loadDependencies = async () => {
>   let stylesheet = document.createElement("link");
>   stylesheet.setAttribute("type", "text/css");
>   stylesheet.setAttribute("rel", "stylesheet");
>   stylesheet.setAttribute(
>     "href",
>     "https://unpkg.com/leaflet@1.5.1/dist/leaflet.css"
>   );
>   stylesheet.setAttribute(
>     "integrity",
>     "sha512-xwE/Az9zrjBIphAcBb3F6JVqxf46+CDLwfLMHloNu6KEQCAWi6HcDUbeOfBIptF7tcCzusKFjFw2yuvEpDL9wQ=="
>   );
>   stylesheet.setAttribute("crossorigin", "anonymous");
> 
>   let script = document.createElement("script");
>   script.src = "https://unpkg.com/leaflet@1.5.1/dist/leaflet.js";
>   script.setAttribute(
>     "integrity",
>     "sha512-GffPMF3RvMeYyc1LWMHtK8EbPv0iNZ8/oTtHPx9/cc2ILxQ+u905qIwdpULaqDkyBKgOaB57QTMg7ztg8Jm2Og=="
>   );
>   script.setAttribute("crossorigin", "anonymous");
>   let loaded = Promise.all([resolveOnLoad(stylesheet), resolveOnLoad(script)]);
>   document.head.appendChild(stylesheet);
>   document.head.appendChild(script);
>   await loaded;
> };
> ```
> 
> In the second case, you'd use it as `loadDependencies().then(()=> console.log("loaded"))`.
> 
> In either case, the principle is the same: if you have a callback based API, write a short wrapper to turn it into a promises API, then use that instead, either with .then or async/await, whichever is more convenient at the time.

html:
<p>For <a href="https://github.com/simonw/datasette-leaflet-geojson">datasette-leaflet-geojson</a> I wanted to dynamically load some external CSS and JavaScript and then execute some code once they had loaded (<a href="https://github.com/simonw/datasette-leaflet-geojson/issues/14">issue 14</a>).</p>
<p>I'm not using any frameworks for this so I wanted to do it with vanilla JavaScript.</p>
<p>I wanted to load the CSS and JavaScript in parallel and fire a callback function when both of them had finished loading.</p>
<p>Here's the pattern I came up with:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">const</span> <span class="pl-en">loadDependencies</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-s1">callback</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-k">let</span> <span class="pl-s1">loaded</span> <span class="pl-c1">=</span> <span class="pl-kos">[</span><span class="pl-kos">]</span><span class="pl-kos">;</span>
    <span class="pl-k">function</span> <span class="pl-en">hasLoaded</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-s1">loaded</span><span class="pl-kos">.</span><span class="pl-en">push</span><span class="pl-kos">(</span><span class="pl-smi">this</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
        <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">loaded</span><span class="pl-kos">.</span><span class="pl-c1">length</span> <span class="pl-c1">==</span> <span class="pl-c1">2</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
            <span class="pl-s1">callback</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
        <span class="pl-kos">}</span>
    <span class="pl-kos">}</span>
    <span class="pl-k">let</span> <span class="pl-s1">stylesheet</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">createElement</span><span class="pl-kos">(</span><span class="pl-s">'link'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'type'</span><span class="pl-kos">,</span> <span class="pl-s">'text/css'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'rel'</span><span class="pl-kos">,</span> <span class="pl-s">'stylesheet'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'href'</span><span class="pl-kos">,</span> <span class="pl-s">'https://unpkg.com/leaflet@1.5.1/dist/leaflet.css'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'integrity'</span><span class="pl-kos">,</span> <span class="pl-s">'sha512-xwE/Az9zrjBIphAcBb3F6JVqxf46+CDLwfLMHloNu6KEQCAWi6HcDUbeOfBIptF7tcCzusKFjFw2yuvEpDL9wQ=='</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'crossorigin'</span><span class="pl-kos">,</span> <span class="pl-s">'anonymous'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-c1">onload</span> <span class="pl-c1">=</span> <span class="pl-s1">hasLoaded</span><span class="pl-kos">;</span>
    <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-c1">head</span><span class="pl-kos">.</span><span class="pl-en">appendChild</span><span class="pl-kos">(</span><span class="pl-s1">stylesheet</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-k">let</span> <span class="pl-s1">script</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">createElement</span><span class="pl-kos">(</span><span class="pl-s">'script'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">script</span><span class="pl-kos">.</span><span class="pl-c1">src</span> <span class="pl-c1">=</span> <span class="pl-s">'https://unpkg.com/leaflet@1.5.1/dist/leaflet.js'</span><span class="pl-kos">;</span>
    <span class="pl-s1">script</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'integrity'</span><span class="pl-kos">,</span> <span class="pl-s">'sha512-GffPMF3RvMeYyc1LWMHtK8EbPv0iNZ8/oTtHPx9/cc2ILxQ+u905qIwdpULaqDkyBKgOaB57QTMg7ztg8Jm2Og=='</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">script</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'crossorigin'</span><span class="pl-kos">,</span> <span class="pl-s">'anonymous'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">script</span><span class="pl-kos">.</span><span class="pl-c1">onload</span> <span class="pl-c1">=</span> <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-c1">onload</span> <span class="pl-c1">=</span> <span class="pl-s1">hasLoaded</span><span class="pl-kos">;</span>
    <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-c1">head</span><span class="pl-kos">.</span><span class="pl-en">appendChild</span><span class="pl-kos">(</span><span class="pl-s1">script</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">;</span></pre></div>
<p>Then called like this:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-en">loadDependencies</span><span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-c">/* Code here can use Leaflet */</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<h2>
<a id="user-content-suggestions-from-carl-johnson" class="anchor" href="#suggestions-from-carl-johnson" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Suggestions from Carl Johnson</h2>
<p><a href="https://github.com/simonw/til/commit/f40b5a6483f2979f4973d8af50ff9e9855f2b388#comments">@carlmjohnson commented</a>:</p>
<blockquote>
<p>Two suggestions.</p>
<p>First, the Javascript is overly complicated with pushing <code>this</code> (?) to an array. I think the reason you're doing this is because you are subconsciously influenced by Python which has broken nonlocals by default. Even ES5 Javascript, however, respects <code>var</code> declarations, and you're using <code>let</code>, which is ES6. So, you can radically simplify this:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">const</span> <span class="pl-en">loadDependencies</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-s1">callback</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-k">let</span> <span class="pl-s1">loaded</span> <span class="pl-c1">=</span> <span class="pl-c1">0</span><span class="pl-kos">;</span>
    <span class="pl-k">function</span> <span class="pl-en">hasLoaded</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-s1">loaded</span><span class="pl-c1">++</span><span class="pl-kos">;</span>
        <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">loaded</span> <span class="pl-c1">==</span> <span class="pl-c1">2</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
            <span class="pl-s1">callback</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
        <span class="pl-kos">}</span>
    <span class="pl-kos">}</span><span class="pl-kos"></span></pre></div>
<p>Two, you can radically complicate this. :-) The idiomatic way in JavaScript to deal with callbacks is to <em>convert them to promises as soon as possible</em>. Again, you're writing ES6 (arrow funcs, <code>const</code>, <code>let</code>) anyway, plus this is a hobby project, so don't let IE11 hold you back. Just use a promise. Again, I think you're influenced by Python, in which async is a big deal architecturally and you can't just drop it in anywhere. In JS, because it's single threaded by the spec, you can just drop in async any time you like and converting callbacks to Promises is trivial. There are two options for converting to promises, first you can keep your existing API:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">const</span> <span class="pl-en">resolveOnLoad</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-s1">el</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span>
  <span class="pl-k">new</span> <span class="pl-v">Promise</span><span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-s1">resolve</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-s1">el</span><span class="pl-kos">.</span><span class="pl-c1">onload</span> <span class="pl-c1">=</span> <span class="pl-s1">resolve</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

<span class="pl-k">const</span> <span class="pl-en">loadDependencies</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-s1">callback</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-k">let</span> <span class="pl-s1">stylesheet</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">createElement</span><span class="pl-kos">(</span><span class="pl-s">'link'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'type'</span><span class="pl-kos">,</span> <span class="pl-s">'text/css'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'rel'</span><span class="pl-kos">,</span> <span class="pl-s">'stylesheet'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'href'</span><span class="pl-kos">,</span> <span class="pl-s">'https://unpkg.com/leaflet@1.5.1/dist/leaflet.css'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'integrity'</span><span class="pl-kos">,</span> <span class="pl-s">'sha512-xwE/Az9zrjBIphAcBb3F6JVqxf46+CDLwfLMHloNu6KEQCAWi6HcDUbeOfBIptF7tcCzusKFjFw2yuvEpDL9wQ=='</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'crossorigin'</span><span class="pl-kos">,</span> <span class="pl-s">'anonymous'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

    <span class="pl-k">let</span> <span class="pl-s1">script</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">createElement</span><span class="pl-kos">(</span><span class="pl-s">'script'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">script</span><span class="pl-kos">.</span><span class="pl-c1">src</span> <span class="pl-c1">=</span> <span class="pl-s">'https://unpkg.com/leaflet@1.5.1/dist/leaflet.js'</span><span class="pl-kos">;</span>
    <span class="pl-s1">script</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'integrity'</span><span class="pl-kos">,</span> <span class="pl-s">'sha512-GffPMF3RvMeYyc1LWMHtK8EbPv0iNZ8/oTtHPx9/cc2ILxQ+u905qIwdpULaqDkyBKgOaB57QTMg7ztg8Jm2Og=='</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-s1">script</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">'crossorigin'</span><span class="pl-kos">,</span> <span class="pl-s">'anonymous'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-v">Promise</span><span class="pl-kos">.</span><span class="pl-en">all</span><span class="pl-kos">(</span><span class="pl-kos">[</span>
        <span class="pl-en">resolveOnLoad</span><span class="pl-kos">(</span><span class="pl-s1">stylesheet</span><span class="pl-kos">)</span><span class="pl-kos">,</span>
        <span class="pl-en">resolveOnLoad</span><span class="pl-kos">(</span><span class="pl-s1">script</span><span class="pl-kos">)</span><span class="pl-kos">,</span>
    <span class="pl-kos">]</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">then</span><span class="pl-kos">(</span><span class="pl-s1">cb</span><span class="pl-kos">)</span>
    <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-c1">head</span><span class="pl-kos">.</span><span class="pl-en">appendChild</span><span class="pl-kos">(</span><span class="pl-s1">stylesheet</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-c1">head</span><span class="pl-kos">.</span><span class="pl-en">appendChild</span><span class="pl-kos">(</span><span class="pl-s1">script</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">;</span></pre></div>
<p>Or second, you just move to all promises/async:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">const</span> <span class="pl-en">resolveOnLoad</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-s1">el</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span>
  <span class="pl-k">new</span> <span class="pl-v">Promise</span><span class="pl-kos">(</span><span class="pl-kos">(</span><span class="pl-s1">resolve</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-s1">el</span><span class="pl-kos">.</span><span class="pl-c1">onload</span> <span class="pl-c1">=</span> <span class="pl-s1">resolve</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

<span class="pl-k">const</span> <span class="pl-en">loadDependencies</span> <span class="pl-c1">=</span> <span class="pl-k">async</span> <span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
  <span class="pl-k">let</span> <span class="pl-s1">stylesheet</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">createElement</span><span class="pl-kos">(</span><span class="pl-s">"link"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">"type"</span><span class="pl-kos">,</span> <span class="pl-s">"text/css"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">"rel"</span><span class="pl-kos">,</span> <span class="pl-s">"stylesheet"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span>
    <span class="pl-s">"href"</span><span class="pl-kos">,</span>
    <span class="pl-s">"https://unpkg.com/leaflet@1.5.1/dist/leaflet.css"</span>
  <span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span>
    <span class="pl-s">"integrity"</span><span class="pl-kos">,</span>
    <span class="pl-s">"sha512-xwE/Az9zrjBIphAcBb3F6JVqxf46+CDLwfLMHloNu6KEQCAWi6HcDUbeOfBIptF7tcCzusKFjFw2yuvEpDL9wQ=="</span>
  <span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-s1">stylesheet</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">"crossorigin"</span><span class="pl-kos">,</span> <span class="pl-s">"anonymous"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

  <span class="pl-k">let</span> <span class="pl-s1">script</span> <span class="pl-c1">=</span> <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">createElement</span><span class="pl-kos">(</span><span class="pl-s">"script"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-s1">script</span><span class="pl-kos">.</span><span class="pl-c1">src</span> <span class="pl-c1">=</span> <span class="pl-s">"https://unpkg.com/leaflet@1.5.1/dist/leaflet.js"</span><span class="pl-kos">;</span>
  <span class="pl-s1">script</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span>
    <span class="pl-s">"integrity"</span><span class="pl-kos">,</span>
    <span class="pl-s">"sha512-GffPMF3RvMeYyc1LWMHtK8EbPv0iNZ8/oTtHPx9/cc2ILxQ+u905qIwdpULaqDkyBKgOaB57QTMg7ztg8Jm2Og=="</span>
  <span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-s1">script</span><span class="pl-kos">.</span><span class="pl-en">setAttribute</span><span class="pl-kos">(</span><span class="pl-s">"crossorigin"</span><span class="pl-kos">,</span> <span class="pl-s">"anonymous"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">let</span> <span class="pl-s1">loaded</span> <span class="pl-c1">=</span> <span class="pl-v">Promise</span><span class="pl-kos">.</span><span class="pl-en">all</span><span class="pl-kos">(</span><span class="pl-kos">[</span><span class="pl-en">resolveOnLoad</span><span class="pl-kos">(</span><span class="pl-s1">stylesheet</span><span class="pl-kos">)</span><span class="pl-kos">,</span> <span class="pl-en">resolveOnLoad</span><span class="pl-kos">(</span><span class="pl-s1">script</span><span class="pl-kos">)</span><span class="pl-kos">]</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-c1">head</span><span class="pl-kos">.</span><span class="pl-en">appendChild</span><span class="pl-kos">(</span><span class="pl-s1">stylesheet</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-c1">head</span><span class="pl-kos">.</span><span class="pl-en">appendChild</span><span class="pl-kos">(</span><span class="pl-s1">script</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
  <span class="pl-k">await</span> <span class="pl-s1">loaded</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">;</span></pre></div>
<p>In the second case, you'd use it as <code>loadDependencies().then(()=&gt; console.log("loaded"))</code>.</p>
<p>In either case, the principle is the same: if you have a callback based API, write a short wrapper to turn it into a promises API, then use that instead, either with .then or async/await, whichever is more convenient at the time.</p>
</blockquote>


path: node_constant-time-compare-strings.md
topic: node
title: Constant-time comparison of strings in Node
url: https://github.com/simonw/til/blob/main/node/constant-time-compare-strings.md
created: 2020-05-01T19:18:32-07:00
created_utc: 2020-05-02T02:18:32+00:00
updated: 2020-05-01T19:19:01-07:00
updated_utc: 2020-05-02T02:19:01+00:00
shot_hash: 8f0e382c05254168f275abc233396722
slug: constant-time-compare-strings
body:
When comparing secrets, passwords etc it's important to use a constant-time compare function to avoid timing attacks.

In Python I use `secrets.compare_digest(a, b)`, [documented here](https://docs.python.org/3/library/secrets.html#secrets.compare_digest).

I needed an equivalent in Node.js today. It has a [crypto.timingSafeEqual() function](https://nodejs.org/api/crypto.html#crypto_crypto_timingsafeequal_a_b) but it's a little tricky to use: it requires arguments that are `Buffer`, `TypedArray` or `DataView` and it throws an exception if they are not the same length.

I figured out this wrapper function so I can operate against strings of varying length:

```javascript
const { timingSafeEqual } = require('crypto');

const compare = (a, b) => {
    try {
        return timingSafeEqual(Buffer.from(a, "utf8"), Buffer.from(b, "utf8"));
    } catch {
        return false;
    }
};
```

html:
<p>When comparing secrets, passwords etc it's important to use a constant-time compare function to avoid timing attacks.</p>
<p>In Python I use <code>secrets.compare_digest(a, b)</code>, <a href="https://docs.python.org/3/library/secrets.html#secrets.compare_digest" rel="nofollow">documented here</a>.</p>
<p>I needed an equivalent in Node.js today. It has a <a href="https://nodejs.org/api/crypto.html#crypto_crypto_timingsafeequal_a_b" rel="nofollow">crypto.timingSafeEqual() function</a> but it's a little tricky to use: it requires arguments that are <code>Buffer</code>, <code>TypedArray</code> or <code>DataView</code> and it throws an exception if they are not the same length.</p>
<p>I figured out this wrapper function so I can operate against strings of varying length:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">const</span> <span class="pl-kos">{</span> timingSafeEqual <span class="pl-kos">}</span> <span class="pl-c1">=</span> <span class="pl-en">require</span><span class="pl-kos">(</span><span class="pl-s">'crypto'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>

<span class="pl-k">const</span> <span class="pl-en">compare</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-s1">a</span><span class="pl-kos">,</span> <span class="pl-s1">b</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-k">try</span> <span class="pl-kos">{</span>
        <span class="pl-k">return</span> <span class="pl-en">timingSafeEqual</span><span class="pl-kos">(</span><span class="pl-v">Buffer</span><span class="pl-kos">.</span><span class="pl-en">from</span><span class="pl-kos">(</span><span class="pl-s1">a</span><span class="pl-kos">,</span> <span class="pl-s">"utf8"</span><span class="pl-kos">)</span><span class="pl-kos">,</span> <span class="pl-v">Buffer</span><span class="pl-kos">.</span><span class="pl-en">from</span><span class="pl-kos">(</span><span class="pl-s1">b</span><span class="pl-kos">,</span> <span class="pl-s">"utf8"</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span> <span class="pl-k">catch</span> <span class="pl-kos">{</span>
        <span class="pl-k">return</span> <span class="pl-c1">false</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span>
<span class="pl-kos">}</span><span class="pl-kos">;</span></pre></div>


path: zsh_custom-zsh-prompt.md
topic: zsh
title: Customizing my zsh prompt
url: https://github.com/simonw/til/blob/main/zsh/custom-zsh-prompt.md
created: 2020-08-12T11:52:30-07:00
created_utc: 2020-08-12T18:52:30+00:00
updated: 2020-08-12T11:52:30-07:00
updated_utc: 2020-08-12T18:52:30+00:00
shot_hash: d614274b96e3b17fb5f7e567e27e8d53
slug: custom-zsh-prompt
body:
I got fed up of the default macOS `zsh` prompt:

    simon@Simons-MacBook-Pro ~ % 

Mainly because I like copying and pasting terminal examples into GitHub issues.

I changed it to this:

    ~ % cd /tmp
    /tmp % 

By adding this line to the top of my `~/.zshrc` file:

    PROMPT='%1~ %# '

html:
<p>I got fed up of the default macOS <code>zsh</code> prompt:</p>
<pre><code>simon@Simons-MacBook-Pro ~ % 
</code></pre>
<p>Mainly because I like copying and pasting terminal examples into GitHub issues.</p>
<p>I changed it to this:</p>
<pre><code>~ % cd /tmp
/tmp % 
</code></pre>
<p>By adding this line to the top of my <code>~/.zshrc</code> file:</p>
<pre><code>PROMPT='%1~ %# '
</code></pre>


path: jq_reformatting-airtable-json.md
topic: jq
title: Converting Airtable JSON for use with sqlite-utils using jq
url: https://github.com/simonw/til/blob/main/jq/reformatting-airtable-json.md
created: 2020-08-28T21:04:44-07:00
created_utc: 2020-08-29T04:04:44+00:00
updated: 2020-08-29T12:01:38-07:00
updated_utc: 2020-08-29T19:01:38+00:00
shot_hash: a06b4c37340ae89e09a9b60c57a284b8
slug: reformatting-airtable-json
body:
The Airtable API outputs JSON that looks like this:

```json
{
  "records": [
    {
      "id": "rec0kmRncBsXrJBSe",
      "fields": {
        "area": "Morro bay",
        "Name": "North Point Beach",
        "body": "North Point Natural Area has some great tide pools at low tide. They are located north of Morro Strand State Beach between the towns of Morro Bay and Cayucos.",
        "website_url": "http://www.californiabeaches.com/beach/north-point-beach/"
      },
      "createdTime": "2017-11-26T01:54:47.000Z"
    },
    {
      "id": "rec0sW0dbM6X1bVRr",
      "fields": {
        "area": "Pismo beach",
        "Name": "Spyglass beach",
        "body": "The Shell Beach area has many tidepool areas along its rugged shoreline. Head to just about any of the beaches here at low tide and you’ll find excellent tide pools. Our favorite access point for tide pools is the one at Spyglass Park.",
        "website_url": "http://www.californiabeaches.com/beach/spyglass-park/"
      },
      "createdTime": "2017-11-26T01:57:33.000Z"
    }
  ]
}
```
The items in the `records` array each have an `id`, a nested `fields` dictionary and a `createdTime`.

I want to pipe these into `sqlite-utils insert` - but that requires a flat JSON list of dictionaries like this:

```json
[
  {
    "id": "rec0kmRncBsXrJBSe",
    "area": "Morro bay",
    "Name": "North Point Beach",
    "body": "North Point Natural Area has some great tide pools at low tide. They are located north of Morro Strand State Beach between the towns of Morro Bay and Cayucos.",
    "website_url": "http://www.californiabeaches.com/beach/north-point-beach/",
    "createdTime": "2017-11-26T01:54:47.000Z"
  },
  {
    "id": "rec0sW0dbM6X1bVRr",
    "area": "Pismo beach",
    "Name": "Spyglass beach",
    "body": "The Shell Beach area has many tidepool areas along its rugged shoreline. Head to just about any of the beaches here at low tide and you’ll find excellent tide pools. Our favorite access point for tide pools is the one at Spyglass Park.",
    "website_url": "http://www.californiabeaches.com/beach/spyglass-park/",
    "createdTime": "2017-11-26T01:57:33.000Z"
  }
]
```
Can I use https://stedolan.github.io/jq/ to convert between the two? it turns out I can:

    cat airtable.json | jq '[.records[] | {id: .id} + .fields + {createdTime: .createdTime}]'

This loops through each item in the `.records` list and for each one creates a new JSON object that concatenates the `id`, all of the `.fields` and the `createdTime`.

I can then pipe that into `sqlite-utils` to create a SQLite table from it. Here's the full recipe:

    curl "https://api.airtable.com/v0/appZOGvNJPXCQ205F/Tidepool%20areas" \
      -H "Authorization: Bearer xxx" | \
      jq '[.records[] | {id: .id} + .fields + {createdTime: .createdTime}]' | \
      sqlite-utils insert tidepools.db tidepools - --pk=id

html:
<p>The Airtable API outputs JSON that looks like this:</p>
<div class="highlight highlight-source-json"><pre>{
  <span class="pl-s"><span class="pl-pds">"</span>records<span class="pl-pds">"</span></span>: [
    {
      <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>rec0kmRncBsXrJBSe<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>fields<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>area<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Morro bay<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>Name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>North Point Beach<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>body<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>North Point Natural Area has some great tide pools at low tide. They are located north of Morro Strand State Beach between the towns of Morro Bay and Cayucos.<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>website_url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>http://www.californiabeaches.com/beach/north-point-beach/<span class="pl-pds">"</span></span>
      },
      <span class="pl-s"><span class="pl-pds">"</span>createdTime<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2017-11-26T01:54:47.000Z<span class="pl-pds">"</span></span>
    },
    {
      <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>rec0sW0dbM6X1bVRr<span class="pl-pds">"</span></span>,
      <span class="pl-s"><span class="pl-pds">"</span>fields<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>area<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Pismo beach<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>Name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Spyglass beach<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>body<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>The Shell Beach area has many tidepool areas along its rugged shoreline. Head to just about any of the beaches here at low tide and you’ll find excellent tide pools. Our favorite access point for tide pools is the one at Spyglass Park.<span class="pl-pds">"</span></span>,
        <span class="pl-s"><span class="pl-pds">"</span>website_url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>http://www.californiabeaches.com/beach/spyglass-park/<span class="pl-pds">"</span></span>
      },
      <span class="pl-s"><span class="pl-pds">"</span>createdTime<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2017-11-26T01:57:33.000Z<span class="pl-pds">"</span></span>
    }
  ]
}</pre></div>
<p>The items in the <code>records</code> array each have an <code>id</code>, a nested <code>fields</code> dictionary and a <code>createdTime</code>.</p>
<p>I want to pipe these into <code>sqlite-utils insert</code> - but that requires a flat JSON list of dictionaries like this:</p>
<div class="highlight highlight-source-json"><pre>[
  {
    <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>rec0kmRncBsXrJBSe<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>area<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Morro bay<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>Name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>North Point Beach<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>body<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>North Point Natural Area has some great tide pools at low tide. They are located north of Morro Strand State Beach between the towns of Morro Bay and Cayucos.<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>website_url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>http://www.californiabeaches.com/beach/north-point-beach/<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>createdTime<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2017-11-26T01:54:47.000Z<span class="pl-pds">"</span></span>
  },
  {
    <span class="pl-s"><span class="pl-pds">"</span>id<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>rec0sW0dbM6X1bVRr<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>area<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Pismo beach<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>Name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Spyglass beach<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>body<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>The Shell Beach area has many tidepool areas along its rugged shoreline. Head to just about any of the beaches here at low tide and you’ll find excellent tide pools. Our favorite access point for tide pools is the one at Spyglass Park.<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>website_url<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>http://www.californiabeaches.com/beach/spyglass-park/<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>createdTime<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2017-11-26T01:57:33.000Z<span class="pl-pds">"</span></span>
  }
]</pre></div>
<p>Can I use <a href="https://stedolan.github.io/jq/" rel="nofollow">https://stedolan.github.io/jq/</a> to convert between the two? it turns out I can:</p>
<pre><code>cat airtable.json | jq '[.records[] | {id: .id} + .fields + {createdTime: .createdTime}]'
</code></pre>
<p>This loops through each item in the <code>.records</code> list and for each one creates a new JSON object that concatenates the <code>id</code>, all of the <code>.fields</code> and the <code>createdTime</code>.</p>
<p>I can then pipe that into <code>sqlite-utils</code> to create a SQLite table from it. Here's the full recipe:</p>
<pre><code>curl "https://api.airtable.com/v0/appZOGvNJPXCQ205F/Tidepool%20areas" \
  -H "Authorization: Bearer xxx" | \
  jq '[.records[] | {id: .id} + .fields + {createdTime: .createdTime}]' | \
  sqlite-utils insert tidepools.db tidepools - --pk=id
</code></pre>


path: django_postgresql-full-text-search-admin.md
topic: django
title: PostgreSQL full-text search in the Django Admin
url: https://github.com/simonw/til/blob/main/django/postgresql-full-text-search-admin.md
created: 2020-07-25T15:36:17-07:00
created_utc: 2020-07-25T22:36:17+00:00
updated: 2020-07-25T15:36:17-07:00
updated_utc: 2020-07-25T22:36:17+00:00
shot_hash: 8239440a6854c5c8b57e7d7f3ca75098
slug: postgresql-full-text-search-admin
body:
Django 3.1 introduces PostgreSQL `search_type="websearch"` - which gives you search with advanced operators like `"phrase search" -excluding`. James Turk [wrote about this here](https://jamesturk.net/posts/websearch-in-django-31/), and it's also in [my weeknotes](https://simonwillison.net/2020/Jul/23/datasette-copyable-datasette-insert-api/).

I decided to add it to my Django Admin interface. It was _really easy_ using the `get_search_results()` model admin method, [documented here](https://docs.djangoproject.com/en/3.0/ref/contrib/admin/#django.contrib.admin.ModelAdmin.get_search_results).

My models already have a `search_document` full-text search column, as described in [Implementing faceted search with Django and PostgreSQL](https://simonwillison.net/2017/Oct/5/django-postgresql-faceted-search/). So all I needed to add to my `ModelAdmin` subclasses was this:

```python
    def get_search_results(self, request, queryset, search_term):
        if not search_term:
            return super().get_search_results(
                request, queryset, search_term
            )
        query = SearchQuery(search_term, search_type="websearch")
        rank = SearchRank(F("search_document"), query)
        queryset = (
            queryset
            .annotate(rank=rank)
            .filter(search_document=query)
            .order_by("-rank")
        )
        return queryset, False
```
Here's [the full implementation](https://github.com/simonw/simonwillisonblog/blob/6c0de9f9976ef831fe92106be662d77dfe80b32a/blog/admin.py) for my personal blog.

html:
<p>Django 3.1 introduces PostgreSQL <code>search_type="websearch"</code> - which gives you search with advanced operators like <code>"phrase search" -excluding</code>. James Turk <a href="https://jamesturk.net/posts/websearch-in-django-31/" rel="nofollow">wrote about this here</a>, and it's also in <a href="https://simonwillison.net/2020/Jul/23/datasette-copyable-datasette-insert-api/" rel="nofollow">my weeknotes</a>.</p>
<p>I decided to add it to my Django Admin interface. It was <em>really easy</em> using the <code>get_search_results()</code> model admin method, <a href="https://docs.djangoproject.com/en/3.0/ref/contrib/admin/#django.contrib.admin.ModelAdmin.get_search_results" rel="nofollow">documented here</a>.</p>
<p>My models already have a <code>search_document</code> full-text search column, as described in <a href="https://simonwillison.net/2017/Oct/5/django-postgresql-faceted-search/" rel="nofollow">Implementing faceted search with Django and PostgreSQL</a>. So all I needed to add to my <code>ModelAdmin</code> subclasses was this:</p>
<div class="highlight highlight-source-python"><pre>    <span class="pl-k">def</span> <span class="pl-en">get_search_results</span>(<span class="pl-s1">self</span>, <span class="pl-s1">request</span>, <span class="pl-s1">queryset</span>, <span class="pl-s1">search_term</span>):
        <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">search_term</span>:
            <span class="pl-k">return</span> <span class="pl-en">super</span>().<span class="pl-en">get_search_results</span>(
                <span class="pl-s1">request</span>, <span class="pl-s1">queryset</span>, <span class="pl-s1">search_term</span>
            )
        <span class="pl-s1">query</span> <span class="pl-c1">=</span> <span class="pl-v">SearchQuery</span>(<span class="pl-s1">search_term</span>, <span class="pl-s1">search_type</span><span class="pl-c1">=</span><span class="pl-s">"websearch"</span>)
        <span class="pl-s1">rank</span> <span class="pl-c1">=</span> <span class="pl-v">SearchRank</span>(<span class="pl-v">F</span>(<span class="pl-s">"search_document"</span>), <span class="pl-s1">query</span>)
        <span class="pl-s1">queryset</span> <span class="pl-c1">=</span> (
            <span class="pl-s1">queryset</span>
            .<span class="pl-en">annotate</span>(<span class="pl-s1">rank</span><span class="pl-c1">=</span><span class="pl-s1">rank</span>)
            .<span class="pl-en">filter</span>(<span class="pl-s1">search_document</span><span class="pl-c1">=</span><span class="pl-s1">query</span>)
            .<span class="pl-en">order_by</span>(<span class="pl-s">"-rank"</span>)
        )
        <span class="pl-k">return</span> <span class="pl-s1">queryset</span>, <span class="pl-c1">False</span></pre></div>
<p>Here's <a href="https://github.com/simonw/simonwillisonblog/blob/6c0de9f9976ef831fe92106be662d77dfe80b32a/blog/admin.py">the full implementation</a> for my personal blog.</p>


path: ics_google-calendar-ics-subscribe-link.md
topic: ics
title: Providing a "subscribe in Google Calendar" link for an ics feed
url: https://github.com/simonw/til/blob/main/ics/google-calendar-ics-subscribe-link.md
created: 2020-08-21T22:42:11-07:00
created_utc: 2020-08-22T05:42:11+00:00
updated: 2022-07-30T17:15:46-07:00
updated_utc: 2022-07-31T00:15:46+00:00
shot_hash: f53fdea874e6579724326415286e19d3
slug: google-calendar-ics-subscribe-link
body:
If you provide your own custom generated ICS file hosted at a URL it's nice to be able to give Google Calendar users an easy way to subscribe to that feed.

As far as I can tell this isn't documented anywhere, but it is possible.

The format is `https://www.google.com/calendar/render?cid=webcal://` followed by your URL.

For example: https://www.google.com/calendar/render?cid=webcal://pretalx.com/pycon-au-2020/schedule/export/schedule.ics

This drops the user into the a Google Calendar interface with a prompt for them to confirm their subscription.

I believe this only works if you are serving your ICS feed over HTTPS.

## The catch: polling frequency

Unfortunately as far as I can tell Google Calendar polls for updates to the feed less than once every 24 hours. This is frustrating, as it makes this feature far less useful than it could be.

html:
<p>If you provide your own custom generated ICS file hosted at a URL it's nice to be able to give Google Calendar users an easy way to subscribe to that feed.</p>
<p>As far as I can tell this isn't documented anywhere, but it is possible.</p>
<p>The format is <code>https://www.google.com/calendar/render?cid=webcal://</code> followed by your URL.</p>
<p>For example: <a href="https://www.google.com/calendar/render?cid=webcal://pretalx.com/pycon-au-2020/schedule/export/schedule.ics" rel="nofollow">https://www.google.com/calendar/render?cid=webcal://pretalx.com/pycon-au-2020/schedule/export/schedule.ics</a></p>
<p>This drops the user into the a Google Calendar interface with a prompt for them to confirm their subscription.</p>
<p>I believe this only works if you are serving your ICS feed over HTTPS.</p>
<h2>
<a id="user-content-the-catch-polling-frequency" class="anchor" href="#the-catch-polling-frequency" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>The catch: polling frequency</h2>
<p>Unfortunately as far as I can tell Google Calendar polls for updates to the feed less than once every 24 hours. This is frustrating, as it makes this feature far less useful than it could be.</p>


path: github-actions_set-environment-for-all-steps.md
topic: github-actions
title: Set environment variables for all steps in a GitHub Action
url: https://github.com/simonw/til/blob/main/github-actions/set-environment-for-all-steps.md
created: 2020-04-19T07:55:08-07:00
created_utc: 2020-04-19T14:55:08+00:00
updated: 2020-04-19T07:55:08-07:00
updated_utc: 2020-04-19T14:55:08+00:00
shot_hash: f81af954d433fbad9fd249ea995abe2c
slug: set-environment-for-all-steps
body:
From [this example](https://github.com/GoogleCloudPlatform/github-actions/blob/20c294aabd5331f9f7b8a26e6075d41c31ce5e0d/example-workflows/cloud-run/.github/workflows/cloud-run.yml) I learned that you can set environment variables such that they will be available in ALL jobs once at the top of a workflow:

```yaml
name: Build and Deploy to Cloud Run

on:
  push:
    branches:
    - master

env:
  PROJECT_ID: ${{ secrets.RUN_PROJECT }}
  RUN_REGION: us-central1
  SERVICE_NAME: helloworld-nodejs
```

I had previously been using this [much more verbose pattern](https://github.com/simonw/big-local-datasette/blob/181de90f1e7b59c7727595ee8cbe7626667fe05a/.github/workflows/deploy.yml#L30-L42):

```yaml
    - name: Fetch projects
      env:
        BIGLOCAL_TOKEN: ${{ secrets.BIGLOCAL_TOKEN }}
      run: python fetch_projects.py dbs/biglocal.db $BIGLOCAL_TOKEN --contact ...
```

html:
<p>From <a href="https://github.com/GoogleCloudPlatform/github-actions/blob/20c294aabd5331f9f7b8a26e6075d41c31ce5e0d/example-workflows/cloud-run/.github/workflows/cloud-run.yml">this example</a> I learned that you can set environment variables such that they will be available in ALL jobs once at the top of a workflow:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Build and Deploy to Cloud Run</span>

<span class="pl-ent">on</span>:
  <span class="pl-ent">push</span>:
    <span class="pl-ent">branches</span>:
    - <span class="pl-s">master</span>

<span class="pl-ent">env</span>:
  <span class="pl-ent">PROJECT_ID</span>: <span class="pl-s">${{ secrets.RUN_PROJECT }}</span>
  <span class="pl-ent">RUN_REGION</span>: <span class="pl-s">us-central1</span>
  <span class="pl-ent">SERVICE_NAME</span>: <span class="pl-s">helloworld-nodejs</span></pre></div>
<p>I had previously been using this <a href="https://github.com/simonw/big-local-datasette/blob/181de90f1e7b59c7727595ee8cbe7626667fe05a/.github/workflows/deploy.yml#L30-L42">much more verbose pattern</a>:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Fetch projects</span>
      <span class="pl-ent">env</span>:
        <span class="pl-ent">BIGLOCAL_TOKEN</span>: <span class="pl-s">${{ secrets.BIGLOCAL_TOKEN }}</span>
      <span class="pl-ent">run</span>: <span class="pl-s">python fetch_projects.py dbs/biglocal.db $BIGLOCAL_TOKEN --contact ...</span></pre></div>


path: github-actions_different-steps-on-a-schedule.md
topic: github-actions
title: Running different steps on a schedule
url: https://github.com/simonw/til/blob/main/github-actions/different-steps-on-a-schedule.md
created: 2020-04-20T07:39:39-07:00
created_utc: 2020-04-20T14:39:39+00:00
updated: 2020-04-20T09:24:01-07:00
updated_utc: 2020-04-20T16:24:01+00:00
shot_hash: cc627abd8d10103171280dad5925be05
slug: different-steps-on-a-schedule
body:
Say you have a workflow that runs hourly, but once a day you want the workflow to run slightly differently - without duplicating the entire workflow.

Thanks to @BrightRan, here's [the solution](https://github.community/t5/GitHub-Actions/Schedule-once-an-hour-but-do-something-different-once-a-day/m-p/54382/highlight/true#M9168). Use the following pattern in an `if:` condition for a step:

    github.event_name == 'schedule' && github.event.schedule == '20 17 * * *'

Longer example:

```yaml
name: Fetch updated data and deploy

on:
  push:
  schedule:
    - cron: '5,35 * * * *'
    - cron: '20 17 * * *'

jobs:
  build_and_deploy:
    runs-on: ubuntu-latest
    steps:
    # ...
    - name: Download existing .db files
      if: |-
        !(github.event_name == 'schedule' && github.event.schedule == '20 17 * * *')
      env:
        DATASETTE_TOKEN: ${{ secrets.DATASETTE_TOKEN }}
      run: |-
        datasette-clone https://biglocal.datasettes.com/ dbs -v --token=$DATASETTE_TOKEN
```
I used this [here](https://github.com/simonw/big-local-datasette/blob/35e1acd4d9859d3af2feb29d0744ce1550e5faec/.github/workflows/deploy.yml), see [#11](https://github.com/simonw/big-local-datasette/issues/11).

html:
<p>Say you have a workflow that runs hourly, but once a day you want the workflow to run slightly differently - without duplicating the entire workflow.</p>
<p>Thanks to @BrightRan, here's <a href="https://github.community/t5/GitHub-Actions/Schedule-once-an-hour-but-do-something-different-once-a-day/m-p/54382/highlight/true#M9168" rel="nofollow">the solution</a>. Use the following pattern in an <code>if:</code> condition for a step:</p>
<pre><code>github.event_name == 'schedule' &amp;&amp; github.event.schedule == '20 17 * * *'
</code></pre>
<p>Longer example:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Fetch updated data and deploy</span>

<span class="pl-ent">on</span>:
  <span class="pl-ent">push</span>:
  <span class="pl-ent">schedule</span>:
    - <span class="pl-ent">cron</span>: <span class="pl-s"><span class="pl-pds">'</span>5,35 * * * *<span class="pl-pds">'</span></span>
    - <span class="pl-ent">cron</span>: <span class="pl-s"><span class="pl-pds">'</span>20 17 * * *<span class="pl-pds">'</span></span>

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">build_and_deploy</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">steps</span>:
    <span class="pl-c"><span class="pl-c">#</span> ...</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Download existing .db files</span>
      <span class="pl-ent">if</span>: <span class="pl-s">|-</span>
<span class="pl-s">        !(github.event_name == 'schedule' &amp;&amp; github.event.schedule == '20 17 * * *')</span>
<span class="pl-s"></span>      <span class="pl-ent">env</span>:
        <span class="pl-ent">DATASETTE_TOKEN</span>: <span class="pl-s">${{ secrets.DATASETTE_TOKEN }}</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        datasette-clone https://biglocal.datasettes.com/ dbs -v --token=$DATASETTE_TOKEN</span></pre></div>
<p>I used this <a href="https://github.com/simonw/big-local-datasette/blob/35e1acd4d9859d3af2feb29d0744ce1550e5faec/.github/workflows/deploy.yml">here</a>, see <a href="https://github.com/simonw/big-local-datasette/issues/11">#11</a>.</p>


path: github-actions_only-master.md
topic: github-actions
title: Only run GitHub Action on push to master / main
url: https://github.com/simonw/til/blob/main/github-actions/only-master.md
created: 2020-04-19T07:46:31-07:00
created_utc: 2020-04-19T14:46:31+00:00
updated: 2022-03-21T18:42:37-07:00
updated_utc: 2022-03-22T01:42:37+00:00
shot_hash: f73b9c6854742fe44497599c9ca6568f
slug: only-master
body:
Spotted in [this Cloud Run example](https://github.com/GoogleCloudPlatform/github-actions/blob/20c294aabd5331f9f7b8a26e6075d41c31ce5e0d/example-workflows/cloud-run/.github/workflows/cloud-run.yml):

```yaml
name: Build and Deploy to Cloud Run

on:
  push:
    branches:
    - master
```

Useful if you don't want people opening pull requests against your repo that inadvertantly trigger a deploy action!

An alternative mechanism I've used is to gate the specific deploy steps in the action, [like this](https://github.com/simonw/cryptozoology/blob/8a86ec283823c91ad42c5f737a912d43791d427f/.github/workflows/deploy.yml#L31-L40).

```yaml
    # Only run the deploy if push was to master
    - name: Set up Cloud Run
      if: github.ref == 'refs/heads/master'
      uses: GoogleCloudPlatform/github-actions/setup-gcloud@v0
      with:
        version: '275.0.0'
        service_account_email: ${{ secrets.GCP_SA_EMAIL }}
        service_account_key: ${{ secrets.GCP_SA_KEY }}
    - name: Deploy to Cloud Run
      if: github.ref == 'refs/heads/master'
      run: |-
        gcloud config set run/region us-central1
```

html:
<p>Spotted in <a href="https://github.com/GoogleCloudPlatform/github-actions/blob/20c294aabd5331f9f7b8a26e6075d41c31ce5e0d/example-workflows/cloud-run/.github/workflows/cloud-run.yml">this Cloud Run example</a>:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Build and Deploy to Cloud Run</span>

<span class="pl-ent">on</span>:
  <span class="pl-ent">push</span>:
    <span class="pl-ent">branches</span>:
    - <span class="pl-s">master</span></pre></div>
<p>Useful if you don't want people opening pull requests against your repo that inadvertantly trigger a deploy action!</p>
<p>An alternative mechanism I've used is to gate the specific deploy steps in the action, <a href="https://github.com/simonw/cryptozoology/blob/8a86ec283823c91ad42c5f737a912d43791d427f/.github/workflows/deploy.yml#L31-L40">like this</a>.</p>
<div class="highlight highlight-source-yaml"><pre>    <span class="pl-c"><span class="pl-c">#</span> Only run the deploy if push was to master</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Set up Cloud Run</span>
      <span class="pl-ent">if</span>: <span class="pl-s">github.ref == 'refs/heads/master'</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">GoogleCloudPlatform/github-actions/setup-gcloud@v0</span>
      <span class="pl-ent">with</span>:
        <span class="pl-ent">version</span>: <span class="pl-s"><span class="pl-pds">'</span>275.0.0<span class="pl-pds">'</span></span>
        <span class="pl-ent">service_account_email</span>: <span class="pl-s">${{ secrets.GCP_SA_EMAIL }}</span>
        <span class="pl-ent">service_account_key</span>: <span class="pl-s">${{ secrets.GCP_SA_KEY }}</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Deploy to Cloud Run</span>
      <span class="pl-ent">if</span>: <span class="pl-s">github.ref == 'refs/heads/master'</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        gcloud config set run/region us-central1</span></pre></div>


path: github-actions_dump-context.md
topic: github-actions
title: Dump out all GitHub Actions context
url: https://github.com/simonw/til/blob/main/github-actions/dump-context.md
created: 2020-04-19T07:50:03-07:00
created_utc: 2020-04-19T14:50:03+00:00
updated: 2020-04-19T07:50:03-07:00
updated_utc: 2020-04-19T14:50:03+00:00
shot_hash: 070e1fa70411ed2f9cd92ea28dc399e2
slug: dump-context
body:
Useful for seeing what's available for `if: ` conditions (see [context and expression syntax](https://help.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions)).

I copied this example action [from here](https://help.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#example-printing-context-information-to-the-log-file) and deployed it [here](https://github.com/simonw/playing-with-actions/blob/master/.github/workflows/dump-context.yml). Here's an [example run](https://github.com/simonw/playing-with-actions/runs/599575180?check_suite_focus=true).

```yaml
on: push

jobs:
  one:
    runs-on: ubuntu-16.04
    steps:
      - name: Dump GitHub context
        env:
          GITHUB_CONTEXT: ${{ toJson(github) }}
        run: echo "$GITHUB_CONTEXT"
      - name: Dump job context
        env:
          JOB_CONTEXT: ${{ toJson(job) }}
        run: echo "$JOB_CONTEXT"
      - name: Dump steps context
        env:
          STEPS_CONTEXT: ${{ toJson(steps) }}
        run: echo "$STEPS_CONTEXT"
      - name: Dump runner context
        env:
          RUNNER_CONTEXT: ${{ toJson(runner) }}
        run: echo "$RUNNER_CONTEXT"
      - name: Dump strategy context
        env:
          STRATEGY_CONTEXT: ${{ toJson(strategy) }}
        run: echo "$STRATEGY_CONTEXT"
      - name: Dump matrix context
        env:
          MATRIX_CONTEXT: ${{ toJson(matrix) }}
        run: echo "$MATRIX_CONTEXT"
```

html:
<p>Useful for seeing what's available for <code>if: </code> conditions (see <a href="https://help.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions">context and expression syntax</a>).</p>
<p>I copied this example action <a href="https://help.github.com/en/actions/reference/context-and-expression-syntax-for-github-actions#example-printing-context-information-to-the-log-file">from here</a> and deployed it <a href="https://github.com/simonw/playing-with-actions/blob/master/.github/workflows/dump-context.yml">here</a>. Here's an <a href="https://github.com/simonw/playing-with-actions/runs/599575180?check_suite_focus=true">example run</a>.</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">on</span>: <span class="pl-s">push</span>

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">one</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-16.04</span>
    <span class="pl-ent">steps</span>:
      - <span class="pl-ent">name</span>: <span class="pl-s">Dump GitHub context</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">GITHUB_CONTEXT</span>: <span class="pl-s">${{ toJson(github) }}</span>
        <span class="pl-ent">run</span>: <span class="pl-s">echo "$GITHUB_CONTEXT"</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Dump job context</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">JOB_CONTEXT</span>: <span class="pl-s">${{ toJson(job) }}</span>
        <span class="pl-ent">run</span>: <span class="pl-s">echo "$JOB_CONTEXT"</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Dump steps context</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">STEPS_CONTEXT</span>: <span class="pl-s">${{ toJson(steps) }}</span>
        <span class="pl-ent">run</span>: <span class="pl-s">echo "$STEPS_CONTEXT"</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Dump runner context</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">RUNNER_CONTEXT</span>: <span class="pl-s">${{ toJson(runner) }}</span>
        <span class="pl-ent">run</span>: <span class="pl-s">echo "$RUNNER_CONTEXT"</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Dump strategy context</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">STRATEGY_CONTEXT</span>: <span class="pl-s">${{ toJson(strategy) }}</span>
        <span class="pl-ent">run</span>: <span class="pl-s">echo "$STRATEGY_CONTEXT"</span>
      - <span class="pl-ent">name</span>: <span class="pl-s">Dump matrix context</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">MATRIX_CONTEXT</span>: <span class="pl-s">${{ toJson(matrix) }}</span>
        <span class="pl-ent">run</span>: <span class="pl-s">echo "$MATRIX_CONTEXT"</span></pre></div>


path: github-actions_markdown-table-of-contents.md
topic: github-actions
title: Updating a Markdown table of contents with a GitHub Action
url: https://github.com/simonw/til/blob/main/github-actions/markdown-table-of-contents.md
created: 2020-07-22T12:42:38-07:00
created_utc: 2020-07-22T19:42:38+00:00
updated: 2020-07-23T11:05:45-07:00
updated_utc: 2020-07-23T18:05:45+00:00
shot_hash: 7def38bdfb8cb037d9197e4df17f644c
slug: markdown-table-of-contents
body:
[markdown-toc](https://github.com/jonschlinkert/markdown-toc) is a Node script that parses a Markdown file and generates a table of contents for it, based on the headings.

You can run it (without installing anything first thanks to the magic of [npx](https://medium.com/@maybekatz/introducing-npx-an-npm-package-runner-55f7d4bd282b)) like so:

    npx markdown-toc README.md

This will output the table of contents to standard out.

You can add the following comment to a markdown file:
  
    <!-- toc -->
    <!-- tocstop -->

And then run the command like this to update a table of contents in place:

    npx markdown-toc -i README.md

I wrote this GitHub Action to apply this command every time the README is updated, then commit the results back to the repository.

Save this as `.github/workflows/readme-toc.yml`:

```yaml
name: Update README table of contents

on:
  workflow_dispatch:
  push:
    branches:
    - main
    - master
    paths:
    - README.md

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Check out repo
      uses: actions/checkout@v2
    - name: Update TOC
      run: npx markdown-toc README.md -i
    - name: Commit and push if README changed
      run: |-
        git diff
        git config --global user.email "readme-bot@example.com"
        git config --global user.name "README-bot"
        git diff --quiet || (git add README.md && git commit -m "Updated README")
        git push
```
You can see this running on the [dogsheep/twitter-to-sqlite](https://github.com/dogsheep/twitter-to-sqlite) repository.

Unfortunately these links don't work on READMEs that are rendered by PyPI yet, e.g. [twitter-to-sqlite](https://pypi.org/project/twitter-to-sqlite/). There's an open issue for that [here](https://github.com/pypa/readme_renderer/issues/169).

html:
<p><a href="https://github.com/jonschlinkert/markdown-toc">markdown-toc</a> is a Node script that parses a Markdown file and generates a table of contents for it, based on the headings.</p>
<p>You can run it (without installing anything first thanks to the magic of <a href="https://medium.com/@maybekatz/introducing-npx-an-npm-package-runner-55f7d4bd282b" rel="nofollow">npx</a>) like so:</p>
<pre><code>npx markdown-toc README.md
</code></pre>
<p>This will output the table of contents to standard out.</p>
<p>You can add the following comment to a markdown file:</p>
<pre><code>&lt;!-- toc --&gt;
&lt;!-- tocstop --&gt;
</code></pre>
<p>And then run the command like this to update a table of contents in place:</p>
<pre><code>npx markdown-toc -i README.md
</code></pre>
<p>I wrote this GitHub Action to apply this command every time the README is updated, then commit the results back to the repository.</p>
<p>Save this as <code>.github/workflows/readme-toc.yml</code>:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Update README table of contents</span>

<span class="pl-ent">on</span>:
  <span class="pl-ent">workflow_dispatch</span>:
  <span class="pl-ent">push</span>:
    <span class="pl-ent">branches</span>:
    - <span class="pl-s">main</span>
    - <span class="pl-s">master</span>
    <span class="pl-ent">paths</span>:
    - <span class="pl-s">README.md</span>

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">build</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">name</span>: <span class="pl-s">Check out repo</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v2</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Update TOC</span>
      <span class="pl-ent">run</span>: <span class="pl-s">npx markdown-toc README.md -i</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Commit and push if README changed</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        git diff</span>
<span class="pl-s">        git config --global user.email "readme-bot@example.com"</span>
<span class="pl-s">        git config --global user.name "README-bot"</span>
<span class="pl-s">        git diff --quiet || (git add README.md &amp;&amp; git commit -m "Updated README")</span>
<span class="pl-s">        git push</span></pre></div>
<p>You can see this running on the <a href="https://github.com/dogsheep/twitter-to-sqlite">dogsheep/twitter-to-sqlite</a> repository.</p>
<p>Unfortunately these links don't work on READMEs that are rendered by PyPI yet, e.g. <a href="https://pypi.org/project/twitter-to-sqlite/" rel="nofollow">twitter-to-sqlite</a>. There's an open issue for that <a href="https://github.com/pypa/readme_renderer/issues/169">here</a>.</p>


path: github-actions_continue-on-error.md
topic: github-actions
title: Skipping a GitHub Actions step without failing
url: https://github.com/simonw/til/blob/main/github-actions/continue-on-error.md
created: 2020-08-22T20:23:51-07:00
created_utc: 2020-08-23T03:23:51+00:00
updated: 2020-11-25T11:44:35-08:00
updated_utc: 2020-11-25T19:44:35+00:00
shot_hash: 30e610ad7045f1fa181666356c86d4a1
slug: continue-on-error
body:
I wanted to have a GitHub Action step run that might fail, but if it failed the rest of the steps should still execute and the overall run should be treated as a success.

`continue-on-error: true` does exactly that:

```yaml
    - name: Download previous database
      run: curl --fail -o tils.db https://til.simonwillison.net/tils.db
      continue-on-error: true
    - name: Build database
      run: python build_database.py
```

[From this workflow](https://github.com/simonw/til/blob/7d799a24921f66e585b8a6b8756b7f8040c899df/.github/workflows/build.yml#L32-L36)

I'm using `curl --fail` here which returns an error code if the file download files (without `--fail` it was writing out a two line error message to a file called `tils.db` which is not what I wanted). Then `continue-on-error: true` to keep on going even if the download failed.

My `build_database.py` script updates the `tils.db` database file if it exists and creates it from scratch if it doesn't.

html:
<p>I wanted to have a GitHub Action step run that might fail, but if it failed the rest of the steps should still execute and the overall run should be treated as a success.</p>
<p><code>continue-on-error: true</code> does exactly that:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Download previous database</span>
      <span class="pl-ent">run</span>: <span class="pl-s">curl --fail -o tils.db https://til.simonwillison.net/tils.db</span>
      <span class="pl-ent">continue-on-error</span>: <span class="pl-c1">true</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Build database</span>
      <span class="pl-ent">run</span>: <span class="pl-s">python build_database.py</span></pre></div>
<p><a href="https://github.com/simonw/til/blob/7d799a24921f66e585b8a6b8756b7f8040c899df/.github/workflows/build.yml#L32-L36">From this workflow</a></p>
<p>I'm using <code>curl --fail</code> here which returns an error code if the file download files (without <code>--fail</code> it was writing out a two line error message to a file called <code>tils.db</code> which is not what I wanted). Then <code>continue-on-error: true</code> to keep on going even if the download failed.</p>
<p>My <code>build_database.py</code> script updates the <code>tils.db</code> database file if it exists and creates it from scratch if it doesn't.</p>


path: github-actions_grep-tests.md
topic: github-actions
title: Using grep to write tests in CI
url: https://github.com/simonw/til/blob/main/github-actions/grep-tests.md
created: 2020-08-19T21:26:05-07:00
created_utc: 2020-08-20T04:26:05+00:00
updated: 2020-08-22T21:18:06-07:00
updated_utc: 2020-08-23T04:18:06+00:00
shot_hash: 3e71efb58ec2d72ce37d6c93d7ace74e
slug: grep-tests
body:
GitHub Actions workflows fail if any of the steps executes something that returns a non-zero exit code.

Today I learned that `grep` returns a non-zero exit code if it fails to find any matches.

This means that piping to grep is a really quick way to write a test as part of an Actions workflow.

I wrote a quick soundness check today using the new `datasette --get /path` option, which runs a fake HTTP request for that path through Datasette and returns the response to standard out. Here's an example:

```yaml
    - name: Build database
      run: scripts/build.sh
    - name: Run tests
      run: |
        datasette . --get /us/pillar-point | grep 'Rocky Beaches'
    - name: Deploy to Vercel
```
I like this pattern a lot: build a database for a custom Datasette deloyment in CI, run one or more quick soundness checks using grep, then deploy if those checks pass.

html:
<p>GitHub Actions workflows fail if any of the steps executes something that returns a non-zero exit code.</p>
<p>Today I learned that <code>grep</code> returns a non-zero exit code if it fails to find any matches.</p>
<p>This means that piping to grep is a really quick way to write a test as part of an Actions workflow.</p>
<p>I wrote a quick soundness check today using the new <code>datasette --get /path</code> option, which runs a fake HTTP request for that path through Datasette and returns the response to standard out. Here's an example:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Build database</span>
      <span class="pl-ent">run</span>: <span class="pl-s">scripts/build.sh</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Run tests</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        datasette . --get /us/pillar-point | grep 'Rocky Beaches'</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Deploy to Vercel</span></pre></div>
<p>I like this pattern a lot: build a database for a custom Datasette deloyment in CI, run one or more quick soundness checks using grep, then deploy if those checks pass.</p>


path: github-actions_commit-if-file-changed.md
topic: github-actions
title: Commit a file if it changed
url: https://github.com/simonw/til/blob/main/github-actions/commit-if-file-changed.md
created: 2020-04-19T10:27:46-07:00
created_utc: 2020-04-19T17:27:46+00:00
updated: 2020-04-28T12:33:00-07:00
updated_utc: 2020-04-28T19:33:00+00:00
shot_hash: 3b4a2012993962434fc8f5853cf5396b
slug: commit-if-file-changed
body:
This recipe runs a Python script to update a README, then commits it back to the parent repo but only if it has changed:

```yaml
on:
  push:
    branches:
    - master
# ...
    - name: Update README
      run: python update_readme.py --rewrite
    - name: Commit README back to the repo
      run: |-
        git config --global user.email "readme-bot@example.com"
        git config --global user.name "README-bot"
        git diff --quiet || (git add README.md && git commit -m "Updated README")
        git push
```
My first attempt threw an error if I tried o run `git commit -m ...` and the README had not changed.

It turns out `git diff --quiet` exits with a 1 exit code if anything has changed, so this recipe adds the file and commits it only if something differs:

```bash
git diff --quiet || (git add README.md && git commit -m "Updated README")
```

Mikeal Rogers has a [publish-to-github-action](https://github.com/mikeal/publish-to-github-action) which uses a [slightly different pattern](https://github.com/mikeal/publish-to-github-action/blob/000c8a4f43e2a7dd4aab81e3fdf8be36d4457ed8/entrypoint.sh#L21-L27):

```bash
# publish any new files
git checkout master
git add -A
timestamp=$(date -u)
git commit -m "Automated publish: ${timestamp} ${GITHUB_SHA}" || exit 0
git pull --rebase publisher master
git push publisher master
```

Cleanest example yet: https://github.com/simonw/coronavirus-data-gov-archive/blob/master/.github/workflows/scheduled.yml

```yaml
name: Fetch latest data

on:
  push:
  repository_dispatch:
  schedule:
    - cron:  '25 * * * *'

jobs:
  scheduled:
    runs-on: ubuntu-latest
    steps:
    - name: Check out this repo
      uses: actions/checkout@v2
    - name: Fetch latest data
      run: |-
        curl https://c19downloads.azureedge.net/downloads/data/data_latest.json | jq . > data_latest.json
        curl https://c19pub.azureedge.net/utlas.geojson | gunzip | jq . > utlas.geojson
        curl https://c19pub.azureedge.net/countries.geojson | gunzip | jq . > countries.geojson
        curl https://c19pub.azureedge.net/regions.geojson | gunzip | jq . > regions.geojson
    - name: Commit and push if it changed
      run: |-
        git config user.name "Automated"
        git config user.email "actions@users.noreply.github.com"
        git add -A
        timestamp=$(date -u)
        git commit -m "Latest data: ${timestamp}" || exit 0
        git push
```

html:
<p>This recipe runs a Python script to update a README, then commits it back to the parent repo but only if it has changed:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">on</span>:
  <span class="pl-ent">push</span>:
    <span class="pl-ent">branches</span>:
    - <span class="pl-s">master</span>
<span class="pl-c"><span class="pl-c">#</span> ...</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Update README</span>
      <span class="pl-ent">run</span>: <span class="pl-s">python update_readme.py --rewrite</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Commit README back to the repo</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        git config --global user.email "readme-bot@example.com"</span>
<span class="pl-s">        git config --global user.name "README-bot"</span>
<span class="pl-s">        git diff --quiet || (git add README.md &amp;&amp; git commit -m "Updated README")</span>
<span class="pl-s">        git push</span></pre></div>
<p>My first attempt threw an error if I tried o run <code>git commit -m ...</code> and the README had not changed.</p>
<p>It turns out <code>git diff --quiet</code> exits with a 1 exit code if anything has changed, so this recipe adds the file and commits it only if something differs:</p>
<div class="highlight highlight-source-shell"><pre>git diff --quiet <span class="pl-k">||</span> (git add README.md <span class="pl-k">&amp;&amp;</span> git commit -m <span class="pl-s"><span class="pl-pds">"</span>Updated README<span class="pl-pds">"</span></span>)</pre></div>
<p>Mikeal Rogers has a <a href="https://github.com/mikeal/publish-to-github-action">publish-to-github-action</a> which uses a <a href="https://github.com/mikeal/publish-to-github-action/blob/000c8a4f43e2a7dd4aab81e3fdf8be36d4457ed8/entrypoint.sh#L21-L27">slightly different pattern</a>:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c"><span class="pl-c">#</span> publish any new files</span>
git checkout master
git add -A
timestamp=<span class="pl-s"><span class="pl-pds">$(</span>date -u<span class="pl-pds">)</span></span>
git commit -m <span class="pl-s"><span class="pl-pds">"</span>Automated publish: <span class="pl-smi">${timestamp}</span> <span class="pl-smi">${GITHUB_SHA}</span><span class="pl-pds">"</span></span> <span class="pl-k">||</span> <span class="pl-c1">exit</span> 0
git pull --rebase publisher master
git push publisher master</pre></div>
<p>Cleanest example yet: <a href="https://github.com/simonw/coronavirus-data-gov-archive/blob/master/.github/workflows/scheduled.yml">https://github.com/simonw/coronavirus-data-gov-archive/blob/master/.github/workflows/scheduled.yml</a></p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Fetch latest data</span>

<span class="pl-ent">on</span>:
  <span class="pl-ent">push</span>:
  <span class="pl-ent">repository_dispatch</span>:
  <span class="pl-ent">schedule</span>:
    - <span class="pl-ent">cron</span>:  <span class="pl-s"><span class="pl-pds">'</span>25 * * * *<span class="pl-pds">'</span></span>

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">scheduled</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">name</span>: <span class="pl-s">Check out this repo</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v2</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Fetch latest data</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        curl https://c19downloads.azureedge.net/downloads/data/data_latest.json | jq . &gt; data_latest.json</span>
<span class="pl-s">        curl https://c19pub.azureedge.net/utlas.geojson | gunzip | jq . &gt; utlas.geojson</span>
<span class="pl-s">        curl https://c19pub.azureedge.net/countries.geojson | gunzip | jq . &gt; countries.geojson</span>
<span class="pl-s">        curl https://c19pub.azureedge.net/regions.geojson | gunzip | jq . &gt; regions.geojson</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Commit and push if it changed</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        git config user.name "Automated"</span>
<span class="pl-s">        git config user.email "actions@users.noreply.github.com"</span>
<span class="pl-s">        git add -A</span>
<span class="pl-s">        timestamp=$(date -u)</span>
<span class="pl-s">        git commit -m "Latest data: ${timestamp}" || exit 0</span>
<span class="pl-s">        git push</span></pre></div>


path: bash_loop-over-csv.md
topic: bash
title: Looping over comma-separated values in Bash
url: https://github.com/simonw/til/blob/main/bash/loop-over-csv.md
created: 2020-09-01T18:48:28-07:00
created_utc: 2020-09-02T01:48:28+00:00
updated: 2020-09-01T18:48:28-07:00
updated_utc: 2020-09-02T01:48:28+00:00
shot_hash: d06963c31326ae773a8e7face614668c
slug: loop-over-csv
body:
Given a file (or a process) that produces comma separated values, here's how to split those into separate variables and use them in a bash script.

The trick is to set the Bash `IFS` to a delimiter, then use `my_array=($my_string)` to split on that delimiter.

Create a text file called `data.txt` containing this:
```
first,1
second,2
```
You can create that by doing:
```bash
echo 'first,1
second,2' > /tmp/data.txt
```
To loop over that file and print each line:
```bash
for line in $(cat /tmp/data.txt);
do
  echo $line
done
```
To split each line into two separate variables in the loop, do this:
```bash
for line in $(cat /tmp/data.txt);
do
  IFS=$','; split=($line); unset IFS;
  # $split is now a bash array
  echo "Column 1: ${split[0]}"
  echo "Column 2: ${split[1]}"
done
```
Outputs:
```
Column 1: first
Column 2: 1
Column 1: second
Column 2: 2
```
Here's a script I wrote using this technique for the TIL [Use labels on Cloud Run services for a billing breakdown](https://til.simonwillison.net/til/til/cloudrun_use-labels-for-billing-breakdown.md):
```bash
#!/bin/bash
for line in $(
  gcloud run services list --platform=managed \
    --format="csv(SERVICE,REGION)" \
    --filter "NOT metadata.labels.service:*" \
  | tail -n +2)
do
  IFS=$','; service_and_region=($line); unset IFS;
  service=${service_and_region[0]}
  region=${service_and_region[1]}
  echo "service: $service    region: $region"
  gcloud run services update $service \
    --region=$region --platform=managed \
    --update-labels service=$service
  echo
done
```

html:
<p>Given a file (or a process) that produces comma separated values, here's how to split those into separate variables and use them in a bash script.</p>
<p>The trick is to set the Bash <code>IFS</code> to a delimiter, then use <code>my_array=($my_string)</code> to split on that delimiter.</p>
<p>Create a text file called <code>data.txt</code> containing this:</p>
<pre><code>first,1
second,2
</code></pre>
<p>You can create that by doing:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">'</span>first,1</span>
<span class="pl-s">second,2<span class="pl-pds">'</span></span> <span class="pl-k">&gt;</span> /tmp/data.txt</pre></div>
<p>To loop over that file and print each line:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-k">for</span> <span class="pl-smi">line</span> <span class="pl-k">in</span> <span class="pl-s"><span class="pl-pds">$(</span>cat /tmp/data.txt<span class="pl-pds">)</span></span><span class="pl-k">;</span>
<span class="pl-k">do</span>
  <span class="pl-c1">echo</span> <span class="pl-smi">$line</span>
<span class="pl-k">done</span></pre></div>
<p>To split each line into two separate variables in the loop, do this:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-k">for</span> <span class="pl-smi">line</span> <span class="pl-k">in</span> <span class="pl-s"><span class="pl-pds">$(</span>cat /tmp/data.txt<span class="pl-pds">)</span></span><span class="pl-k">;</span>
<span class="pl-k">do</span>
  IFS=<span class="pl-s"><span class="pl-pds">$'</span>,<span class="pl-pds">'</span></span><span class="pl-k">;</span> split=(<span class="pl-smi">$line</span>)<span class="pl-k">;</span> <span class="pl-c1">unset</span> IFS<span class="pl-k">;</span>
  <span class="pl-c"><span class="pl-c">#</span> $split is now a bash array</span>
  <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>Column 1: <span class="pl-smi">${split[0]}</span><span class="pl-pds">"</span></span>
  <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>Column 2: <span class="pl-smi">${split[1]}</span><span class="pl-pds">"</span></span>
<span class="pl-k">done</span></pre></div>
<p>Outputs:</p>
<pre><code>Column 1: first
Column 2: 1
Column 1: second
Column 2: 2
</code></pre>
<p>Here's a script I wrote using this technique for the TIL <a href="https://til.simonwillison.net/til/til/cloudrun_use-labels-for-billing-breakdown.md" rel="nofollow">Use labels on Cloud Run services for a billing breakdown</a>:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c"><span class="pl-c">#!</span>/bin/bash</span>
<span class="pl-k">for</span> <span class="pl-smi">line</span> <span class="pl-k">in</span> <span class="pl-s"><span class="pl-pds">$(</span></span>
<span class="pl-s">  gcloud run services list --platform=managed \</span>
<span class="pl-s">    --format=<span class="pl-s"><span class="pl-pds">"</span>csv(SERVICE,REGION)<span class="pl-pds">"</span></span> \</span>
<span class="pl-s">    --filter <span class="pl-s"><span class="pl-pds">"</span>NOT metadata.labels.service:*<span class="pl-pds">"</span></span> \</span>
<span class="pl-s">  <span class="pl-k">|</span> tail -n +2<span class="pl-pds">)</span></span>
<span class="pl-k">do</span>
  IFS=<span class="pl-s"><span class="pl-pds">$'</span>,<span class="pl-pds">'</span></span><span class="pl-k">;</span> service_and_region=(<span class="pl-smi">$line</span>)<span class="pl-k">;</span> <span class="pl-c1">unset</span> IFS<span class="pl-k">;</span>
  service=<span class="pl-smi">${service_and_region[0]}</span>
  region=<span class="pl-smi">${service_and_region[1]}</span>
  <span class="pl-c1">echo</span> <span class="pl-s"><span class="pl-pds">"</span>service: <span class="pl-smi">$service</span>    region: <span class="pl-smi">$region</span><span class="pl-pds">"</span></span>
  gcloud run services update <span class="pl-smi">$service</span> \
    --region=<span class="pl-smi">$region</span> --platform=managed \
    --update-labels service=<span class="pl-smi">$service</span>
  <span class="pl-c1">echo</span>
<span class="pl-k">done</span></pre></div>


path: cloudrun_gcloud-run-services-list.md
topic: cloudrun
title: Using the gcloud run services list command
url: https://github.com/simonw/til/blob/main/cloudrun/gcloud-run-services-list.md
created: 2020-09-01T21:40:04-07:00
created_utc: 2020-09-02T04:40:04+00:00
updated: 2020-09-01T21:40:25-07:00
updated_utc: 2020-09-02T04:40:25+00:00
shot_hash: 18f4f503f9530d61e27ad5f3c77d9bdd
slug: gcloud-run-services-list
body:
The `gcloud run services list` command lists your services running on Google Cloud Run:

```
~ % gcloud run services list --platform=managed
   SERVICE                        REGION       URL                                                            LAST DEPLOYED BY                                              LAST DEPLOYED AT
✔  calands                        us-central1  https://calands-j7hipcg4aq-uc.a.run.app                        ...@gmail.com                                           2020-09-02T00:15:29.563846Z
✔  cloud-run-hello                us-central1  https://cloud-run-hello-j7hipcg4aq-uc.a.run.app                ...@gmail.com                                           2020-09-02T00:16:07.835843Z
✔  covid-19                       us-central1  https://covid-19-j7hipcg4aq-uc.a.run.app                       ...@gmail.com                                           2020-09-02T00:16:46.979188Z
...
```

It has two useful but under-documented options: `--filter` which filters based on a special filter language, and `--format` which customizes the output format.

## --filter

I found the `--filter` option really hard to figure out. It has [documentation here](https://cloud.google.com/sdk/gcloud/reference/topic/filters) describing the predicate language it uses, but I had to apply trial and error to find options that worked for `gcloud run services`. Here are a few I found.

To see data for just one specific service by name, use `--filter=SERVICE:covid-19`. Lowercase `service` doesn't work for some reason.

```
~ % gcloud run services list --platform=managed --filter=SERVICE:covid-19
   SERVICE   REGION       URL                                       LAST DEPLOYED BY     LAST DEPLOYED AT
✔  covid-19  us-central1  https://covid-19-j7hipcg4aq-uc.a.run.app  ...@gmail.com  2020-09-02T00:16:46.979188Z
```

To filter by labels that you have set on your services, use `--filter="metadata.labels.name=value"`. It took me a while to figure out I needed the `metadata.` prefix here.

Here's a filter for every service that does NOT have a label called `service` (I apply a label of 'service' to all of my services so I can [see them separately in billing](https://til.simonwillison.net/til/til/cloudrun_use-labels-for-billing-breakdown.md)).

```
gcloud run services list --platform=managed --filter='NOT metadata.labels.service:*'
```

## --format

The `--format` option is [documented here](https://cloud.google.com/sdk/gcloud/reference/topic/formats). Here's what I've worked out.

`gcloud run services list --platform=managed --format=json` outputs JSON. `--format=yaml` outputs YAML.

Two more interesting ones are `--format='table(colums go here)'` and `--format='csv(columns go here)'`. For example:
```
~ % gcloud run services list --platform=managed --format='table(SERVICE,URL)'
SERVICE                        URL
calands                        https://calands-j7hipcg4aq-uc.a.run.app
covid-19                       https://covid-19-j7hipcg4aq-uc.a.run.app
```

Or for CSV:
```
~ % gcloud run services list --platform=managed --format='csv(SERVICE,URL)'
service,url
calands,https://calands-j7hipcg4aq-uc.a.run.app
covid-19,https://covid-19-j7hipcg4aq-uc.a.run.app
```
Other values I found that work include `REGION`, `labels` and `metadata`. The latter two output a not-valid-JSON (possibly Python `repr()`) nested structure:
```
~ % gcloud run services list --platform=managed --format='table(SERVICE,labels,metadata)' --filter=SERVICE:covid-19
SERVICE   LABELS                                                                   METADATA
covid-19  {'service': 'covid-19', 'cloud.googleapis.com/location': 'us-central1'}  {'annotations': {'client.knative.dev/user-image': 'gcr.io/datasette-222320/datasette', 'run.googleapis.com/client-name': 'gcloud', 'run.googleapis.com/client-version': '302.0.0', 'serving.knative.dev/creator': 'swillison@gmail.com', 'serving.knative.dev/lastModifier': 'swillison@gmail.com'}, 'creationTimestamp': '2020-03-10T18:47:04.923274Z', 'generation': 670, 'labels': {'cloud.googleapis.com/location': 'us-central1', 'service': 'covid-19'}, 'name': 'covid-19', 'namespace': '99025868001', 'resourceVersion': 'AAWuSY0eaHQ', 'selfLink': '/apis/serving.knative.dev/v1/namespaces/99025868001/services/covid-19', 'uid': '6ff64723-a38d-4784-ac5e-07a745061845'}
```

html:
<p>The <code>gcloud run services list</code> command lists your services running on Google Cloud Run:</p>
<pre><code>~ % gcloud run services list --platform=managed
   SERVICE                        REGION       URL                                                            LAST DEPLOYED BY                                              LAST DEPLOYED AT
✔  calands                        us-central1  https://calands-j7hipcg4aq-uc.a.run.app                        ...@gmail.com                                           2020-09-02T00:15:29.563846Z
✔  cloud-run-hello                us-central1  https://cloud-run-hello-j7hipcg4aq-uc.a.run.app                ...@gmail.com                                           2020-09-02T00:16:07.835843Z
✔  covid-19                       us-central1  https://covid-19-j7hipcg4aq-uc.a.run.app                       ...@gmail.com                                           2020-09-02T00:16:46.979188Z
...
</code></pre>
<p>It has two useful but under-documented options: <code>--filter</code> which filters based on a special filter language, and <code>--format</code> which customizes the output format.</p>
<h2>
<a id="user-content---filter" class="anchor" href="#--filter" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>--filter</h2>
<p>I found the <code>--filter</code> option really hard to figure out. It has <a href="https://cloud.google.com/sdk/gcloud/reference/topic/filters" rel="nofollow">documentation here</a> describing the predicate language it uses, but I had to apply trial and error to find options that worked for <code>gcloud run services</code>. Here are a few I found.</p>
<p>To see data for just one specific service by name, use <code>--filter=SERVICE:covid-19</code>. Lowercase <code>service</code> doesn't work for some reason.</p>
<pre><code>~ % gcloud run services list --platform=managed --filter=SERVICE:covid-19
   SERVICE   REGION       URL                                       LAST DEPLOYED BY     LAST DEPLOYED AT
✔  covid-19  us-central1  https://covid-19-j7hipcg4aq-uc.a.run.app  ...@gmail.com  2020-09-02T00:16:46.979188Z
</code></pre>
<p>To filter by labels that you have set on your services, use <code>--filter="metadata.labels.name=value"</code>. It took me a while to figure out I needed the <code>metadata.</code> prefix here.</p>
<p>Here's a filter for every service that does NOT have a label called <code>service</code> (I apply a label of 'service' to all of my services so I can <a href="https://til.simonwillison.net/til/til/cloudrun_use-labels-for-billing-breakdown.md" rel="nofollow">see them separately in billing</a>).</p>
<pre><code>gcloud run services list --platform=managed --filter='NOT metadata.labels.service:*'
</code></pre>
<h2>
<a id="user-content---format" class="anchor" href="#--format" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>--format</h2>
<p>The <code>--format</code> option is <a href="https://cloud.google.com/sdk/gcloud/reference/topic/formats" rel="nofollow">documented here</a>. Here's what I've worked out.</p>
<p><code>gcloud run services list --platform=managed --format=json</code> outputs JSON. <code>--format=yaml</code> outputs YAML.</p>
<p>Two more interesting ones are <code>--format='table(colums go here)'</code> and <code>--format='csv(columns go here)'</code>. For example:</p>
<pre><code>~ % gcloud run services list --platform=managed --format='table(SERVICE,URL)'
SERVICE                        URL
calands                        https://calands-j7hipcg4aq-uc.a.run.app
covid-19                       https://covid-19-j7hipcg4aq-uc.a.run.app
</code></pre>
<p>Or for CSV:</p>
<pre><code>~ % gcloud run services list --platform=managed --format='csv(SERVICE,URL)'
service,url
calands,https://calands-j7hipcg4aq-uc.a.run.app
covid-19,https://covid-19-j7hipcg4aq-uc.a.run.app
</code></pre>
<p>Other values I found that work include <code>REGION</code>, <code>labels</code> and <code>metadata</code>. The latter two output a not-valid-JSON (possibly Python <code>repr()</code>) nested structure:</p>
<pre><code>~ % gcloud run services list --platform=managed --format='table(SERVICE,labels,metadata)' --filter=SERVICE:covid-19
SERVICE   LABELS                                                                   METADATA
covid-19  {'service': 'covid-19', 'cloud.googleapis.com/location': 'us-central1'}  {'annotations': {'client.knative.dev/user-image': 'gcr.io/datasette-222320/datasette', 'run.googleapis.com/client-name': 'gcloud', 'run.googleapis.com/client-version': '302.0.0', 'serving.knative.dev/creator': 'swillison@gmail.com', 'serving.knative.dev/lastModifier': 'swillison@gmail.com'}, 'creationTimestamp': '2020-03-10T18:47:04.923274Z', 'generation': 670, 'labels': {'cloud.googleapis.com/location': 'us-central1', 'service': 'covid-19'}, 'name': 'covid-19', 'namespace': '99025868001', 'resourceVersion': 'AAWuSY0eaHQ', 'selfLink': '/apis/serving.knative.dev/v1/namespaces/99025868001/services/covid-19', 'uid': '6ff64723-a38d-4784-ac5e-07a745061845'}
</code></pre>


path: python_debug-click-with-pdb.md
topic: python
title: Debugging a Click application using pdb
url: https://github.com/simonw/til/blob/main/python/debug-click-with-pdb.md
created: 2020-09-03T10:48:51-07:00
created_utc: 2020-09-03T17:48:51+00:00
updated: 2020-09-03T12:00:52-07:00
updated_utc: 2020-09-03T19:00:52+00:00
shot_hash: e87c2de5504719d8f7b464770eaed7db
slug: debug-click-with-pdb
body:
This tip is for when you are working on a Python command-line application that runs using that program's name, as opposed to typing `python my_script.py`. I usually need this when I'm working on applications built using [Click](https://click.palletsprojects.com/), e.g. projects I start using my [click-app](https://github.com/simonw/click-app) cookiecutter template.

So you're working on one of these apps and it throws an exception:

```
% dogsheep-beta index beta.db dogsheep-beta.yml 
Traceback (most recent call last):
...
  File "/Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/utils.py", line 46, in run_indexer
    columns = derive_columns(other_db, sql)
  File "/Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/utils.py", line 63, in derive_columns
    cursor = db.conn.execute(sql + " limit 0")
sqlite3.OperationalError: near "from": syntax error
```

You can run the Python debugger against it using `python -i $(which dogsheep-beta)` to get an interactive prompt, then `import pdb; pdb.pm()` to open the debugger at the last exception:

```
% python -i $(which dogsheep-beta) index beta.db dogsheep-beta.yml
Traceback (most recent call last):
  File "/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/bin/dogsheep-beta", line 11, in <module>
    load_entry_point('dogsheep-beta', 'console_scripts', 'dogsheep-beta')()
  File "/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/cli.py", line 30, in index
    run_indexer(db_path, rules, tokenize=None if tokenize == "none" else tokenize)
  File "/Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/utils.py", line 46, in run_indexer
    columns = derive_columns(other_db, sql)
  File "/Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/utils.py", line 63, in derive_columns
    cursor = db.conn.execute(sql + " limit 0")
sqlite3.OperationalError: near "from": syntax error
>>> import pdb; pdb.pm()
> /Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/utils.py(63)derive_columns()
-> cursor = db.conn.execute(sql + " limit 0")
(Pdb) print(sql)
...
```

html:
<p>This tip is for when you are working on a Python command-line application that runs using that program's name, as opposed to typing <code>python my_script.py</code>. I usually need this when I'm working on applications built using <a href="https://click.palletsprojects.com/" rel="nofollow">Click</a>, e.g. projects I start using my <a href="https://github.com/simonw/click-app">click-app</a> cookiecutter template.</p>
<p>So you're working on one of these apps and it throws an exception:</p>
<pre><code>% dogsheep-beta index beta.db dogsheep-beta.yml 
Traceback (most recent call last):
...
  File "/Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/utils.py", line 46, in run_indexer
    columns = derive_columns(other_db, sql)
  File "/Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/utils.py", line 63, in derive_columns
    cursor = db.conn.execute(sql + " limit 0")
sqlite3.OperationalError: near "from": syntax error
</code></pre>
<p>You can run the Python debugger against it using <code>python -i $(which dogsheep-beta)</code> to get an interactive prompt, then <code>import pdb; pdb.pm()</code> to open the debugger at the last exception:</p>
<pre><code>% python -i $(which dogsheep-beta) index beta.db dogsheep-beta.yml
Traceback (most recent call last):
  File "/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/bin/dogsheep-beta", line 11, in &lt;module&gt;
    load_entry_point('dogsheep-beta', 'console_scripts', 'dogsheep-beta')()
  File "/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/simon/.local/share/virtualenvs/dogsheep-beta-u_po4Rpj/lib/python3.8/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/cli.py", line 30, in index
    run_indexer(db_path, rules, tokenize=None if tokenize == "none" else tokenize)
  File "/Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/utils.py", line 46, in run_indexer
    columns = derive_columns(other_db, sql)
  File "/Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/utils.py", line 63, in derive_columns
    cursor = db.conn.execute(sql + " limit 0")
sqlite3.OperationalError: near "from": syntax error
&gt;&gt;&gt; import pdb; pdb.pm()
&gt; /Users/simon/Dropbox/Development/dogsheep-beta/dogsheep_beta/utils.py(63)derive_columns()
-&gt; cursor = db.conn.execute(sql + " limit 0")
(Pdb) print(sql)
...
</code></pre>


path: typescript_basic-tsc.md
topic: typescript
title: Very basic tsc usage
url: https://github.com/simonw/til/blob/main/typescript/basic-tsc.md
created: 2020-09-06T09:36:03-07:00
created_utc: 2020-09-06T16:36:03+00:00
updated: 2020-10-09T19:52:44-07:00
updated_utc: 2020-10-10T02:52:44+00:00
shot_hash: 00bb53e8490810fb4c0fd25c691a93cb
slug: basic-tsc
body:
I guess I [have to learn TypeScript](https://twitter.com/simonw/status/1302517496767938561) now.

Here's how I got started in as few steps as possible, with the help of [Get Started With Typescript in 2019](https://www.robertcooper.me/get-started-with-typescript-in-2019) by Robert Cooper.

## Installation using npm

I created a new project:

    mkdir -p ~/Dropbox/Learning-TypeScript/first-typescript
    cd ~/Dropbox/Learning-TypeScript/first-typescript

Then installed the TypeScript compiler:

    npm install --save-dev typescript

Using `--global` instead of `--save-dev` would have installed in globally, but I'm not ready for that kind of commitment yet!

Apparently I need a `tsconfig.json` file. Running this command creates one for me containing some suggested defaults:

    % ./node_modules/.bin/tsc --init

Next step: create a `.ts` file to start testing it out. I put the following in `greetings.ts`:

```typescript
const greeting = (person: string) => {
  console.log("Hello " + person);
};

greeting("Simon");
```

Next, compile it! Thanks to `npm install --save-dev typescript` the `tsc` compiler is now available here:

    % ./node_modules/.bin/tsc

Run without any arguments it seeks out the `tsconfig.json` file, compiles any `.ts` files and produces matching `.js` files.

That seems to have worked:

    % node greetings.js 
    Good day Simon
    % cat greetings.js
    "use strict";
    var greeting = function (person) {
        console.log("Good day " + person);
    };
    greeting("Simon");

## Running tsc --watch

The `--watch` command continues to run and automatically compiles files when they are saved:

    % ./node_modules/.bin/tsc --watch
    [9:32:44 AM] Starting compilation in watch mode...

    [9:32:44 AM] Found 0 errors. Watching for file changes.

I changed the last line of my `greetings.ts` file to `greeting(1)` (a type error) to see what happened:

    [9:33:56 AM] File change detected. Starting incremental compilation...

    greetings.ts:5:10 - error TS2345: Argument of type 'number' is not assignable to parameter of type 'string'.

    5 greeting(1);
               ~

    [9:33:56 AM] Found 1 error. Watching for file changes.


## Using npx

Tip [from @Benjie](https://twitter.com/Benjie/status/1302655836163014656): you can use [npx](https://www.npmjs.com/package/npx) to avoid the `./node_modules/.bin` prefix. I had thought that `npx` installed and ran a new global version, but it turns out it will notice your `node_modules` folder and run from that instead if one exists:

    % npx tsc --watch

## Running this in Visual Studio Code

VSCode has built-in TypeScript support. Hit Shift+Command+B and select the `tsc: watch` option and it runs that watch command in a embedded terminal pane inside the editor itself.

html:
<p>I guess I <a href="https://twitter.com/simonw/status/1302517496767938561" rel="nofollow">have to learn TypeScript</a> now.</p>
<p>Here's how I got started in as few steps as possible, with the help of <a href="https://www.robertcooper.me/get-started-with-typescript-in-2019" rel="nofollow">Get Started With Typescript in 2019</a> by Robert Cooper.</p>
<h2>
<a id="user-content-installation-using-npm" class="anchor" href="#installation-using-npm" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installation using npm</h2>
<p>I created a new project:</p>
<pre><code>mkdir -p ~/Dropbox/Learning-TypeScript/first-typescript
cd ~/Dropbox/Learning-TypeScript/first-typescript
</code></pre>
<p>Then installed the TypeScript compiler:</p>
<pre><code>npm install --save-dev typescript
</code></pre>
<p>Using <code>--global</code> instead of <code>--save-dev</code> would have installed in globally, but I'm not ready for that kind of commitment yet!</p>
<p>Apparently I need a <code>tsconfig.json</code> file. Running this command creates one for me containing some suggested defaults:</p>
<pre><code>% ./node_modules/.bin/tsc --init
</code></pre>
<p>Next step: create a <code>.ts</code> file to start testing it out. I put the following in <code>greetings.ts</code>:</p>
<div class="highlight highlight-source-ts"><pre><span class="pl-k">const</span> <span class="pl-en">greeting</span> <span class="pl-c1">=</span> <span class="pl-kos">(</span><span class="pl-s1">person</span>: <span class="pl-smi">string</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
  <span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s">"Hello "</span> <span class="pl-c1">+</span> <span class="pl-s1">person</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">;</span>

<span class="pl-en">greeting</span><span class="pl-kos">(</span><span class="pl-s">"Simon"</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<p>Next, compile it! Thanks to <code>npm install --save-dev typescript</code> the <code>tsc</code> compiler is now available here:</p>
<pre><code>% ./node_modules/.bin/tsc
</code></pre>
<p>Run without any arguments it seeks out the <code>tsconfig.json</code> file, compiles any <code>.ts</code> files and produces matching <code>.js</code> files.</p>
<p>That seems to have worked:</p>
<pre><code>% node greetings.js 
Good day Simon
% cat greetings.js
"use strict";
var greeting = function (person) {
    console.log("Good day " + person);
};
greeting("Simon");
</code></pre>
<h2>
<a id="user-content-running-tsc---watch" class="anchor" href="#running-tsc---watch" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running tsc --watch</h2>
<p>The <code>--watch</code> command continues to run and automatically compiles files when they are saved:</p>
<pre><code>% ./node_modules/.bin/tsc --watch
[9:32:44 AM] Starting compilation in watch mode...

[9:32:44 AM] Found 0 errors. Watching for file changes.
</code></pre>
<p>I changed the last line of my <code>greetings.ts</code> file to <code>greeting(1)</code> (a type error) to see what happened:</p>
<pre><code>[9:33:56 AM] File change detected. Starting incremental compilation...

greetings.ts:5:10 - error TS2345: Argument of type 'number' is not assignable to parameter of type 'string'.

5 greeting(1);
           ~

[9:33:56 AM] Found 1 error. Watching for file changes.
</code></pre>
<h2>
<a id="user-content-using-npx" class="anchor" href="#using-npx" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Using npx</h2>
<p>Tip <a href="https://twitter.com/Benjie/status/1302655836163014656" rel="nofollow">from @Benjie</a>: you can use <a href="https://www.npmjs.com/package/npx" rel="nofollow">npx</a> to avoid the <code>./node_modules/.bin</code> prefix. I had thought that <code>npx</code> installed and ran a new global version, but it turns out it will notice your <code>node_modules</code> folder and run from that instead if one exists:</p>
<pre><code>% npx tsc --watch
</code></pre>
<h2>
<a id="user-content-running-this-in-visual-studio-code" class="anchor" href="#running-this-in-visual-studio-code" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running this in Visual Studio Code</h2>
<p>VSCode has built-in TypeScript support. Hit Shift+Command+B and select the <code>tsc: watch</code> option and it runs that watch command in a embedded terminal pane inside the editor itself.</p>


path: aws_instance-costs-per-month.md
topic: aws
title: Display EC2 instance costs per month
url: https://github.com/simonw/til/blob/main/aws/instance-costs-per-month.md
created: 2020-09-06T19:43:29-07:00
created_utc: 2020-09-07T02:43:29+00:00
updated: 2020-09-06T19:43:29-07:00
updated_utc: 2020-09-07T02:43:29+00:00
shot_hash: 47d56d5c931266cac22ae86df8d494cf
slug: instance-costs-per-month
body:
The [EC2 pricing page](https://aws.amazon.com/ec2/pricing/on-demand/) shows cost per hour, which is pretty much useless. I want cost per month. The following JavaScript, pasted into the browser developer console, modifies the page to show cost-per-month instead.

```javascript
Array.from(
    document.querySelectorAll('td')
).filter(
    el => el.textContent.toLowerCase().includes('per hour')
).forEach(
    el => el.textContent = '$' + (parseFloat(
        /\d+\.\d+/.exec(el.textContent)[0]
    ) * 24 * 30).toFixed(2) + ' per month'
)
```

html:
<p>The <a href="https://aws.amazon.com/ec2/pricing/on-demand/" rel="nofollow">EC2 pricing page</a> shows cost per hour, which is pretty much useless. I want cost per month. The following JavaScript, pasted into the browser developer console, modifies the page to show cost-per-month instead.</p>
<div class="highlight highlight-source-js"><pre><span class="pl-v">Array</span><span class="pl-kos">.</span><span class="pl-en">from</span><span class="pl-kos">(</span>
    <span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">querySelectorAll</span><span class="pl-kos">(</span><span class="pl-s">'td'</span><span class="pl-kos">)</span>
<span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">filter</span><span class="pl-kos">(</span>
    <span class="pl-s1">el</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">el</span><span class="pl-kos">.</span><span class="pl-c1">textContent</span><span class="pl-kos">.</span><span class="pl-en">toLowerCase</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">includes</span><span class="pl-kos">(</span><span class="pl-s">'per hour'</span><span class="pl-kos">)</span>
<span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">forEach</span><span class="pl-kos">(</span>
    <span class="pl-s1">el</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">el</span><span class="pl-kos">.</span><span class="pl-c1">textContent</span> <span class="pl-c1">=</span> <span class="pl-s">'$'</span> <span class="pl-c1">+</span> <span class="pl-kos">(</span><span class="pl-en">parseFloat</span><span class="pl-kos">(</span>
        <span class="pl-pds">/<span class="pl-cce">\d</span><span class="pl-c1">+</span><span class="pl-cce">\.</span><span class="pl-cce">\d</span><span class="pl-c1">+</span>/</span><span class="pl-kos">.</span><span class="pl-en">exec</span><span class="pl-kos">(</span><span class="pl-s1">el</span><span class="pl-kos">.</span><span class="pl-c1">textContent</span><span class="pl-kos">)</span><span class="pl-kos">[</span><span class="pl-c1">0</span><span class="pl-kos">]</span>
    <span class="pl-kos">)</span> * <span class="pl-c1">24</span> * <span class="pl-c1">30</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">toFixed</span><span class="pl-kos">(</span><span class="pl-c1">2</span><span class="pl-kos">)</span> <span class="pl-c1">+</span> <span class="pl-s">' per month'</span>
<span class="pl-kos">)</span></pre></div>


path: linux_basic-strace.md
topic: linux
title: Basic strace to see what a process is doing
url: https://github.com/simonw/til/blob/main/linux/basic-strace.md
created: 2020-09-07T16:14:00-07:00
created_utc: 2020-09-07T23:14:00+00:00
updated: 2020-09-07T16:19:43-07:00
updated_utc: 2020-09-07T23:19:43+00:00
shot_hash: 2d9c231b458b0aefcf4edceeaa90dde7
slug: basic-strace
body:
I had a long-running process and I wanted to check that it was at least doing _something_.

I know its PID (32425 in this case) - here's how to use `strace` to see what it's doing right now. This command outputs a lot of lines very quickly - hit `Ctrl+C` to stop it.
```
$ sudo strace -p 32425
strace: Process 32425 attached
lseek(4, 923287936, SEEK_SET)           = 923287936
write(4, "\r\0\0\0\1\0\"\0\0\"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
lseek(3, 919687168, SEEK_SET)           = 919687168
read(3, "\r\0\0\0\1\0\"\0\0\"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
lseek(4, 923292032, SEEK_SET)           = 923292032
write(4, "\0\3kf\0\0\0\0\265\17I\247\326\307U\263xF\34\305\336\231\352a", 24) = 24
lseek(4, 923292056, SEEK_SET)           = 923292056
```
So at least there's stuff happening!

Running `strace -c` does a count of syscalls and provides a summary. The command appears not to do anything, but when you hit `Ctrl+C` after a few seconds it will dump out the counts so far:
```
$ sudo strace -c -p 32425
strace: Process 32425 attached
^Cstrace: Process 32425 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 56.67    0.048823          28      1741           write
 29.60    0.025496          10      2613           lseek
 13.73    0.011828          14       871           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.086147                  5225           total
```
The `-k` option adds stack traces to each syscall which give a better idea of where in the code the syscalls are being made from:
```
$ sudo strace -k -p 32425
strace: Process 32425 attached
lseek(4, 974466576, SEEK_SET)           = 974466576
 > /lib/x86_64-linux-gnu/libc-2.27.so(llseek+0x7) [0x110327]
 > /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(unixWrite+0x5d) [0x393cd]
 > /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(pagerWalFrames+0x5e5) [0xa8e75]
 > /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(pagerStress+0x13f) [0xa9cff]
 > /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(getPageNormal+0x33a) [0xadeea]
 > /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(backupOnePage.isra.666+0x102) [0xa86c2]
 > /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(sqlite3_backup_step+0x2f2) [0xb7a12]
 > /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(sqlite3RunVacuum+0x95c) [0x10534c]
 > /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(sqlite3VdbeExec+0x1dbf) [0xe230f]
 > /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(sqlite3_step+0x210) [0xec3d0]
 > /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(pysqlite_step+0x1e) [0x23fae]
 > /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(_pysqlite_query_execute+0x359) [0x22309]
 > /usr/bin/python3.6(PyCFunction_Call+0x52) [0x1670e2]
 > /usr/bin/python3.6(PyObject_Call+0x3e) [0x19fe1e]
```

html:
<p>I had a long-running process and I wanted to check that it was at least doing <em>something</em>.</p>
<p>I know its PID (32425 in this case) - here's how to use <code>strace</code> to see what it's doing right now. This command outputs a lot of lines very quickly - hit <code>Ctrl+C</code> to stop it.</p>
<pre><code>$ sudo strace -p 32425
strace: Process 32425 attached
lseek(4, 923287936, SEEK_SET)           = 923287936
write(4, "\r\0\0\0\1\0\"\0\0\"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
lseek(3, 919687168, SEEK_SET)           = 919687168
read(3, "\r\0\0\0\1\0\"\0\0\"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 4096) = 4096
lseek(4, 923292032, SEEK_SET)           = 923292032
write(4, "\0\3kf\0\0\0\0\265\17I\247\326\307U\263xF\34\305\336\231\352a", 24) = 24
lseek(4, 923292056, SEEK_SET)           = 923292056
</code></pre>
<p>So at least there's stuff happening!</p>
<p>Running <code>strace -c</code> does a count of syscalls and provides a summary. The command appears not to do anything, but when you hit <code>Ctrl+C</code> after a few seconds it will dump out the counts so far:</p>
<pre><code>$ sudo strace -c -p 32425
strace: Process 32425 attached
^Cstrace: Process 32425 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 56.67    0.048823          28      1741           write
 29.60    0.025496          10      2613           lseek
 13.73    0.011828          14       871           read
------ ----------- ----------- --------- --------- ----------------
100.00    0.086147                  5225           total
</code></pre>
<p>The <code>-k</code> option adds stack traces to each syscall which give a better idea of where in the code the syscalls are being made from:</p>
<pre><code>$ sudo strace -k -p 32425
strace: Process 32425 attached
lseek(4, 974466576, SEEK_SET)           = 974466576
 &gt; /lib/x86_64-linux-gnu/libc-2.27.so(llseek+0x7) [0x110327]
 &gt; /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(unixWrite+0x5d) [0x393cd]
 &gt; /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(pagerWalFrames+0x5e5) [0xa8e75]
 &gt; /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(pagerStress+0x13f) [0xa9cff]
 &gt; /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(getPageNormal+0x33a) [0xadeea]
 &gt; /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(backupOnePage.isra.666+0x102) [0xa86c2]
 &gt; /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(sqlite3_backup_step+0x2f2) [0xb7a12]
 &gt; /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(sqlite3RunVacuum+0x95c) [0x10534c]
 &gt; /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(sqlite3VdbeExec+0x1dbf) [0xe230f]
 &gt; /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(sqlite3_step+0x210) [0xec3d0]
 &gt; /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(pysqlite_step+0x1e) [0x23fae]
 &gt; /home/ubuntu/datasette-venv/lib/python3.6/site-packages/pysqlite3/_sqlite3.cpython-36m-x86_64-linux-gnu.so(_pysqlite_query_execute+0x359) [0x22309]
 &gt; /usr/bin/python3.6(PyCFunction_Call+0x52) [0x1670e2]
 &gt; /usr/bin/python3.6(PyObject_Call+0x3e) [0x19fe1e]
</code></pre>


path: github-actions_debug-tmate.md
topic: github-actions
title: Open a debugging shell in GitHub Actions with tmate
url: https://github.com/simonw/til/blob/main/github-actions/debug-tmate.md
created: 2020-09-14T15:25:36-07:00
created_utc: 2020-09-14T22:25:36+00:00
updated: 2022-02-17T15:30:51-08:00
updated_utc: 2022-02-17T23:30:51+00:00
shot_hash: 64a5eae4fd60080f0d219ecc7b9ccd05
slug: debug-tmate
body:
> :warning: **17 Feb 2022: There have been reports of running tmate causing account suspensions**. See [this issue](https://github.com/mxschmitt/action-tmate/issues/104) for details. Continue with caution.

Thanks to [this Twitter conversation](https://twitter.com/harrymarr/status/1304820879268950021) I found out about [mxschmitt/action-tmate](https://github.com/mxschmitt/action-tmate), which uses https://tmate.io/ to open an interactive shell session running inside the GitHub Actions environment.

I created a `.github/workflows/tmate.yml` file in my repo containing the following:

```yaml
name: tmate session

on:
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Setup tmate session
      uses: mxschmitt/action-tmate@v3
```
Clicking the "Run workflow" button in the GitHub Actions interface then gave me the following in the GitHub Actions log output:
```
WebURL: https://tmate.io/t/JA69KaB2avRPRZSkRb8JPa9Gd

SSH: ssh JA69KaB2avRPRZSkRb8JPa9Gd@nyc1.tmate.io
```
I ran `ssh JA69KaB2avRPRZSkRb8JPa9Gd@nyc1.tmate.io` and got a direction connection to the Action, with my project files all available thanks to the `- uses: actions/checkout@v2` step.

Once I'd finish testing things out in that environment, I typed `touch continue` and the session ended itself.

## Starting a shell just for test failures on manual runs

I had a tricky test failure that I wanted to debug interactively. Here's a recipe for starting a tmate shell ONLY if the previous step failed, and only if the run was triggered manually (using `workflow_dispatch`) - because I don't want an accidental test opening up a shell and burning up my GitHub Actions minutes allowance.

```yaml
    steps:
    - name: Run tests
      run: pytest
    - name: tmate session if tests fail
      if: failure() && github.event_name == 'workflow_dispatch'
      uses: mxschmitt/action-tmate@v3
```

html:
<blockquote>
<p><g-emoji class="g-emoji" alias="warning" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/26a0.png">⚠️</g-emoji> <strong>17 Feb 2022: There have been reports of running tmate causing account suspensions</strong>. See <a href="https://github.com/mxschmitt/action-tmate/issues/104">this issue</a> for details. Continue with caution.</p>
</blockquote>
<p>Thanks to <a href="https://twitter.com/harrymarr/status/1304820879268950021" rel="nofollow">this Twitter conversation</a> I found out about <a href="https://github.com/mxschmitt/action-tmate">mxschmitt/action-tmate</a>, which uses <a href="https://tmate.io/" rel="nofollow">https://tmate.io/</a> to open an interactive shell session running inside the GitHub Actions environment.</p>
<p>I created a <code>.github/workflows/tmate.yml</code> file in my repo containing the following:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">tmate session</span>

<span class="pl-ent">on</span>:
  <span class="pl-ent">workflow_dispatch</span>:

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">build</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v2</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Setup tmate session</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">mxschmitt/action-tmate@v3</span></pre></div>
<p>Clicking the "Run workflow" button in the GitHub Actions interface then gave me the following in the GitHub Actions log output:</p>
<pre><code>WebURL: https://tmate.io/t/JA69KaB2avRPRZSkRb8JPa9Gd

SSH: ssh JA69KaB2avRPRZSkRb8JPa9Gd@nyc1.tmate.io
</code></pre>
<p>I ran <code>ssh JA69KaB2avRPRZSkRb8JPa9Gd@nyc1.tmate.io</code> and got a direction connection to the Action, with my project files all available thanks to the <code>- uses: actions/checkout@v2</code> step.</p>
<p>Once I'd finish testing things out in that environment, I typed <code>touch continue</code> and the session ended itself.</p>
<h2>
<a id="user-content-starting-a-shell-just-for-test-failures-on-manual-runs" class="anchor" href="#starting-a-shell-just-for-test-failures-on-manual-runs" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Starting a shell just for test failures on manual runs</h2>
<p>I had a tricky test failure that I wanted to debug interactively. Here's a recipe for starting a tmate shell ONLY if the previous step failed, and only if the run was triggered manually (using <code>workflow_dispatch</code>) - because I don't want an accidental test opening up a shell and burning up my GitHub Actions minutes allowance.</p>
<div class="highlight highlight-source-yaml"><pre>    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">name</span>: <span class="pl-s">Run tests</span>
      <span class="pl-ent">run</span>: <span class="pl-s">pytest</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">tmate session if tests fail</span>
      <span class="pl-ent">if</span>: <span class="pl-s">failure() &amp;&amp; github.event_name == 'workflow_dispatch'</span>
      <span class="pl-ent">uses</span>: <span class="pl-s">mxschmitt/action-tmate@v3</span></pre></div>


path: github-actions_service-containers-docker.md
topic: github-actions
title: Talking to a PostgreSQL service container from inside a Docker container
url: https://github.com/simonw/til/blob/main/github-actions/service-containers-docker.md
created: 2020-09-18T11:43:19-07:00
created_utc: 2020-09-18T18:43:19+00:00
updated: 2020-09-18T11:43:19-07:00
updated_utc: 2020-09-18T18:43:19+00:00
shot_hash: 51077fe6e173bfb07e064112a7303b53
slug: service-containers-docker
body:
I have a Django application which uses PostgreSQL. I build the Django application into its own Docker container, push that built container to the GitHub package registery and then deploy that container to production.

I wanted to run the tests inside the container as part of the deployment process, to make sure the container that I build is ready to be deployed (via continuous deployment).

In production I'm using Digital Ocean PostgreSQL rather than running PostgreSQL in a container. For running the tests I decided to use GitHub's [PostgreSQL service containers](https://docs.github.com/en/actions/guides/creating-postgresql-service-containers) to run the tests.

But how do you set it up so tests running inside a Docker container can talk to the PostgreSQL service container provided by the GitHub Actions environment?

This took a while to figure out. The key insight was that Docker containers (at least on Linux) have a magic IP address, `172.17.0.1`, which can be used to access their host environment - and GitHub's PostgreSQL container is available to that host environment on localhost port 5432.

So here's the recipe I ended up using:

```yaml
name: Build, test and deploy

on:
  push:

jobs:
  build_test_deploy:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres
        env:
          POSTGRES_USER: postgres
          POSTGRES_DB: postgres
          POSTGRES_PASSWORD: postgres
        ports:
          - 5432:5432
        # Health checks to wait until postgres has started
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
    steps:
    - uses: actions/checkout@v2
    - name: Build the tagged Docker image
      run: |-
        docker build -t my-tag .
    - name: Run tests
      run: |-
        docker run \
          -e DATABASE_URL="postgres://postgres:postgres@172.17.0.1:5432/postgres" \
          --entrypoint=/app/github-actions-runtests.sh \
          my-tag
```
My `github-actions-runtests.sh` file uses [django-pytest](https://pytest-django.readthedocs.io/) and looks like this:
```bash
#!/bin/bash
cd /app
pytest --ds=config.test_settings
```

html:
<p>I have a Django application which uses PostgreSQL. I build the Django application into its own Docker container, push that built container to the GitHub package registery and then deploy that container to production.</p>
<p>I wanted to run the tests inside the container as part of the deployment process, to make sure the container that I build is ready to be deployed (via continuous deployment).</p>
<p>In production I'm using Digital Ocean PostgreSQL rather than running PostgreSQL in a container. For running the tests I decided to use GitHub's <a href="https://docs.github.com/en/actions/guides/creating-postgresql-service-containers">PostgreSQL service containers</a> to run the tests.</p>
<p>But how do you set it up so tests running inside a Docker container can talk to the PostgreSQL service container provided by the GitHub Actions environment?</p>
<p>This took a while to figure out. The key insight was that Docker containers (at least on Linux) have a magic IP address, <code>172.17.0.1</code>, which can be used to access their host environment - and GitHub's PostgreSQL container is available to that host environment on localhost port 5432.</p>
<p>So here's the recipe I ended up using:</p>
<div class="highlight highlight-source-yaml"><pre><span class="pl-ent">name</span>: <span class="pl-s">Build, test and deploy</span>

<span class="pl-ent">on</span>:
  <span class="pl-ent">push</span>:

<span class="pl-ent">jobs</span>:
  <span class="pl-ent">build_test_deploy</span>:
    <span class="pl-ent">runs-on</span>: <span class="pl-s">ubuntu-latest</span>
    <span class="pl-ent">services</span>:
      <span class="pl-ent">postgres</span>:
        <span class="pl-ent">image</span>: <span class="pl-s">postgres</span>
        <span class="pl-ent">env</span>:
          <span class="pl-ent">POSTGRES_USER</span>: <span class="pl-s">postgres</span>
          <span class="pl-ent">POSTGRES_DB</span>: <span class="pl-s">postgres</span>
          <span class="pl-ent">POSTGRES_PASSWORD</span>: <span class="pl-s">postgres</span>
        <span class="pl-ent">ports</span>:
          - <span class="pl-c1">5432:5432</span>
        <span class="pl-c"><span class="pl-c">#</span> Health checks to wait until postgres has started</span>
        <span class="pl-ent">options</span>: <span class="pl-s">&gt;-</span>
<span class="pl-s">          --health-cmd pg_isready</span>
<span class="pl-s">          --health-interval 10s</span>
<span class="pl-s">          --health-timeout 5s</span>
<span class="pl-s">          --health-retries 5</span>
<span class="pl-s"></span>    <span class="pl-ent">steps</span>:
    - <span class="pl-ent">uses</span>: <span class="pl-s">actions/checkout@v2</span>
    - <span class="pl-ent">name</span>: <span class="pl-s">Build the tagged Docker image</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        docker build -t my-tag .</span>
<span class="pl-s"></span>    - <span class="pl-ent">name</span>: <span class="pl-s">Run tests</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        docker run \</span>
<span class="pl-s">          -e DATABASE_URL="postgres://postgres:postgres@172.17.0.1:5432/postgres" \</span>
<span class="pl-s">          --entrypoint=/app/github-actions-runtests.sh \</span>
<span class="pl-s">          my-tag</span></pre></div>
<p>My <code>github-actions-runtests.sh</code> file uses <a href="https://pytest-django.readthedocs.io/" rel="nofollow">django-pytest</a> and looks like this:</p>
<div class="highlight highlight-source-shell"><pre><span class="pl-c"><span class="pl-c">#!</span>/bin/bash</span>
<span class="pl-c1">cd</span> /app
pytest --ds=config.test_settings</pre></div>


path: jinja_autoescape-template.md
topic: jinja
title: Turning on Jinja autoescaping when using Template() directly
url: https://github.com/simonw/til/blob/main/jinja/autoescape-template.md
created: 2020-09-18T16:13:44-07:00
created_utc: 2020-09-18T23:13:44+00:00
updated: 2020-09-18T16:13:44-07:00
updated_utc: 2020-09-18T23:13:44+00:00
shot_hash: 2e68ed8f90d94f0ea50ed833e69dda78
slug: autoescape-template
body:
Jinja autoescaping is turned off by default. Coming from Django this frequently catches me out.

You can turn on autoescaping for your Jinja environment using:

```python
from jinja2 import Environment, FileSystemLoader

env = Environment(
    loader=FileSystemLoader("/path/to/templates"),
    autoescape=True
)
```

But what about if you are using `Template` directly? TIL that the `Template` class takes all of the same options as `Environment` does, so you can do this:

```python
from jinja2 import Template

template = Template("""
<p>Hello {{ name }}</p>
""", autoescape=True)

print(template.render({"name": "Simon & Cleo"}))
# Output: <p>Hello Simon &amp; Cleo</p>
```
Here's the [Template class constructor](https://github.com/pallets/jinja/blob/2.11.2/src/jinja2/environment.py#L984-L1005).

html:
<p>Jinja autoescaping is turned off by default. Coming from Django this frequently catches me out.</p>
<p>You can turn on autoescaping for your Jinja environment using:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">jinja2</span> <span class="pl-k">import</span> <span class="pl-v">Environment</span>, <span class="pl-v">FileSystemLoader</span>

<span class="pl-s1">env</span> <span class="pl-c1">=</span> <span class="pl-v">Environment</span>(
    <span class="pl-s1">loader</span><span class="pl-c1">=</span><span class="pl-v">FileSystemLoader</span>(<span class="pl-s">"/path/to/templates"</span>),
    <span class="pl-s1">autoescape</span><span class="pl-c1">=</span><span class="pl-c1">True</span>
)</pre></div>
<p>But what about if you are using <code>Template</code> directly? TIL that the <code>Template</code> class takes all of the same options as <code>Environment</code> does, so you can do this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">jinja2</span> <span class="pl-k">import</span> <span class="pl-v">Template</span>

<span class="pl-s1">template</span> <span class="pl-c1">=</span> <span class="pl-v">Template</span>(<span class="pl-s">"""</span>
<span class="pl-s">&lt;p&gt;Hello {{ name }}&lt;/p&gt;</span>
<span class="pl-s">"""</span>, <span class="pl-s1">autoescape</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)

<span class="pl-en">print</span>(<span class="pl-s1">template</span>.<span class="pl-en">render</span>({<span class="pl-s">"name"</span>: <span class="pl-s">"Simon &amp; Cleo"</span>}))
<span class="pl-c"># Output: &lt;p&gt;Hello Simon &amp;amp; Cleo&lt;/p&gt;</span></pre></div>
<p>Here's the <a href="https://github.com/pallets/jinja/blob/2.11.2/src/jinja2/environment.py#L984-L1005">Template class constructor</a>.</p>


path: sqlite_compile-spellfix-osx.md
topic: sqlite
title: Compiling the SQLite spellfix.c module on macOS
url: https://github.com/simonw/til/blob/main/sqlite/compile-spellfix-osx.md
created: 2020-09-19T16:11:56-07:00
created_utc: 2020-09-19T23:11:56+00:00
updated: 2020-09-19T16:11:56-07:00
updated_utc: 2020-09-19T23:11:56+00:00
shot_hash: 26bfbd99ff003e0cc4c06c0ff8563929
slug: compile-spellfix-osx
body:
I wanted to browse a backup copy of my Plex database, which is a SQLite file. I tried this:

```
$ datasette databaseBackup.db94acbe6d-7442-4361-9663-61f1e97fe930
Usage: datasette serve [OPTIONS] [FILES]...

Error: Connection to databaseBackup.db94acbe6d-7442-4361-9663-61f1e97fe930 failed check: no such module: spellfix1
```
The `spellfix1` module is an optional module for SQLite. It's not shipped as part of the SQLite amalgamation distribution, so you have to compile it separately.

Here's how I did that.

I needed to know my SQLite version, in order to download the correct C module:

```
$ sqlite3 --version
3.28.0 2019-04-15 14:49:49 378230ae7f4b721c8b8d83c8ceb891449685cd23b1702a57841f1be40b5daapl
```
Now I need the source code for `spellfix.c` for that version. I navigated to https://github.com/sqlite/sqlite/blob/master/ext/misc/spellfix.c and then selected the `version-3.28.0` tag from the tag dropdown. This gave me the following page:

https://github.com/sqlite/sqlite/blob/version-3.28.0/ext/misc/spellfix.c

I downloaded the code using the "Raw" link:

```
cd /tmp
wget 'https://raw.githubusercontent.com/sqlite/sqlite/version-3.28.0/ext/misc/spellfix.c'
```
Next step - compile it:

```
gcc -I. -g -fPIC -shared spellfix.c -o spellfix.so
```
The `spellfix.so` module can now be used with Datasette like this:
```
datasette databaseBackup.db94acbe6d-7442-4361-9663-61f1e97fe930 --load-extension=spellfix.so
```

html:
<p>I wanted to browse a backup copy of my Plex database, which is a SQLite file. I tried this:</p>
<pre><code>$ datasette databaseBackup.db94acbe6d-7442-4361-9663-61f1e97fe930
Usage: datasette serve [OPTIONS] [FILES]...

Error: Connection to databaseBackup.db94acbe6d-7442-4361-9663-61f1e97fe930 failed check: no such module: spellfix1
</code></pre>
<p>The <code>spellfix1</code> module is an optional module for SQLite. It's not shipped as part of the SQLite amalgamation distribution, so you have to compile it separately.</p>
<p>Here's how I did that.</p>
<p>I needed to know my SQLite version, in order to download the correct C module:</p>
<pre><code>$ sqlite3 --version
3.28.0 2019-04-15 14:49:49 378230ae7f4b721c8b8d83c8ceb891449685cd23b1702a57841f1be40b5daapl
</code></pre>
<p>Now I need the source code for <code>spellfix.c</code> for that version. I navigated to <a href="https://github.com/sqlite/sqlite/blob/master/ext/misc/spellfix.c">https://github.com/sqlite/sqlite/blob/master/ext/misc/spellfix.c</a> and then selected the <code>version-3.28.0</code> tag from the tag dropdown. This gave me the following page:</p>
<p><a href="https://github.com/sqlite/sqlite/blob/version-3.28.0/ext/misc/spellfix.c">https://github.com/sqlite/sqlite/blob/version-3.28.0/ext/misc/spellfix.c</a></p>
<p>I downloaded the code using the "Raw" link:</p>
<pre><code>cd /tmp
wget 'https://raw.githubusercontent.com/sqlite/sqlite/version-3.28.0/ext/misc/spellfix.c'
</code></pre>
<p>Next step - compile it:</p>
<pre><code>gcc -I. -g -fPIC -shared spellfix.c -o spellfix.so
</code></pre>
<p>The <code>spellfix.so</code> module can now be used with Datasette like this:</p>
<pre><code>datasette databaseBackup.db94acbe6d-7442-4361-9663-61f1e97fe930 --load-extension=spellfix.so
</code></pre>


path: python_click-option-names.md
topic: python
title: Understanding option names in Click
url: https://github.com/simonw/til/blob/main/python/click-option-names.md
created: 2020-09-22T08:49:08-07:00
created_utc: 2020-09-22T15:49:08+00:00
updated: 2020-09-22T08:49:36-07:00
updated_utc: 2020-09-22T15:49:36+00:00
shot_hash: 7fc29350f7e31b4933f501426b2f5cad
slug: click-option-names
body:
I hit [a bug today](https://github.com/simonw/datasette/issues/973) where I had defined a Click option called `open` but in doing so I replaced the Python bulit-in `open()` function:

```python
@click.command()
# ...
@click.option("-o", "--open", is_flag=True, help="Open Datasette in your web browser")
def my_command(open):
    # Now open() is no longer available
```
This inspired me to finally figure out how Click function argument names work. It's documented here: https://click.palletsprojects.com/en/7.x/options/#name-your-options

Short version: you can do this:

```python
@click.command()
# ...
@click.option("-o", "--open", "open_browser", is_flag=True, help="Open Datasette in your web browser")
def my_command(open_browser):
    # Now open() can be used safely
```
Click will use the positional argument without any hyphen prefixes as the name of the argument. If none is provided it will use the first `--` one. If that's not defined it will use the `-o` one - all with the hypens stripped.

html:
<p>I hit <a href="https://github.com/simonw/datasette/issues/973">a bug today</a> where I had defined a Click option called <code>open</code> but in doing so I replaced the Python bulit-in <code>open()</code> function:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">click</span>.<span class="pl-s1">command</span>()</span>
<span class="pl-c"># ...</span>
<span class="pl-en">@<span class="pl-s1">click</span>.<span class="pl-s1">option</span>(<span class="pl-s">"-o"</span>, <span class="pl-s">"--open"</span>, <span class="pl-s1">is_flag</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">help</span><span class="pl-c1">=</span><span class="pl-s">"Open Datasette in your web browser"</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">my_command</span>(<span class="pl-s1">open</span>):
    <span class="pl-c"># Now open() is no longer available</span></pre></div>
<p>This inspired me to finally figure out how Click function argument names work. It's documented here: <a href="https://click.palletsprojects.com/en/7.x/options/#name-your-options" rel="nofollow">https://click.palletsprojects.com/en/7.x/options/#name-your-options</a></p>
<p>Short version: you can do this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">click</span>.<span class="pl-s1">command</span>()</span>
<span class="pl-c"># ...</span>
<span class="pl-en">@<span class="pl-s1">click</span>.<span class="pl-s1">option</span>(<span class="pl-s">"-o"</span>, <span class="pl-s">"--open"</span>, <span class="pl-s">"open_browser"</span>, <span class="pl-s1">is_flag</span><span class="pl-c1">=</span><span class="pl-c1">True</span>, <span class="pl-s1">help</span><span class="pl-c1">=</span><span class="pl-s">"Open Datasette in your web browser"</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">my_command</span>(<span class="pl-s1">open_browser</span>):
    <span class="pl-c"># Now open() can be used safely</span></pre></div>
<p>Click will use the positional argument without any hyphen prefixes as the name of the argument. If none is provided it will use the first <code>--</code> one. If that's not defined it will use the <code>-o</code> one - all with the hypens stripped.</p>


path: sqlite_text-value-is-integer-or-float.md
topic: sqlite
title: Figuring out if a text value in SQLite is a valid integer or float
url: https://github.com/simonw/til/blob/main/sqlite/text-value-is-integer-or-float.md
created: 2020-09-27T13:40:22-07:00
created_utc: 2020-09-27T20:40:22+00:00
updated: 2020-12-29T13:55:23-08:00
updated_utc: 2020-12-29T21:55:23+00:00
shot_hash: cdc8bbe0eb4f58fc60de70430a4e98b1
slug: text-value-is-integer-or-float
body:
Given a table with a `TEXT` column in SQLite I want to figure out if every value in that table is actually the text representation of an integer or floating point value, so I can decide if it's a good idea to change the type of the column (using [sqlite-utils transform](https://sqlite-utils.datasette.io/en/stable/python-api.html#transforming-a-table)).

To do this efficiently, I want a SQLite idiom that will tell me if a string value is a valid integer or floating point number.

After [much tinkering](https://github.com/simonw/sqlite-utils/issues/179) I've found two recipes for this that seem to work well.

This evaluates to true if value contains a valid integer representation:
```sql
cast(cast(value AS INTEGER) AS TEXT) = value
```
And this does the same thing for floating point numbers:
```sql
cast(cast(value AS REAL) AS TEXT) in (value, value || '.0')
```
The `|| '.0'` bit there is needed because `cast('1' as REAL)` returns `1.0`, not just `1`.

(Note that `1.200` will not pass this test and will be incorrectly considered an invalid floating point representation)

## Demos

The float version:

```sql
select
  value,
  cast(cast(value AS REAL) AS TEXT) in (value, value || '.0') as is_valid_float
from
  (
    select
      '1' as value
    union
    select
      '1.1' as value
    union
    select
      'dog' as value
    union
    select
      null as value
  )
```
[Try that here](https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+REAL%29+AS+TEXT%29+in+%28value%2C+value+%7C%7C+%27.0%27%29+as+is_valid_float%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29)

| value | is_valid_float |
| ----- | -------------- |
| null  | null           |
| 1     | 1              |
| 1.1   | 1              |
| dog   | 0              |

The integer version:
```sql
select
  value,
  cast(cast(value AS INTEGER) AS TEXT) = value as is_valid_int
from
  (
    select
      '1' as value
    union
    select
      '1.1' as value
    union
    select
      'dog' as value
    union
    select
      null as value
  )
```
[Try that here](https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+INTEGER%29+AS+TEXT%29+%3D+value+as+is_valid_int%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29)

| value | is_valid_int |
| ----- | ------------ |
| null  | null         |
| 1     | 1            |
| 1.1   | 0            |
| dog   | 0            |

html:
<p>Given a table with a <code>TEXT</code> column in SQLite I want to figure out if every value in that table is actually the text representation of an integer or floating point value, so I can decide if it's a good idea to change the type of the column (using <a href="https://sqlite-utils.datasette.io/en/stable/python-api.html#transforming-a-table" rel="nofollow">sqlite-utils transform</a>).</p>
<p>To do this efficiently, I want a SQLite idiom that will tell me if a string value is a valid integer or floating point number.</p>
<p>After <a href="https://github.com/simonw/sqlite-utils/issues/179">much tinkering</a> I've found two recipes for this that seem to work well.</p>
<p>This evaluates to true if value contains a valid integer representation:</p>
<div class="highlight highlight-source-sql"><pre>cast(cast(value <span class="pl-k">AS</span> <span class="pl-k">INTEGER</span>) <span class="pl-k">AS</span> <span class="pl-k">TEXT</span>) <span class="pl-k">=</span> value</pre></div>
<p>And this does the same thing for floating point numbers:</p>
<div class="highlight highlight-source-sql"><pre>cast(cast(value <span class="pl-k">AS</span> <span class="pl-k">REAL</span>) <span class="pl-k">AS</span> <span class="pl-k">TEXT</span>) <span class="pl-k">in</span> (value, value <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>.0<span class="pl-pds">'</span></span>)</pre></div>
<p>The <code>|| '.0'</code> bit there is needed because <code>cast('1' as REAL)</code> returns <code>1.0</code>, not just <code>1</code>.</p>
<p>(Note that <code>1.200</code> will not pass this test and will be incorrectly considered an invalid floating point representation)</p>
<h2>
<a id="user-content-demos" class="anchor" href="#demos" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Demos</h2>
<p>The float version:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  value,
  cast(cast(value <span class="pl-k">AS</span> <span class="pl-k">REAL</span>) <span class="pl-k">AS</span> <span class="pl-k">TEXT</span>) <span class="pl-k">in</span> (value, value <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>.0<span class="pl-pds">'</span></span>) <span class="pl-k">as</span> is_valid_float
<span class="pl-k">from</span>
  (
    <span class="pl-k">select</span>
      <span class="pl-s"><span class="pl-pds">'</span>1<span class="pl-pds">'</span></span> <span class="pl-k">as</span> value
    <span class="pl-k">union</span>
    <span class="pl-k">select</span>
      <span class="pl-s"><span class="pl-pds">'</span>1.1<span class="pl-pds">'</span></span> <span class="pl-k">as</span> value
    <span class="pl-k">union</span>
    <span class="pl-k">select</span>
      <span class="pl-s"><span class="pl-pds">'</span>dog<span class="pl-pds">'</span></span> <span class="pl-k">as</span> value
    <span class="pl-k">union</span>
    <span class="pl-k">select</span>
      <span class="pl-k">null</span> <span class="pl-k">as</span> value
  )</pre></div>
<p><a href="https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+REAL%29+AS+TEXT%29+in+%28value%2C+value+%7C%7C+%27.0%27%29+as+is_valid_float%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29" rel="nofollow">Try that here</a></p>
<table>
<thead>
<tr>
<th>value</th>
<th>is_valid_float</th>
</tr>
</thead>
<tbody>
<tr>
<td>null</td>
<td>null</td>
</tr>
<tr>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>1.1</td>
<td>1</td>
</tr>
<tr>
<td>dog</td>
<td>0</td>
</tr>
</tbody>
</table>
<p>The integer version:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  value,
  cast(cast(value <span class="pl-k">AS</span> <span class="pl-k">INTEGER</span>) <span class="pl-k">AS</span> <span class="pl-k">TEXT</span>) <span class="pl-k">=</span> value <span class="pl-k">as</span> is_valid_int
<span class="pl-k">from</span>
  (
    <span class="pl-k">select</span>
      <span class="pl-s"><span class="pl-pds">'</span>1<span class="pl-pds">'</span></span> <span class="pl-k">as</span> value
    <span class="pl-k">union</span>
    <span class="pl-k">select</span>
      <span class="pl-s"><span class="pl-pds">'</span>1.1<span class="pl-pds">'</span></span> <span class="pl-k">as</span> value
    <span class="pl-k">union</span>
    <span class="pl-k">select</span>
      <span class="pl-s"><span class="pl-pds">'</span>dog<span class="pl-pds">'</span></span> <span class="pl-k">as</span> value
    <span class="pl-k">union</span>
    <span class="pl-k">select</span>
      <span class="pl-k">null</span> <span class="pl-k">as</span> value
  )</pre></div>
<p><a href="https://latest.datasette.io/fixtures?sql=select%0D%0A++value%2C%0D%0A++cast%28cast%28value+AS+INTEGER%29+AS+TEXT%29+%3D+value+as+is_valid_int%0D%0Afrom%0D%0A++%28%0D%0A++++select%0D%0A++++++%271%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%271.1%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++%27dog%27+as+value%0D%0A++++union%0D%0A++++select%0D%0A++++++null+as+value%0D%0A++%29" rel="nofollow">Try that here</a></p>
<table>
<thead>
<tr>
<th>value</th>
<th>is_valid_int</th>
</tr>
</thead>
<tbody>
<tr>
<td>null</td>
<td>null</td>
</tr>
<tr>
<td>1</td>
<td>1</td>
</tr>
<tr>
<td>1.1</td>
<td>0</td>
</tr>
<tr>
<td>dog</td>
<td>0</td>
</tr>
</tbody>
</table>


path: bash_escaping-a-string.md
topic: bash
title: Escaping strings in Bash using !:q
url: https://github.com/simonw/til/blob/main/bash/escaping-a-string.md
created: 2020-10-01T13:32:02-07:00
created_utc: 2020-10-01T20:32:02+00:00
updated: 2020-10-03T22:30:04-07:00
updated_utc: 2020-10-04T05:30:04+00:00
shot_hash: 7b7a5e24ac848f6bd511817576928928
slug: escaping-a-string
body:
TIL this trick, [via Pascal Hirsch](https://twitter.com/phphys/status/1311727268398465029) on Twitter. Enter a line of Bash starting with a `#` comment, then run `!:q` on the next line to see what that would be with proper Bash escaping applied.

```
bash-3.2$ # This string 'has single' "and double" quotes and a $
bash-3.2$ !:q
'# This string '\''has single'\'' "and double" quotes and a $'
bash: # This string 'has single' "and double" quotes and a $: command not found
```
How does this work? [James Coglan explains](https://twitter.com/mountain_ghosts/status/1311767073933099010):

> The `!` character begins a history expansion; `!string` produces the last command beginning with `string`, and `:q` is a modifier that quotes the result; so I'm guessing this is equivalent to `!string` where `string` is `""`, so it produces the most recent command, just like `!!` does

A bunch more useful tips in the [thread about this on Hacker News](https://news.ycombinator.com/item?id=24659282).

html:
<p>TIL this trick, <a href="https://twitter.com/phphys/status/1311727268398465029" rel="nofollow">via Pascal Hirsch</a> on Twitter. Enter a line of Bash starting with a <code>#</code> comment, then run <code>!:q</code> on the next line to see what that would be with proper Bash escaping applied.</p>
<pre><code>bash-3.2$ # This string 'has single' "and double" quotes and a $
bash-3.2$ !:q
'# This string '\''has single'\'' "and double" quotes and a $'
bash: # This string 'has single' "and double" quotes and a $: command not found
</code></pre>
<p>How does this work? <a href="https://twitter.com/mountain_ghosts/status/1311767073933099010" rel="nofollow">James Coglan explains</a>:</p>
<blockquote>
<p>The <code>!</code> character begins a history expansion; <code>!string</code> produces the last command beginning with <code>string</code>, and <code>:q</code> is a modifier that quotes the result; so I'm guessing this is equivalent to <code>!string</code> where <code>string</code> is <code>""</code>, so it produces the most recent command, just like <code>!!</code> does</p>
</blockquote>
<p>A bunch more useful tips in the <a href="https://news.ycombinator.com/item?id=24659282" rel="nofollow">thread about this on Hacker News</a>.</p>


path: selenium_selenium-python-macos.md
topic: selenium
title: Installing Selenium for Python on macOS with ChromeDriver
url: https://github.com/simonw/til/blob/main/selenium/selenium-python-macos.md
created: 2020-10-02T13:06:54-07:00
created_utc: 2020-10-02T20:06:54+00:00
updated: 2020-10-02T15:23:48-07:00
updated_utc: 2020-10-02T22:23:48+00:00
shot_hash: 07893e37d7de129a16663e74d4491aac
slug: selenium-python-macos
body:
I needed to run Selenium on macOS for the first time today. Here's how I got it working.

## Install the chromedriver binary

### If you have homebrew

This is by far the easiest option:

    brew cask install chromedriver

This also ensures `chromedriver` is on your path, which means you don't need to use an explicit `chromedriver_path` later on.

You still need to run it once in the terminal `chromedriver` to get the macOS error, then allow it in the `Security & Privacy` preferences - see below.

### Without using homebrew

ChromeDriver is available from the official website here: https://sites.google.com/a/chromium.org/chromedriver/downloads

I have Chrome 85 so I downloaded the `chromedriver_mac64.zip` file from https://chromedriver.storage.googleapis.com/index.html?path=85.0.4183.87/

Unzipping this gave me a `chromedriver` binary file. I decided to put this in my `~/bin` directory.

### Skipping the error on macOS

The first time I ran it I got an error complaining that the binary has not been signed:

    ~/bin/chromedriver
    # A window displayed on macOS with an error

To fix this, go to `System Preferences -> Security & Privacy` - there was a prompt there about the binary, with an "open this anyway" button. Clicking that worked around the signing issue.

## Installing the Selenium Python driver

I installed Selenium using `pip` for Python 3:

    pip install selenium

Since I was planning to use it from a Jupyter Notebook I actually installed it by running the following in a cell in a notebook:

    %pip install selenium

The benefit of running this in the notebook is that you don't need to know the exact path to pip running in the same virtual environment as Jupyter, so I use this trick a lot.

## Demonstrating that it works

I ran this in a notebook cell:

```python
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

# This is not needed if chromedriver is already on your path:
chromedriver_path = "/Users/simon/bin/chromedriver"

options = Options()
options.add_argument("--window-size=1920x1080")
options.add_argument("--verbose")
# options.add_argument("--headless")

driver = webdriver.Chrome(options=options, chromedriver_path=executable_path)
driver.get("https://www.example.com")
```
This opened a visible Chrome window to `https://www.example.com/`
```python
print(driver.find_element_by_css_selector('body').text)
```
This output the following, showing that Selenium is fully working:
```
Example Domain
This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.
More information...
```

## Installing geckodriver for Firefox

I got Firefox support working by downloading the `geckodriver` binary from https://github.com/mozilla/geckodriver and copying that to my `~/bin/` directory. Then this worked:
```python
from selenium import webdriver

firefox = webdriver.Firefox(executable_path="/Users/simon/bin/geckodriver")
firefox.get('http://seleniumhq.org/')
print(firefox.find_element_by_css_selector('body').text)
```
I used `wget` for the download (rather than clicking the link in my browser) thanks to the warning here: https://firefox-source-docs.mozilla.org/testing/geckodriver/Notarization.html

An easier option: install it with Homebrew:

    brew install geckodriver

This puts it on the PATH and ensures the code is already signed and does not show a warning. You can then use it like this:

```python
from selenium import webdriver

firefox = webdriver.Firefox()
firefox.get('http://seleniumhq.org/')
```
You can close the Firefox window (and terminate the Firefox process) later like this:
```python
firefox.close()
```

html:
<p>I needed to run Selenium on macOS for the first time today. Here's how I got it working.</p>
<h2>
<a id="user-content-install-the-chromedriver-binary" class="anchor" href="#install-the-chromedriver-binary" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Install the chromedriver binary</h2>
<h3>
<a id="user-content-if-you-have-homebrew" class="anchor" href="#if-you-have-homebrew" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>If you have homebrew</h3>
<p>This is by far the easiest option:</p>
<pre><code>brew cask install chromedriver
</code></pre>
<p>This also ensures <code>chromedriver</code> is on your path, which means you don't need to use an explicit <code>chromedriver_path</code> later on.</p>
<p>You still need to run it once in the terminal <code>chromedriver</code> to get the macOS error, then allow it in the <code>Security &amp; Privacy</code> preferences - see below.</p>
<h3>
<a id="user-content-without-using-homebrew" class="anchor" href="#without-using-homebrew" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Without using homebrew</h3>
<p>ChromeDriver is available from the official website here: <a href="https://sites.google.com/a/chromium.org/chromedriver/downloads" rel="nofollow">https://sites.google.com/a/chromium.org/chromedriver/downloads</a></p>
<p>I have Chrome 85 so I downloaded the <code>chromedriver_mac64.zip</code> file from <a href="https://chromedriver.storage.googleapis.com/index.html?path=85.0.4183.87/" rel="nofollow">https://chromedriver.storage.googleapis.com/index.html?path=85.0.4183.87/</a></p>
<p>Unzipping this gave me a <code>chromedriver</code> binary file. I decided to put this in my <code>~/bin</code> directory.</p>
<h3>
<a id="user-content-skipping-the-error-on-macos" class="anchor" href="#skipping-the-error-on-macos" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Skipping the error on macOS</h3>
<p>The first time I ran it I got an error complaining that the binary has not been signed:</p>
<pre><code>~/bin/chromedriver
# A window displayed on macOS with an error
</code></pre>
<p>To fix this, go to <code>System Preferences -&gt; Security &amp; Privacy</code> - there was a prompt there about the binary, with an "open this anyway" button. Clicking that worked around the signing issue.</p>
<h2>
<a id="user-content-installing-the-selenium-python-driver" class="anchor" href="#installing-the-selenium-python-driver" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing the Selenium Python driver</h2>
<p>I installed Selenium using <code>pip</code> for Python 3:</p>
<pre><code>pip install selenium
</code></pre>
<p>Since I was planning to use it from a Jupyter Notebook I actually installed it by running the following in a cell in a notebook:</p>
<pre><code>%pip install selenium
</code></pre>
<p>The benefit of running this in the notebook is that you don't need to know the exact path to pip running in the same virtual environment as Jupyter, so I use this trick a lot.</p>
<h2>
<a id="user-content-demonstrating-that-it-works" class="anchor" href="#demonstrating-that-it-works" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Demonstrating that it works</h2>
<p>I ran this in a notebook cell:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">selenium</span> <span class="pl-k">import</span> <span class="pl-s1">webdriver</span>
<span class="pl-k">from</span> <span class="pl-s1">selenium</span>.<span class="pl-s1">webdriver</span>.<span class="pl-s1">chrome</span>.<span class="pl-s1">options</span> <span class="pl-k">import</span> <span class="pl-v">Options</span>

<span class="pl-c"># This is not needed if chromedriver is already on your path:</span>
<span class="pl-s1">chromedriver_path</span> <span class="pl-c1">=</span> <span class="pl-s">"/Users/simon/bin/chromedriver"</span>

<span class="pl-s1">options</span> <span class="pl-c1">=</span> <span class="pl-v">Options</span>()
<span class="pl-s1">options</span>.<span class="pl-en">add_argument</span>(<span class="pl-s">"--window-size=1920x1080"</span>)
<span class="pl-s1">options</span>.<span class="pl-en">add_argument</span>(<span class="pl-s">"--verbose"</span>)
<span class="pl-c"># options.add_argument("--headless")</span>

<span class="pl-s1">driver</span> <span class="pl-c1">=</span> <span class="pl-s1">webdriver</span>.<span class="pl-v">Chrome</span>(<span class="pl-s1">options</span><span class="pl-c1">=</span><span class="pl-s1">options</span>, <span class="pl-s1">chromedriver_path</span><span class="pl-c1">=</span><span class="pl-s1">executable_path</span>)
<span class="pl-s1">driver</span>.<span class="pl-en">get</span>(<span class="pl-s">"https://www.example.com"</span>)</pre></div>
<p>This opened a visible Chrome window to <code>https://www.example.com/</code></p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">print</span>(<span class="pl-s1">driver</span>.<span class="pl-en">find_element_by_css_selector</span>(<span class="pl-s">'body'</span>).<span class="pl-s1">text</span>)</pre></div>
<p>This output the following, showing that Selenium is fully working:</p>
<pre><code>Example Domain
This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.
More information...
</code></pre>
<h2>
<a id="user-content-installing-geckodriver-for-firefox" class="anchor" href="#installing-geckodriver-for-firefox" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Installing geckodriver for Firefox</h2>
<p>I got Firefox support working by downloading the <code>geckodriver</code> binary from <a href="https://github.com/mozilla/geckodriver">https://github.com/mozilla/geckodriver</a> and copying that to my <code>~/bin/</code> directory. Then this worked:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">selenium</span> <span class="pl-k">import</span> <span class="pl-s1">webdriver</span>

<span class="pl-s1">firefox</span> <span class="pl-c1">=</span> <span class="pl-s1">webdriver</span>.<span class="pl-v">Firefox</span>(<span class="pl-s1">executable_path</span><span class="pl-c1">=</span><span class="pl-s">"/Users/simon/bin/geckodriver"</span>)
<span class="pl-s1">firefox</span>.<span class="pl-en">get</span>(<span class="pl-s">'http://seleniumhq.org/'</span>)
<span class="pl-en">print</span>(<span class="pl-s1">firefox</span>.<span class="pl-en">find_element_by_css_selector</span>(<span class="pl-s">'body'</span>).<span class="pl-s1">text</span>)</pre></div>
<p>I used <code>wget</code> for the download (rather than clicking the link in my browser) thanks to the warning here: <a href="https://firefox-source-docs.mozilla.org/testing/geckodriver/Notarization.html" rel="nofollow">https://firefox-source-docs.mozilla.org/testing/geckodriver/Notarization.html</a></p>
<p>An easier option: install it with Homebrew:</p>
<pre><code>brew install geckodriver
</code></pre>
<p>This puts it on the PATH and ensures the code is already signed and does not show a warning. You can then use it like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">selenium</span> <span class="pl-k">import</span> <span class="pl-s1">webdriver</span>

<span class="pl-s1">firefox</span> <span class="pl-c1">=</span> <span class="pl-s1">webdriver</span>.<span class="pl-v">Firefox</span>()
<span class="pl-s1">firefox</span>.<span class="pl-en">get</span>(<span class="pl-s">'http://seleniumhq.org/'</span>)</pre></div>
<p>You can close the Firefox window (and terminate the Firefox process) later like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">firefox</span>.<span class="pl-en">close</span>()</pre></div>


path: selenium_async-javascript-in-selenium.md
topic: selenium
title: Using async/await in JavaScript in Selenium
url: https://github.com/simonw/til/blob/main/selenium/async-javascript-in-selenium.md
created: 2020-10-02T14:19:41-07:00
created_utc: 2020-10-02T21:19:41+00:00
updated: 2020-10-02T14:19:41-07:00
updated_utc: 2020-10-02T21:19:41+00:00
shot_hash: 992eac8c26a8aaa7cf7fc833e872275d
slug: async-javascript-in-selenium
body:
Thanks [Stuart Langridge](https://twitter.com/sil/status/1312137808111304704) for showing me how to do this:

```python
from selenium import webdriver

chromedriver_path = "/Users/simon/bin/chromedriver"
driver = webdriver.Chrome(executable_path=chromedriver_path)

script = """
done = arguments[arguments.length-1];
a1 = async () => {
    return 42
};
a2 = async() => {
    return await a1()+1
};
a2().then(done);
"""
output = driver.execute_async_script(script)
# output is now the Python integer 43
```

html:
<p>Thanks <a href="https://twitter.com/sil/status/1312137808111304704" rel="nofollow">Stuart Langridge</a> for showing me how to do this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">selenium</span> <span class="pl-k">import</span> <span class="pl-s1">webdriver</span>

<span class="pl-s1">chromedriver_path</span> <span class="pl-c1">=</span> <span class="pl-s">"/Users/simon/bin/chromedriver"</span>
<span class="pl-s1">driver</span> <span class="pl-c1">=</span> <span class="pl-s1">webdriver</span>.<span class="pl-v">Chrome</span>(<span class="pl-s1">executable_path</span><span class="pl-c1">=</span><span class="pl-s1">chromedriver_path</span>)

<span class="pl-s1">script</span> <span class="pl-c1">=</span> <span class="pl-s">"""</span>
<span class="pl-s">done = arguments[arguments.length-1];</span>
<span class="pl-s">a1 = async () =&gt; {</span>
<span class="pl-s">    return 42</span>
<span class="pl-s">};</span>
<span class="pl-s">a2 = async() =&gt; {</span>
<span class="pl-s">    return await a1()+1</span>
<span class="pl-s">};</span>
<span class="pl-s">a2().then(done);</span>
<span class="pl-s">"""</span>
<span class="pl-s1">output</span> <span class="pl-c1">=</span> <span class="pl-s1">driver</span>.<span class="pl-en">execute_async_script</span>(<span class="pl-s1">script</span>)
<span class="pl-c"># output is now the Python integer 43</span></pre></div>


path: javascript_manipulating-query-params.md
topic: javascript
title: Manipulating query strings with URLSearchParams
url: https://github.com/simonw/til/blob/main/javascript/manipulating-query-params.md
created: 2020-10-04T11:35:21-07:00
created_utc: 2020-10-04T18:35:21+00:00
updated: 2020-10-04T11:35:21-07:00
updated_utc: 2020-10-04T18:35:21+00:00
shot_hash: e641c68a9fe41d1235785d7c0e0c68c7
slug: manipulating-query-params
body:
The `URLSearchParams` class, in [every modern browser](https://caniuse.com/?search=URLSearchParams) since IE 11, provides a sensible API for manipulating query string parameters in JavaScript. I first used it to build Datasette's column action menu, see [table.js](https://github.com/simonw/datasette/blob/0.50a0/datasette/static/table.js) and [issue 981](https://github.com/simonw/datasette/issues/981).

Here's how it handles multiple parameters with the same name, e.g. `?foo=bar&foo=baz`:

```javascript
var params = new URLSearchParams('?foo=bar&foo=baz')
console.log(params.get("foo"))
// Outputs "bar"
console.log(params.getAll("foo"))
// Outputs ["bar", "baz"]
console.log(params.get("foo2"))
// Outputs null
console.log(params.getAll("foo2"))
// Outputs []
```

It can also be used to add, remove and append values - then turned back into a query string using `params.toString()`:

```javascript
var params = new URLSearchParams('?foo=bar&foo=baz')
params.append("foo", "another");
params.toString()
// "foo=bar&foo=baz&foo=another"
params.set("foo", "over-written")
// "foo=over-written"
params.delete("foo")
params.toString()
// ""
```
To construct a parameters object from the query string used on the current page, do this:
```javascript
var params = new URLSearchParams(location.search);
```
It doesn't matter if the string passed to `new URLSearchParams()` has a leading question mark or not - the result is the same.

html:
<p>The <code>URLSearchParams</code> class, in <a href="https://caniuse.com/?search=URLSearchParams" rel="nofollow">every modern browser</a> since IE 11, provides a sensible API for manipulating query string parameters in JavaScript. I first used it to build Datasette's column action menu, see <a href="https://github.com/simonw/datasette/blob/0.50a0/datasette/static/table.js">table.js</a> and <a href="https://github.com/simonw/datasette/issues/981">issue 981</a>.</p>
<p>Here's how it handles multiple parameters with the same name, e.g. <code>?foo=bar&amp;foo=baz</code>:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">var</span> <span class="pl-s1">params</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">URLSearchParams</span><span class="pl-kos">(</span><span class="pl-s">'?foo=bar&amp;foo=baz'</span><span class="pl-kos">)</span>
<span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">params</span><span class="pl-kos">.</span><span class="pl-en">get</span><span class="pl-kos">(</span><span class="pl-s">"foo"</span><span class="pl-kos">)</span><span class="pl-kos">)</span>
<span class="pl-c">// Outputs "bar"</span>
<span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">params</span><span class="pl-kos">.</span><span class="pl-en">getAll</span><span class="pl-kos">(</span><span class="pl-s">"foo"</span><span class="pl-kos">)</span><span class="pl-kos">)</span>
<span class="pl-c">// Outputs ["bar", "baz"]</span>
<span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">params</span><span class="pl-kos">.</span><span class="pl-en">get</span><span class="pl-kos">(</span><span class="pl-s">"foo2"</span><span class="pl-kos">)</span><span class="pl-kos">)</span>
<span class="pl-c">// Outputs null</span>
<span class="pl-smi">console</span><span class="pl-kos">.</span><span class="pl-en">log</span><span class="pl-kos">(</span><span class="pl-s1">params</span><span class="pl-kos">.</span><span class="pl-en">getAll</span><span class="pl-kos">(</span><span class="pl-s">"foo2"</span><span class="pl-kos">)</span><span class="pl-kos">)</span>
<span class="pl-c">// Outputs []</span></pre></div>
<p>It can also be used to add, remove and append values - then turned back into a query string using <code>params.toString()</code>:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">var</span> <span class="pl-s1">params</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">URLSearchParams</span><span class="pl-kos">(</span><span class="pl-s">'?foo=bar&amp;foo=baz'</span><span class="pl-kos">)</span>
<span class="pl-s1">params</span><span class="pl-kos">.</span><span class="pl-en">append</span><span class="pl-kos">(</span><span class="pl-s">"foo"</span><span class="pl-kos">,</span> <span class="pl-s">"another"</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-s1">params</span><span class="pl-kos">.</span><span class="pl-en">toString</span><span class="pl-kos">(</span><span class="pl-kos">)</span>
<span class="pl-c">// "foo=bar&amp;foo=baz&amp;foo=another"</span>
<span class="pl-s1">params</span><span class="pl-kos">.</span><span class="pl-en">set</span><span class="pl-kos">(</span><span class="pl-s">"foo"</span><span class="pl-kos">,</span> <span class="pl-s">"over-written"</span><span class="pl-kos">)</span>
<span class="pl-c">// "foo=over-written"</span>
<span class="pl-s1">params</span><span class="pl-kos">.</span><span class="pl-en">delete</span><span class="pl-kos">(</span><span class="pl-s">"foo"</span><span class="pl-kos">)</span>
<span class="pl-s1">params</span><span class="pl-kos">.</span><span class="pl-en">toString</span><span class="pl-kos">(</span><span class="pl-kos">)</span>
<span class="pl-c">// ""</span></pre></div>
<p>To construct a parameters object from the query string used on the current page, do this:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">var</span> <span class="pl-s1">params</span> <span class="pl-c1">=</span> <span class="pl-k">new</span> <span class="pl-v">URLSearchParams</span><span class="pl-kos">(</span><span class="pl-s1">location</span><span class="pl-kos">.</span><span class="pl-c1">search</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<p>It doesn't matter if the string passed to <code>new URLSearchParams()</code> has a leading question mark or not - the result is the same.</p>


path: digitalocean_datasette-on-digitalocean-app-platform.md
topic: digitalocean
title: Running Datasette on DigitalOcean App Platform
url: https://github.com/simonw/til/blob/main/digitalocean/datasette-on-digitalocean-app-platform.md
created: 2020-10-06T19:45:25-07:00
created_utc: 2020-10-07T02:45:25+00:00
updated: 2020-10-07T07:29:46-07:00
updated_utc: 2020-10-07T14:29:46+00:00
shot_hash: 412787a6bcb503088eddbc6cfbd2114f
slug: datasette-on-digitalocean-app-platform
body:
[App Platform](https://www.digitalocean.com/docs/app-platform/) is the new PaaS from DigitalOcean. I figured out how to run Datasette on it.

The bare minimum needed is a GitHub repository with two files: `requirements.txt` and `Procfile`.

`requirements.txt` can contain a single line:
```
datasette
```
`Procfile` needs this:
```
web: datasette . -h 0.0.0.0 -p $PORT --cors
```

Your web process needs to listen on `0.0.0.0` and on the port in the `$PORT` environment variable.

Connect this GitHub repository up to DigitalOcean App Platform and it will deploy the application - detecting that it's a Python application (due to the `requirements.txt` file), installing those requirements and then starting up the process in the `Procfile`.

Any SQLite `.db` files that you add to the root of the GitHub repository will be automatically served by Datasette when it starts up.

Because Datasette is run using `datasette .` it will also automatically pick up a `metadata.json` file or anything in custom `templates/` or `plugins/` folders, as described in [Configuration directory mode](https://docs.datasette.io/en/stable/config.html#configuration-directory-mode) in the documentation.

## Building database files

I don't particularly like putting binary SQLite files in a GitHub repository - I prefer to store CSV files or SQL text files and build them into a database file as part of the deployment process.

The best way I've found to do this in a DigitalOcean App is to create a `build.sh` script that builds the database, then execute it using a `Build Command`.

One way to do this is to visit the "Components" tab end click "Edit" in the Commands section, then set the "Build Command" to `. build.sh`. Now any code you add to a `build.sh` script in your repo will be executed as part of the deployment.

A better way (thanks, [Kamal Nasser](https://www.digitalocean.com/community/questions/configure-a-build-command-for-a-python-project-without-using-the-web-ui?comment=92105)) is to use a `bin/pre_compile` or `bin/post_compile` script in your repository.

I started with a `build.sh` script that looked like this:

```
wget https://latest.datasette.io/fixtures.db
```
And this resulted in the `fixtures.db` folder being served at `/fixtures` under my app's subdomain.

html:
<p><a href="https://www.digitalocean.com/docs/app-platform/" rel="nofollow">App Platform</a> is the new PaaS from DigitalOcean. I figured out how to run Datasette on it.</p>
<p>The bare minimum needed is a GitHub repository with two files: <code>requirements.txt</code> and <code>Procfile</code>.</p>
<p><code>requirements.txt</code> can contain a single line:</p>
<pre><code>datasette
</code></pre>
<p><code>Procfile</code> needs this:</p>
<pre><code>web: datasette . -h 0.0.0.0 -p $PORT --cors
</code></pre>
<p>Your web process needs to listen on <code>0.0.0.0</code> and on the port in the <code>$PORT</code> environment variable.</p>
<p>Connect this GitHub repository up to DigitalOcean App Platform and it will deploy the application - detecting that it's a Python application (due to the <code>requirements.txt</code> file), installing those requirements and then starting up the process in the <code>Procfile</code>.</p>
<p>Any SQLite <code>.db</code> files that you add to the root of the GitHub repository will be automatically served by Datasette when it starts up.</p>
<p>Because Datasette is run using <code>datasette .</code> it will also automatically pick up a <code>metadata.json</code> file or anything in custom <code>templates/</code> or <code>plugins/</code> folders, as described in <a href="https://docs.datasette.io/en/stable/config.html#configuration-directory-mode" rel="nofollow">Configuration directory mode</a> in the documentation.</p>
<h2>
<a id="user-content-building-database-files" class="anchor" href="#building-database-files" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Building database files</h2>
<p>I don't particularly like putting binary SQLite files in a GitHub repository - I prefer to store CSV files or SQL text files and build them into a database file as part of the deployment process.</p>
<p>The best way I've found to do this in a DigitalOcean App is to create a <code>build.sh</code> script that builds the database, then execute it using a <code>Build Command</code>.</p>
<p>One way to do this is to visit the "Components" tab end click "Edit" in the Commands section, then set the "Build Command" to <code>. build.sh</code>. Now any code you add to a <code>build.sh</code> script in your repo will be executed as part of the deployment.</p>
<p>A better way (thanks, <a href="https://www.digitalocean.com/community/questions/configure-a-build-command-for-a-python-project-without-using-the-web-ui?comment=92105" rel="nofollow">Kamal Nasser</a>) is to use a <code>bin/pre_compile</code> or <code>bin/post_compile</code> script in your repository.</p>
<p>I started with a <code>build.sh</code> script that looked like this:</p>
<pre><code>wget https://latest.datasette.io/fixtures.db
</code></pre>
<p>And this resulted in the <code>fixtures.db</code> folder being served at <code>/fixtures</code> under my app's subdomain.</p>


path: github_graphql-search-topics.md
topic: github
title: Searching for repositories by topic using the GitHub GraphQL API
url: https://github.com/simonw/til/blob/main/github/graphql-search-topics.md
created: 2020-10-09T19:50:43-07:00
created_utc: 2020-10-10T02:50:43+00:00
updated: 2020-10-09T19:50:43-07:00
updated_utc: 2020-10-10T02:50:43+00:00
shot_hash: 8105336c7d99543e11e3772b25b443d6
slug: graphql-search-topics
body:
I wanted to use the GitHub GraphQL API to return all of the repositories on the https://github.com/topics/git-scraping page.

At first glance there isn't a GraphQL field for that page - but it turns out you can access it using a GitHub search:

    topic:git-scraping sort:updated-desc

An oddity of GitHub search is that sort order can be defined using tokens that form part of the search query!

Here's a GraphQL query [tested here](https://developer.github.com/v4/explorer/) that returns the most recent 100 `git-scraping` tagged repos, sorted by most recently updated.

```graphql
{
  search(query: "topic:git-scraping sort:updated-desc", type: REPOSITORY, first: 100) {
    repositoryCount
    nodes {
      ... on Repository {
        nameWithOwner
        description
        updatedAt
        createdAt
        diskUsage
      }
    }
  }
}
```

html:
<p>I wanted to use the GitHub GraphQL API to return all of the repositories on the <a href="https://github.com/topics/git-scraping">https://github.com/topics/git-scraping</a> page.</p>
<p>At first glance there isn't a GraphQL field for that page - but it turns out you can access it using a GitHub search:</p>
<pre><code>topic:git-scraping sort:updated-desc
</code></pre>
<p>An oddity of GitHub search is that sort order can be defined using tokens that form part of the search query!</p>
<p>Here's a GraphQL query <a href="https://developer.github.com/v4/explorer/">tested here</a> that returns the most recent 100 <code>git-scraping</code> tagged repos, sorted by most recently updated.</p>
<div class="highlight highlight-source-graphql"><pre>{
  <span class="pl-v">search</span>(<span class="pl-v">query</span>: <span class="pl-s"><span class="pl-pds">"</span>topic:git-scraping sort:updated-desc<span class="pl-pds">"</span></span>, <span class="pl-v">type</span>:<span class="pl-c1"> REPOSITORY</span>, <span class="pl-v">first</span>: <span class="pl-c1">100</span>) {
    <span class="pl-v">repositoryCount</span>
    <span class="pl-v">nodes</span> {
      <span class="pl-k">...</span> <span class="pl-k">on</span> <span class="pl-c1">Repository</span> {
        <span class="pl-v">nameWithOwner</span>
        <span class="pl-v">description</span>
        <span class="pl-v">updatedAt</span>
        <span class="pl-v">createdAt</span>
        <span class="pl-v">diskUsage</span>
      }
    }
  }
}</pre></div>


path: homebrew_upgrading-python-homebrew-packages.md
topic: homebrew
title: Upgrading Python Homebrew packages using pip
url: https://github.com/simonw/til/blob/main/homebrew/upgrading-python-homebrew-packages.md
created: 2020-10-14T09:21:01-07:00
created_utc: 2020-10-14T16:21:01+00:00
updated: 2020-10-14T09:21:01-07:00
updated_utc: 2020-10-14T16:21:01+00:00
shot_hash: 2a7e13c236f5b5373a18c74c96c518cd
slug: upgrading-python-homebrew-packages
body:
[VisiData 2.0](https://www.visidata.org/) came out today. I previously installed VisiData using Homebrew, but the VisiData tap has not yet been updated with the latest version.

Homebrew Python packages (including the packages for [Datasette](https://formulae.brew.sh/formula/datasette) and [sqlite-utils](https://formulae.brew.sh/formula/sqlite-utils)) work by setting up their own package-specific virtual environments. This means you can upgrade them without waiting for the tap.

To find the virtual environment, run `head -n 1` against the Homebrew-providid executable. VisiData is `vd`, so this works:
```
% head -n 1 $(which vd)
#!/usr/local/Cellar/visidata/1.5.2/libexec/bin/python3.8
```
Now you can call `pip` within that virtual directory to perform the upgrade like so:
```
/usr/local/Cellar/visidata/1.5.2/libexec/bin/pip install -U visidata
```

html:
<p><a href="https://www.visidata.org/" rel="nofollow">VisiData 2.0</a> came out today. I previously installed VisiData using Homebrew, but the VisiData tap has not yet been updated with the latest version.</p>
<p>Homebrew Python packages (including the packages for <a href="https://formulae.brew.sh/formula/datasette" rel="nofollow">Datasette</a> and <a href="https://formulae.brew.sh/formula/sqlite-utils" rel="nofollow">sqlite-utils</a>) work by setting up their own package-specific virtual environments. This means you can upgrade them without waiting for the tap.</p>
<p>To find the virtual environment, run <code>head -n 1</code> against the Homebrew-providid executable. VisiData is <code>vd</code>, so this works:</p>
<pre><code>% head -n 1 $(which vd)
#!/usr/local/Cellar/visidata/1.5.2/libexec/bin/python3.8
</code></pre>
<p>Now you can call <code>pip</code> within that virtual directory to perform the upgrade like so:</p>
<pre><code>/usr/local/Cellar/visidata/1.5.2/libexec/bin/pip install -U visidata
</code></pre>


path: python_click-file-encoding.md
topic: python
title: Explicit file encodings using click.File
url: https://github.com/simonw/til/blob/main/python/click-file-encoding.md
created: 2020-10-16T13:39:08-07:00
created_utc: 2020-10-16T20:39:08+00:00
updated: 2020-10-16T16:15:49-07:00
updated_utc: 2020-10-16T23:15:49+00:00
shot_hash: 2137d96980a2ad9ec1684cd51e858131
slug: click-file-encoding
body:
I wanted to add a `--encoding` option to `sqlite-utils insert` which could be used to change the file encoding used to read the incoming CSV or TSV file - see [sqlite-utils #182](https://github.com/simonw/sqlite-utils/issues/182).

Just one problem: the Click file argument was defined using `click.File()`, which has useful features like automatically handling `-` for standard input. The code looked like this:

```python
@click.argument("json_file", type=click.File(), required=True)
# ...
def command(json_file):
    # ...
```
`click.File()` takes an optional `encoding=` parameter, but that requires you to know the file encoding in advance. In my case I wanted to use the encoding optionally provided by an `--encoding=` option.

The workaround I came up with was to switch to using `click.File("rb")`, which opened the incoming file (or stdin stream) in binary mode. I could then wrap it in a `codecs.getreader()` object that could convert those bytes into a Python string using the user-supplied encoding.

```python
@click.argument("json_file", type=click.File("rb"), required=True)
@click.option("--encoding", help="Character encoding for input, defaults to utf-8")
def command(json_file, encoding):
    encoding = encoding or "utf-8"
    json_file = codecs.getreader(encoding)(json_file)
    reader = csv.reader(json_file)
    # ...
```

html:
<p>I wanted to add a <code>--encoding</code> option to <code>sqlite-utils insert</code> which could be used to change the file encoding used to read the incoming CSV or TSV file - see <a href="https://github.com/simonw/sqlite-utils/issues/182">sqlite-utils #182</a>.</p>
<p>Just one problem: the Click file argument was defined using <code>click.File()</code>, which has useful features like automatically handling <code>-</code> for standard input. The code looked like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">click</span>.<span class="pl-s1">argument</span>(<span class="pl-s">"json_file"</span>, <span class="pl-s1">type</span><span class="pl-c1">=</span><span class="pl-s1">click</span>.<span class="pl-v">File</span>(), <span class="pl-s1">required</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)</span>
<span class="pl-c"># ...</span>
<span class="pl-k">def</span> <span class="pl-en">command</span>(<span class="pl-s1">json_file</span>):
    <span class="pl-c"># ...</span></pre></div>
<p><code>click.File()</code> takes an optional <code>encoding=</code> parameter, but that requires you to know the file encoding in advance. In my case I wanted to use the encoding optionally provided by an <code>--encoding=</code> option.</p>
<p>The workaround I came up with was to switch to using <code>click.File("rb")</code>, which opened the incoming file (or stdin stream) in binary mode. I could then wrap it in a <code>codecs.getreader()</code> object that could convert those bytes into a Python string using the user-supplied encoding.</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">click</span>.<span class="pl-s1">argument</span>(<span class="pl-s">"json_file"</span>, <span class="pl-s1">type</span><span class="pl-c1">=</span><span class="pl-s1">click</span>.<span class="pl-v">File</span>(<span class="pl-s">"rb"</span>), <span class="pl-s1">required</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)</span>
<span class="pl-en">@<span class="pl-s1">click</span>.<span class="pl-s1">option</span>(<span class="pl-s">"--encoding"</span>, <span class="pl-s1">help</span><span class="pl-c1">=</span><span class="pl-s">"Character encoding for input, defaults to utf-8"</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">command</span>(<span class="pl-s1">json_file</span>, <span class="pl-s1">encoding</span>):
    <span class="pl-s1">encoding</span> <span class="pl-c1">=</span> <span class="pl-s1">encoding</span> <span class="pl-c1">or</span> <span class="pl-s">"utf-8"</span>
    <span class="pl-s1">json_file</span> <span class="pl-c1">=</span> <span class="pl-s1">codecs</span>.<span class="pl-en">getreader</span>(<span class="pl-s1">encoding</span>)(<span class="pl-s1">json_file</span>)
    <span class="pl-s1">reader</span> <span class="pl-c1">=</span> <span class="pl-s1">csv</span>.<span class="pl-en">reader</span>(<span class="pl-s1">json_file</span>)
    <span class="pl-c"># ...</span></pre></div>


path: javascript_javascript-that-responds-to-media-queries.md
topic: javascript
title: Writing JavaScript that responds to media queries
url: https://github.com/simonw/til/blob/main/javascript/javascript-that-responds-to-media-queries.md
created: 2020-10-21T20:06:08-07:00
created_utc: 2020-10-22T03:06:08+00:00
updated: 2020-10-22T08:04:15-07:00
updated_utc: 2020-10-22T15:04:15+00:00
shot_hash: 98138c7261b30bc084720a511f7aa93c
slug: javascript-that-responds-to-media-queries
body:
I wanted to change the layout of [my blog](https://simonwillison.net/) on mobile screens such that the content from the "Elsewhere" right hand column combined with the main column in the correct order (issue [#165](https://github.com/simonw/simonwillisonblog/issues/165)). I couldn't find a way to do this in pure CSS without duplicating a bunch of content, so I decided to do it with JavaScript.

I needed to apply some changes to the DOM if the users window was smaller than 800px - or if the user resized their window to less than 800px after they had first loaded the page.

Here's the JavaScript I came up with, using [Window.matchMedia()](https://developer.mozilla.org/en-US/docs/Web/API/Window/matchMedia):
```javascript
window.hasBeenRearrangedForMobile = false;
function rearrangeForMobile() {
  // Make changes to the DOM here
}
function conditionalRearrange(m) {
  if (m.matches && !window.hasBeenRearrangedForMobile) {
    rearrangeForMobile();
    window.hasBeenRearrangedForMobile = true;
  }
}
var mediaMatcher = window.matchMedia('(max-width: 800px)');
conditionalRearrange(mediaMatcher);
mediaMatcher.addListener(conditionalRearrange);
```
[Full code is here](https://github.com/simonw/simonwillisonblog/blob/13c287336675e3b4f8e3f3d0f4fdeff738e87ad0/templates/homepage.html#L45-L88).

That `rearrangeForMobile()` function makes changes to the DOM, by copying some content from the sidebar into the main column. Media queries in my CSS then control the display of that content at different screen sizes - so if the user changes the width of the page things will continue to display correctly based on the current width, without needing to execute extra JavaScript.

Here's [the accompanying CSS](https://github.com/simonw/simonwillisonblog/blob/13c287336675e3b4f8e3f3d0f4fdeff738e87ad0/static/css/all.css#L795-L812):

```css
.elsewhere-date {
    display: none;
}

#primary .elsewhere-in-primary {
    display: none;
}

@media (max-width: 800px) {
  #primary .elsewhere-in-primary {
    display: block;
  }
  .elsewhere-date {
    display: inline;
  }
  .hide-secondary-on-mobile {
    display: none;
  }
  // ...
}
```

html:
<p>I wanted to change the layout of <a href="https://simonwillison.net/" rel="nofollow">my blog</a> on mobile screens such that the content from the "Elsewhere" right hand column combined with the main column in the correct order (issue <a href="https://github.com/simonw/simonwillisonblog/issues/165">#165</a>). I couldn't find a way to do this in pure CSS without duplicating a bunch of content, so I decided to do it with JavaScript.</p>
<p>I needed to apply some changes to the DOM if the users window was smaller than 800px - or if the user resized their window to less than 800px after they had first loaded the page.</p>
<p>Here's the JavaScript I came up with, using <a href="https://developer.mozilla.org/en-US/docs/Web/API/Window/matchMedia" rel="nofollow">Window.matchMedia()</a>:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-smi">window</span><span class="pl-kos">.</span><span class="pl-c1">hasBeenRearrangedForMobile</span> <span class="pl-c1">=</span> <span class="pl-c1">false</span><span class="pl-kos">;</span>
<span class="pl-k">function</span> <span class="pl-en">rearrangeForMobile</span><span class="pl-kos">(</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
  <span class="pl-c">// Make changes to the DOM here</span>
<span class="pl-kos">}</span>
<span class="pl-k">function</span> <span class="pl-en">conditionalRearrange</span><span class="pl-kos">(</span><span class="pl-s1">m</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
  <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">m</span><span class="pl-kos">.</span><span class="pl-c1">matches</span> <span class="pl-c1">&amp;&amp;</span> !<span class="pl-smi">window</span><span class="pl-kos">.</span><span class="pl-c1">hasBeenRearrangedForMobile</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-en">rearrangeForMobile</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
    <span class="pl-smi">window</span><span class="pl-kos">.</span><span class="pl-c1">hasBeenRearrangedForMobile</span> <span class="pl-c1">=</span> <span class="pl-c1">true</span><span class="pl-kos">;</span>
  <span class="pl-kos">}</span>
<span class="pl-kos">}</span>
<span class="pl-k">var</span> <span class="pl-s1">mediaMatcher</span> <span class="pl-c1">=</span> <span class="pl-smi">window</span><span class="pl-kos">.</span><span class="pl-en">matchMedia</span><span class="pl-kos">(</span><span class="pl-s">'(max-width: 800px)'</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-en">conditionalRearrange</span><span class="pl-kos">(</span><span class="pl-s1">mediaMatcher</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-s1">mediaMatcher</span><span class="pl-kos">.</span><span class="pl-en">addListener</span><span class="pl-kos">(</span><span class="pl-s1">conditionalRearrange</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<p><a href="https://github.com/simonw/simonwillisonblog/blob/13c287336675e3b4f8e3f3d0f4fdeff738e87ad0/templates/homepage.html#L45-L88">Full code is here</a>.</p>
<p>That <code>rearrangeForMobile()</code> function makes changes to the DOM, by copying some content from the sidebar into the main column. Media queries in my CSS then control the display of that content at different screen sizes - so if the user changes the width of the page things will continue to display correctly based on the current width, without needing to execute extra JavaScript.</p>
<p>Here's <a href="https://github.com/simonw/simonwillisonblog/blob/13c287336675e3b4f8e3f3d0f4fdeff738e87ad0/static/css/all.css#L795-L812">the accompanying CSS</a>:</p>
<div class="highlight highlight-source-css"><pre>.<span class="pl-c1">elsewhere-date</span> {
    <span class="pl-c1">display</span>: none;
}

<span class="pl-kos">#</span><span class="pl-c1">primary</span> .<span class="pl-c1">elsewhere-in-primary</span> {
    <span class="pl-c1">display</span>: none;
}

<span class="pl-k">@media</span> (<span class="pl-c1">max-width</span>: <span class="pl-c1">800<span class="pl-smi">px</span></span>) {
  <span class="pl-kos">#</span><span class="pl-c1">primary</span> .<span class="pl-c1">elsewhere-in-primary</span> {
    <span class="pl-c1">display</span>: block;
  }
  .<span class="pl-c1">elsewhere-date</span> {
    <span class="pl-c1">display</span>: inline;
  }
  .<span class="pl-c1">hide-secondary-on-mobile</span> {
    <span class="pl-c1">display</span>: none;
  }
  <span class="pl-c1">/</span><span class="pl-c1">/</span> ...
}</pre></div>


path: python_decorators-with-optional-arguments.md
topic: python
title: Decorators with optional arguments
url: https://github.com/simonw/til/blob/main/python/decorators-with-optional-arguments.md
created: 2020-10-28T14:54:48-07:00
created_utc: 2020-10-28T21:54:48+00:00
updated: 2020-12-29T13:55:23-08:00
updated_utc: 2020-12-29T21:55:23+00:00
shot_hash: 4334df4ececb51704cc6332f3e01f059
slug: decorators-with-optional-arguments
body:
[sqlite-utils](https://sqlite-utils.datasette.io/) provides [a decorator](https://sqlite-utils.datasette.io/en/stable/python-api.html#registering-custom-sql-functions) for registering custom Python functions that looks like this:

```python
from sqlite_utils import Database

db = Database(memory=True)

@db.register_function
def reverse_string(s):
    return "".join(reversed(list(s)))
```

I wanted to add an [optional deterministic=True parameter](https://github.com/simonw/sqlite-utils/issues/191) to the decorator.

Problem: the decorator currently does not take any arguments. But I wanted this to work as well:

```python
@db.register_function(deterministic=True)
def reverse_string(s):
    return "".join(reversed(list(s)))
```

I don't want to break backwards compatibility with existing code.

First lesson: any time you are designing a decorator that might accept arguments in the future, design it to work like this!

```python
@db.register_function()
def reverse_string(s):
    return "".join(reversed(list(s)))
```

Since I hadn't done that, I needed an alternative pattern that could differentiate between the two ways in which the decorator might be called. I ended up going with this:
```python
    def register_function(self, fn=None, deterministic=None):
        def register(fn):
            name = fn.__name__
            arity = len(inspect.signature(fn).parameters)
            kwargs = {}
            if deterministic and sys.version_info >= (3, 8):
                kwargs["deterministic"] = True
            self.conn.create_function(name, arity, fn, **kwargs)
            return fn

        if fn is None:
            return register
        else:
            register(fn)
```

html:
<p><a href="https://sqlite-utils.datasette.io/" rel="nofollow">sqlite-utils</a> provides <a href="https://sqlite-utils.datasette.io/en/stable/python-api.html#registering-custom-sql-functions" rel="nofollow">a decorator</a> for registering custom Python functions that looks like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">sqlite_utils</span> <span class="pl-k">import</span> <span class="pl-v">Database</span>

<span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-v">Database</span>(<span class="pl-s1">memory</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)

<span class="pl-en">@<span class="pl-s1">db</span>.<span class="pl-s1">register_function</span></span>
<span class="pl-k">def</span> <span class="pl-en">reverse_string</span>(<span class="pl-s1">s</span>):
    <span class="pl-k">return</span> <span class="pl-s">""</span>.<span class="pl-en">join</span>(<span class="pl-en">reversed</span>(<span class="pl-en">list</span>(<span class="pl-s1">s</span>)))</pre></div>
<p>I wanted to add an <a href="https://github.com/simonw/sqlite-utils/issues/191">optional deterministic=True parameter</a> to the decorator.</p>
<p>Problem: the decorator currently does not take any arguments. But I wanted this to work as well:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">db</span>.<span class="pl-s1">register_function</span>(<span class="pl-s1">deterministic</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">reverse_string</span>(<span class="pl-s1">s</span>):
    <span class="pl-k">return</span> <span class="pl-s">""</span>.<span class="pl-en">join</span>(<span class="pl-en">reversed</span>(<span class="pl-en">list</span>(<span class="pl-s1">s</span>)))</pre></div>
<p>I don't want to break backwards compatibility with existing code.</p>
<p>First lesson: any time you are designing a decorator that might accept arguments in the future, design it to work like this!</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">db</span>.<span class="pl-s1">register_function</span>()</span>
<span class="pl-k">def</span> <span class="pl-en">reverse_string</span>(<span class="pl-s1">s</span>):
    <span class="pl-k">return</span> <span class="pl-s">""</span>.<span class="pl-en">join</span>(<span class="pl-en">reversed</span>(<span class="pl-en">list</span>(<span class="pl-s1">s</span>)))</pre></div>
<p>Since I hadn't done that, I needed an alternative pattern that could differentiate between the two ways in which the decorator might be called. I ended up going with this:</p>
<div class="highlight highlight-source-python"><pre>    <span class="pl-k">def</span> <span class="pl-en">register_function</span>(<span class="pl-s1">self</span>, <span class="pl-s1">fn</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">deterministic</span><span class="pl-c1">=</span><span class="pl-c1">None</span>):
        <span class="pl-k">def</span> <span class="pl-en">register</span>(<span class="pl-s1">fn</span>):
            <span class="pl-s1">name</span> <span class="pl-c1">=</span> <span class="pl-s1">fn</span>.<span class="pl-s1">__name__</span>
            <span class="pl-s1">arity</span> <span class="pl-c1">=</span> <span class="pl-en">len</span>(<span class="pl-s1">inspect</span>.<span class="pl-en">signature</span>(<span class="pl-s1">fn</span>).<span class="pl-s1">parameters</span>)
            <span class="pl-s1">kwargs</span> <span class="pl-c1">=</span> {}
            <span class="pl-k">if</span> <span class="pl-s1">deterministic</span> <span class="pl-c1">and</span> <span class="pl-s1">sys</span>.<span class="pl-s1">version_info</span> <span class="pl-c1">&gt;=</span> (<span class="pl-c1">3</span>, <span class="pl-c1">8</span>):
                <span class="pl-s1">kwargs</span>[<span class="pl-s">"deterministic"</span>] <span class="pl-c1">=</span> <span class="pl-c1">True</span>
            <span class="pl-s1">self</span>.<span class="pl-s1">conn</span>.<span class="pl-en">create_function</span>(<span class="pl-s1">name</span>, <span class="pl-s1">arity</span>, <span class="pl-s1">fn</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>)
            <span class="pl-k">return</span> <span class="pl-s1">fn</span>

        <span class="pl-k">if</span> <span class="pl-s1">fn</span> <span class="pl-c1">is</span> <span class="pl-c1">None</span>:
            <span class="pl-k">return</span> <span class="pl-s1">register</span>
        <span class="pl-k">else</span>:
            <span class="pl-en">register</span>(<span class="pl-s1">fn</span>)</pre></div>


path: javascript_dropdown-menu-with-details-summary.md
topic: javascript
title: Dropdown menu with details summary
url: https://github.com/simonw/til/blob/main/javascript/dropdown-menu-with-details-summary.md
created: 2020-10-31T20:12:10-07:00
created_utc: 2020-11-01T03:12:10+00:00
updated: 2020-10-31T20:12:10-07:00
updated_utc: 2020-11-01T03:12:10+00:00
shot_hash: 86b44044f2dc117902e0406636959c79
slug: dropdown-menu-with-details-summary
body:
I added dropdown menus to [Datasette 0.51](https://docs.datasette.io/en/stable/changelog.html#v0-51) - see [#1064](https://github.com/simonw/datasette/issues/1064).

I implemented them using the HTML `<details><summary>` element. The HTML looked like this:

```html
<details class="nav-menu">
    <summary><svg aria-labelledby="nav-menu-svg-title" role="img"
        fill="currentColor" stroke="currentColor" xmlns="http://www.w3.org/2000/svg"
        viewBox="0 0 16 16" width="16" height="16">
            <title id="nav-menu-svg-title">Menu</title>
            <path fill-rule="evenodd" d="M1 2.75A.75.75 0 011.75 2h12.5a.75.75 0 110 1.5H1.75A.75.75 0 011 2.75zm0 5A.75.75 0 011.75 7h12.5a.75.75 0 110 1.5H1.75A.75.75 0 011 7.75zM1.75 12a.75.75 0 100 1.5h12.5a.75.75 0 100-1.5H1.75z"></path>
    </svg></summary>
    <div class="nav-menu-inner">
        <ul>
            <li><a href="/">Item one</a></li>
            <li><a href="/">Item two</a></li>
            <li><a href="/">Item three</a></li>
        </ul>
    </div>
</details>
```
See the top right corner of https://latest-with-plugins.datasette.io/ for a demo.

This displays an SVG icon which, when clicked, expands to show the menu. The SVG icon uses `aria-labelledby="nav-menu-svg-title" role="img"` and a `<title id="nav-menu-svg-title">` element for accessibility.

I styled the menu using a variant of the following CSS:

```css
details.nav-menu > summary {
    list-style: none;
    display: inline;
    position: relative;
    cursor: pointer;
}
details.nav-menu > summary::-webkit-details-marker {
    display: none;
}
details .nav-menu-inner {
    position: absolute;
    top: 2rem;
    left: 10px;
    width: 180px;
    z-index: 1000;
    border: 1px solid black;
}
.nav-menu-inner a {
    display: block;
}
```
`list-style: none;` hides the default reveal arrow from most browsers. `::-webkit-details-marker { display:none }` handles the rest.

The `summary` element uses `position: relative;` and the `details .nav-menu-inner` uses `position: absolute` - this positions the open dropdown menu in the right place.

## Click outside the box to close the menu

The above uses no JavaScript at all, but comes with one downside: it's usual with menus to clear them if you click outside the menu, but here you need to click on the exact icon again to hide it.

I solved that with the following JavaScript, run at the bottom of the page:

```javascript
document.body.addEventListener('click', (ev) => {
    /* Close any open details elements that this click is outside of */
    var target = ev.target;
    var detailsClickedWithin = null;
    while (target && target.tagName != 'DETAILS') {
        target = target.parentNode;
    }
    if (target && target.tagName == 'DETAILS') {
        detailsClickedWithin = target;
    }
    Array.from(document.getElementsByTagName('details')).filter(
        (details) => details.open && details != detailsClickedWithin
    ).forEach(details => details.open = false);
});
```

html:
<p>I added dropdown menus to <a href="https://docs.datasette.io/en/stable/changelog.html#v0-51" rel="nofollow">Datasette 0.51</a> - see <a href="https://github.com/simonw/datasette/issues/1064">#1064</a>.</p>
<p>I implemented them using the HTML <code>&lt;details&gt;&lt;summary&gt;</code> element. The HTML looked like this:</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">details</span> <span class="pl-c1">class</span>="<span class="pl-s">nav-menu</span>"<span class="pl-kos">&gt;</span>
    <span class="pl-kos">&lt;</span><span class="pl-ent">summary</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;</span><span class="pl-ent">svg</span> <span class="pl-c1">aria-labelledby</span>="<span class="pl-s">nav-menu-svg-title</span>" <span class="pl-c1">role</span>="<span class="pl-s">img</span>"
        <span class="pl-c1">fill</span>="<span class="pl-s">currentColor</span>" <span class="pl-c1">stroke</span>="<span class="pl-s">currentColor</span>" <span class="pl-c1">xmlns</span>="<span class="pl-s">http://www.w3.org/2000/svg</span>"
        <span class="pl-c1">viewBox</span>="<span class="pl-s">0 0 16 16</span>" <span class="pl-c1">width</span>="<span class="pl-s">16</span>" <span class="pl-c1">height</span>="<span class="pl-s">16</span>"<span class="pl-kos">&gt;</span>
            <span class="pl-kos">&lt;</span><span class="pl-ent">title</span> <span class="pl-c1">id</span>="<span class="pl-s">nav-menu-svg-title</span>"<span class="pl-kos">&gt;</span>Menu<span class="pl-kos">&lt;/</span><span class="pl-ent">title</span><span class="pl-kos">&gt;</span>
            <span class="pl-kos">&lt;</span><span class="pl-ent">path</span> <span class="pl-c1">fill-rule</span>="<span class="pl-s">evenodd</span>" <span class="pl-c1">d</span>="<span class="pl-s">M1 2.75A.75.75 0 011.75 2h12.5a.75.75 0 110 1.5H1.75A.75.75 0 011 2.75zm0 5A.75.75 0 011.75 7h12.5a.75.75 0 110 1.5H1.75A.75.75 0 011 7.75zM1.75 12a.75.75 0 100 1.5h12.5a.75.75 0 100-1.5H1.75z</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">path</span><span class="pl-kos">&gt;</span>
    <span class="pl-kos">&lt;/</span><span class="pl-ent">svg</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">summary</span><span class="pl-kos">&gt;</span>
    <span class="pl-kos">&lt;</span><span class="pl-ent">div</span> <span class="pl-c1">class</span>="<span class="pl-s">nav-menu-inner</span>"<span class="pl-kos">&gt;</span>
        <span class="pl-kos">&lt;</span><span class="pl-ent">ul</span><span class="pl-kos">&gt;</span>
            <span class="pl-kos">&lt;</span><span class="pl-ent">li</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;</span><span class="pl-ent">a</span> <span class="pl-c1">href</span>="<span class="pl-s">/</span>"<span class="pl-kos">&gt;</span>Item one<span class="pl-kos">&lt;/</span><span class="pl-ent">a</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">li</span><span class="pl-kos">&gt;</span>
            <span class="pl-kos">&lt;</span><span class="pl-ent">li</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;</span><span class="pl-ent">a</span> <span class="pl-c1">href</span>="<span class="pl-s">/</span>"<span class="pl-kos">&gt;</span>Item two<span class="pl-kos">&lt;/</span><span class="pl-ent">a</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">li</span><span class="pl-kos">&gt;</span>
            <span class="pl-kos">&lt;</span><span class="pl-ent">li</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;</span><span class="pl-ent">a</span> <span class="pl-c1">href</span>="<span class="pl-s">/</span>"<span class="pl-kos">&gt;</span>Item three<span class="pl-kos">&lt;/</span><span class="pl-ent">a</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">li</span><span class="pl-kos">&gt;</span>
        <span class="pl-kos">&lt;/</span><span class="pl-ent">ul</span><span class="pl-kos">&gt;</span>
    <span class="pl-kos">&lt;/</span><span class="pl-ent">div</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;/</span><span class="pl-ent">details</span><span class="pl-kos">&gt;</span></pre></div>
<p>See the top right corner of <a href="https://latest-with-plugins.datasette.io/" rel="nofollow">https://latest-with-plugins.datasette.io/</a> for a demo.</p>
<p>This displays an SVG icon which, when clicked, expands to show the menu. The SVG icon uses <code>aria-labelledby="nav-menu-svg-title" role="img"</code> and a <code>&lt;title id="nav-menu-svg-title"&gt;</code> element for accessibility.</p>
<p>I styled the menu using a variant of the following CSS:</p>
<div class="highlight highlight-source-css"><pre><span class="pl-ent">details</span>.<span class="pl-c1">nav-menu</span> <span class="pl-c1">&gt;</span> <span class="pl-ent">summary</span> {
    <span class="pl-c1">list-style</span>: none;
    <span class="pl-c1">display</span>: inline;
    <span class="pl-c1">position</span>: relative;
    <span class="pl-c1">cursor</span>: pointer;
}
<span class="pl-ent">details</span>.<span class="pl-c1">nav-menu</span> <span class="pl-c1">&gt;</span> <span class="pl-ent">summary</span>::<span class="pl-ent">-webkit-details-marker</span> {
    <span class="pl-c1">display</span>: none;
}
<span class="pl-ent">details</span> .<span class="pl-c1">nav-menu-inner</span> {
    <span class="pl-c1">position</span>: absolute;
    <span class="pl-c1">top</span>: <span class="pl-c1">2<span class="pl-smi">rem</span></span>;
    <span class="pl-c1">left</span>: <span class="pl-c1">10<span class="pl-smi">px</span></span>;
    <span class="pl-c1">width</span>: <span class="pl-c1">180<span class="pl-smi">px</span></span>;
    <span class="pl-c1">z-index</span>: <span class="pl-c1">1000</span>;
    <span class="pl-c1">border</span>: <span class="pl-c1">1<span class="pl-smi">px</span></span> solid black;
}
.<span class="pl-c1">nav-menu-inner</span> <span class="pl-ent">a</span> {
    <span class="pl-c1">display</span>: block;
}</pre></div>
<p><code>list-style: none;</code> hides the default reveal arrow from most browsers. <code>::-webkit-details-marker { display:none }</code> handles the rest.</p>
<p>The <code>summary</code> element uses <code>position: relative;</code> and the <code>details .nav-menu-inner</code> uses <code>position: absolute</code> - this positions the open dropdown menu in the right place.</p>
<h2>
<a id="user-content-click-outside-the-box-to-close-the-menu" class="anchor" href="#click-outside-the-box-to-close-the-menu" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Click outside the box to close the menu</h2>
<p>The above uses no JavaScript at all, but comes with one downside: it's usual with menus to clear them if you click outside the menu, but here you need to click on the exact icon again to hide it.</p>
<p>I solved that with the following JavaScript, run at the bottom of the page:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-c1">body</span><span class="pl-kos">.</span><span class="pl-en">addEventListener</span><span class="pl-kos">(</span><span class="pl-s">'click'</span><span class="pl-kos">,</span> <span class="pl-kos">(</span><span class="pl-s1">ev</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-kos">{</span>
    <span class="pl-c">/* Close any open details elements that this click is outside of */</span>
    <span class="pl-k">var</span> <span class="pl-s1">target</span> <span class="pl-c1">=</span> <span class="pl-s1">ev</span><span class="pl-kos">.</span><span class="pl-c1">target</span><span class="pl-kos">;</span>
    <span class="pl-k">var</span> <span class="pl-s1">detailsClickedWithin</span> <span class="pl-c1">=</span> <span class="pl-c1">null</span><span class="pl-kos">;</span>
    <span class="pl-k">while</span> <span class="pl-kos">(</span><span class="pl-s1">target</span> <span class="pl-c1">&amp;&amp;</span> <span class="pl-s1">target</span><span class="pl-kos">.</span><span class="pl-c1">tagName</span> != <span class="pl-s">'DETAILS'</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-s1">target</span> <span class="pl-c1">=</span> <span class="pl-s1">target</span><span class="pl-kos">.</span><span class="pl-c1">parentNode</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span>
    <span class="pl-k">if</span> <span class="pl-kos">(</span><span class="pl-s1">target</span> <span class="pl-c1">&amp;&amp;</span> <span class="pl-s1">target</span><span class="pl-kos">.</span><span class="pl-c1">tagName</span> <span class="pl-c1">==</span> <span class="pl-s">'DETAILS'</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
        <span class="pl-s1">detailsClickedWithin</span> <span class="pl-c1">=</span> <span class="pl-s1">target</span><span class="pl-kos">;</span>
    <span class="pl-kos">}</span>
    <span class="pl-v">Array</span><span class="pl-kos">.</span><span class="pl-en">from</span><span class="pl-kos">(</span><span class="pl-smi">document</span><span class="pl-kos">.</span><span class="pl-en">getElementsByTagName</span><span class="pl-kos">(</span><span class="pl-s">'details'</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">filter</span><span class="pl-kos">(</span>
        <span class="pl-kos">(</span><span class="pl-s1">details</span><span class="pl-kos">)</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">details</span><span class="pl-kos">.</span><span class="pl-c1">open</span> <span class="pl-c1">&amp;&amp;</span> <span class="pl-s1">details</span> != <span class="pl-s1">detailsClickedWithin</span>
    <span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">forEach</span><span class="pl-kos">(</span><span class="pl-s1">details</span> <span class="pl-c1">=&gt;</span> <span class="pl-s1">details</span><span class="pl-kos">.</span><span class="pl-c1">open</span> <span class="pl-c1">=</span> <span class="pl-c1">false</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>


path: datasette_redirects-for-datasette.md
topic: datasette
title: Redirects for Datasette
url: https://github.com/simonw/til/blob/main/datasette/redirects-for-datasette.md
created: 2020-11-25T11:53:32-08:00
created_utc: 2020-11-25T19:53:32+00:00
updated: 2020-11-25T11:53:32-08:00
updated_utc: 2020-11-25T19:53:32+00:00
shot_hash: d8510c8f4cb6c43f65afd4a6acb5d643
slug: redirects-for-datasette
body:
I made some changes to my https://til.simonwillison.net/ site that resulted in cleaner URL designs, so I needed to setup some redirects. I configured the redirects using a one-off Datasette plugin called `redirects.py` which I dropped into the `plugins/` directory for the Datasette instance:

```python
from datasette import hookimpl
from datasette.utils.asgi import Response


@hookimpl
def register_routes():
    return (
        (r"^/til/til/(?P<topic>[^_]+)_(?P<slug>[^\.]+)\.md$", lambda request: Response.redirect(
            "/{topic}/{slug}".format(**request.url_vars), status=301
        )),
        ("^/til/feed.atom$", lambda: Response.redirect("/tils/feed.atom", status=301)),
        (
            "^/til/search$",
            lambda request: Response.redirect(
                "/tils/search"
                + (("?" + request.query_string) if request.query_string else ""),
                status=301,
            ),
        ),
    )
```

html:
<p>I made some changes to my <a href="https://til.simonwillison.net/" rel="nofollow">https://til.simonwillison.net/</a> site that resulted in cleaner URL designs, so I needed to setup some redirects. I configured the redirects using a one-off Datasette plugin called <code>redirects.py</code> which I dropped into the <code>plugins/</code> directory for the Datasette instance:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">datasette</span> <span class="pl-k">import</span> <span class="pl-s1">hookimpl</span>
<span class="pl-k">from</span> <span class="pl-s1">datasette</span>.<span class="pl-s1">utils</span>.<span class="pl-s1">asgi</span> <span class="pl-k">import</span> <span class="pl-v">Response</span>


<span class="pl-en">@<span class="pl-s1">hookimpl</span></span>
<span class="pl-k">def</span> <span class="pl-en">register_routes</span>():
    <span class="pl-k">return</span> (
        (<span class="pl-s">r"^/til/til/(?P&lt;topic&gt;[^_]+)_(?P&lt;slug&gt;[^\.]+)\.md$"</span>, <span class="pl-k">lambda</span> <span class="pl-s1">request</span>: <span class="pl-v">Response</span>.<span class="pl-en">redirect</span>(
            <span class="pl-s">"/{topic}/{slug}"</span>.<span class="pl-en">format</span>(<span class="pl-c1">**</span><span class="pl-s1">request</span>.<span class="pl-s1">url_vars</span>), <span class="pl-s1">status</span><span class="pl-c1">=</span><span class="pl-c1">301</span>
        )),
        (<span class="pl-s">"^/til/feed.atom$"</span>, <span class="pl-k">lambda</span>: <span class="pl-v">Response</span>.<span class="pl-en">redirect</span>(<span class="pl-s">"/tils/feed.atom"</span>, <span class="pl-s1">status</span><span class="pl-c1">=</span><span class="pl-c1">301</span>)),
        (
            <span class="pl-s">"^/til/search$"</span>,
            <span class="pl-k">lambda</span> <span class="pl-s1">request</span>: <span class="pl-v">Response</span>.<span class="pl-en">redirect</span>(
                <span class="pl-s">"/tils/search"</span>
                <span class="pl-c1">+</span> ((<span class="pl-s">"?"</span> <span class="pl-c1">+</span> <span class="pl-s1">request</span>.<span class="pl-s1">query_string</span>) <span class="pl-k">if</span> <span class="pl-s1">request</span>.<span class="pl-s1">query_string</span> <span class="pl-k">else</span> <span class="pl-s">""</span>),
                <span class="pl-s1">status</span><span class="pl-c1">=</span><span class="pl-c1">301</span>,
            ),
        ),
    )</pre></div>


path: python_subprocess-time-limit.md
topic: python
title: Running Python code in a subprocess with a time limit
url: https://github.com/simonw/til/blob/main/python/subprocess-time-limit.md
created: 2020-12-06T17:03:30-08:00
created_utc: 2020-12-07T01:03:30+00:00
updated: 2020-12-06T23:35:41-08:00
updated_utc: 2020-12-07T07:35:41+00:00
shot_hash: 48d1ddaa155ff9bf18b96008227f6ce8
slug: subprocess-time-limit
body:
I figured out how to run a subprocess with a time limit for [datasette-ripgrep](https://github.com/simonw/datasette-ripgrep), using the `asyncio.create_subprocess_exec()` method. The pattern looks like this:

```python
import asyncio

proc = await asyncio.create_subprocess_exec(
    "rg",
    "-e",
    ".*",
    stdout=asyncio.subprocess.PIPE,
    stdin=asyncio.subprocess.PIPE,
)

try:
    stdout, stderr = await asyncio.wait_for(proc.communicate(), timeout=0.1)
    print(stdout)
except asyncio.exceptions.TimeoutError:
    print("Command timed out")

# If it timed out we should terminate the process
try:
    proc.kill()
except OSError:
    # Ignore 'no such process' error
    pass
```
For [datasette-seaborn](https://github.com/simonw/datasette-seaborn) I wanted to render a chart using the Python seaborn library with a time limit of five seconds for the render.

I realized I could do this by launching Python itself as the subprocess executable (using `sys.executable`) and sending Python code to `stdin` to be executed in a process, using the same time limit mechanism.

It seems to work! Here's the pattern wrapped up in a function:

```python
import asyncio, sys


async def execute_python_with_time_limit(code, time_limit):
    proc = await asyncio.create_subprocess_exec(
        sys.executable,
        "-",
        stdout=asyncio.subprocess.PIPE,
        stdin=asyncio.subprocess.PIPE,
    )
    try:
        stdout, stderr = await asyncio.wait_for(
            proc.communicate(code.encode("utf-8")), timeout=time_limit
        )
    except asyncio.exceptions.TimeoutError:
        try:
            proc.kill()
        except OSError:
            # Ignore 'no such process' error
            pass
        raise
    return stdout, stderr
```
Example of using it (pasting into the shell you get from `python3 -m asyncio` in Python 3.8+):

```
>>> await execute_python_with_time_limit('print("hello")', 1)
(b'hello\n', None)
>>> await execute_python_with_time_limit('import time\ntime.sleep(1)', 0.7)
Traceback (most recent call last):
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "<console>", line 1, in <module>
  File "<console>", line 9, in execute_python_with_time_limit
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py", line 498, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError
```
It returns the stdout output of the code, so to use this you'll need to figure out some kind of serialization format for the data that is returned from the subprocess. JSON or pickle should work fine.

`sys.executable` is the path to the current Python interpreter. This ensures that any imports will take place in the correct Python virtual environment. Passing `-` as the first argument causes Python to execute code from standard input, which is then provided using the `.communicate()` method.

Is this a good idea? I think so, but I'd love to hear from you if there's a simpler, cleaner way to do this.

html:
<p>I figured out how to run a subprocess with a time limit for <a href="https://github.com/simonw/datasette-ripgrep">datasette-ripgrep</a>, using the <code>asyncio.create_subprocess_exec()</code> method. The pattern looks like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">asyncio</span>

<span class="pl-s1">proc</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">asyncio</span>.<span class="pl-en">create_subprocess_exec</span>(
    <span class="pl-s">"rg"</span>,
    <span class="pl-s">"-e"</span>,
    <span class="pl-s">".*"</span>,
    <span class="pl-s1">stdout</span><span class="pl-c1">=</span><span class="pl-s1">asyncio</span>.<span class="pl-s1">subprocess</span>.<span class="pl-v">PIPE</span>,
    <span class="pl-s1">stdin</span><span class="pl-c1">=</span><span class="pl-s1">asyncio</span>.<span class="pl-s1">subprocess</span>.<span class="pl-v">PIPE</span>,
)

<span class="pl-k">try</span>:
    <span class="pl-s1">stdout</span>, <span class="pl-s1">stderr</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">asyncio</span>.<span class="pl-en">wait_for</span>(<span class="pl-s1">proc</span>.<span class="pl-en">communicate</span>(), <span class="pl-s1">timeout</span><span class="pl-c1">=</span><span class="pl-c1">0.1</span>)
    <span class="pl-en">print</span>(<span class="pl-s1">stdout</span>)
<span class="pl-k">except</span> <span class="pl-s1">asyncio</span>.<span class="pl-s1">exceptions</span>.<span class="pl-v">TimeoutError</span>:
    <span class="pl-en">print</span>(<span class="pl-s">"Command timed out"</span>)

<span class="pl-c"># If it timed out we should terminate the process</span>
<span class="pl-k">try</span>:
    <span class="pl-s1">proc</span>.<span class="pl-en">kill</span>()
<span class="pl-k">except</span> <span class="pl-v">OSError</span>:
    <span class="pl-c"># Ignore 'no such process' error</span>
    <span class="pl-k">pass</span></pre></div>
<p>For <a href="https://github.com/simonw/datasette-seaborn">datasette-seaborn</a> I wanted to render a chart using the Python seaborn library with a time limit of five seconds for the render.</p>
<p>I realized I could do this by launching Python itself as the subprocess executable (using <code>sys.executable</code>) and sending Python code to <code>stdin</code> to be executed in a process, using the same time limit mechanism.</p>
<p>It seems to work! Here's the pattern wrapped up in a function:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">asyncio</span>, <span class="pl-s1">sys</span>


<span class="pl-k">async</span> <span class="pl-k">def</span> <span class="pl-en">execute_python_with_time_limit</span>(<span class="pl-s1">code</span>, <span class="pl-s1">time_limit</span>):
    <span class="pl-s1">proc</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">asyncio</span>.<span class="pl-en">create_subprocess_exec</span>(
        <span class="pl-s1">sys</span>.<span class="pl-s1">executable</span>,
        <span class="pl-s">"-"</span>,
        <span class="pl-s1">stdout</span><span class="pl-c1">=</span><span class="pl-s1">asyncio</span>.<span class="pl-s1">subprocess</span>.<span class="pl-v">PIPE</span>,
        <span class="pl-s1">stdin</span><span class="pl-c1">=</span><span class="pl-s1">asyncio</span>.<span class="pl-s1">subprocess</span>.<span class="pl-v">PIPE</span>,
    )
    <span class="pl-k">try</span>:
        <span class="pl-s1">stdout</span>, <span class="pl-s1">stderr</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">asyncio</span>.<span class="pl-en">wait_for</span>(
            <span class="pl-s1">proc</span>.<span class="pl-en">communicate</span>(<span class="pl-s1">code</span>.<span class="pl-en">encode</span>(<span class="pl-s">"utf-8"</span>)), <span class="pl-s1">timeout</span><span class="pl-c1">=</span><span class="pl-s1">time_limit</span>
        )
    <span class="pl-k">except</span> <span class="pl-s1">asyncio</span>.<span class="pl-s1">exceptions</span>.<span class="pl-v">TimeoutError</span>:
        <span class="pl-k">try</span>:
            <span class="pl-s1">proc</span>.<span class="pl-en">kill</span>()
        <span class="pl-k">except</span> <span class="pl-v">OSError</span>:
            <span class="pl-c"># Ignore 'no such process' error</span>
            <span class="pl-k">pass</span>
        <span class="pl-k">raise</span>
    <span class="pl-k">return</span> <span class="pl-s1">stdout</span>, <span class="pl-s1">stderr</span></pre></div>
<p>Example of using it (pasting into the shell you get from <code>python3 -m asyncio</code> in Python 3.8+):</p>
<pre><code>&gt;&gt;&gt; await execute_python_with_time_limit('print("hello")', 1)
(b'hello\n', None)
&gt;&gt;&gt; await execute_python_with_time_limit('import time\ntime.sleep(1)', 0.7)
Traceback (most recent call last):
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
  File "&lt;console&gt;", line 1, in &lt;module&gt;
  File "&lt;console&gt;", line 9, in execute_python_with_time_limit
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/tasks.py", line 498, in wait_for
    raise exceptions.TimeoutError()
asyncio.exceptions.TimeoutError
</code></pre>
<p>It returns the stdout output of the code, so to use this you'll need to figure out some kind of serialization format for the data that is returned from the subprocess. JSON or pickle should work fine.</p>
<p><code>sys.executable</code> is the path to the current Python interpreter. This ensures that any imports will take place in the correct Python virtual environment. Passing <code>-</code> as the first argument causes Python to execute code from standard input, which is then provided using the <code>.communicate()</code> method.</p>
<p>Is this a good idea? I think so, but I'd love to hear from you if there's a simpler, cleaner way to do this.</p>


path: readthedocs_custom-sphinx-templates.md
topic: readthedocs
title: Using custom Sphinx templates on Read the Docs
url: https://github.com/simonw/til/blob/main/readthedocs/custom-sphinx-templates.md
created: 2020-12-07T11:07:43-08:00
created_utc: 2020-12-07T19:07:43+00:00
updated: 2020-12-07T11:07:43-08:00
updated_utc: 2020-12-07T19:07:43+00:00
shot_hash: 4dbb7377e69efb5298b81b3ca9106a97
slug: custom-sphinx-templates
body:
I wanted to make a small customization to one of my documentation templates on Read the Docs.

It turns out you can over-ride the default templates by creating a `_templates` directory in your documentation folder and dropping new files into it - no additional configuration changes required.

I wanted to customize a small part of the `layout.html` template. The original template is here: https://github.com/simonw/sphinx_rtd_theme/blob/master/sphinx_rtd_theme/layout.html

It has named blocks in it, which means I can over-ride just the specific named block I'm interested in. I created my own `{% block sidebartitle %}` block, then saved it in a file in `_templates/layout.html`.

To inherit from the default layout template, I used `{% extends "!layout.html" %}` - note the `!` there which causes it to use the default template (without that character the build script throws an infinite recursion error).

Here's the `layout.html` template I used:
```html+jinja
{%- extends "!layout.html" %}

{% block sidebartitle %}

<a href="https://datasette.io/">
  <img src="{{ pathto('_static/' + logo, 1) }}" class="logo" alt="{{ _('Logo') }}"/>
</a>

{% if theme_display_version %}
  {%- set nav_version = version %}
  {% if READTHEDOCS and current_version %}
    {%- set nav_version = current_version %}
  {% endif %}
  {% if nav_version %}
    <div class="version">
      {{ nav_version }}
    </div>
  {% endif %}
{% endif %}

{% include "searchbox.html" %}

{% endblock %}
```
See https://github.com/simonw/datasette/tree/main/docs for the full documentation layout - the only thing I had to change to get this custom template was adding the new `_templates` folder and the `layout.html` file.

html:
<p>I wanted to make a small customization to one of my documentation templates on Read the Docs.</p>
<p>It turns out you can over-ride the default templates by creating a <code>_templates</code> directory in your documentation folder and dropping new files into it - no additional configuration changes required.</p>
<p>I wanted to customize a small part of the <code>layout.html</code> template. The original template is here: <a href="https://github.com/simonw/sphinx_rtd_theme/blob/master/sphinx_rtd_theme/layout.html">https://github.com/simonw/sphinx_rtd_theme/blob/master/sphinx_rtd_theme/layout.html</a></p>
<p>It has named blocks in it, which means I can over-ride just the specific named block I'm interested in. I created my own <code>{% block sidebartitle %}</code> block, then saved it in a file in <code>_templates/layout.html</code>.</p>
<p>To inherit from the default layout template, I used <code>{% extends "!layout.html" %}</code> - note the <code>!</code> there which causes it to use the default template (without that character the build script throws an infinite recursion error).</p>
<p>Here's the <code>layout.html</code> template I used:</p>
<div class="highlight highlight-text-html-django"><pre><span class="pl-e">{%</span>- <span class="pl-k">extends</span> <span class="pl-s">"!layout.html"</span> <span class="pl-e">%}</span>

<span class="pl-e">{%</span> <span class="pl-k">block</span> <span class="pl-s">sidebartitle</span> <span class="pl-e">%}</span>

&lt;<span class="pl-ent">a</span> <span class="pl-e">href</span>=<span class="pl-s"><span class="pl-pds">"</span>https://datasette.io/<span class="pl-pds">"</span></span>&gt;
  &lt;<span class="pl-ent">img</span> <span class="pl-e">src</span>=<span class="pl-s"><span class="pl-pds">"</span>{{ pathto('_static/' + logo, 1) }}<span class="pl-pds">"</span></span> <span class="pl-e">class</span>=<span class="pl-s"><span class="pl-pds">"</span>logo<span class="pl-pds">"</span></span> <span class="pl-e">alt</span>=<span class="pl-s"><span class="pl-pds">"</span>{{ _('Logo') }}<span class="pl-pds">"</span></span>/&gt;
&lt;/<span class="pl-ent">a</span>&gt;

<span class="pl-e">{%</span> <span class="pl-k">if</span> <span class="pl-s">theme_display_version</span> <span class="pl-e">%}</span>
  <span class="pl-e">{%</span>- <span class="pl-s">set</span> <span class="pl-s">nav_version</span> = <span class="pl-s">version</span> <span class="pl-e">%}</span>
  <span class="pl-e">{%</span> <span class="pl-k">if</span> <span class="pl-s">READTHEDOCS</span> <span class="pl-k">and</span> <span class="pl-s">current_version</span> <span class="pl-e">%}</span>
    <span class="pl-e">{%</span>- <span class="pl-s">set</span> <span class="pl-s">nav_version</span> = <span class="pl-s">current_version</span> <span class="pl-e">%}</span>
  <span class="pl-e">{%</span> <span class="pl-k">endif</span> <span class="pl-e">%}</span>
  <span class="pl-e">{%</span> <span class="pl-k">if</span> <span class="pl-s">nav_version</span> <span class="pl-e">%}</span>
    &lt;<span class="pl-ent">div</span> <span class="pl-e">class</span>=<span class="pl-s"><span class="pl-pds">"</span>version<span class="pl-pds">"</span></span>&gt;
      {{ nav_version }}
    &lt;/<span class="pl-ent">div</span>&gt;
  <span class="pl-e">{%</span> <span class="pl-k">endif</span> <span class="pl-e">%}</span>
<span class="pl-e">{%</span> <span class="pl-k">endif</span> <span class="pl-e">%}</span>

<span class="pl-e">{%</span> <span class="pl-k">include</span> <span class="pl-s">"searchbox.html"</span> <span class="pl-e">%}</span>

<span class="pl-e">{%</span> <span class="pl-k">endblock</span> <span class="pl-e">%}</span></pre></div>
<p>See <a href="https://github.com/simonw/datasette/tree/main/docs">https://github.com/simonw/datasette/tree/main/docs</a> for the full documentation layout - the only thing I had to change to get this custom template was adding the new <code>_templates</code> folder and the <code>layout.html</code> file.</p>


path: python_style-yaml-dump.md
topic: python
title: Controlling the style of dumped YAML using PyYAML
url: https://github.com/simonw/til/blob/main/python/style-yaml-dump.md
created: 2020-12-07T12:48:39-08:00
created_utc: 2020-12-07T20:48:39+00:00
updated: 2020-12-07T12:53:40-08:00
updated_utc: 2020-12-07T20:53:40+00:00
shot_hash: fa3a59904c567cfafeae53b77a58e2e7
slug: style-yaml-dump
body:
I had a list of Python dictionaries I wanted to output as YAML, but I wanted to control the style of the output.

Here's the data:
```python
items = [
    {
        "date": "2020-11-28",
        "body": "[Datasette 0.52](https://docs.datasette.io/en/stable/changelog.html#v0-52) - `--config` is now `--setting`, new `database_actions` plugin hook, `datasette publish cloudrun --apt-get-install` option and several bug fixes.",
    },
    {
        "date": "2020-10-31",
        "body": "[Datasette 0.51](https://docs.datasette.io/en/stable/changelog.html#v0-51) - A new visual design, plugin hooks for adding navigation options, better handling of binary data, URL building utility methods and better support for running Datasette behind a proxy. [Annotated release notes](https://simonwillison.net/2020/Nov/1/datasette-0-51/).",
    },
]
```
By default, the YAML output by `import yaml; print(yaml.dump(items))` looks like this:
```yaml
- body: '[Datasette 0.52](https://docs.datasette.io/en/stable/changelog.html#v0-52)
    - `--config` is now `--setting`, new `database_actions` plugin hook, `datasette
    publish cloudrun --apt-get-install` option and several bug fixes.'
  date: '2020-11-28'
- body: '[Datasette 0.51](https://docs.datasette.io/en/stable/changelog.html#v0-51)
    - A new visual design, plugin hooks for adding navigation options, better handling
    of binary data, URL building utility methods and better support for running Datasette
    behind a proxy. [Annotated release notes](https://simonwillison.net/2020/Nov/1/datasette-0-51/).'
  date: '2020-10-31'
```
I wanted to list the `date` key first, and I wanted the `body` key to use `>-` YAML multi-line syntax rather than a single quoted string.

I ended up combining [these](https://stackoverflow.com/a/8641732) [two](https://stackoverflow.com/a/16782282) recipes from Stack Overflow. First I registered new representers with PyYaml:

```python
import yaml
from collections import OrderedDict

class literal(str):
    pass

def literal_presenter(dumper, data):
    return dumper.represent_scalar("tag:yaml.org,2002:str", data, style=">")


yaml.add_representer(literal, literal_presenter)

def represent_ordereddict(dumper, data):
    value = []

    for item_key, item_value in data.items():
        node_key = dumper.represent_data(item_key)
        node_value = dumper.represent_data(item_value)

        value.append((node_key, node_value))

    return yaml.nodes.MappingNode(u"tag:yaml.org,2002:map", value)

yaml.add_representer(OrderedDict, represent_ordereddict)
```
Then I used the following Python code to output my YAML in the desired key order:
```python
print(yaml.dump([OrderedDict([
    ("date", item["date"]),
    ("body", literal(item["body"]))
]) for item in items], width=100))
```
The result was:
```yaml
- date: '2020-11-28'
  body: >-
    [Datasette 0.52](https://docs.datasette.io/en/stable/changelog.html#v0-52) - `--config` is now `--setting`,
    new `database_actions` plugin hook, `datasette publish cloudrun --apt-get-install` option and several
    bug fixes.
- date: '2020-10-31'
  body: >-
    [Datasette 0.51](https://docs.datasette.io/en/stable/changelog.html#v0-51) - A new visual design,
    plugin hooks for adding navigation options, better handling of binary data, URL building utility methods
    and better support for running Datasette behind a proxy. [Annotated release notes](https://simonwillison.net/2020/Nov/1/datasette-0-51/).
```
Using `>` as the line style caused the `width=100` argument to be respected. When I tried this with `|` as the line style the indentation was not applied.

html:
<p>I had a list of Python dictionaries I wanted to output as YAML, but I wanted to control the style of the output.</p>
<p>Here's the data:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">items</span> <span class="pl-c1">=</span> [
    {
        <span class="pl-s">"date"</span>: <span class="pl-s">"2020-11-28"</span>,
        <span class="pl-s">"body"</span>: <span class="pl-s">"[Datasette 0.52](https://docs.datasette.io/en/stable/changelog.html#v0-52) - `--config` is now `--setting`, new `database_actions` plugin hook, `datasette publish cloudrun --apt-get-install` option and several bug fixes."</span>,
    },
    {
        <span class="pl-s">"date"</span>: <span class="pl-s">"2020-10-31"</span>,
        <span class="pl-s">"body"</span>: <span class="pl-s">"[Datasette 0.51](https://docs.datasette.io/en/stable/changelog.html#v0-51) - A new visual design, plugin hooks for adding navigation options, better handling of binary data, URL building utility methods and better support for running Datasette behind a proxy. [Annotated release notes](https://simonwillison.net/2020/Nov/1/datasette-0-51/)."</span>,
    },
]</pre></div>
<p>By default, the YAML output by <code>import yaml; print(yaml.dump(items))</code> looks like this:</p>
<div class="highlight highlight-source-yaml"><pre>- <span class="pl-ent">body</span>: <span class="pl-s"><span class="pl-pds">'</span>[Datasette 0.52](https://docs.datasette.io/en/stable/changelog.html#v0-52)</span>
<span class="pl-s">    - `--config` is now `--setting`, new `database_actions` plugin hook, `datasette</span>
<span class="pl-s">    publish cloudrun --apt-get-install` option and several bug fixes.<span class="pl-pds">'</span></span>
  <span class="pl-ent">date</span>: <span class="pl-s"><span class="pl-pds">'</span>2020-11-28<span class="pl-pds">'</span></span>
- <span class="pl-ent">body</span>: <span class="pl-s"><span class="pl-pds">'</span>[Datasette 0.51](https://docs.datasette.io/en/stable/changelog.html#v0-51)</span>
<span class="pl-s">    - A new visual design, plugin hooks for adding navigation options, better handling</span>
<span class="pl-s">    of binary data, URL building utility methods and better support for running Datasette</span>
<span class="pl-s">    behind a proxy. [Annotated release notes](https://simonwillison.net/2020/Nov/1/datasette-0-51/).<span class="pl-pds">'</span></span>
  <span class="pl-ent">date</span>: <span class="pl-s"><span class="pl-pds">'</span>2020-10-31<span class="pl-pds">'</span></span></pre></div>
<p>I wanted to list the <code>date</code> key first, and I wanted the <code>body</code> key to use <code>&gt;-</code> YAML multi-line syntax rather than a single quoted string.</p>
<p>I ended up combining <a href="https://stackoverflow.com/a/8641732" rel="nofollow">these</a> <a href="https://stackoverflow.com/a/16782282" rel="nofollow">two</a> recipes from Stack Overflow. First I registered new representers with PyYaml:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">yaml</span>
<span class="pl-k">from</span> <span class="pl-s1">collections</span> <span class="pl-k">import</span> <span class="pl-v">OrderedDict</span>

<span class="pl-k">class</span> <span class="pl-s1">literal</span>(<span class="pl-s1">str</span>):
    <span class="pl-k">pass</span>

<span class="pl-k">def</span> <span class="pl-en">literal_presenter</span>(<span class="pl-s1">dumper</span>, <span class="pl-s1">data</span>):
    <span class="pl-k">return</span> <span class="pl-s1">dumper</span>.<span class="pl-en">represent_scalar</span>(<span class="pl-s">"tag:yaml.org,2002:str"</span>, <span class="pl-s1">data</span>, <span class="pl-s1">style</span><span class="pl-c1">=</span><span class="pl-s">"&gt;"</span>)


<span class="pl-s1">yaml</span>.<span class="pl-en">add_representer</span>(<span class="pl-s1">literal</span>, <span class="pl-s1">literal_presenter</span>)

<span class="pl-k">def</span> <span class="pl-en">represent_ordereddict</span>(<span class="pl-s1">dumper</span>, <span class="pl-s1">data</span>):
    <span class="pl-s1">value</span> <span class="pl-c1">=</span> []

    <span class="pl-k">for</span> <span class="pl-s1">item_key</span>, <span class="pl-s1">item_value</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>.<span class="pl-en">items</span>():
        <span class="pl-s1">node_key</span> <span class="pl-c1">=</span> <span class="pl-s1">dumper</span>.<span class="pl-en">represent_data</span>(<span class="pl-s1">item_key</span>)
        <span class="pl-s1">node_value</span> <span class="pl-c1">=</span> <span class="pl-s1">dumper</span>.<span class="pl-en">represent_data</span>(<span class="pl-s1">item_value</span>)

        <span class="pl-s1">value</span>.<span class="pl-en">append</span>((<span class="pl-s1">node_key</span>, <span class="pl-s1">node_value</span>))

    <span class="pl-k">return</span> <span class="pl-s1">yaml</span>.<span class="pl-s1">nodes</span>.<span class="pl-v">MappingNode</span>(<span class="pl-s">u"tag:yaml.org,2002:map"</span>, <span class="pl-s1">value</span>)

<span class="pl-s1">yaml</span>.<span class="pl-en">add_representer</span>(<span class="pl-v">OrderedDict</span>, <span class="pl-s1">represent_ordereddict</span>)</pre></div>
<p>Then I used the following Python code to output my YAML in the desired key order:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">print</span>(<span class="pl-s1">yaml</span>.<span class="pl-en">dump</span>([<span class="pl-v">OrderedDict</span>([
    (<span class="pl-s">"date"</span>, <span class="pl-s1">item</span>[<span class="pl-s">"date"</span>]),
    (<span class="pl-s">"body"</span>, <span class="pl-en">literal</span>(<span class="pl-s1">item</span>[<span class="pl-s">"body"</span>]))
]) <span class="pl-k">for</span> <span class="pl-s1">item</span> <span class="pl-c1">in</span> <span class="pl-s1">items</span>], <span class="pl-s1">width</span><span class="pl-c1">=</span><span class="pl-c1">100</span>))</pre></div>
<p>The result was:</p>
<div class="highlight highlight-source-yaml"><pre>- <span class="pl-ent">date</span>: <span class="pl-s"><span class="pl-pds">'</span>2020-11-28<span class="pl-pds">'</span></span>
  <span class="pl-ent">body</span>: <span class="pl-s">&gt;-</span>
<span class="pl-s">    [Datasette 0.52](https://docs.datasette.io/en/stable/changelog.html#v0-52) - `--config` is now `--setting`,</span>
<span class="pl-s">    new `database_actions` plugin hook, `datasette publish cloudrun --apt-get-install` option and several</span>
<span class="pl-s">    bug fixes.</span>
<span class="pl-s"></span>- <span class="pl-ent">date</span>: <span class="pl-s"><span class="pl-pds">'</span>2020-10-31<span class="pl-pds">'</span></span>
  <span class="pl-ent">body</span>: <span class="pl-s">&gt;-</span>
<span class="pl-s">    [Datasette 0.51](https://docs.datasette.io/en/stable/changelog.html#v0-51) - A new visual design,</span>
<span class="pl-s">    plugin hooks for adding navigation options, better handling of binary data, URL building utility methods</span>
<span class="pl-s">    and better support for running Datasette behind a proxy. [Annotated release notes](https://simonwillison.net/2020/Nov/1/datasette-0-51/).</span></pre></div>
<p>Using <code>&gt;</code> as the line style caused the <code>width=100</code> argument to be respected. When I tried this with <code>|</code> as the line style the indentation was not applied.</p>


